Assessing Eating Behaviour Using Upper Limb Mounted Motion Sensors: A Systematic Review

Detecting Eating Episodes From Wrist Motion Using Daily Pattern Analysis

This paper presents new methods to detect eating from wrist motion. Our main novelty is that we analyze a full day of wrist motion data as a single sample so that the detection of eating occurrences can benefit from diurnal context. We develop a two-stage framework to facilitate a feasible full-day analysis. The first-stage model calculates local probabilities of eating P(Ew) within windows of data, and the second-stage model calculates enhanced probabilities of eating P(Ed) by treating all P(Ew) within a single day as one sample. The framework also incorporates an augmentation technique, which involves the iterative retraining of the first-stage model. This allows us to generate a sufficient number of day-length samples from datasets of limited size. We test our methods on the publicly available Clemson All-Day (CAD) dataset and FreeFIC dataset, and find that the inclusion of day-length analysis substantially improves accuracy in detecting eating episodes. We also benchmark our results against several state-of-the-art methods. Our approach achieved an eating episode true positive rate (TPR) of 89% with 1.4 false positives per true positive (FP/TP), and a time weighted accuracy of 84%, which are the highest accuracies reported on the CAD dataset. Our results show that the daily pattern classifier substantially improves meal detections and in particular reduces transient false detections that tend to occur when relying on shorter windows to look for individual ingestion or consumption events.

E-Health Interventions for Older Adults With Frailty: A Systematic Review

OBJECTIVE

: To systematically review the efficacy of e-Health interventions on physical performance, activity and quality of life in older adults with sarcopenia or frailty.

METHODS

: A systematic review was conducted by searching the MEDLINE, Embase, Cochrane Library, CINHAL, Web of Science, and the Physiotherapy Evidence Database for experimental studies published in English from 1990 to 2021. E-Health studies investigating physical activity, physical performance, quality of life, and activity of daily living assessment in adults aged ≥65 years with sarcopenia or frailty were selected.

RESULTS

: Among the 3,164 identified articles screened, a total of 4 studies complied with the inclusion criteria. The studies were heterogeneous by participant characteristics, type of e-Health intervention, and outcome measurement. Age criteria for participant selection and sex distribution were different between studies. Each study used different criteria for frailty, and no study used sarcopenia as a selection criteria. E-Health interventions were various across studies. Two studies used frailty status as an outcome measure and showed conflicting results. Muscle strength was assessed in 2 studies, and meta-analysis showed statistically significant improvement after intervention (standardized mean difference, 0.51; 95% confidence interval, 0.07-0.94; p=0.80, I2=0%).

CONCLUSION

: This systematic review found insufficient evidence to support the efficacy of e-Health interventions. Nevertheless, the studies included in this review showed positive effects of e-Health interventions on improving muscle strength, physical activity, and quality of life in older adults with frailty.

Factors associated with eating rate: a systematic review and narrative synthesis informed by socio-ecological model

Accumulating evidence shows associations between rapid eating and overweight. Modifying eating rate might be a potential weight management strategy without imposing additional dietary restrictions. A comprehensive understanding of factors associated with eating speed will help with designing effective interventions. The aim of this review was to synthesise the current state of knowledge on the factors associated with eating rate. The socio-ecological model (SEM) was utilised to scaffold the identified factors. A comprehensive literature search of eleven databases was conducted to identify factors associated with eating rate. The 104 studies that met the inclusion criteria were heterogeneous in design and methods of eating rate measurement. We identified thirty-nine factors that were independently linked to eating speed and mapped them onto the individual, social and environmental levels of the SEM. The majority of the reported factors pertained to the individual characteristics (n = 20) including demographics, cognitive/psychological factors and habitual food oral processing behaviours. Social factors (n = 11) included eating companions, social and cultural norms, and family structure. Environmental factors (n = 8) included food texture and presentation, methods of consumption or background sounds. Measures of body weight, food form and characteristics, food oral processing behaviours and gender, age and ethnicity were the most researched and consistent factors associated with eating rate. A number of other novel and underresearched factors emerged, but these require replication and further research. We highlight directions for further research in this space and potential evidence-based candidates for interventions targeting eating rate.

Technology to Automatically Record Eating Behavior in Real Life: A Systematic Review

To monitor adherence to diets and to design and evaluate nutritional interventions, it is essential to obtain objective knowledge about eating behavior. In most research, measures of eating behavior are based on self-reporting, such as 24-h recalls, food records (food diaries) and food frequency questionnaires. Self-reporting is prone to inaccuracies due to inaccurate and subjective recall and other biases. Recording behavior using nonobtrusive technology in daily life would overcome this. Here, we provide an up-to-date systematic overview encompassing all (close-to) publicly or commercially available technologies to automatically record eating behavior in real-life settings. A total of 1328 studies were screened and, after applying defined inclusion and exclusion criteria, 122 studies were included for in-depth evaluation. Technologies in these studies were categorized by what type of eating behavior they measure and which type of sensor technology they use. In general, we found that relatively simple sensors are often used. Depending on the purpose, these are mainly motion sensors, microphones, weight sensors and photo cameras. While several of these technologies are commercially available, there is still a lack of publicly available algorithms that are needed to process and interpret the resulting data. We argue that future work should focus on developing robust algorithms and validating these technologies in real-life settings. Combining technologies (e.g., prompting individuals for self-reports at sensed, opportune moments) is a promising route toward ecologically valid studies of eating behavior.

Validity and Feasibility of the Monitoring and Modeling Family Eating Dynamics System to Automatically Detect In-field Family Eating Behavior: Observational Study

Background

The field of dietary assessment has a long history, marked by both controversies and advances. Emerging technologies may be a potential solution to address the limitations of self-report dietary assessment methods. The Monitoring and Modeling Family Eating Dynamics (M2FED) study uses wrist-worn smartwatches to automatically detect real-time eating activity in the field. The ecological momentary assessment (EMA) methodology was also used to confirm whether eating occurred (ie, ground truth) and to measure other contextual information, including positive and negative affect, hunger, satiety, mindful eating, and social context.

Objective

This study aims to report on participant compliance (feasibility) to the 2 distinct EMA protocols of the M2FED study (hourly time-triggered and eating event–triggered assessments) and on the performance (validity) of the smartwatch algorithm in automatically detecting eating events in a family-based study.

Methods

In all, 20 families (58 participants) participated in the 2-week, observational, M2FED study. All participants wore a smartwatch on their dominant hand and responded to time-triggered and eating event–triggered mobile questionnaires via EMA while at home. Compliance to EMA was calculated overall, for hourly time-triggered mobile questionnaires, and for eating event–triggered mobile questionnaires. The predictors of compliance were determined using a logistic regression model. The number of true and false positive eating events was calculated, as well as the precision of the smartwatch algorithm. The Mann-Whitney U test, Kruskal-Wallis test, and Spearman rank correlation were used to determine whether there were differences in the detection of eating events by participant age, gender, family role, and height.

Results

The overall compliance rate across the 20 deployments was 89.26% (3723/4171) for all EMAs, 89.7% (3328/3710) for time-triggered EMAs, and 85.7% (395/461) for eating event–triggered EMAs. Time of day (afternoon odds ratio [OR] 0.60, 95% CI 0.42-0.85; evening OR 0.53, 95% CI 0.38-0.74) and whether other family members had also answered an EMA (OR 2.07, 95% CI 1.66-2.58) were significant predictors of compliance to time-triggered EMAs. Weekend status (OR 2.40, 95% CI 1.25-4.91) and deployment day (OR 0.92, 95% CI 0.86-0.97) were significant predictors of compliance to eating event–triggered EMAs. Participants confirmed that 76.5% (302/395) of the detected events were true eating events (ie, true positives), and the precision was 0.77. The proportion of correctly detected eating events did not significantly differ by participant age, gender, family role, or height (P>.05).

Conclusions

This study demonstrates that EMA is a feasible tool to collect ground-truth eating activity and thus evaluate the performance of wearable sensors in the field. The combination of a wrist-worn smartwatch to automatically detect eating and a mobile device to capture ground-truth eating activity offers key advantages for the user and makes mobile health technologies more accessible to nonengineering behavioral researchers.

Top-Down Detection of Eating Episodes by Analyzing Large Windows of Wrist Motion Using a Convolutional Neural Network

In this work, we describe a new method to detect periods of eating by tracking wrist motion during everyday life. Eating uses hand-to-mouth gestures for ingestion, each of which lasts a few seconds. Previous works have detected these gestures individually and then aggregated them to identify meals. The novelty of our approach is that we analyze a much longer window (0.5–15 min) using a convolutional neural network. Longer windows can contain other gestures related to eating, such as cutting or manipulating food, preparing foods for consumption, and resting between ingestion events. The context of these other gestures can improve the detection of periods of eating. We test our methods on the public Clemson all-day dataset, which consists of 354 recordings containing 1063 eating episodes. We found that accuracy at detecting eating increased by 15% in ≥4 min windows compared to ≤15 s windows. Using a 6 min window, we detected 89% of eating episodes, with 1.7 false positives for every true positive (FP/TP). These are the best results achieved to date on this dataset.

Artificial intelligence in nutrition research: perspectives on current and future applications

Artificial intelligence (AI) is a rapidly evolving area that offers unparalleled opportunities of progress and applications in many healthcare fields. In this review, we provide an overview of the main and latest applications of AI in nutrition research and identify gaps to address to potentialize this emerging field. AI algorithms may help better understand and predict the complex and non-linear interactions between nutrition-related data and health outcomes, particularly when large amounts of data need to be structured and integrated, such as in metabolomics. AI-based approaches, including image recognition, may also improve dietary assessment by maximizing efficiency and addressing systematic and random errors associated with self-reported measurements of dietary intakes. Finally, AI applications can extract, structure and analyze large amounts of data from social media platforms to better understand dietary behaviours and perceptions among the population. In summary, AI-based approaches will likely improve and advance nutrition research as well as help explore new applications. However, further research is needed to identify areas where AI does deliver added value compared with traditional approaches, and other areas where AI is simply not likely to advance the field. Novelty: Artificial intelligence offers unparalleled opportunities of progress and applications in nutrition. There remain gaps to address to potentialize this emerging field.

Single-Stage Intake Gesture Detection Using CTC Loss and Extended Prefix Beam Search

Accurate detection of individual intake gestures is a key step towards automatic dietary monitoring. Both inertial sensor data of wrist movements and video data depicting the upper body have been used for this purpose. The most advanced approaches to date use a two-stage approach, in which (i) frame-level intake probabilities are learned from the sensor data using a deep neural network, and then (ii) sparse intake events are detected by finding the maxima of the frame-level probabilities. In this study, we propose a single-stage approach which directly decodes the probabilities learned from sensor data into sparse intake detections. This is achieved by weakly supervised training using Connectionist Temporal Classification (CTC) loss, and decoding using a novel extended prefix beam search decoding algorithm. Benefits of this approach include (i) end-to-end training for detections, (ii) simplified timing requirements for intake gesture labels, and (iii) improved detection performance compared to existing approaches. Across two separate datasets, we achieve relative F₁ score improvements between 1.9% and 6.2% over the two-stage approach for intake detection and eating/drinking detection tasks, for both video and inertial sensors.

Fluid Intake Monitoring Systems for the Elderly: A Review of the Literature

Fluid intake monitoring is an essential component in preventing dehydration and overhydration, especially for the senior population. Numerous critical health problems are associated with poor or excessive drinking such as swelling of the brain and heart failure. Real-time systems for monitoring fluid intake will not only measure the exact amount consumed by the users, but could also motivate people to maintain a healthy lifestyle by providing feedback to encourage them to hydrate regularly throughout the day. This paper reviews the most recent solutions to automatic fluid intake monitoring both commercially and in the literature. The available technologies are divided into four categories: wearables, surfaces with embedded sensors, vision- and environmental-based solutions, and smart containers. A detailed performance evaluation was carried out considering detection accuracy, usability and availability. It was observed that the most promising results came from studies that used data fusion from multiple technologies, compared to using an individual technology. The areas that need further research and the challenges for each category are discussed in detail.

Current practice, perceived barriers and resource needs related to measurement of dietary intake, analysis and interpretation of data: A survey of Australian nutrition and dietetics practitioners and researchers

AIM

To inform future training and professional development for individuals who measure, analyse and interpret dietary intake data.

METHODS

A cross-sectional online survey was distributed via e-newsletter to members of Dietitians Australia, Dietitian Connection and Nutrition Society Australia. The survey included 37 questions on three key areas of practice: (a) methods used to assess dietary intake, (b) barriers faced when conducting dietary intake assessment and (c) resources needed to optimise collection, analysis and interpretation of dietary intake data.

RESULTS

Of 173 responses, 103 respondents provided complete data over 2 weeks. Of these, 76% were APDs. The majority (90%) indicated that dietary assessment was important in their role. Respondents (63%) undertook dietary assessments to inform individual/patient care. When assessing intakes, the majority (79%) were interested in examining food/food group intakes. Paper based methods were most commonly used and diet histories, food frequency questionnaires and 24-hour recalls were the most frequently used methods. The biggest barrier identified to implementing dietary assessment methods into practice was participant burden. Over a third of respondents reported they had received specific training on selecting an appropriate dietary assessment method. The majority of respondents (83%) believed having access to a dietary assessment methods toolkit would be useful.

CONCLUSION

Survey findings provide insight into the need for further capacity building strategies, including professional development to improve collection, analysis and interpretation of dietary intake for Australian nutritionists and dietitians. The creation of online resources could help overcome identified barriers and provide a link to best practice methodologies and contemporary tools.

Deep Learning-Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors

BACKGROUND

Multimodal wearable technologies have brought forward wide possibilities in human activity recognition, and more specifically personalized monitoring of eating habits. The emerging challenge now is the selection of most discriminative information from high-dimensional data collected from multiple sources. The available fusion algorithms with their complex structure are poorly adopted to the computationally constrained environment which requires integrating information directly at the source. As a result, more simple low-level fusion methods are needed.

OBJECTIVE

In the absence of a data combining process, the cost of directly applying high-dimensional raw data to a deep classifier would be computationally expensive with regard to the response time, energy consumption, and memory requirement. Taking this into account, we aimed to develop a data fusion technique in a computationally efficient way to achieve a more comprehensive insight of human activity dynamics in a lower dimension. The major objective was considering statistical dependency of multisensory data and exploring intermodality correlation patterns for different activities.

METHODS

In this technique, the information in time (regardless of the number of sources) is transformed into a 2D space that facilitates classification of eating episodes from others. This is based on a hypothesis that data captured by various sensors are statistically associated with each other and the covariance matrix of all these signals has a unique distribution correlated with each activity which can be encoded on a contour representation. These representations are then used as input of a deep model to learn specific patterns associated with specific activity.

RESULTS

In order to show the generalizability of the proposed fusion algorithm, 2 different scenarios were taken into account. These scenarios were different in terms of temporal segment size, type of activity, wearable device, subjects, and deep learning architecture. The first scenario used a data set in which a single participant performed a limited number of activities while wearing the Empatica E4 wristband. In the second scenario, a data set related to the activities of daily living was used where 10 different participants wore inertial measurement units while performing a more complex set of activities. The precision metric obtained from leave-one-subject-out cross-validation for the second scenario reached 0.803. The impact of missing data on performance degradation was also evaluated.

CONCLUSIONS

To conclude, the proposed fusion technique provides the possibility of embedding joint variability information over different modalities in just a single 2D representation which results in obtaining a more global view of different aspects of daily human activities at hand, and yet preserving the desired performance level in activity recognition.

Artificial Intelligence in Nutrients Science Research: A Review

Artificial intelligence (AI) as a branch of computer science, the purpose of which is to imitate thought processes, learning abilities and knowledge management, finds more and more applications in experimental and clinical medicine. In recent decades, there has been an expansion of AI applications in biomedical sciences. The possibilities of artificial intelligence in the field of medical diagnostics, risk prediction and support of therapeutic techniques are growing rapidly. The aim of the article is to analyze the current use of AI in nutrients science research. The literature review was conducted in PubMed. A total of 399 records published between 1987 and 2020 were obtained, of which, after analyzing the titles and abstracts, 261 were rejected. In the next stages, the remaining records were analyzed using the full-text versions and, finally, 55 papers were selected. These papers were divided into three areas: AI in biomedical nutrients research (20 studies), AI in clinical nutrients research (22 studies) and AI in nutritional epidemiology (13 studies). It was found that the artificial neural network (ANN) methodology was dominant in the group of research on food composition study and production of nutrients. However, machine learning (ML) algorithms were widely used in studies on the influence of nutrients on the functioning of the human body in health and disease and in studies on the gut microbiota. Deep learning (DL) algorithms prevailed in a group of research works on clinical nutrients intake. The development of dietary systems using AI technology may lead to the creation of a global network that will be able to both actively support and monitor the personalized supply of nutrients.

A Data Driven End-to-End Approach for In-the-Wild Monitoring of Eating Behavior Using Smartwatches

The increased worldwide prevalence of obesity has sparked the interest of the scientific community towards tools that objectively and automatically monitor eating behavior. Despite the study of obesity being in the spotlight, such tools can also be used to study eating disorders (e.g. anorexia nervosa) or provide a personalized monitoring platform for patients or athletes. This paper presents a complete framework towards the automated i) modeling of in-meal eating behavior and ii) temporal localization of meals, from raw inertial data collected in-the-wild using commercially available smartwatches. Initially, we present an end-to-end Neural Network which detects food intake events (i.e. bites). The proposed network uses both convolutional and recurrent layers that are trained simultaneously. Subsequently, we show how the distribution of the detected bites throughout the day can be used to estimate the start and end points of meals, using signal processing algorithms. We perform extensive evaluation on each framework part individually. Leave-one-subject-out (LOSO) evaluation shows that our bite detection approach outperforms four state-of-the-art algorithms towards the detection of bites during the course of a meal (0.923 F1 score). Furthermore, LOSO and held-out set experiments regarding the estimation of meal start/end points reveal that the proposed approach outperforms a relevant approach found in the literature (Jaccard Index of 0.820 and 0.821 for the LOSO and held-out experiments, respectively). Experiments are performed using our publicly available FIC and the newly introduced FreeFIC datasets.

Validation of motion tracking as tool for observational toothbrushing studies

Video observation (VO) is an established tool for observing toothbrushing behaviour, however, it is a subjective method requiring thorough calibration and training, and the toothbrush position is not always clearly visible. As automated tracking of motions may overcome these disadvantages, the study aimed to compare observational data of habitual toothbrushing as well as of post-instruction toothbrushing obtained from motion tracking (MT) to observational data obtained from VO. One-hundred-three subjects (37.4±14.7 years) were included and brushed their teeth with a manual (MB; n = 51) or a powered toothbrush (PB; n = 52) while being simultaneously video-filmed and tracked. Forty-six subjects were then instructed how to brush their teeth systematically and were filmed/tracked for a second time. Videos were analysed with INTERACT (Mangold, Germany); parameters of interest were toothbrush position, brushing time, changes between areas (events) and the Toothbrushing Systematic Index (TSI). Overall, the median proportion (min; max) of identically classified toothbrush positions (both sextant/surface correct) in a brushing session was 87.8% (50.0; 96.9), which was slightly higher for MB compared to PB (90.3 (50.0; 96.9) vs 86.5 (63.7; 96.5) resp.; p = 0.005). The number of events obtained from MT was higher than from VO (p < 0.001) with a moderate to high correlation between them (MB: ρ = 0.52, p < 0.001; PB: ρ = 0.87; p < 0.001). After instruction, both methods revealed a significant increase of the TSI regardless of the toothbrush type (p < 0.001 each). Motion tracking is a suitable tool for observing toothbrushing behaviour, is able to measure improvements after instruction, and can be used with both manual and powered toothbrushes.

A Real-Time Eating Detection System for Capturing Eating Moments and Triggering Ecological Momentary Assessments to Obtain Further Context: System Development and Validation Study

BACKGROUND

Eating behavior has a high impact on the well-being of an individual. Such behavior involves not only when an individual is eating, but also various contextual factors such as with whom and where an individual is eating and what kind of food the individual is eating. Despite the relevance of such factors, most automated eating detection systems are not designed to capture contextual factors.

OBJECTIVE

The aims of this study were to (1) design and build a smartwatch-based eating detection system that can detect meal episodes based on dominant hand movements, (2) design ecological momentary assessment (EMA) questions to capture meal contexts upon detection of a meal by the eating detection system, and (3) validate the meal detection system that triggers EMA questions upon passive detection of meal episodes.

METHODS

The meal detection system was deployed among 28 college students at a US institution over a period of 3 weeks. The participants reported various contextual data through EMAs triggered when the eating detection system correctly detected a meal episode. The EMA questions were designed after conducting a survey study with 162 students from the same campus. Responses from EMAs were used to define exclusion criteria.

RESULTS

Among the total consumed meals, 89.8% (264/294) of breakfast, 99.0% (406/410) of lunch, and 98.0% (589/601) of dinner episodes were detected by our novel meal detection system. The eating detection system showed a high accuracy by capturing 96.48% (1259/1305) of the meals consumed by the participants. The meal detection classifier showed a precision of 80%, recall of 96%, and F1 of 87.3%. We found that over 99% (1248/1259) of the detected meals were consumed with distractions. Such eating behavior is considered "unhealthy" and can lead to overeating and uncontrolled weight gain. A high proportion of meals was consumed alone (680/1259, 54.01%). Our participants self-reported 62.98% (793/1259) of their meals as healthy. Together, these results have implications for designing technologies to encourage healthy eating behavior.

CONCLUSIONS

The presented eating detection system is the first of its kind to leverage EMAs to capture the eating context, which has strong implications for well-being research. We reflected on the contextual data gathered by our system and discussed how these insights can be used to design individual-specific interventions.

Future Directions for Integrative Objective Assessment of Eating Using Wearable Sensing Technology

Established methods for nutritional assessment suffer from a number of important limitations. Diaries are burdensome to complete, food frequency questionnaires only capture average food intake, and both suffer from difficulties in self estimation of portion size and biases resulting from misreporting. Online and app versions of these methods have been developed, but issues with misreporting and portion size estimation remain. New methods utilizing passive data capture are required that address reporting bias, extend timescales for data collection, and transform what is possible for measuring habitual intakes. Digital and sensing technologies are enabling the development of innovative and transformative new methods in this area that will provide a better understanding of eating behavior and associations with health. In this article we describe how wrist-worn wearables, on-body cameras, and body-mounted biosensors can be used to capture data about when, what, and how much people eat and drink. We illustrate how these new techniques can be integrated to provide complete solutions for the passive, objective assessment of a wide range of traditional dietary factors, as well as novel measures of eating architecture, within person variation in intakes, and food/nutrient combinations within meals. We also discuss some of the challenges these new approaches will bring.

Challenges in the interpretation and therapeutic manipulation of human ingestive microstructure

This minireview focuses on the interpretative value of ingestive microstructure by summarizing observations from both rodent and human studies. Preliminary data on the therapeutic manipulation of distinct microstructural components of eating are also outlined. In rodents, the interpretative framework of ingestive microstructure mainly concentrates on deprivation state, palatability, satiation, and the role of learning from previous experiences. In humans, however, the control of eating is further influenced by genetic, psychosocial, cultural, and environmental factors, which add complexity and challenges to the interpretation of the microstructure of meal intake. Nevertheless, the presented findings stress the importance of microstructural analyses of ingestion, as a method to investigate specific behavioral variables that underlie the regulation of appetite control.

Automatic, wearable-based, in-field eating detection approaches for public health research: a scoping review

Dietary intake, eating behaviors, and context are important in chronic disease development, yet our ability to accurately assess these in research settings can be limited by biased traditional self-reporting tools. Objective measurement tools, specifically, wearable sensors, present the opportunity to minimize the major limitations of self-reported eating measures by generating supplementary sensor data that can improve the validity of self-report data in naturalistic settings. This scoping review summarizes the current use of wearable devices/sensors that automatically detect eating-related activity in naturalistic research settings. Five databases were searched in December 2019, and 618 records were retrieved from the literature search. This scoping review included N = 40 studies (from 33 articles) that reported on one or more wearable sensors used to automatically detect eating activity in the field. The majority of studies (N = 26, 65%) used multi-sensor systems (incorporating > 1 wearable sensors), and accelerometers were the most commonly utilized sensor (N = 25, 62.5%). All studies (N = 40, 100.0%) used either self-report or objective ground-truth methods to validate the inferred eating activity detected by the sensor(s). The most frequently reported evaluation metrics were Accuracy (N = 12) and F1-score (N = 10). This scoping review highlights the current state of wearable sensors' ability to improve upon traditional eating assessment methods by passively detecting eating activity in naturalistic settings, over long periods of time, and with minimal user interaction. A key challenge in this field, wide variation in eating outcome measures and evaluation metrics, demonstrates the need for the development of a standardized form of comparability among sensors/multi-sensor systems and multidisciplinary collaboration.

Learning Deep Representations for Video-Based Intake Gesture Detection

Automatic detection of individual intake gestures during eating occasions has the potential to improve dietary monitoring and support dietary recommendations. Existing studies typically make use of on-body solutions such as inertial and audio sensors, while video is used as ground truth. Intake gesture detection directly based on video has rarely been attempted. In this study, we address this gap and show that deep learning architectures can successfully be applied to the problem of video-based detection of intake gestures. For this purpose, we collect and label video data of eating occasions using 360-degree video of 102 participants. Applying state-of-the-art approaches from video action recognition, our results show that (1) the best model achieves an F₁ score of 0.858, (2) appearance features contribute more than motion features, and (3) temporal context in form of multiple video frames is essential for top model performance.