"Automatic Ingestion Monitor Version 2" - A Novel Wearable Device for Automatic Food Intake Detection and Passive Capture of Food Images

Evaluating the Quality and Comparative Validity of Manual Food Logging and Artificial Intelligence-Enabled Food Image Recognition in Apps for Nutrition Care

For artificial intelligence (AI) to support nutrition care, high quality and accuracy of its features within smartphone applications (apps) are essential. This study evaluated popular apps' features, quality, behaviour change potential, and comparative validity of dietary assessment via manual logging and AI. The top 200 free and paid nutrition-related apps from Australia's Apple App and Google Play stores were screened (n = 800). Apps were assessed using MARS (quality) and ABACUS (behaviour change potential). Nutritional outputs from manual food logging and AI-enabled food-image recognition apps were compared with food records for Western, Asian, and Recommended diets. Among 18 apps, Noom scored highest on MARS (mean = 4.44) and ABACUS (21/21). From 16 manual food-logging apps, energy was overestimated for Western (mean: 1040 kJ) but underestimated for Asian (mean: -1520 kJ) diets. MyFitnessPal and Fastic had the highest accuracy (97% and 92%, respectively) out of seven AI-enabled food image recognition apps. Apps with more AI integration demonstrated better functionality, but automatic energy estimations from AI-enabled food image recognition were inaccurate. To enhance the integration of apps into nutrition care, collaborating with dietitians is essential for improving their credibility and comparative validity by expanding food databases. Moreover, training AI models are needed to improve AI-enabled food recognition, especially for mixed dishes and culturally diverse foods.

"How difficult it is to change dietary behaviour" experience of older people with sarcopenic obesity: a qualitative study

BACKGROUND

Dietary intervention is an important method to manage sarcopenic obesity, but the implementation in real world is difficult to achieve an ideal condition. This study aimed to the experiences of older people with sarcopenic obesity during the implementation of dietary behavioural change (DBC) intervention.

METHODS

This study is a semi-structured individual interview embedded within a pilot randomized controlled trial on community-dwelling older people with sarcopenic obesity. Purposive sampling was applied to invite 21 participants who had received a 15-week DBC intervention. The interviews were audio-recorded and transcribed verbatim. Content analysis was performed to analyze the data.

RESULTS

The themes for facilitators included: (a) Attach importance to self's health; (b) Family's support; (c) Concern self's body shape; (d) Instructor's support; (e) Regular food diary taken. The themes for barriers included: (a) Difficulties of taking food diary; (b) Difficulties of calculating the food amount; (c) Yield to offspring's appetite; (d) Misjudging self's or family's appetite.

CONCLUSION

Support from family members and instructor, caring about self's health and body image facilitated the intervention implementation. The complication of food amount estimation and diary record, personal sacrifice for next generations, and previous living experience were barriers for implementing the intervention. Overall, the older people with sarcopenic obesity can accept the design of DBC intervention program and have great willing to join.

sEMG-based automatic characterization of swallowed materials

Monitoring of ingestive activities is critically important for managing the health and wellness of individuals with various health conditions, including the elderly, diabetics, and individuals seeking better weight control. Monitoring swallowing events can be an ideal surrogate for developing streamlined methods for effective monitoring and quantification of eating or drinking events. Swallowing is an essential process for maintaining life. This seemingly simple process is the result of coordinated actions of several muscles and nerves in a complex fashion. In this study, we introduce automated methods for the detection and quantification of various eating and drinking activities. Wireless surface electromyography (sEMG) was used to detect chewing and swallowing from sEMG signals obtained from the sternocleidomastoid muscle, in addition to signals obtained from a wrist-mounted IMU sensor. A total of 4675 swallows were collected from 55 participants in the study. Multiple methods were employed to estimate bolus volumes in the case of fluid intake, including regression and classification models. Among the tested models, neural networks-based regression achieved an R2 of 0.88 and a root mean squared error of 0.2 (minimum bolus volume was 10 ml). Convolutional neural networks-based classification (when considering each bolus volume as a separate class) achieved an accuracy of over 99% using random cross-validation and around 66% using cross-subject validation. Multiple classification methods were also used for solid bolus type detection, including SVM and decision trees (DT), which achieved an accuracy above 99% with random validation and above 94% in cross-subject validation. Finally, regression models with both random and cross-subject validation were used for estimating the solid bolus volume with an R2 value that approached 1 and root mean squared error values as low as 0.00037 (minimum solid bolus weight was 3 gm). These reported results lay the foundation for a cost-effective and non-invasive method for monitoring swallowing activities which can be extremely beneficial in managing various chronic health conditions, such as diabetes and obesity.

A CNN Model for Physical Activity Recognition and Energy Expenditure Estimation from an Eyeglass-Mounted Wearable Sensor

Metabolic syndrome poses a significant health challenge worldwide, prompting the need for comprehensive strategies integrating physical activity monitoring and energy expenditure. Wearable sensor devices have been used both for energy intake and energy expenditure (EE) estimation. Traditionally, sensors are attached to the hip or wrist. The primary aim of this research is to investigate the use of an eyeglass-mounted wearable energy intake sensor (Automatic Ingestion Monitor v2, AIM-2) for simultaneous recognition of physical activity (PAR) and estimation of steady-state EE as compared to a traditional hip-worn device. Study data were collected from six participants performing six structured activities, with the reference EE measured using indirect calorimetry (COSMED K5) and reported as metabolic equivalents of tasks (METs). Next, a novel deep convolutional neural network-based multitasking model (Multitasking-CNN) was developed for PAR and EE estimation. The Multitasking-CNN was trained with a two-step progressive training approach for higher accuracy, where in the first step the model for PAR was trained, and in the second step the model was fine-tuned for EE estimation. Finally, the performance of Multitasking-CNN on AIM-2 attached to eyeglasses was compared to the ActiGraph GT9X (AG) attached to the right hip. On the AIM-2 data, Multitasking-CNN achieved a maximum of 95% testing accuracy of PAR, a minimum of 0.59 METs mean square error (MSE), and 11% mean absolute percentage error (MAPE) in EE estimation. Conversely, on AG data, the Multitasking-CNN model achieved a maximum of 82% testing accuracy in PAR, a minimum of 0.73 METs MSE, and 13% MAPE in EE estimation. These results suggest the feasibility of using an eyeglass-mounted sensor for both PAR and EE estimation.

Transforming Big Data into AI-ready data for nutrition and obesity research

OBJECTIVE

Big Data are increasingly used in obesity and nutrition research to gain new insights and derive personalized guidance; however, this data in raw form are often not usable. Substantial preprocessing, which requires machine learning (ML), human judgment, and specialized software, is required to transform Big Data into artificial intelligence (AI)- and ML-ready data. These preprocessing steps are the most complex part of the entire modeling pipeline. Understanding the complexity of these steps by the end user is critical for reducing misunderstanding, faulty interpretation, and erroneous downstream conclusions.

METHODS

We reviewed three popular obesity/nutrition Big Data sources: microbiome, metabolomics, and accelerometry. The preprocessing pipelines, specialized software, challenges, and how decisions impact final AI- and ML-ready products were detailed.

RESULTS

Opportunities for advances to improve quality control, speed of preprocessing, and intelligent end user consumption were presented.

CONCLUSIONS

Big Data have the exciting potential for identifying new modifiable factors that impact obesity research. However, to ensure accurate interpretation of conclusions arising from Big Data, the choices involved in preparing AI- and ML-ready data need to be transparent to investigators and clinicians relying on the conclusions.

Contextualizing remote fall risk: Video data capture and implementing ethical AI

Wearable inertial measurement units (IMUs) are being used to quantify gait characteristics that are associated with increased fall risk, but the current limitation is the lack of contextual information that would clarify IMU data. Use of wearable video-based cameras would provide a comprehensive understanding of an individual's habitual fall risk, adding context to clarify abnormal IMU data. Generally, there is taboo when suggesting the use of wearable cameras to capture real-world video, clinical and patient apprehension due to ethical and privacy concerns. This perspective proposes that routine use of wearable cameras could be realized within digital medicine through AI-based computer vision models to obfuscate/blur/shade sensitive information while preserving helpful contextual information for a comprehensive patient assessment. Specifically, no person sees the raw video data to understand context, rather AI interprets the raw video data first to blur sensitive objects and uphold privacy. That may be more routinely achieved than one imagines as contemporary resources exist. Here, to showcase/display the potential an exemplar model is suggested via off-the-shelf methods to detect and blur sensitive objects (e.g., people) with an accuracy of 88%. Here, the benefit of the proposed approach includes a more comprehensive understanding of an individual's free-living fall risk (from free-living IMU-based gait) without compromising privacy. More generally, the video and AI approach could be used beyond fall risk to better inform habitual experiences and challenges across a range of clinical cohorts. Medicine is becoming more receptive to wearables as a helpful toolbox, camera-based devices should be plausible instruments.

Detecting Eating Episodes From Wrist Motion Using Daily Pattern Analysis

This paper presents new methods to detect eating from wrist motion. Our main novelty is that we analyze a full day of wrist motion data as a single sample so that the detection of eating occurrences can benefit from diurnal context. We develop a two-stage framework to facilitate a feasible full-day analysis. The first-stage model calculates local probabilities of eating P(Ew) within windows of data, and the second-stage model calculates enhanced probabilities of eating P(Ed) by treating all P(Ew) within a single day as one sample. The framework also incorporates an augmentation technique, which involves the iterative retraining of the first-stage model. This allows us to generate a sufficient number of day-length samples from datasets of limited size. We test our methods on the publicly available Clemson All-Day (CAD) dataset and FreeFIC dataset, and find that the inclusion of day-length analysis substantially improves accuracy in detecting eating episodes. We also benchmark our results against several state-of-the-art methods. Our approach achieved an eating episode true positive rate (TPR) of 89% with 1.4 false positives per true positive (FP/TP), and a time weighted accuracy of 84%, which are the highest accuracies reported on the CAD dataset. Our results show that the daily pattern classifier substantially improves meal detections and in particular reduces transient false detections that tend to occur when relying on shorter windows to look for individual ingestion or consumption events.

Eat-Radar: Continuous Fine-Grained Intake Gesture Detection Using FMCW Radar and 3D Temporal Convolutional Network With Attention

Unhealthy dietary habits are considered as the primary cause of various chronic diseases, including obesity and diabetes. The automatic food intake monitoring system has the potential to improve the quality of life (QoL) of people with diet-related diseases through dietary assessment. In this work, we propose a novel contactless radar-based approach for food intake monitoring. Specifically, a Frequency Modulated Continuous Wave (FMCW) radar sensor is employed to recognize fine-grained eating and drinking gestures. The fine-grained eating/drinking gesture contains a series of movements from raising the hand to the mouth until putting away the hand from the mouth. A 3D temporal convolutional network with self-attention (3D-TCN-Att) is developed to detect and segment eating and drinking gestures in meal sessions by processing the Range-Doppler Cube (RD Cube). Unlike previous radar-based research, this work collects data in continuous meal sessions (more realistic scenarios). We create a public dataset comprising 70 meal sessions (4,132 eating gestures and 893 drinking gestures) from 70 participants with a total duration of 1,155 minutes. Four eating styles (fork & knife, chopsticks, spoon, hand) are included in this dataset. To validate the performance of the proposed approach, seven-fold cross-validation method is applied. The 3D-TCN-Att model achieves a segmental F1-score of 0.896 and 0.868 for eating and drinking gestures, respectively. The results of the proposed approach indicate the feasibility of using radar for fine-grained eating and drinking gesture detection and segmentation in meal sessions.

Integrated image and sensor-based food intake detection in free-living

The first step in any dietary monitoring system is the automatic detection of eating episodes. To detect eating episodes, either sensor data or images can be used, and either method can result in false-positive detection. This study aims to reduce the number of false positives in the detection of eating episodes by a wearable sensor, Automatic Ingestion Monitor v2 (AIM-2). Thirty participants wore the AIM-2 for two days each (pseudo-free-living and free-living). The eating episodes were detected by three methods: (1) recognition of solid foods and beverages in images captured by AIM-2; (2) recognition of chewing from the AIM-2 accelerometer sensor; and (3) hierarchical classification to combine confidence scores from image and accelerometer classifiers. The integration of image- and sensor-based methods achieved 94.59% sensitivity, 70.47% precision, and 80.77% F1-score in the free-living environment, which is significantly better than either of the original methods (8% higher sensitivity). The proposed method successfully reduces the number of false positives in the detection of eating episodes.

Estimating Amount of Food in a Circular Dining Bowl from a Single Image

Unhealthy diet is a top risk factor causing obesity and numerous chronic diseases. To help the public adopt healthy diet, nutrition scientists need user-friendly tools to conduct Dietary Assessment (DA). In recent years, new DA tools have been developed using a smartphone or a wearable device which acquires images during a meal. These images are then processed to estimate calories and nutrients of the consumed food. Although considerable progress has been made, 2D food images lack scale reference and 3D volumetric information. In addition, food must be sufficiently observable from the image. This basic condition can be met when the food is stand-alone (no food container is used) or it is contained in a shallow plate. However, the condition cannot be met easily when a bowl is used. The food is often occluded by the bowl edge, and the shape of the bowl may not be fully determined from the image. However, bowls are the most utilized food containers by billions of people in many parts of the world, especially in Asia and Africa. In this work, we propose to premeasure plates and bowls using a marked adhesive strip before a dietary study starts. This simple procedure eliminates the use of a scale reference throughout the DA study. In addition, we use mathematical models and image processing to reconstruct the bowl in 3D. Our key idea is to estimate how full the bowl is rather than how much food is (in either volume or weight) in the bowl. This idea reduces the effect of occlusion. The experimental data have shown satisfactory results of our methods which enable accurate DA studies using both plates and bowls with reduced burden on research participants.

Evaluation of Acceptability, Functionality, and Validity of a Passive Image-Based Dietary Intake Assessment Method in Adults and Children of Ghanaian and Kenyan Origin Living in London, UK

BACKGROUND

Accurate estimation of dietary intake is challenging. However, whilst some progress has been made in high-income countries, low- and middle-income countries (LMICs) remain behind, contributing to critical nutritional data gaps. This study aimed to validate an objective, passive image-based dietary intake assessment method against weighed food records in London, UK, for onward deployment to LMICs.

METHODS

Wearable camera devices were used to capture food intake on eating occasions in 18 adults and 17 children of Ghanaian and Kenyan origin living in London. Participants were provided pre-weighed meals of Ghanaian and Kenyan cuisine and camera devices to automatically capture images of the eating occasions. Food images were assessed for portion size, energy, nutrient intake, and the relative validity of the method compared to the weighed food records.

RESULTS

The Pearson and Intraclass correlation coefficients of estimates of intakes of food, energy, and 19 nutrients ranged from 0.60 to 0.95 and 0.67 to 0.90, respectively. Bland-Altman analysis showed good agreement between the image-based method and the weighed food record. Under-estimation of dietary intake by the image-based method ranged from 4 to 23%.

CONCLUSIONS

Passive food image capture and analysis provides an objective assessment of dietary intake comparable to weighed food records.

Technology to Automatically Record Eating Behavior in Real Life: A Systematic Review

To monitor adherence to diets and to design and evaluate nutritional interventions, it is essential to obtain objective knowledge about eating behavior. In most research, measures of eating behavior are based on self-reporting, such as 24-h recalls, food records (food diaries) and food frequency questionnaires. Self-reporting is prone to inaccuracies due to inaccurate and subjective recall and other biases. Recording behavior using nonobtrusive technology in daily life would overcome this. Here, we provide an up-to-date systematic overview encompassing all (close-to) publicly or commercially available technologies to automatically record eating behavior in real-life settings. A total of 1328 studies were screened and, after applying defined inclusion and exclusion criteria, 122 studies were included for in-depth evaluation. Technologies in these studies were categorized by what type of eating behavior they measure and which type of sensor technology they use. In general, we found that relatively simple sensors are often used. Depending on the purpose, these are mainly motion sensors, microphones, weight sensors and photo cameras. While several of these technologies are commercially available, there is still a lack of publicly available algorithms that are needed to process and interpret the resulting data. We argue that future work should focus on developing robust algorithms and validating these technologies in real-life settings. Combining technologies (e.g., prompting individuals for self-reports at sensed, opportune moments) is a promising route toward ecologically valid studies of eating behavior.

I2N: image to nutrients, a sensor guided semi-automated tool for annotation of images for nutrition analysis of eating episodes

Introduction

Dietary assessment is important for understanding nutritional status. Traditional methods of monitoring food intake through self-report such as diet diaries, 24-hour dietary recall, and food frequency questionnaires may be subject to errors and can be time-consuming for the user.

Methods

This paper presents a semi-automatic dietary assessment tool we developed - a desktop application called Image to Nutrients (I2N) - to process sensor-detected eating events and images captured during these eating events by a wearable sensor. I2N has the capacity to offer multiple food and nutrient databases (e.g., USDA-SR, FNDDS, USDA Global Branded Food Products Database) for annotating eating episodes and food items. I2N estimates energy intake, nutritional content, and the amount consumed. The components of I2N are three-fold: 1) sensor-guided image review, 2) annotation of food images for nutritional analysis, and 3) access to multiple food databases. Two studies were used to evaluate the feasibility and usefulness of I2N: 1) a US-based study with 30 participants and a total of 60 days of data and 2) a Ghana-based study with 41 participants and a total of 41 days of data).

Results

In both studies, a total of 314 eating episodes were annotated using at least three food databases. Using I2N's sensor-guided image review, the number of images that needed to be reviewed was reduced by 93% and 85% for the two studies, respectively, compared to reviewing all the images.

Discussion

I2N is a unique tool that allows for simultaneous viewing of food images, sensor-guided image review, and access to multiple databases in one tool, making nutritional analysis of food images efficient. The tool is flexible, allowing for nutritional analysis of images if sensor signals aren't available.

Vision-Based Methods for Food and Fluid Intake Monitoring: A Literature Review

Food and fluid intake monitoring are essential for reducing the risk of dehydration, malnutrition, and obesity. The existing research has been preponderantly focused on dietary monitoring, while fluid intake monitoring, on the other hand, is often neglected. Food and fluid intake monitoring can be based on wearable sensors, environmental sensors, smart containers, and the collaborative use of multiple sensors. Vision-based intake monitoring methods have been widely exploited with the development of visual devices and computer vision algorithms. Vision-based methods provide non-intrusive solutions for monitoring. They have shown promising performance in food/beverage recognition and segmentation, human intake action detection and classification, and food volume/fluid amount estimation. However, occlusion, privacy, computational efficiency, and practicality pose significant challenges. This paper reviews the existing work (253 articles) on vision-based intake (food and fluid) monitoring methods to assess the size and scope of the available literature and identify the current challenges and research gaps. This paper uses tables and graphs to depict the patterns of device selection, viewing angle, tasks, algorithms, experimental settings, and performance of the existing monitoring systems.

The spectrum of eating environments encountered in free living adults documented using a passive capture food intake wearable device

Introduction

The aim of this feasibility and proof-of-concept study was to examine the use of a novel wearable device for automatic food intake detection to capture the full range of free-living eating environments of adults with overweight and obesity. In this paper, we document eating environments of individuals that have not been thoroughly described previously in nutrition software as current practices rely on participant self-report and methods with limited eating environment options.

Methods

Data from 25 participants and 116 total days (7 men, 18 women, Mage = 44 ± 12 years, BMI 34.3 ± 5.2 kg/mm2), who wore the passive capture device for at least 7 consecutive days (≥12h waking hours/d) were analyzed. Data were analyzed at the participant level and stratified amongst meal type into breakfast, lunch, dinner, and snack categories. Out of 116 days, 68.1% included breakfast, 71.5% included lunch, 82.8% included dinner, and 86.2% included at least one snack.

Results

The most prevalent eating environment among all eating occasions was at home and with one or more screens in use (breakfast: 48.1%, lunch: 42.2%, dinner: 50%, and snacks: 55%), eating alone (breakfast: 75.9%, lunch: 89.2%, dinner: 74.3%, snacks: 74.3%), in the dining room (breakfast: 36.7%, lunch: 30.1%, dinner: 45.8%) or living room (snacks: 28.0%), and in multiple locations (breakfast: 44.3%, lunch: 28.8%, dinner: 44.8%, snacks: 41.3%).

Discussion

Results suggest a passive capture device can provide accurate detection of food intake in multiple eating environments. To our knowledge, this is the first study to classify eating occasions in multiple eating environments and may be a useful tool for future behavioral research studies to accurately codify eating environments.

Capturing children food exposure using wearable cameras and deep learning

Children's dietary habits are influenced by complex factors within their home, school and neighborhood environments. Identifying such influencers and assessing their effects is traditionally based on self-reported data which can be prone to recall bias. We developed a culturally acceptable machine-learning-based data-collection system to objectively capture school-children's exposure to food (including food items, food advertisements, and food outlets) in two urban Arab centers: Greater Beirut, in Lebanon, and Greater Tunis, in Tunisia. Our machine-learning-based system consists of 1) a wearable camera that captures continuous footage of children's environment during a typical school day, 2) a machine learning model that automatically identifies images related to food from the collected data and discards any other footage, 3) a second machine learning model that classifies food-related images into images that contain actual food items, images that contain food advertisements, and images that contain food outlets, and 4) a third machine learning model that classifies images that contain food items into two classes, corresponding to whether the food items are being consumed by the child wearing the camera or whether they are consumed by others. This manuscript reports on a user-centered design study to assess the acceptability of using wearable cameras to capture food exposure among school children in Greater Beirut and Greater Tunis. We then describe how we trained our first machine learning model to detect food exposure images using data collected from the Web and utilizing the latest trends in deep learning for computer vision. Next, we describe how we trained our other machine learning models to classify food-related images into their respective categories using a combination of public data and data acquired via crowdsourcing. Finally, we describe how the different components of our system were packed together and deployed in a real-world case study and we report on its performance.

Estimation of Plate and Bowl Dimensions for Food Portion Size Assessment in a Wearable Sensor System

Automatic food portion size estimation (FPSE) with minimal user burden is a challenging task. Most of the existing FPSE methods use fiducial markers and/or virtual models as dimensional references. An alternative approach is to estimate the dimensions of the eating containers prior to estimating the portion size. In this article, we propose a wearable sensor system (the automatic ingestion monitor integrated with a ranging sensor) and a related method for the estimation of dimensions of plates and bowls. The contributions of this study are: 1) the model eliminates the need for fiducial markers; 2) the camera system [automatic ingestion monitor version 2 (AIM-2)] is not restricted in terms of positioning relative to the food item; 3) our model accounts for radial lens distortion caused due to lens aberrations; 4) a ranging sensor directly gives the distance between the sensor and the eating surface; 5) the model is not restricted to circular plates; and 6) the proposed system implements a passive method that can be used for assessment of container dimensions with minimum user interaction. The error rates (mean ± std. dev) for dimension estimation were 2.01% ± 4.10% for plate widths/diameters, 2.75% ± 38.11% for bowl heights, and 4.58% ± 6.78% for bowl diameters.

A Review of Digital Innovations for Diet Monitoring and Precision Nutrition

This article provides an up-to-date review of technological advances in 3 key areas related to diet monitoring and precision nutrition. First, we review developments in mobile applications, with a focus on food photography and artificial intelligence to facilitate the process of diet monitoring. Second, we review advances in 2 types of wearable and handheld sensors that can potentially be used to fully automate certain aspects of diet logging: physical sensors to detect moments of dietary intake, and chemical sensors to estimate the composition of diets and meals. Finally, we review new programs that can generate personalized/precision nutrition recommendations based on measurements of gut microbiota and continuous glucose monitors with artificial intelligence. The article concludes with a discussion of potential pitfalls of some of these technologies.

Machine learning modeling practices to support the principles of AI and ethics in nutrition research

BACKGROUND

Nutrition research is relying more on artificial intelligence and machine learning models to understand, diagnose, predict, and explain data. While artificial intelligence and machine learning models provide powerful modeling tools, failure to use careful and well-thought-out modeling processes can lead to misleading conclusions and concerns surrounding ethics and bias.

METHODS

Based on our experience as reviewers and journal editors in nutrition and obesity, we identified the most frequently omitted best practices from statistical modeling and how these same practices extend to machine learning models. We next addressed areas required for implementation of machine learning that are not included in commercial software packages.

RESULTS

Here, we provide a tutorial on best artificial intelligence and machine learning modeling practices that can reduce potential ethical problems with a checklist and guiding principles to aid nutrition researchers in developing, evaluating, and implementing artificial intelligence and machine learning models in nutrition research.

CONCLUSION

The quality of AI/ML modeling in nutrition research requires iterative and tailored processes to mitigate against potential ethical problems or to predict conclusions that are free of bias.

Thought on Food: A Systematic Review of Current Approaches and Challenges for Food Intake Detection

Nowadays, individuals have very stressful lifestyles, affecting their nutritional habits. In the early stages of life, teenagers begin to exhibit bad habits and inadequate nutrition. Likewise, other people with dementia, Alzheimer's disease, or other conditions may not take food or medicine regularly. Therefore, the ability to monitor could be beneficial for them and for the doctors that can analyze the patterns of eating habits and their correlation with overall health. Many sensors help accurately detect food intake episodes, including electrogastrography, cameras, microphones, and inertial sensors. Accurate detection may provide better control to enable healthy nutrition habits. This paper presents a systematic review of the use of technology for food intake detection, focusing on the different sensors and methodologies used. The search was performed with a Natural Language Processing (NLP) framework that helps screen irrelevant studies while following the PRISMA methodology. It automatically searched and filtered the research studies in different databases, including PubMed, Springer, ACM, IEEE Xplore, MDPI, and Elsevier. Then, the manual analysis selected 30 papers based on the results of the framework for further analysis, which support the interest in using sensors for food intake detection and nutrition assessment. The mainly used sensors are cameras, inertial, and acoustic sensors that handle the recognition of food intake episodes with artificial intelligence techniques. This research identifies the most used sensors and data processing methodologies to detect food intake.

Non-Invasive Screen Exposure Time Assessment Using Wearable Sensor and Object Detection

Cumulative screen exposure has been increased due to the explosion of digital technology ownership in the past decade for all people, including children who face exposure related risks such as obesity, eye problems, and disrupted sleep. Screen exposure is linked to physical and mental health risks among both children and adults. Current methods of screen exposure assessment have their limitations, mostly in the prospective of objectiveness, robustness, and invasiveness. In this paper, we propose a novel method to measure screen exposure time using a wearable sensor and computer vision technology. We use a customized, lightweight, wearable senor to capture egocentric images and use deep learning-based object detection module to identify the existence of electronic screens. The duration of screen exposure is further estimated using post-processing technology to filter consecutive frames regarding to the screen usage. Our method is non-invasive and robust, providing an objective and accurate means to screen exposure measurement. We conduct experiments on various environments to identify the existence of three types of screens and duration of screen exposure. The experimental results demonstrate the feasibility of automatically assessing screen time exposure and great potential to be applied in large scale experiments for behavioral study.

A Comparative Study of Deep Learning Algorithms for Detecting Food Intake

The choice of appropriate machine learning algorithms is crucial for classification problems. This study compares the performance of state-of-the-art time-series deep learning algorithms for classifying food intake using sensor signals. The sensor signals were collected with the help of a wearable sensor system (the Automatic Ingestion Monitor v2, or AIM-2). AIM-2 used an optical and 3-axis accelerometer sensor to capture temporalis muscle activation. Raw signals from those sensors were used to train five classifiers (multilayer perceptron (MLP), time Convolutional Neural Network (time-CNN), Fully Convolutional Neural Network (FCN), Residual Neural Network (ResNet), and Inception network) to differentiate food intake (eating and drinking) from other activities. Data were collected from 17 pilot subjects over the course of 23 days in free-living conditions. A leave one subject out cross-validation scheme was used for training and testing. Time-CNN, FCN, ResNet, and Inception achieved average balanced classification accuracy of 88.84%, 90.18%, 93.47%, and 92.15%, respectively. The results indicate that ResNet outperforms other state-of-the-art deep learning algorithms for this specific problem.

Improvement of Methodology for Manual Energy Intake Estimation From Passive Capture Devices

Objective

To describe best practices for manual nutritional analyses of data from passive capture wearable devices in free-living conditions.

Method

18 participants (10 female) with a mean age of 45 ± 10 years and mean BMI of 34.2 ± 4.6 kg/m² consumed usual diet for 3 days in a free-living environment while wearing an automated passive capture device. This wearable device facilitates capture of images without manual input from the user. Data from the first nine participants were used by two trained nutritionists to identify sources contributing to inter-nutritionist variance in nutritional analyses. The nutritionists implemented best practices to mitigate these sources of variance in the next nine participants. The three best practices to reduce variance in analysis of energy intake (EI) estimation were: (1) a priori standardized food selection, (2) standardized nutrient database selection, and (3) increased number of images captured around eating episodes.

Results

Inter-rater repeatability for EI, using intraclass correlation coefficient (ICC), improved by 0.39 from pre-best practices to post-best practices (0.14 vs 0.85, 95% CI, respectively), Bland–Altman analysis indicated strongly improved agreement between nutritionists for limits of agreement (LOA) post-best practices.

Conclusion

Significant improvement of ICC and LOA for estimation of EI following implementation of best practices demonstrates that these practices improve the reproducibility of dietary analysis from passive capture device images in free-living environments.

Enhancing Nutrition Care Through Real-Time, Sensor-Based Capture of Eating Occasions: A Scoping Review

As food intake patterns become less structured, different methods of dietary assessment may be required to capture frequently omitted snacks, smaller meals, and the time of day when they are consumed. Incorporating sensors that passively and objectively detect eating behavior may assist in capturing these eating occasions into dietary assessment methods. The aim of this study was to identify and collate sensor-based technologies that are feasible for dietitians to use to assist with performing dietary assessments in real-world practice settings. A scoping review was conducted using the PRISMA extension for scoping reviews (PRISMA-ScR) framework. Studies were included if they were published between January 2016 and December 2021 and evaluated the performance of sensor-based devices for identifying and recording the time of food intake. Devices from included studies were further evaluated against a set of feasibility criteria to determine whether they could potentially be used to assist dietitians in conducting dietary assessments. The feasibility criteria were, in brief, consisting of an accuracy ≥80%; tested in settings where subjects were free to choose their own foods and activities; social acceptability and comfort; a long battery life; and a relatively rapid detection of an eating episode. Fifty-four studies describing 53 unique devices and 4 device combinations worn on the wrist (n = 18), head (n = 16), neck (n = 9), and other locations (n = 14) were included. Whilst none of the devices strictly met all feasibility criteria currently, continuous refinement and testing of device software and hardware are likely given the rapidly changing nature of this emerging field. The main reasons devices failed to meet the feasibility criteria were: an insufficient or lack of reporting on battery life (91%), the use of a limited number of foods and behaviors to evaluate device performance (63%), and the device being socially unacceptable or uncomfortable to wear for long durations (46%). Until sensor-based dietary assessment tools have been designed into more inconspicuous prototypes and are able to detect most food and beverage consumption throughout the day, their use will not be feasible for dietitians in practice settings.

Top-Down Detection of Eating Episodes by Analyzing Large Windows of Wrist Motion Using a Convolutional Neural Network

In this work, we describe a new method to detect periods of eating by tracking wrist motion during everyday life. Eating uses hand-to-mouth gestures for ingestion, each of which lasts a few seconds. Previous works have detected these gestures individually and then aggregated them to identify meals. The novelty of our approach is that we analyze a much longer window (0.5–15 min) using a convolutional neural network. Longer windows can contain other gestures related to eating, such as cutting or manipulating food, preparing foods for consumption, and resting between ingestion events. The context of these other gestures can improve the detection of periods of eating. We test our methods on the public Clemson all-day dataset, which consists of 354 recordings containing 1063 eating episodes. We found that accuracy at detecting eating increased by 15% in ≥4 min windows compared to ≤15 s windows. Using a 6 min window, we detected 89% of eating episodes, with 1.7 false positives for every true positive (FP/TP). These are the best results achieved to date on this dataset.

Perspective: Opportunities and Challenges of Technology Tools in Dietary and Activity Assessment: Bridging Stakeholder Viewpoints

The science and tools of measuring energy intake and output in humans have rapidly advanced in the last decade. Engineered devices such as wearables and sensors, software applications, and Web-based tools are now ubiquitous in both research and consumer environments. The assessment of energy expenditure in particular has progressed from reliance on self-report instruments to advanced technologies requiring collaboration across multiple disciplines, from optics to accelerometry. In contrast, assessing energy intake still heavily relies on self-report mechanisms. Although these tools have improved, moving from paper-based to online reporting, considerable room for refinement remains in existing tools, and great opportunities exist for novel, transformational tools, including those using spectroscopy and chemo-sensing. This report reviews the state of the science, and the opportunities and challenges in existing and emerging technologies, from the perspectives of 3 key stakeholders: researchers, users, and developers. Each stakeholder approaches these tools with unique requirements: researchers are concerned with validity, accuracy, data detail and abundance, and ethical use; users with ease of use and privacy; and developers with high adherence and utilization, intellectual property, licensing rights, and monetization. Cross-cutting concerns include frequent updating and integration of the food and nutrient databases on which assessments rely, improving accessibility and reducing disparities in use, and maintaining reliable technical assistance. These contextual challenges are discussed in terms of opportunities and further steps in the direction of personalized health.

Detection of Food Intake Sensor's Wear Compliance in Free-Living

Objective detection of periods of wear and non-wear is critical for human studies that rely on information from wearable sensors, such as food intake sensors. In this paper, we present a novel method of compliance detection on the example of the Automatic Ingestion Monitor v2 (AIM-2) sensor, containing a tri-axial accelerometer, a still camera, and a chewing sensor. The method was developed and validated using data from a study of 30 participants aged 18-39, each wearing the AIM-2 for two days (a day in pseudo-free-living and a day in free-living). Four types of wear compliance were analyzed: 'normal-wear', 'non-compliant-wear', 'non-wear-carried', and 'non-wear-stationary'. The ground truth of those four types of compliance was obtained by reviewing the images of the egocentric camera. The features for compliance detection were the standard deviation of acceleration, average pitch, and roll angles, and mean square error of two consecutive images. These were used to train three random forest classifiers 1) accelerometer-based, 2) image-based, and 3) combined accelerometer and image-based. Satisfactory wear compliance measurement accuracy was obtained using the combined classifier (89.24%) on leave one subject out cross-validation. The average duration of compliant wear in the study was 9h with a standard deviation of 2h or 70.96% of total on-time. This method can be used to calculate the wear and non-wear time of AIM-2, and potentially be extended to other devices. The study also included assessments of sensor burden and privacy concerns. The survey results suggest recommendations that may be used to increase wear compliance.

A Dietary Assessment Training Course Path: The Italian IV SCAI Study on Children Food Consumption

The eating patterns in a population can be estimated through dietary surveys in which open-ended assessment methods, such as diaries and interviews, or semi-quantitative food frequency questionnaires are administered. A harmonized dietary survey methodology, together with a standardized operational procedure, in conducting the study is crucial to ensure the comparability of the results and the accuracy of information, thus reducing uncertainty and increasing the reliability of the results. Dietary patterns (i) include several target variables (foods, energy and nutrients, other food components), (ii) require several explanatory variables (age, gender, anthropometric measurements, socio-cultural and economic characteristics, lifestyle, preferences, attitudes, beliefs, organization of food-related activities, etc.), and (iii) have impacts in several domains: imbalance diets; acute and chronic exposures affect health, specifically non-communicable diseases; and then sanitary expenditure. On the other hand, food demand has impacts on the food system: production, distribution, and food services system; food wastes and other wastes generated by food-related activities of the households (e.g., packaging disposal) have consequences on the “health of the planet” which in turn can have effects on human health. Harmonization and standardization of measurement methods and procedures in such a complex context require an ad hoc structured information system made by databases (food nomenclatures, portion sizes, food atlas, recipes) and methodological tools (quantification methods, food coding systems, assessment of nutritional status, data processing to extrapolate what we consider validated dietary data). Establishing a community of professionals specialized in dietary data management could lead to build a surveillance system for monitoring eating habits in the short term, thus reducing costs, and to arrange a training re-training system. Creating and maintaining the dietary data managers community is challenging but possible. In this context, the cooperation between the CREA Research Centre for Food and Nutrition and the Italian National Health Institute (ISS) promoted and supported by the Italian Ministry of Health may represent a model of best practice that can ensure a continuous training for the professional community carrying out a nutritional study.