Unobtrusive and Wearable Systems for Automatic Dietary Monitoring

AI4FoodDB: a database for personalized e-Health nutrition and lifestyle through wearable devices and artificial intelligence

The increasing prevalence of diet-related diseases calls for an improvement in nutritional advice. Personalized nutrition aims to solve this problem by adapting dietary and lifestyle guidelines to the unique circumstances of each individual. With the latest advances in technology and data science, researchers can now automatically collect and analyze large amounts of data from a variety of sources, including wearable and smart devices. By combining these diverse data, more comprehensive insights of the human body and its diseases can be achieved. However, there are still major challenges to overcome, including the need for more robust data and standardization of methodologies for better subject monitoring and assessment. Here, we present the AI4Food database (AI4FoodDB), which gathers data from a nutritional weight loss intervention monitoring 100 overweight and obese participants during 1 month. Data acquisition involved manual traditional approaches, novel digital methods and the collection of biological samples, obtaining: (i) biological samples at the beginning and the end of the intervention, (ii) anthropometric measurements every 2 weeks, (iii) lifestyle and nutritional questionnaires at two different time points and (iv) continuous digital measurements for 2 weeks. To the best of our knowledge, AI4FoodDB is the first public database that centralizes food images, wearable sensors, validated questionnaires and biological samples from the same intervention. AI4FoodDB thus has immense potential for fostering the advancement of automatic and novel artificial intelligence techniques in the field of personalized care. Moreover, the collected information will yield valuable insights into the relationships between different variables and health outcomes, allowing researchers to generate and test new hypotheses, identify novel biomarkers and digital endpoints, and explore how different lifestyle, biological and digital factors impact health. The aim of this article is to describe the datasets included in AI4FoodDB and to outline the potential that they hold for precision health research. Database URL https://github.com/AI4Food/AI4FoodDB.

Automated Diet Capture Using Voice Alerts and Speech Recognition on Smartphones: Pilot Usability and Acceptability Study

BACKGROUND

Effective monitoring of dietary habits is critical for promoting healthy lifestyles and preventing or delaying the onset and progression of diet-related diseases, such as type 2 diabetes. Recent advances in speech recognition technologies and natural language processing present new possibilities for automated diet capture; however, further exploration is necessary to assess the usability and acceptability of such technologies for diet logging.

OBJECTIVE

This study explores the usability and acceptability of speech recognition technologies and natural language processing for automated diet logging.

METHODS

We designed and developed base2Diet-an iOS smartphone application that prompts users to log their food intake using voice or text. To compare the effectiveness of the 2 diet logging modes, we conducted a 28-day pilot study with 2 arms and 2 phases. A total of 18 participants were included in the study, with 9 participants in each arm (text: n=9, voice: n=9). During phase I of the study, all 18 participants received reminders for breakfast, lunch, and dinner at preselected times. At the beginning of phase II, all participants were given the option to choose 3 times during the day to receive 3 times daily reminders to log their food intake for the remainder of the phase, with the ability to modify the selected times at any point before the end of the study.

RESULTS

The total number of distinct diet logging events per participant was 1.7 times higher in the voice arm than in the text arm (P=.03, unpaired t test). Similarly, the total number of active days per participant was 1.5 times higher in the voice arm than in the text arm (P=.04, unpaired t test). Furthermore, the text arm had a higher attrition rate than the voice arm, with only 1 participant dropping out of the study in the voice arm, while 5 participants dropped out in the text arm.

CONCLUSIONS

The results of this pilot study demonstrate the potential of voice technologies in automated diet capturing using smartphones. Our findings suggest that voice-based diet logging is more effective and better received by users compared to traditional text-based methods, underscoring the need for further research in this area. These insights carry significant implications for the development of more effective and accessible tools for monitoring dietary habits and promoting healthy lifestyle choices.

A Conceptual Model for Mobile Health-enabled Slow Eating Strategies

Ingestive behaviors (IBs) (eg, bites, chews, oral processing, swallows, pauses) have meaningful roles in enhancing satiety, promoting fullness, and decreasing food consumption, and thus may be an underused strategy for obesity prevention and treatment. Limited IB monitoring research has been conducted because of a lack of accurate automated measurement capabilities outside laboratory settings. Self-report methods are used, but they have questionable validity and reliability. This paper aimed to present a conceptual model in which IB, specifically slow eating, supported by technological advancements, contributes to controlling hedonic and homeostatic processes, providing an opportunity to reduce energy intake, and improve health outcomes.

Perspective: Opportunities and Challenges of Technology Tools in Dietary and Activity Assessment: Bridging Stakeholder Viewpoints

The science and tools of measuring energy intake and output in humans have rapidly advanced in the last decade. Engineered devices such as wearables and sensors, software applications, and Web-based tools are now ubiquitous in both research and consumer environments. The assessment of energy expenditure in particular has progressed from reliance on self-report instruments to advanced technologies requiring collaboration across multiple disciplines, from optics to accelerometry. In contrast, assessing energy intake still heavily relies on self-report mechanisms. Although these tools have improved, moving from paper-based to online reporting, considerable room for refinement remains in existing tools, and great opportunities exist for novel, transformational tools, including those using spectroscopy and chemo-sensing. This report reviews the state of the science, and the opportunities and challenges in existing and emerging technologies, from the perspectives of 3 key stakeholders: researchers, users, and developers. Each stakeholder approaches these tools with unique requirements: researchers are concerned with validity, accuracy, data detail and abundance, and ethical use; users with ease of use and privacy; and developers with high adherence and utilization, intellectual property, licensing rights, and monetization. Cross-cutting concerns include frequent updating and integration of the food and nutrient databases on which assessments rely, improving accessibility and reducing disparities in use, and maintaining reliable technical assistance. These contextual challenges are discussed in terms of opportunities and further steps in the direction of personalized health.

Fluid Intake Monitoring Systems for the Elderly: A Review of the Literature

Fluid intake monitoring is an essential component in preventing dehydration and overhydration, especially for the senior population. Numerous critical health problems are associated with poor or excessive drinking such as swelling of the brain and heart failure. Real-time systems for monitoring fluid intake will not only measure the exact amount consumed by the users, but could also motivate people to maintain a healthy lifestyle by providing feedback to encourage them to hydrate regularly throughout the day. This paper reviews the most recent solutions to automatic fluid intake monitoring both commercially and in the literature. The available technologies are divided into four categories: wearables, surfaces with embedded sensors, vision- and environmental-based solutions, and smart containers. A detailed performance evaluation was carried out considering detection accuracy, usability and availability. It was observed that the most promising results came from studies that used data fusion from multiple technologies, compared to using an individual technology. The areas that need further research and the challenges for each category are discussed in detail.

Wearable sensors for continuous oral cavity and dietary monitoring toward personalized healthcare and digital medicine

Oral monitoring plays an essential role in preventing and diagnosing systemic diseases through saliva in the mouth. Dietary monitoring is also crucial to reduce the likelihood of chronic diseases such as hypertension and diabetes by analyzing food types, amounts and diet patterns. Therefore, the oral cavity and dietary monitoring are vital for accurate personalized healthcare management that can improve healthcare. To perform continuous oral cavity and dietary monitoring in real-time, a wearable sensing system capable of continuous analysis is necessary. In this review, we classify chemical and physical biosensing methods and summarize recent progress in wearable sensor development for oral cavity and dietary monitoring for personalized healthcare and digital medicine. We also discuss future perspectives and the potential of wearable sensors to provide robust data for food-intake monitoring and the saliva analysis of super-aged/aging societies, non-face-to-face social life, and global pandemic disease issues. We believe that this review will result in a paradigm shift toward personalized healthcare and digital medicine using wearable sensors through the analysis of massively parallel healthcare data.

Deep Learning-Based Multimodal Data Fusion: Case Study in Food Intake Episodes Detection Using Wearable Sensors

BACKGROUND

Multimodal wearable technologies have brought forward wide possibilities in human activity recognition, and more specifically personalized monitoring of eating habits. The emerging challenge now is the selection of most discriminative information from high-dimensional data collected from multiple sources. The available fusion algorithms with their complex structure are poorly adopted to the computationally constrained environment which requires integrating information directly at the source. As a result, more simple low-level fusion methods are needed.

OBJECTIVE

In the absence of a data combining process, the cost of directly applying high-dimensional raw data to a deep classifier would be computationally expensive with regard to the response time, energy consumption, and memory requirement. Taking this into account, we aimed to develop a data fusion technique in a computationally efficient way to achieve a more comprehensive insight of human activity dynamics in a lower dimension. The major objective was considering statistical dependency of multisensory data and exploring intermodality correlation patterns for different activities.

METHODS

In this technique, the information in time (regardless of the number of sources) is transformed into a 2D space that facilitates classification of eating episodes from others. This is based on a hypothesis that data captured by various sensors are statistically associated with each other and the covariance matrix of all these signals has a unique distribution correlated with each activity which can be encoded on a contour representation. These representations are then used as input of a deep model to learn specific patterns associated with specific activity.

RESULTS

In order to show the generalizability of the proposed fusion algorithm, 2 different scenarios were taken into account. These scenarios were different in terms of temporal segment size, type of activity, wearable device, subjects, and deep learning architecture. The first scenario used a data set in which a single participant performed a limited number of activities while wearing the Empatica E4 wristband. In the second scenario, a data set related to the activities of daily living was used where 10 different participants wore inertial measurement units while performing a more complex set of activities. The precision metric obtained from leave-one-subject-out cross-validation for the second scenario reached 0.803. The impact of missing data on performance degradation was also evaluated.

CONCLUSIONS

To conclude, the proposed fusion technique provides the possibility of embedding joint variability information over different modalities in just a single 2D representation which results in obtaining a more global view of different aspects of daily human activities at hand, and yet preserving the desired performance level in activity recognition.

"Everyone can take photos." Feasibility and relative validity of phone photography-based assessment of children's diets - a mixed methods study

Background

Dietary assessment methods that are user-friendly, simple, yet valid are of interest to both researchers and participants, particularly for use in disadvantaged settings, where language barriers and low levels of education are often present. We tested if parents taking photos of what children ate, using mobile phones, would be a feasible, acceptable method that could still provide information with adequate relative validity.

Methods

We used a mixed-methods design, with parents of 21 5- to 7-year-olds from disadvantaged areas in Sweden. Parents reported all dietary intake, during non-school hours, on three days (two weekdays) using a photo method (PM). The PM consisted of simple instructions and a fiduciary card, but no training, equipment or software. Text messages could be sent if necessary. As a reference method, parents completed three 24-h recalls (24HRs) with an interviewer each following day. The next week, parents completed a 9-item semi-FFQ regarding the preceding week. The outcomes were intakes (in dl) of 9 food groups, categorised as fruits and vegetables, energy-dense sweet/salty foods, and sweet drinks. Agreement with the reference 24HRs was assessed using correlations, median differences and Bland-Altman plots. Parents completed an open-ended questionnaire on barriers and facilitators. Data collectors provided complementary information. Qualitative data was analysed using qualitative manifest analysis.

Results

Nineteen parents (90%) provided complete data. The majority (n = 13) spoke Swedish as a second language, few (n = 4) were proficient. Compared to 24HRs, intakes measured by PM correlated well for all categories (Spearman’s rho = 0.609–0.845). However, intakes were underreported, significantly so for fruits and vegetables; Bland-Altman plots indicated that the underestimation was fairly constant across intake levels. When the FFQ was compared to the 24HRs, parameters of agreement were generally inferior than for the PM. Parents found the PM a positive experience, primarily facilitated by its simplicity and familiarity. Barriers, mainly related to time and logistics, can inform further methodological refinements.

Conclusions

The PM was an acceptable and feasible way to measure children’s diet outside of school hours in this population of parents from disadvantaged areas. While the absolute validity should be evaluated further, this relatively simple method has potential for assessing intakes of well-defined foods at group level.

Assessing Eating Behaviour Using Upper Limb Mounted Motion Sensors: A Systematic Review

Wearable motion tracking sensors are now widely used to monitor physical activity, and have recently gained more attention in dietary monitoring research. The aim of this review is to synthesise research to date that utilises upper limb motion tracking sensors, either individually or in combination with other technologies (e.g., cameras, microphones), to objectively assess eating behaviour. Eleven electronic databases were searched in January 2019, and 653 distinct records were obtained. Including 10 studies found in backward and forward searches, a total of 69 studies met the inclusion criteria, with 28 published since 2017. Fifty studies were conducted exclusively in laboratory settings, 13 exclusively in free-living settings, and three in both settings. The most commonly used motion sensor was an accelerometer (64) worn on the wrist (60) or lower arm (5), while in most studies (45), accelerometers were used in combination with gyroscopes. Twenty-six studies used commercial-grade smartwatches or fitness bands, 11 used professional grade devices, and 32 used standalone sensor chipsets. The most used machine learning approaches were Support Vector Machine (SVM, n = 21), Random Forest (n = 19), Decision Tree (n = 16), Hidden Markov Model (HMM, n = 10) algorithms, and from 2017 Deep Learning (n = 5). While comparisons of the detection models are not valid due to the use of different datasets, the models that consider the sequential context of data across time, such as HMM and Deep Learning, show promising results for eating activity detection. We discuss opportunities for future research and emerging applications in the context of dietary assessment and monitoring.

Cognitive reappraisal of low-calorie food predicts real-world craving and consumption of high- and low-calorie foods in daily life

In an increasingly obesogenic environment, an individual's regulatory capacity to pursue nutrient-rich, low-calorie foods over palatable, energy-dense items is essential to maintaining a healthy weight and preventing the detrimental health risks of obesity. Cognitive reappraisal, the process by which one changes the meaning of a stimulus by altering its emotional impact (or in this case, its appetitive value) demonstrates promise as a regulatory strategy to decrease obesogenic food consumption, but little research has directly addressed the relationship between cognitive reappraisal of food cravings and real-world eating behaviors. Additionally, research examining self-regulation of eating has typically focused exclusively on diminishing cravings and consumption of unhealthy, high-calorie foods, rather than examining, in tandem, ways to strengthen (or, up-regulate) cravings for healthier, low-calorie alternatives. In the present study, fifty-seven college aged participants first completed a cognitive reappraisal task in the laboratory in which they practiced regulating their craving responses to high- and low-calorie food items by focusing on the long-term health consequences of repeatedly consuming the pictured foods. Next, for a week following the laboratory session, participants reported daily eating behaviors via ecological momentary assessment. Participants who reported greater up-regulatory success during the reappraisal task also reported increased craving strength for low-calorie foods as well as decreased consumption of high-calorie foods in their daily lives. Greater overall regulation success also predicted more frequent consumption of craved low-calorie foods. These findings substantiate the association between cognitive reappraisal ability and real-world appetitive behaviors, and suggest that future interventions may benefit from specifically targeting individuals' evaluations of low-calorie foods.

Dietary Assessment with a Wearable Camera among Children: Feasibility and Intercoder Reliability

BACKGROUND

The eButton, a multisensor device worn on the chest, uses a camera to passively capture images of everything in front of the child throughout the day. These images can be analyzed to provide a passive method of dietary intake assessment.

OBJECTIVE

This study assessed the eButton's feasibility and intercoder reliability for dietary intake assessment.

DESIGN

Children were recruited in the summer and fall of 2015, in Houston, TX, to wear the eButton to take 2 full days of dietary images, and the child-parent dyad participated in a following-day interview to verify what dietitians recorded from the images.

PARTICIPANTS/SETTING

Thirty 9- to 13-year-old children participated during days convenient to them.

MAIN OUTCOME MEASURES

Two dietitians independently manually reviewed the images to identify eating events, foods in those events, and portion sizes.

STATISTICAL ANALYSES PERFORMED

Descriptive statistics of agreements and disagreements were calculated between dietitians and with children; t tests and Bland-Altman plots of differences in total kilocalories were calculated between dietitians and between initial dietitian estimates and those finalized after the verification interviews.

RESULTS

The dietitians agreed on the identity of 60.5% of the 1,026 foods but disagreed on 28.6% of the foods and on the names for 10.8% of the foods. After the verification interviews, the dietitians agreed with the child-parent dyads on the identity of 77.0% of the 921 foods; the child-parent dyad identified 12.4% of the day's foods when images were not available or not clear; the child-parent dyad clarified that 5.4% of the foods identified were not consumed by the child; and the child-parent dyad clarified the identity of 5.2% of the foods. A software-based approach (three-dimensional wire mesh) could be used to estimate portion size on 24% of the foods, and professional judgment was required for 67.8%. Mean caloric intakes per day were not statistically significantly different between dietitians but were different between dietitians and child-parent dyads in total and on day 2.

CONCLUSIONS

An early test of intercoder reliability of an all-day image method of dietary intake assessment obtained intercoder agreement between the two dietitians processing these images of intraclass correlation coefficient=0.67. A following-day verification interview with the child and parent was necessary to ensure completeness of estimates. Several feasibility problems occurred, which may be remedied with additional participant and dietitian training and further technological development.

BioMEMS for biosensors and closed-loop drug delivery

The efficacy of pharmaceutical treatments can be greatly enhanced by physiological feedback from the patient using biosensors, though this is often invasive or infeasible. By adapting microelectromechanical systems (MEMS) technology to miniaturize such biosensors, previously inaccessible signals can be obtained, often from inside the patient. This is enabled by the device's extremely small footprint which minimizes both power consumption and implantation trauma, as well as the transport time for chemical analytes, in turn decreasing the sensor's response time. MEMS fabrication also allows mass production which can be easily scaled without sacrificing its high reproducibility and reliability, and allows seamless integration with control circuitry and telemetry which is already produced using the same materials and fabrication steps. By integrating these systems with drug delivery devices, many of which are also MEMS-based, closed loop drug delivery can be achieved. This paper surveys the types of signal transduction devices available for biosensing-primarily electrochemical, optical, and mechanical-looking at their implementation via MEMS technology. The impact of MEMS technology on the challenges of biosensor development, particularly safety, power consumption, degradation, fouling, and foreign body response, are also discussed.

Detecting Smoking Events Using Accelerometer Data Collected Via Smartwatch Technology: Validation Study

BACKGROUND

Smoking is the leading cause of preventable death in the world today. Ecological research on smoking in context currently relies on self-reported smoking behavior. Emerging smartwatch technology may more objectively measure smoking behavior by automatically detecting smoking sessions using robust machine learning models.

OBJECTIVE

This study aimed to examine the feasibility of detecting smoking behavior using smartwatches. The second aim of this study was to compare the success of observing smoking behavior with smartwatches to that of conventional self-reporting.

METHODS

A convenience sample of smokers was recruited for this study. Participants (N=10) recorded 12 hours of accelerometer data using a mobile phone and smartwatch. During these 12 hours, they engaged in various daily activities, including smoking, for which they logged the beginning and end of each smoking session. Raw data were classified as either smoking or nonsmoking using a machine learning model for pattern recognition. The accuracy of the model was evaluated by comparing the output with a detailed description of a modeled smoking session.

RESULTS

In total, 120 hours of data were collected from participants and analyzed. The accuracy of self-reported smoking was approximately 78% (96/123). Our model was successful in detecting 100 of 123 (81%) smoking sessions recorded by participants. After eliminating sessions from the participants that did not adhere to study protocols, the true positive detection rate of the smartwatch based-detection increased to more than 90%. During the 120 hours of combined observation time, only 22 false positive smoking sessions were detected resulting in a 2.8% false positive rate.

CONCLUSIONS

Smartwatch technology can provide an accurate, nonintrusive means of monitoring smoking behavior in natural contexts. The use of machine learning algorithms for passively detecting smoking sessions may enrich ecological momentary assessment protocols and cessation intervention studies that often rely on self-reported behaviors and may not allow for targeted data collection and communications around smoking events.

EarBit: Using Wearable Sensors to Detect Eating Episodes in Unconstrained Environments

Chronic and widespread diseases such as obesity, diabetes, and hypercholesterolemia require patients to monitor their food intake, and food journaling is currently the most common method for doing so. However, food journaling is subject to self-bias and recall errors, and is poorly adhered to by patients. In this paper, we propose an alternative by introducing EarBit, a wearable system that detects eating moments. We evaluate the performance of inertial, optical, and acoustic sensing modalities and focus on inertial sensing, by virtue of its recognition and usability performance. Using data collected in a simulated home setting with minimum restrictions on participants' behavior, we build our models and evaluate them with an unconstrained outside-the-lab study. For both studies, we obtained video footage as ground truth for participants activities. Using leave-one-user-out validation, EarBit recognized all the eating episodes in the semi-controlled lab study, and achieved an accuracy of 90.1% and an F₁-score of 90.9% in detecting chewing instances. In the unconstrained, outside-the-lab evaluation, EarBit obtained an accuracy of 93% and an F₁-score of 80.1% in detecting chewing instances. It also accurately recognized all but one recorded eating episodes. These episodes ranged from a 2 minute snack to a 30 minute meal.

A Cardiac Early Warning System with Multi Channel SCG and ECG Monitoring for Mobile Health

Use of information and communication technology such as smart phone, smart watch, smart glass and portable health monitoring devices for healthcare services has made Mobile Health (mHealth) an emerging research area. Coronary Heart Disease (CHD) is considered as a leading cause of death world wide and an increasing number of people die prematurely due to CHD. Under such circumstances, there is a growing demand for a reliable cardiac monitoring system to catch the intermittent abnormalities and detect critical cardiac behaviors which lead to sudden death. Use of mobile devices to collect Electrocardiography (ECG), Seismocardiography (SCG) data and efficient analysis of those data can monitor a patient’s cardiac activities for early warning. This paper presents a novel cardiac data acquisition method and combined analysis of Electrocardiography (ECG) and multi channel Seismocardiography (SCG) data. An early warning system is implemented to monitor the cardiac activities of a person and accuracy assessment of the early warning system is conducted for the ECG data only. The assessment shows 88% accuracy and effectiveness of our proposed analysis, which implies the viability and applicability of the proposed early warning system.