1
|
Crystal AA, Valero M, Nino V, Ingram KH. Empowering Diabetics: Advancements in Smartphone-Based Food Classification, Volume Measurement, and Nutritional Estimation. SENSORS (BASEL, SWITZERLAND) 2024; 24:4089. [PMID: 39000868 PMCID: PMC11244259 DOI: 10.3390/s24134089] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/23/2024] [Revised: 06/06/2024] [Accepted: 06/14/2024] [Indexed: 07/16/2024]
Abstract
Diabetes has emerged as a worldwide health crisis, affecting approximately 537 million adults. Maintaining blood glucose requires careful observation of diet, physical activity, and adherence to medications if necessary. Diet monitoring historically involves keeping food diaries; however, this process can be labor-intensive, and recollection of food items may introduce errors. Automated technologies such as food image recognition systems (FIRS) can make use of computer vision and mobile cameras to reduce the burden of keeping diaries and improve diet tracking. These tools provide various levels of diet analysis, and some offer further suggestions for improving the nutritional quality of meals. The current study is a systematic review of mobile computer vision-based approaches for food classification, volume estimation, and nutrient estimation. Relevant articles published over the last two decades are evaluated, and both future directions and issues related to FIRS are explored.
Collapse
Affiliation(s)
- Afnan Ahmed Crystal
- Department of Computer Science, Kennesaw State University, Kennesaw, GA 30060, USA
| | - Maria Valero
- Department of Information Technology, Kennesaw State University, Kennesaw, GA 30060, USA
| | - Valentina Nino
- Departement of Industrial and Systems Engineering, Kennesaw State University, Kennesaw, GA 30060, USA
| | - Katherine H Ingram
- Department of Exercise Science and Sport Management, Kennesaw State University, Kennesaw, GA 30060, USA
| |
Collapse
|
2
|
Chen X, Kamavuako EN. Vision-Based Methods for Food and Fluid Intake Monitoring: A Literature Review. SENSORS (BASEL, SWITZERLAND) 2023; 23:6137. [PMID: 37447988 PMCID: PMC10346353 DOI: 10.3390/s23136137] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 05/28/2023] [Accepted: 06/28/2023] [Indexed: 07/15/2023]
Abstract
Food and fluid intake monitoring are essential for reducing the risk of dehydration, malnutrition, and obesity. The existing research has been preponderantly focused on dietary monitoring, while fluid intake monitoring, on the other hand, is often neglected. Food and fluid intake monitoring can be based on wearable sensors, environmental sensors, smart containers, and the collaborative use of multiple sensors. Vision-based intake monitoring methods have been widely exploited with the development of visual devices and computer vision algorithms. Vision-based methods provide non-intrusive solutions for monitoring. They have shown promising performance in food/beverage recognition and segmentation, human intake action detection and classification, and food volume/fluid amount estimation. However, occlusion, privacy, computational efficiency, and practicality pose significant challenges. This paper reviews the existing work (253 articles) on vision-based intake (food and fluid) monitoring methods to assess the size and scope of the available literature and identify the current challenges and research gaps. This paper uses tables and graphs to depict the patterns of device selection, viewing angle, tasks, algorithms, experimental settings, and performance of the existing monitoring systems.
Collapse
Affiliation(s)
- Xin Chen
- Department of Engineering, King’s College London, London WC2R 2LS, UK;
| | - Ernest N. Kamavuako
- Department of Engineering, King’s College London, London WC2R 2LS, UK;
- Faculté de Médecine, Université de Kindu, Site de Lwama II, Kindu, Maniema, Democratic Republic of the Congo
| |
Collapse
|
3
|
Amugongo LM, Kriebitz A, Boch A, Lütge C. Mobile Computer Vision-Based Applications for Food Recognition and Volume and Calorific Estimation: A Systematic Review. Healthcare (Basel) 2022; 11:healthcare11010059. [PMID: 36611519 PMCID: PMC9818870 DOI: 10.3390/healthcare11010059] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 12/20/2022] [Accepted: 12/21/2022] [Indexed: 12/28/2022] Open
Abstract
The growing awareness of the influence of "what we eat" on lifestyle and health has led to an increase in the use of embedded food analysis and recognition systems. These solutions aim to effectively monitor daily food consumption, and therefore provide dietary recommendations to enable and support lifestyle changes. Mobile applications, due to their high accessibility, are ideal for real-life food recognition, volume estimation and calorific estimation. In this study, we conducted a systematic review based on articles that proposed mobile computer vision-based solutions for food recognition, volume estimation and calorific estimation. In addition, we assessed the extent to which these applications provide explanations to aid the users to understand the related classification and/or predictions. Our results show that 90.9% of applications do not distinguish between food and non-food. Similarly, only one study that proposed a mobile computer vision-based application for dietary intake attempted to provide explanations of features that contribute towards classification. Mobile computer vision-based applications are attracting a lot of interest in healthcare. They have the potential to assist in the management of chronic illnesses such as diabetes, ensuring that patients eat healthily and reducing complications associated with unhealthy food. However, to improve trust, mobile computer vision-based applications in healthcare should provide explanations of how they derive their classifications or volume and calorific estimations.
Collapse
|
4
|
Dalakleidi KV, Papadelli M, Kapolos I, Papadimitriou K. Applying Image-Based Food-Recognition Systems on Dietary Assessment: A Systematic Review. Adv Nutr 2022; 13:2590-2619. [PMID: 35803496 PMCID: PMC9776640 DOI: 10.1093/advances/nmac078] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2022] [Revised: 06/06/2022] [Accepted: 07/06/2022] [Indexed: 01/29/2023] Open
Abstract
Dietary assessment can be crucial for the overall well-being of humans and, at least in some instances, for the prevention and management of chronic, life-threatening diseases. Recall and manual record-keeping methods for food-intake monitoring are available, but often inaccurate when applied for a long period of time. On the other hand, automatic record-keeping approaches that adopt mobile cameras and computer vision methods seem to simplify the process and can improve current human-centric diet-monitoring methods. Here we present an extended critical literature overview of image-based food-recognition systems (IBFRS) combining a camera of the user's mobile device with computer vision methods and publicly available food datasets (PAFDs). In brief, such systems consist of several phases, such as the segmentation of the food items on the plate, the classification of the food items in a specific food category, and the estimation phase of volume, calories, or nutrients of each food item. A total of 159 studies were screened in this systematic review of IBFRS. A detailed overview of the methods adopted in each of the 78 included studies of this systematic review of IBFRS is provided along with their performance on PAFDs. Studies that included IBFRS without presenting their performance in at least 1 of the above-mentioned phases were excluded. Among the included studies, 45 (58%) studies adopted deep learning methods and especially convolutional neural networks (CNNs) in at least 1 phase of the IBFRS with input PAFDs. Among the implemented techniques, CNNs outperform all other approaches on the PAFDs with a large volume of data, since the richness of these datasets provides adequate training resources for such algorithms. We also present evidence for the benefits of application of IBFRS in professional dietetic practice. Furthermore, challenges related to the IBFRS presented here are also thoroughly discussed along with future directions.
Collapse
Affiliation(s)
- Kalliopi V Dalakleidi
- Department of Food Science and Technology, University of the Peloponnese, Kalamata, Greece
| | - Marina Papadelli
- Department of Food Science and Technology, University of the Peloponnese, Kalamata, Greece
| | - Ioannis Kapolos
- Department of Food Science and Technology, University of the Peloponnese, Kalamata, Greece
| | - Konstantinos Papadimitriou
- Laboratory of Food Quality Control and Hygiene, Department of Food Science and Human Nutrition, Agricultural University of Athens, Athens, Greece
| |
Collapse
|
5
|
Zhang Q, He C, Qin W, Liu D, Yin J, Long Z, He H, Sun HC, Xu H. Eliminate the hardware: Mobile terminals-oriented food recognition and weight estimation system. Front Nutr 2022; 9:965801. [PMID: 36466396 PMCID: PMC9709194 DOI: 10.3389/fnut.2022.965801] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Accepted: 10/24/2022] [Indexed: 09/29/2023] Open
Abstract
Food recognition and weight estimation based on image methods have always been hotspots in the field of computer vision and medical nutrition, and have good application prospects in digital nutrition therapy and health detection. With the development of deep learning technology, image-based recognition technology has also rapidly extended to various fields, such as agricultural pests, disease identification, tumor marker recognition, wound severity judgment, road wear recognition, and food safety detection. This article proposes a non-wearable food recognition and weight estimation system (nWFWS) to identify the food type and food weight in the target recognition area via smartphones, so to assist clinical patients and physicians in monitoring diet-related health conditions. In addition, the system is mainly designed for mobile terminals; it can be installed on a mobile phone with an Android system or an iOS system. This can lower the cost and burden of additional wearable health monitoring equipment while also greatly simplifying the automatic estimation of food intake via mobile phone photography and image collection. Based on the system's ability to accurately identify 1,455 food pictures with an accuracy rate of 89.60%, we used a deep convolutional neural network and visual-inertial system to collect image pixels, and 612 high-resolution food images with different traits after systematic training, to obtain a preliminary relationship model between the area of food pixels and the measured weight was obtained, and the weight of untested food images was successfully determined. There was a high correlation between the predicted and actual values. In a word, this system is feasible and relatively accurate for one automated dietary monitoring and nutritional assessment.
Collapse
Affiliation(s)
- Qinqiu Zhang
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
- Sichuan Key Laboratory of Fruit and Vegetable Postharvest Physiology, College of Food Science, Sichuan Agricultural University, Ya’an, China
| | - Chengyuan He
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| | - Wen Qin
- Sichuan Key Laboratory of Fruit and Vegetable Postharvest Physiology, College of Food Science, Sichuan Agricultural University, Ya’an, China
| | - Decai Liu
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| | - Jun Yin
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| | - Zhiwen Long
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| | - Huimin He
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| | - Ho Ching Sun
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| | - Huilin Xu
- Chengdu Shangyi Information Technology Co., Ltd., Chengdu, China
| |
Collapse
|
6
|
Neves PA, Simões J, Costa R, Pimenta L, Gonçalves NJ, Albuquerque C, Cunha C, Zdravevski E, Lameski P, Garcia NM, Pires IM. Thought on Food: A Systematic Review of Current Approaches and Challenges for Food Intake Detection. SENSORS (BASEL, SWITZERLAND) 2022; 22:6443. [PMID: 36080901 PMCID: PMC9460522 DOI: 10.3390/s22176443] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/12/2022] [Revised: 08/14/2022] [Accepted: 08/24/2022] [Indexed: 06/15/2023]
Abstract
Nowadays, individuals have very stressful lifestyles, affecting their nutritional habits. In the early stages of life, teenagers begin to exhibit bad habits and inadequate nutrition. Likewise, other people with dementia, Alzheimer's disease, or other conditions may not take food or medicine regularly. Therefore, the ability to monitor could be beneficial for them and for the doctors that can analyze the patterns of eating habits and their correlation with overall health. Many sensors help accurately detect food intake episodes, including electrogastrography, cameras, microphones, and inertial sensors. Accurate detection may provide better control to enable healthy nutrition habits. This paper presents a systematic review of the use of technology for food intake detection, focusing on the different sensors and methodologies used. The search was performed with a Natural Language Processing (NLP) framework that helps screen irrelevant studies while following the PRISMA methodology. It automatically searched and filtered the research studies in different databases, including PubMed, Springer, ACM, IEEE Xplore, MDPI, and Elsevier. Then, the manual analysis selected 30 papers based on the results of the framework for further analysis, which support the interest in using sensors for food intake detection and nutrition assessment. The mainly used sensors are cameras, inertial, and acoustic sensors that handle the recognition of food intake episodes with artificial intelligence techniques. This research identifies the most used sensors and data processing methodologies to detect food intake.
Collapse
Affiliation(s)
- Paulo Alexandre Neves
- School of Technology, Polytechnic Institute of Castelo Branco, 6000-767 Castelo Branco, Portugal
| | - João Simões
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal
| | - Ricardo Costa
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal
| | - Luís Pimenta
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal
| | - Norberto Jorge Gonçalves
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal
| | - Carlos Albuquerque
- Health Sciences Research Unit: Nursing (UICISA: E), Nursing School of Coimbra (ESEnfC), 3046-851 Coimbra, Portugal
- Higher School of Health, Polytechnic Institute of Viseu, 3504-510 Viseu, Portugal
- Child Studies Research Center (CIEC), University of Minho, 4710-057 Braga, Portugal
| | - Carlos Cunha
- CISeD—Research Centre in Digital Services, Polytechnic Institute of Viseu, 3504-510 Viseu, Portugal
| | - Eftim Zdravevski
- Faculty of Computer Science and Engineering, University Ss Cyril and Methodius, 1000 Skopje, North Macedonia
| | - Petre Lameski
- Faculty of Computer Science and Engineering, University Ss Cyril and Methodius, 1000 Skopje, North Macedonia
| | - Nuno M. Garcia
- Instituto de Telecomunicações, Universidade da Beira Interior, 6200-001 Covilhã, Portugal
| | - Ivan Miguel Pires
- Instituto de Telecomunicações, Universidade da Beira Interior, 6200-001 Covilhã, Portugal
| |
Collapse
|
7
|
Vasiloglou MF, Marcano I, Lizama S, Papathanail I, Spanakis EK, Mougiakakou S. Multimedia Data-Based Mobile Applications for Dietary Assessment. J Diabetes Sci Technol 2022:19322968221085026. [PMID: 35348398 DOI: 10.1177/19322968221085026] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/03/2023]
Abstract
Diabetes mellitus (DM) and obesity are chronic medical conditions associated with significant morbidity and mortality. Accurate macronutrient and energy estimation could be beneficial in attempts to manage DM and obesity, leading to improved glycemic control and weight reduction, respectively. Existing dietary assessment methods are subject to major errors in measurement, are time consuming, are costly, and do not provide real-time feedback. The increasing adoption of smartphones and artificial intelligence, along with the advances in algorithms and hardware, allowed the development of technologies executed in smartphones that use food/beverage multimedia data as an input, and output information about the nutrient content in almost real time. Scope of this review was to explore the various image-based and video-based systems designed for dietary assessment. We identified 22 different systems and divided these into three categories on the basis of their setting for evaluation: laboratory (12), preclinical (7), and clinical (3). The major findings of the review are that there is still a number of open research questions and technical challenges to be addressed and end users-including health care professionals and patients-need to be involved in the design and development of such innovative solutions. Last, there is a clear need that these systems should be validated under unconstrained real-life conditions and that they should be compared with conventional methods for dietary assessment.
Collapse
Affiliation(s)
- Maria F Vasiloglou
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - Isabel Marcano
- Division of Endocrinology, Diabetes, and Nutrition, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Sergio Lizama
- Division of Endocrinology, Diabetes, and Nutrition, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Ioannis Papathanail
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - Elias K Spanakis
- Division of Endocrinology, Diabetes, and Nutrition, University of Maryland School of Medicine, Baltimore, MD, USA
- Division of Endocrinology, Baltimore Veterans Affairs Medical Center, Baltimore, MD, USA
| | - Stavroula Mougiakakou
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
- Department of Emergency Medicine, Bern University Hospital, Bern, Switzerland
| |
Collapse
|
8
|
Matsuzaka Y, Hosaka T, Ogaito A, Yoshinari K, Uesawa Y. Prediction Model of Aryl Hydrocarbon Receptor Activation by a Novel QSAR Approach, DeepSnap-Deep Learning. Molecules 2020; 25:molecules25061317. [PMID: 32183141 PMCID: PMC7144728 DOI: 10.3390/molecules25061317] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2020] [Revised: 03/05/2020] [Accepted: 03/09/2020] [Indexed: 12/31/2022] Open
Abstract
The aryl hydrocarbon receptor (AhR) is a ligand-dependent transcription factor that senses environmental exogenous and endogenous ligands or xenobiotic chemicals. In particular, exposure of the liver to environmental metabolism-disrupting chemicals contributes to the development and propagation of steatosis and hepatotoxicity. However, the mechanisms for AhR-induced hepatotoxicity and tumor propagation in the liver remain to be revealed, due to the wide variety of AhR ligands. Recently, quantitative structure–activity relationship (QSAR) analysis using deep neural network (DNN) has shown superior performance for the prediction of chemical compounds. Therefore, this study proposes a novel QSAR analysis using deep learning (DL), called the DeepSnap–DL method, to construct prediction models of chemical activation of AhR. Compared with conventional machine learning (ML) techniques, such as the random forest, XGBoost, LightGBM, and CatBoost, the proposed method achieves high-performance prediction of AhR activation. Thus, the DeepSnap–DL method may be considered a useful tool for achieving high-throughput in silico evaluation of AhR-induced hepatotoxicity.
Collapse
Affiliation(s)
- Yasunari Matsuzaka
- Department of Medical Molecular Informatics, Meiji Pharmaceutical University, 204-8588 Tokyo, Japan;
| | - Takuomi Hosaka
- Laboratory of Molecular Toxicology, School of Pharmaceutical Sciences, University of Shizuoka, Shizuoka 422-8529, Japan; (T.H.); (A.O.); (K.Y.)
| | - Anna Ogaito
- Laboratory of Molecular Toxicology, School of Pharmaceutical Sciences, University of Shizuoka, Shizuoka 422-8529, Japan; (T.H.); (A.O.); (K.Y.)
| | - Kouichi Yoshinari
- Laboratory of Molecular Toxicology, School of Pharmaceutical Sciences, University of Shizuoka, Shizuoka 422-8529, Japan; (T.H.); (A.O.); (K.Y.)
| | - Yoshihiro Uesawa
- Department of Medical Molecular Informatics, Meiji Pharmaceutical University, 204-8588 Tokyo, Japan;
- Correspondence:
| |
Collapse
|