A Hierarchical Deep Fusion Framework for Egocentric Activity Recognition Using a Wearable Hybrid Sensor System

Intelligent ADL Recognition via IoT-Based Multimodal Deep Learning Framework

Smart home monitoring systems via internet of things (IoT) are required for taking care of elders at home. They provide the flexibility of monitoring elders remotely for their families and caregivers. Activities of daily living are an efficient way to effectively monitor elderly people at home and patients at caregiving facilities. The monitoring of such actions depends largely on IoT-based devices, either wireless or installed at different places. This paper proposes an effective and robust layered architecture using multisensory devices to recognize the activities of daily living from anywhere. Multimodality refers to the sensory devices of multiple types working together to achieve the objective of remote monitoring. Therefore, the proposed multimodal-based approach includes IoT devices, such as wearable inertial sensors and videos recorded during daily routines, fused together. The data from these multi-sensors have to be processed through a pre-processing layer through different stages, such as data filtration, segmentation, landmark detection, and 2D stick model. In next layer called the features processing, we have extracted, fused, and optimized different features from multimodal sensors. The final layer, called classification, has been utilized to recognize the activities of daily living via a deep learning technique known as convolutional neural network. It is observed from the proposed IoT-based multimodal layered system's results that an acceptable mean accuracy rate of 84.14% has been achieved.

Human Activity Recognition through Recurrent Neural Networks for Human–Robot Interaction in Agriculture

The present study deals with human awareness, which is a very important aspect of human–robot interaction. This feature is particularly essential in agricultural environments, owing to the information-rich setup that they provide. The objective of this investigation was to recognize human activities associated with an envisioned synergistic task. In order to attain this goal, a data collection field experiment was designed that derived data from twenty healthy participants using five wearable sensors (embedded with tri-axial accelerometers, gyroscopes, and magnetometers) attached to them. The above task involved several sub-activities, which were carried out by agricultural workers in real field conditions, concerning load lifting and carrying. Subsequently, the obtained signals from on-body sensors were processed for noise-removal purposes and fed into a Long Short-Term Memory neural network, which is widely used in deep learning for feature recognition in time-dependent data sequences. The proposed methodology demonstrated considerable efficacy in predicting the defined sub-activities with an average accuracy of 85.6%. Moreover, the trained model properly classified the defined sub-activities in a range of 74.1–90.4% for precision and 71.0–96.9% for recall. It can be inferred that the combination of all sensors can achieve the highest accuracy in human activity recognition, as concluded from a comparative analysis for each sensor’s impact on the model’s performance. These results confirm the applicability of the proposed methodology for human awareness purposes in agricultural environments, while the dataset was made publicly available for future research.

A Review of the Recent Developments in Integrating Machine Learning Models with Sensor Devices in the Smart Buildings Sector with a View to Attaining Enhanced Sensing, Energy Efficiency, and Optimal Building Management

Lately, many scientists have focused their research on subjects like smart buildings, sensor devices, virtual sensing, buildings management, Internet of Things (IoT), artificial intelligence in the smart buildings sector, improving life quality within smart homes, assessing the occupancy status information, detecting human behavior with a view to assisted living, maintaining environmental health, and preserving natural resources. The main purpose of our review consists of surveying the current state of the art regarding the recent developments in integrating supervised and unsupervised machine learning models with sensor devices in the smart building sector with a view to attaining enhanced sensing, energy efficiency and optimal building management. We have devised the research methodology with a view to identifying, filtering, categorizing, and analyzing the most important and relevant scientific articles regarding the targeted topic. To this end, we have used reliable sources of scientific information, namely the Elsevier Scopus and the Clarivate Analytics Web of Science international databases, in order to assess the interest regarding the above-mentioned topic within the scientific literature. After processing the obtained papers, we finally obtained, on the basis of our devised methodology, a reliable, eloquent and representative pool of 146 papers scientific works that would be useful for developing our survey. Our approach provides a useful up-to-date overview for researchers from different fields, which can be helpful when submitting project proposals or when studying complex topics such those reviewed in this paper. Meanwhile, the current study offers scientists the possibility of identifying future research directions that have not yet been addressed in the scientific literature or improving the existing approaches based on the body of knowledge. Moreover, the conducted review creates the premises for identifying in the scientific literature the main purposes for integrating Machine Learning techniques with sensing devices in smart environments, as well as purposes that have not been investigated yet.

A multisource fusion framework driven by user-defined knowledge for egocentric activity recognition

Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its widespread applicability to human systems, including the evaluation of dietary and physical activity and the monitoring of patients and older adults. In this paper, we present a knowledge-driven multisource fusion framework for the recognition of egocentric activities in daily living (ADL). This framework employs Dezert-Smarandache theory across three information sources: the wearer's knowledge, images acquired by a wearable camera, and sensor data from wearable inertial measurement units and GPS. A simple likelihood table is designed to provide routine ADL information for each individual. A well-trained convolutional neural network is then used to produce a set of textual tags that, along with routine information and other sensor data, are used to recognize ADLs based on information theory-based statistics and a support vector machine. Our experiments show that the proposed method accurately recognizes 15 predefined ADL classes, including a variety of sedentary activities that have previously been difficult to recognize. When applied to real-life data recorded using a self-constructed wearable device, our method outperforms previous approaches, and an average accuracy of 85.4% is achieved for the 15 ADLs.