Seamless tracing of human behavior using complementary wearable and house-embedded sensors

Touchless Heart Rate Monitoring from an Unmanned Aerial Vehicle Using Videoplethysmography

Motivation: The advancement of preventive medicine and, subsequently, telemedicine drives the need for noninvasive and remote measurements in patients' natural environments. Heart rate (HR) measurements are particularly promising and extensively researched due to their quick assessment and comprehensive representation of patients' conditions. However, in scenarios such as endurance training or emergencies, where HR measurement was not anticipated and direct access to victims is limited, no method enables obtaining HR results that are suitable even for triage. Methods: This paper presents the possibility of remotely measuring of human HR from a series of in-flight videos using videoplethysmography (VPG) along with skin detection, human pose estimation and image stabilization methods. An unmanned aerial vehicle (UAV) equipped with a camera captured ten segments of video footage featuring volunteers engaged in free walking and running activities in natural sunlight. The human pose was determined using the OpenPose algorithm, and subsequently, skin areas on the face and forearms were identified and tracked in consecutive frames. Ultimately, HR was estimated using several VPG methods: the green channel (G), green-red difference (GR), excess green (ExG), independent component analysis (ICA), and a plane orthogonal to the skin (POS). Results: When compared to simultaneous readings from a reference ECG-based wearable recorder, the root-mean-squared error ranged from 17.7 (G) to 27.7 (POS), with errors of less than 3.5 bpm achieved for the G and GR methods. Conclusions: These results demonstrate the acceptable accuracy of touchless human pulse measurement with the accompanying UAV-mounted camera. The method bridges the gap between HR-transmitting wearables and emergency HR recorders, and it has the potential to be advantageous in training or rescue scenarios in mountain, water, disaster, or battlefield settings.

Movement Tube Detection Network Integrating 3D CNN and Object Detection Framework to Detect Fall

Unlike most of the existing neural network-based fall detection methods, which only detect fall at the time range, the algorithm proposed in this paper detect fall in both spatial and temporal dimension. A movement tube detection network integrating 3D CNN and object detection framework such as SSD is proposed to detect human fall with constrained movement tubes. The constrained movement tube, which encapsulates the person with a sequence of bounding boxes, has the merits of encapsulating the person closely and avoiding peripheral interference. A 3D convolutional neural network is used to encode the motion and appearance features of a video clip, which are fed into the tube anchors generation layer, softmax classification, and movement tube regression layer. The movement tube regression layer fine tunes the tube anchors to the constrained movement tubes. A large-scale spatio-temporal (LSST) fall dataset is constructed using self-collected data to evaluate the fall detection in both spatial and temporal dimensions. LSST has three characteristics of large scale, annotation, and posture and viewpoint diversities. Furthermore, the comparative experiments on a public dataset demonstrate that the proposed algorithm achieved sensitivity, specificity an accuracy of 100%, 97.04%, and 97.23%, respectively, outperforms the existing methods.

Assisted Living System with Adaptive Sensor's Contribution

Multimodal sensing and data processing have become a common approach in modern assisted living systems. This is widely justified by the complementary properties of sensors based on different sensing paradigms. However, all previous proposals assume data fusion to be made based on fixed criteria. We proved that particular sensors show different performance depending on the subject's activity and consequently present the concept of an adaptive sensor's contribution. In the proposed prototype architecture, the sensor information is first unified and then modulated to prefer the most reliable sensors. We also take into consideration the dynamics of the subject's behavior and propose two algorithms for the adaptation of sensors' contribution, and discuss their advantages and limitations based on case studies.

Consistency of Outputs of the Selected Motion Acquisition Methods for Human Activity Recognition

The aim of this paper is to choose the optimal motion sensor for the selected human activity recognition. In the described studies, different human motion measurement methods are used simultaneously such as optoelectronics, video, electromyographic, accelerometric, and pressure sensors. Several analyses of activity recognition were performed: recognition correctness for all activities together, matrices of the recognition errors of the individual activities for all volunteers for the individual sensors, and recognition correctness of all activities for each volunteer and each sensor. The experiments enabled to find a range of interchangeability and to choose the most appropriate sensor for recognition of the selected motion.

Recognition of Sedentary Behavior by Machine Learning Analysis of Wearable Sensors during Activities of Daily Living for Telemedical Assessment of Cardiovascular Risk

With the recent advancement in wearable computing, sensor technologies, and data processing approaches, it is possible to develop smart clothing that integrates sensors into garments. The main objective of this study was to develop the method of automatic recognition of sedentary behavior related to cardiovascular risk based on quantitative measurement of physical activity. The solution is based on the designed prototype of the smart shirt equipped with a processor, wearable sensors, power supply and telemedical interface. The data derived from wearable sensors were used to create feature vector that consisted of the estimation of the user-specific relative intensity and the variance of filtered accelerometer data. The method was validated using an experimental protocol which was designed to be safe for the elderly and was based on clinically validated short physical performance battery (SPPB) test tasks. To obtain the recognition model six classifiers were examined and compared including Linear Discriminant Analysis, Support Vector Machines, K-Nearest Neighbors, Naive Bayes, Binary Decision Trees and Artificial Neural Networks. The classification models were able to identify the sedentary behavior with an accuracy of 95.00% ± 2.11%. Experimental results suggested that high accuracy can be obtained by estimating sedentary behavior pattern using the smart shirt and machine learning approach. The main advantage of the developed method to continuously monitor patient activities in a free-living environment and could potentially be used for early detection of increased cardiovascular risk.

Graph-based representation of behavior in detection and prediction of daily living activities

Various surveillance systems capture signs of human activities of daily living (ADLs) and store multimodal information as time line behavioral records. In this paper, we present a novel approach to the analysis of a behavioral record used in a surveillance system designed for use in elderly smart homes. The description of a subject's activity is first decomposed into elementary poses - easily detectable by dedicated intelligent sensors - and represented by the share coefficients. Then, the activity is represented in the form of an attributed graph, where nodes correspond to elementary poses. As share coefficients of poses are expressed as attributes assigned to graph nodes, their change corresponding to a subject's action is represented by flow in graph edges. The behavioral record is thus a time series of graphs, which tiny size facilitates storage and management of long-term monitoring results. At the system learning stage, the contribution of elementary poses is accumulated, discretized and probability-ordered leading to a finite list representing the possible transitions between states. Such a list is independently built for each room in the supervised residence, and employed for assessment of the current action in the context of subject's habits and a room purpose. The proposed format of a behavioral record, applied to an adaptive surveillance system, is particularly advantageous for representing new activities not known at the setup stage, for providing a quantitative measure of transitions between poses and for expressing the difference between a predicted and actual action in a numerical way.

An Engineering Perspective of External Cardiac Loop Recorder: A Systematic Review

External cardiac loop recorder (ELR) is a kind of ECG monitoring system that records cardiac activities of a subject continuously for a long time. When the heart palpitations are not the frequent and nonspecific character, it is difficult to diagnose the disease. In such a case, ELR is used for long-term monitoring of heart signal of the patient. But the cost of ELR is very high. Therefore, it is not prominently available in developing countries like India. Since the design of ELR includes the ECG electrodes, instrumentation amplifier, analog to digital converter, and signal processing unit, a comparative review of each part of the ELR is presented in this paper in order to design a cost effective, low power, and compact kind of ELR. This review will also give different choices available for selecting and designing each part of the ELR system. Finally, the review will suggest the better choice for designing a cost effective external cardiac loop recorder that helps to make it available even for rural people in India.

Monitoring activities of daily living based on wearable wireless body sensor network

With recent advances in microprocessor chip technology, wireless communication, and biomedical engineering it is possible to develop miniaturized ubiquitous health monitoring devices that are capable of recording physiological and movement signals during daily life activities. The aim of the research is to implement and test the prototype of health monitoring system. The system consists of the body central unit with Bluetooth module and wearable sensors: the custom-designed ECG sensor, the temperature sensor, the skin humidity sensor and accelerometers placed on the human body or integrated with clothes and a network gateway to forward data to a remote medical server. The system includes custom-designed transmission protocol and remote web-based graphical user interface for remote real time data analysis. Experimental results for a group of humans who performed various activities (eg. working, running, etc.) showed maximum 5% absolute error compared to certified medical devices. The results are promising and indicate that developed wireless wearable monitoring system faces challenges of multi-sensor human health monitoring during performing daily activities and opens new opportunities in developing novel healthcare services.

Mining personal data using smartphones and wearable devices: a survey

The staggering growth in smartphone and wearable device use has led to a massive scale generation of personal (user-specific) data. To explore, analyze, and extract useful information and knowledge from the deluge of personal data, one has to leverage these devices as the data-mining platforms in ubiquitous, pervasive, and big data environments. This study presents the personal ecosystem where all computational resources, communication facilities, storage and knowledge management systems are available in user proximity. An extensive review on recent literature has been conducted and a detailed taxonomy is presented. The performance evaluation metrics and their empirical evidences are sorted out in this paper. Finally, we have highlighted some future research directions and potentially emerging application areas for personal data mining using smartphones and wearable devices.