A machine learning approach to improve contactless heart rate monitoring using a webcam

Simultaneous, Non-Contact and Motion-Based Monitoring of Respiratory Rate in Sheep Under Experimental Condition Using Visible and Near-Infrared Videos

The validation of methods for understanding the effects of many diseases and treatments requires the use of animal models in translational research. In this context, sheep have been employed extensively in scientific studies. However, the imposition of experimental conditions upon these animals may result in the experience of discomfort, pain, and stress. The ethical debates surrounding the use of animals in research have resulted in the adoption of Directive 2010/63/EU. The present study proposes a non-contact method for monitoring the respiration rate of sheep based on video processing. The Detecron2 model was trained to segment the sheep's body, abdominal, and facial regions in the video frames. A motion-tracking algorithm was developed to assess abdominal movement associated with the sheep's respiratory cycle. The method was applied to videos of Rhön sheep under experimental and housing conditions, utilising two types of cameras to assess the effectiveness of the proposed approach. The mean average error (MAE) obtained was 0.79 breaths/minute for the visible and 1.83 breaths/minute for the near-infrared (NIR) method. This study demonstrates the feasibility of video technology for simultaneous and non-invasive respiration monitoring, being a crucial parameter for assessing the health deterioration of multiple laboratory animals.

Deep learning and remote photoplethysmography powered advancements in contactless physiological measurement

In recent decades, there has been ongoing development in the application of computer vision (CV) in the medical field. As conventional contact-based physiological measurement techniques often restrict a patient's mobility in the clinical environment, the ability to achieve continuous, comfortable and convenient monitoring is thus a topic of interest to researchers. One type of CV application is remote imaging photoplethysmography (rPPG), which can predict vital signs using a video or image. While contactless physiological measurement techniques have an excellent application prospect, the lack of uniformity or standardization of contactless vital monitoring methods limits their application in remote healthcare/telehealth settings. Several methods have been developed to improve this limitation and solve the heterogeneity of video signals caused by movement, lighting, and equipment. The fundamental algorithms include traditional algorithms with optimization and developing deep learning (DL) algorithms. This article aims to provide an in-depth review of current Artificial Intelligence (AI) methods using CV and DL in contactless physiological measurement and a comprehensive summary of the latest development of contactless measurement techniques for skin perfusion, respiratory rate, blood oxygen saturation, heart rate, heart rate variability, and blood pressure.

Cross-Attention Enhanced Pyramid Multi-Scale Networks for Sensor-Based Human Activity Recognition

Human Activity Recognition (HAR) has recently attracted widespread attention, with the effective application of this technology helping people in areas such as healthcare, smart homes, and gait analysis. Deep learning methods have shown remarkable performance in HAR. A pivotal challenge is the trade-off between recognition accuracy and computational efficiency, especially in resource-constrained mobile devices. This challenge necessitates the development of models that enhance feature representation capabilities without imposing additional computational burdens. Addressing this, we introduce a novel HAR model leveraging deep learning, ingeniously designed to navigate the accuracy-efficiency trade-off. The model comprises two innovative modules: 1) Pyramid Multi-scale Convolutional Network (PMCN), which is designed with a symmetric structure and is capable of obtaining a rich receptive field at a finer level through its multiscale representation capability; 2) Cross-Attention Mechanism, which establishes interrelationships among sensor dimensions, temporal dimensions, and channel dimensions, and effectively enhances useful information while suppressing irrelevant data. The proposed model is rigorously evaluated across four diverse datasets: UCI, WISDM, PAMAP2, and OPPORTUNITY. Additional ablation and comparative studies are conducted to comprehensively assess the performance of the model. Experimental results demonstrate that the proposed model achieves superior activity recognition accuracy while maintaining low computational overhead.

Enhanced Contactless Heart Rate Monitoring Using Camera with Motion Artifact Removal During Physical Activities

Contactless monitoring of heart rate (HR) can improve passive and continuous tracking of cardiovascular activities and overall people's health. Remote photoplethysmography (rPPG) using a camera eliminates the need for a wearable device. rPPG-based HR has shown promising results to be accurate and comparable to conventional methods such as contact PPG. Most experiments use stationary subjects while motion is known to affect the accuracy of remote PPG. In this paper, a novel methodology is introduced to enhance the accuracy and reliability of HR monitoring based on rPPG in the presence of physical activities like Yoga. This method quickly and accurately tracks HR and analyzes head motion to exclude unreliable data within short windows of rPPG signals. The method was tested with smartphone video data collected from 60 subjects when they are doing activities with varying levels of movement. Results show that our method without motion removal improves the accuracy of the HR readings by 0.7 bpm, reaching 3.57 bpm on average for a 30-sec-window. The accuracy is further improved by another 1.3 bpm after removing the motion artifacts, and reaches 2.29 bpm.Clinical relevance- The enhancement of HR readings from shorter rPPG signal with motion tolerance during physical activities can ultimately help with a more reliable HR tracking of people in uncontrolled settings like home which is a critical step towards remote health-care or wellness tracking.

NeoCam: An Edge-Cloud Platform for Non-Invasive Real-Time Monitoring in Neonatal Intensive Care Units

In this work we introduce NeoCam, an open source hardware-software platform for video-based monitoring of preterms infants in Neonatal Intensive Care Units (NICUs). NeoCam includes an edge computing device that performs video acquisition and processing in real-time. Compared to other proposed solutions, it has the advantage of handling data more efficiently by performing most of the processing on the device, including proper anonymisation for better compliance with privacy regulations. In addition, it allows to perform various video analysis tasks of clinical interest in parallel at speeds of between 20 and 30 frames-per-second. We introduce algorithms to measure without contact the breathing rate, motor activity, body pose and emotional status of the infants. For breathing rate, our system shows good agreement with existing methods provided there is sufficient light and proper imaging conditions. Models for motor activity and stress detection are new to the best of our knowledge. NeoCam has been tested on preterms in the NICU of the University Hospital Puerta del Mar (Cádiz, Spain), and we report the lessons learned from this trial.

Two-Stage Motion Artifact Reduction Algorithm for rPPG Signals Obtained from Facial Video Recordings

Recent years have witnessed the publication of many research articles regarding the contactless measurement and monitoring of heart rate signals deduced from facial video recordings. The techniques presented in these articles, such as examining the changes in the heart rate of an infant, provide a noninvasive assessment in many cases where the direct placement of any hardware equipment is undesirable. However, performing accurate measurements in cases that include noise motion artifacts still presents an obstacle to overcome. In this research article, a two-stage method for noise reduction in facial video recording is proposed. The first stage of the system consists of dividing each (30) seconds of the acquired signal into (60) partitions and then shifting each partition to the mean level before recombining them to form the estimated heart rate signal. The second stage utilizes the wavelet transform for denoising the signal obtained from the first stage. The denoised signal is compared to a reference signal acquired from a pulse oximeter, resulting in the mean bias error (0.13), root mean square error (3.41) and correlation coefficient (0.97). The proposed algorithm is applied to (33) individuals being subjected to a normal webcam for acquiring their video recording, which can easily be performed at homes, hospitals, or any other environment. Finally, it is worth noting that this noninvasive remote technique is useful for acquiring the heart signal while preserving social distancing, which is a desirable feature in the current period of COVID-19.

Benchmarking of Contactless Heart Rate Measurement Systems in ARM-Based Embedded Platforms

Heart rate monitoring is especially important for aging individuals because it is associated with longevity and cardiovascular risk. Typically, this vital parameter can be measured using wearable sensors, which are widely available commercially. However, wearable sensors have some disadvantages in terms of acceptability, especially when used by elderly people. Thus, contactless solutions have increasingly attracted the scientific community in recent years. Camera-based photoplethysmography (also known as remote photoplethysmography) is an emerging method of contactless heart rate monitoring that uses a camera and a processing unit on the hardware side, and appropriate image processing methodologies on the software side. This paper describes the design and implementation of a novel pipeline for heart rate estimation using a commercial and low-cost camera as the input device. The pipeline's performance was tested and compared on a desktop PC, a laptop, and three different ARM-based embedded platforms (Raspberry Pi 4, Odroid N2+, and Jetson Nano). The results showed that the designed and implemented pipeline achieved an average accuracy of about 96.7% for heart rate estimation, with very low variance (between 1.5% and 2.5%) across processing platforms, user distances from the camera, and frame resolutions. Furthermore, benchmark analysis showed that the Odroid N2+ platform was the most convenient in terms of CPU load, RAM usage, and average execution time of the algorithmic pipeline.

On the improvement of heart rate prediction using the combination of singular spectrum analysis and copula-based analysis approach

In recent years, many people have been working from home due to the exceptional circumstances concerning the coronavirus disease 2019 (COVID-19) pandemic. It has also negatively influenced general health and quality of life. Therefore, physical activity has been gaining much attention in preventing the spread of Severe Acute Respiratory Syndrome Coronavirus. For planning an effective physical activity for different clients, physical activity intensity and load degree needs to be appropriately adjusted depending on the individual's physical/health conditions. Heart rate (HR) is one of the most critical health indicators for monitoring exercise intensity and load degree because it is closely related to the heart rate. Heart rate prediction estimates the heart rate at the next moment based on now and other influencing factors. Therefore, an accurate short-term HR prediction technique can deliver efficient early warning for human health and decrease the happening of harmful events. The work described in this article aims to introduce a novel hybrid approach to model and predict the heart rate dynamics for different exercises. The results indicate that the combination of singular spectrum analysis (SSA) and the Clayton Copula model can accurately predict HR for the short term.

Intelligent Remote Photoplethysmography-Based Methods for Heart Rate Estimation from Face Videos: A Survey

Over the last few years, a rich amount of research has been conducted on remote vital sign monitoring of the human body. Remote photoplethysmography (rPPG) is a camera-based, unobtrusive technology that allows continuous monitoring of changes in vital signs and thereby helps to diagnose and treat diseases earlier in an effective manner. Recent advances in computer vision and its extensive applications have led to rPPG being in high demand. This paper specifically presents a survey on different remote photoplethysmography methods and investigates all facets of heart rate analysis. We explore the investigation of the challenges of the video-based rPPG method and extend it to the recent advancements in the literature. We discuss the gap within the literature and suggestions for future directions.

Automated Remote Pulse Oximetry System (ARPOS)

Current methods of measuring heart rate (HR) and oxygen levels (SPO2) require physical contact, are individualised, and for accurate oxygen levels may also require a blood test. No-touch or non-invasive technologies are not currently commercially available for use in healthcare settings. To date, there has been no assessment of a system that measures HR and SPO2 using commercial off-the-shelf camera technology that utilises R, G, B, and IR data. Moreover, no formal remote photoplethysmography studies have been performed in real-life scenarios with participants at home with different demographic characteristics. This novel study addresses all these objectives by developing, optimising, and evaluating a system that measures the HR and SPO2 of 40 participants. HR and SPO2 are determined by measuring the frequencies from different wavelength band regions using FFT and radiometric measurements after pre-processing face regions of interest (forehead, lips, and cheeks) from colour, IR, and depth data. Detrending, interpolating, hamming, and normalising the signal with FastICA produced the lowest RMSE of 7.8 for HR with the r-correlation value of 0.85 and RMSE 2.3 for SPO2. This novel system could be used in several critical care settings, including in care homes and in hospitals and prompt clinical intervention as required.

A Predictive Analysis of Heart Rates Using Machine Learning Techniques

Heart disease, caused by low heart rate, is one of the most significant causes of mortality in the world today. Therefore, it is critical to monitor heart health by identifying the deviation in the heart rate very early, which makes it easier to detect and manage the heart’s function irregularities at a very early stage. The fast-growing use of advanced technology such as the Internet of Things (IoT), wearable monitoring systems and artificial intelligence (AI) in the healthcare systems has continued to play a vital role in the analysis of huge amounts of health-based data for early and accurate disease detection and diagnosis for personalized treatment and prognosis evaluation. It is then important to analyze the effectiveness of using data analytics and machine learning to monitor and predict heart rates using wearable device (accelerometer)-generated data. Hence, in this study, we explored a number of powerful data-driven models including the autoregressive integrated moving average (ARIMA) model, linear regression, support vector regression (SVR), k-nearest neighbor (KNN) regressor, decision tree regressor, random forest regressor and long short-term memory (LSTM) recurrent neural network algorithm for the analysis of accelerometer data to make future HR predictions from the accelerometer’s univariant HR time-series data from healthy people. The performances of the models were evaluated under different durations. Evaluated on a very recently created data set, our experimental results demonstrate the effectiveness of using an ARIMA model with a walk-forward validation and linear regression for predicting heart rate under all durations and other models for durations longer than 1 min. The results of this study show that employing these data analytics techniques can be used to predict future HR more accurately using accelerometers.

Non-contact physiological monitoring of post-operative patients in the intensive care unit

Prolonged non-contact camera-based monitoring in critically ill patients presents unique challenges, but may facilitate safe recovery. A study was designed to evaluate the feasibility of introducing a non-contact video camera monitoring system into an acute clinical setting. We assessed the accuracy and robustness of the video camera-derived estimates of the vital signs against the electronically-recorded reference values in both day and night environments. We demonstrated non-contact monitoring of heart rate and respiratory rate for extended periods of time in 15 post-operative patients. Across day and night, heart rate was estimated for up to 53.2% (103.0 h) of the total valid camera data with a mean absolute error (MAE) of 2.5 beats/min in comparison to two reference sensors. We obtained respiratory rate estimates for 63.1% (119.8 h) of the total valid camera data with a MAE of 2.4 breaths/min against the reference value computed from the chest impedance pneumogram. Non-contact estimates detected relevant changes in the vital-sign values between routine clinical observations. Pivotal respiratory events in a post-operative patient could be identified from the analysis of video-derived respiratory information. Continuous vital-sign monitoring supported by non-contact video camera estimates could be used to track early signs of physiological deterioration during post-operative care.

Camera-based heart rate estimation for hospitalized newborns in the presence of motion artifacts

Background

Heart rate (HR) is an important vital sign for evaluating the physiological condition of a newborn infant. Recently, for measuring HR, novel RGB camera-based non-contact techniques have demonstrated their specific superiority compared with other techniques, such as dopplers and thermal cameras. However, they still suffered poor robustness in infants’ HR measurements due to frequent body movement.

Methods

This paper introduces a framework to improve the robustness of infants’ HR measurements by solving motion artifact problems. Our solution is based on the following steps: morphology-based filtering, region-of-interest (ROI) dividing, Eulerian video magnification and majority voting. In particular, ROI dividing improves ROI information utilization. The majority voting scheme improves the statistical robustness by choosing the HR with the highest probability. Additionally, we determined the dividing parameter that leads to the most accurate HR measurements. In order to examine the performance of the proposed method, we collected 4 hours of videos and recorded the corresponding electrocardiogram (ECG) of 9 hospitalized neonates under two different conditions—rest still and visible movements.

Results

Experimental results indicate a promising performance: the mean absolute error during rest still and visible movements are 3.39 beats per minute (BPM) and 4.34 BPM, respectively, which improves at least 2.00 and 1.88 BPM compared with previous works. The Bland-Altman plots also show the remarkable consistency of our results and the HR derived from the ground-truth ECG.

Conclusions

To the best of our knowledge, this is the first study aimed at improving the robustness of neonatal HR measurement under motion artifacts using an RGB camera. The preliminary results have shown the promising prospects of the proposed method, which hopefully reduce neonatal mortality in hospitals.

A review on utilizing machine learning technology in the fields of electronic emergency triage and patient priority systems in telemedicine: Coherent taxonomy, motivations, open research challenges and recommendations for intelligent future work

BACKGROUND

With the remarkable increasing in the numbers of patients, the triaging and prioritizing patients into multi-emergency level is required to accommodate all the patients, save more lives, and manage the medical resources effectively. Triaging and prioritizing patients becomes particularly challenging especially for the patients who are far from hospital and use telemedicine system. To this end, the researchers exploiting the useful tool of machine learning to address this challenge. Hence, carrying out an intensive investigation and in-depth study in the field of using machine learning in E-triage and patient priority are essential and required.

OBJECTIVES

This research aims to (1) provide a literature review and an in-depth study on the roles of machine learning in the fields of electronic emergency triage (E-triage) and prioritize patients for fast healthcare services in telemedicine applications. (2) highlight the effectiveness of machine learning methods in terms of algorithms, medical input data, output results, and machine learning goals in remote healthcare telemedicine systems. (3) present the relationship between machine learning goals and the electronic triage processes specifically on the: triage levels, medical features for input, outcome results as outputs, and the relevant diseases. (4), the outcomes of our analyses are subjected to organize and propose a cross-over taxonomy between machine learning algorithms and telemedicine structure. (5) present lists of motivations, open research challenges and recommendations for future intelligent work for both academic and industrial sectors in telemedicine and remote healthcare applications.

METHODS

An intensive research is carried out by reviewing all articles related to the field of E-triage and remote priority systems that utilise machine learning algorithms and sensors. We have searched all related keywords to investigate the databases of Science Direct, IEEE Xplore, Web of Science, PubMed, and Medline for the articles, which have been published from January 2012 up to date.

RESULTS

A new crossover matching between machine learning methods and telemedicine taxonomy is proposed. The crossover-taxonomy is developed in this study to identify the relationship between machine learning algorithm and the equivalent telemedicine categories whereas the machine learning algorithm has been utilized. The impact of utilizing machine learning is composed in proposing the telemedicine architecture based on synchronous (real-time/ online) and asynchronous (store-and-forward / offline) structure. In addition to that, list of machine learning algorithms, list of the performance metrics, list of inputs data and outputs results are presented. Moreover, open research challenges, the benefits of utilizing machine learning and the recommendations for new research opportunities that need to be addressed for the synergistic integration of multidisciplinary works are organized and presented accordingly.

DISCUSSION

The state-of-the-art studies on the E-triage and priority systems that utilise machine learning algorithms in telemedicine architecture are discussed. This approach allows the researchers to understand the modernisation of healthcare systems and the efficient use of artificial intelligence and machine learning. In particular, the growing worldwide population and various chronic diseases such as heart chronic diseases, blood pressure and diabetes, require smart health monitoring systems in E-triage and priority systems, in which machine learning algorithms could be greatly beneficial.

CONCLUSIONS

Although research directions on E-triage and priority systems that use machine learning algorithms in telemedicine vary, they are equally essential and should be considered. Hence, we provide a comprehensive review to emphasise the advantages of the existing research in multidisciplinary works of artificial intelligence, machine learning and healthcare services.

A Review of Deep Learning-Based Contactless Heart Rate Measurement Methods

The interest in contactless or remote heart rate measurement has been steadily growing in healthcare and sports applications. Contactless methods involve the utilization of a video camera and image processing algorithms. Recently, deep learning methods have been used to improve the performance of conventional contactless methods for heart rate measurement. After providing a review of the related literature, a comparison of the deep learning methods whose codes are publicly available is conducted in this paper. The public domain UBFC dataset is used to compare the performance of these deep learning methods for heart rate measurement. The results obtained show that the deep learning method PhysNet generates the best heart rate measurement outcome among these methods, with a mean absolute error value of 2.57 beats per minute and a mean square error value of 7.56 beats per minute.

Contact-Free Pulse Signal Extraction from Human Face Videos: A Review and New Optimized Filtering Approach

In this chapter, we review methods for video-based heart monitoring, from classical signal processing approaches to modern deep learning methods. In addition, we propose a new method for learning an optimal filter that can overcome many of the problems that can affect classical approaches, such as light reflection and subject's movements, at a fraction of the training cost of deep learning approaches. Following the usual procedures for region of interest extraction and tracking, robust skin color estimation and signal pre-processing, we introduce a least-squares error optimal filter, learnt using an established training dataset to estimate the photoplethysmographic (PPG) signal more accurately from the measured color changes over time. This method not only improves the accuracy of heart rate measurement but also resulted in the extraction of a cleaner pulse signal, which could be integrated into many other useful applications such as human biometric recognition or recognition of emotional state. The method was tested on the DEAP dataset and showed improved performance over the best previous classical method on that dataset. The results obtained show that our proposed contact-free heart rate measurement method has significantly improved on existing methods.

Modulation Model of the Photoplethysmography Signal for Vital Sign Extraction

This paper introduces an amplitude and frequency modulation (AM-FM) model to characterize the photoplethysmography (PPG) signal. The model indicates that the PPG signal spectrum contains one dominant frequency component - the heart rate (HR), which is guarded by two weaker frequency components on both sides; the distance from the dominant component to the guard components represents the respiratory rate (RR). Based on this model, an efficient algorithm is proposed to estimate both HR and RR by searching for the dominant frequency component and two guard components. The proposed method is performed in the frequency domain to estimate RR, which is more robust to additive noise than the prior art based on temporal features. Experiments were conducted on two types of PPG signals collected with a contact sensor (an oximeter) and a contactless visible imaging sensor (a color camera), respectively. The PPG signal from the contactless sensor is much noisier than the signal from the contact sensor. The experimental results demonstrate the effectiveness of the proposed algorithm, including under relatively noisy scenarios.

Jaya Spider Monkey Optimization-driven Deep Convolutional LSTM for the prediction of COVID’19

COVID’19 is an emerging disease and the precise epidemiological profile does not exist in the world. Hence, the COVID’19 outbreak is treated as a Public Health Emergency of the International Concern by the World Health Organization (WHO). Hence, an effective and optimal prediction of COVID’19 mechanism, named Jaya Spider Monkey Optimization-based Deep Convolutional long short-term classifier (JayaSMO-based Deep ConvLSTM) is proposed in this research to predict the rate of confirmed, death, and recovered cases from the time series data. The proposed COVID’19 prediction method uses the COVID’19 data, which is the trending domain of research at the current era of fighting the COVID’19 attacks thereby, to reduce the death toll. However, the proposed JayaSMO algorithm is designed by integrating the Spider Monkey Optimization (SMO) with the Jaya algorithm, respectively. The Deep ConvLSTM classifier facilitates to predict the COVID’19 from the time series data based on the fitness function. Besides, the technical indicators, such as Relative Strength Index (RSI), Rate of Change (ROCR), Exponential Moving Average (EMA), Williams %R, Double Exponential Moving Average (DEMA), and Stochastic %K, are extracted effectively for further processing. Thus, the resulted output of the proposed JayaSMO-based Deep ConvLSTM is employed for COVID’19 prediction. Moreover, the developed model obtained the better performance using the metrics, like Mean Square Error (MSE), and Root Mean Square Error (RMSE) by considering confirmed, death, and the recovered cases of COVID’19 for China and Oman. Thus, the proposed JayaSMO-based Deep ConvLSTM showed improved results with a minimal MSE of 1.791, and the minimal RMSE of 1.338 based on confirmed cases in Oman. In addition, the developed model achieved the death cases with the values of 1.609, and 1.268 for MSE and RMSE, whereas the MSE and the RMSE value of 1.945, and 1.394 is achieved by the developed model using recovered cases in China.

Business Process Management and Digital Innovations: A Systematic Literature Review

Emerging technologies have capabilities to reshape business process management (BPM) from its traditional version to a more explorative variant. However, to exploit the full benefits of new IT, it is essential to reveal BPM’s research potential and to detect recent trends in practice. Therefore, this work presents a systematic literature review (SLR) with 231 recent academic articles (from 2014 until May 2019) that integrate BPM with digital innovations (DI). We position those articles against seven future BPM-DI trends that were inductively derived from an expert panel. By complementing the expected trends in practice with a state-of-the-art literature review, we are able to derive covered and uncovered themes in order to help bridge a rigor-relevance gap. The major technological impacts within the BPM field seem to focus on value creation, customer engagement and managing human-centric and knowledge-intensive business processes. Finally, our findings are categorized into specific calls for research and for action to let scholars and organizations better prepare for future digital needs.

Is Continuous Heart Rate Monitoring of Livestock a Dream or Is It Realistic? A Review

For all homoeothermic living organisms, heart rate (HR) is a core variable to control the metabolic energy production in the body, which is crucial to realize essential bodily functions. Consequently, HR monitoring is becoming increasingly important in research of farm animals, not only for production efficiency, but also for animal welfare. Real-time HR monitoring for humans has become feasible though there are still shortcomings for continuously accurate measuring. This paper is an effort to estimate whether it is realistic to get a continuous HR sensor for livestock that can be used for long term monitoring. The review provides the reported techniques to monitor HR of living organisms by emphasizing their principles, advantages, and drawbacks. Various properties and capabilities of these techniques are compared to check the potential to transfer the mostly adequate sensor technology of humans to livestock in term of application. Based upon this review, we conclude that the photoplethysmographic (PPG) technique seems feasible for implementation in livestock. Therefore, we present the contributions to overcome challenges to evolve to better solutions. Our study indicates that it is realistic today to develop a PPG sensor able to be integrated into an ear tag for mid-sized and larger farm animals for continuously and accurately monitoring their HRs.

Motion Assessment for Accelerometric and Heart Rate Cycling Data Analysis

Motion analysis is an important topic in the monitoring of physical activities and recognition of neurological disorders. The present paper is devoted to motion assessment using accelerometers inside mobile phones located at selected body positions and the records of changes in the heart rate during cycling, under different body loads. Acquired data include 1293 signal segments recorded by the mobile phone and the Garmin device for uphill and downhill cycling. The proposed method is based upon digital processing of the heart rate and the mean power in different frequency bands of accelerometric data. The classification of the resulting features was performed by the support vector machine, Bayesian methods, k-nearest neighbor method, and neural networks. The proposed criterion is then used to find the best positions for the sensors with the highest discrimination abilities. The results suggest the sensors be positioned on the spine for the classification of uphill and downhill cycling, yielding an accuracy of 96.5% and a cross-validation error of 0.04 evaluated by a two-layer neural network system for features based on the mean power in the frequency bands 〈 3 , 8 〉 and 〈 8 , 15 〉 Hz. This paper shows the possibility of increasing this accuracy to 98.3% by the use of more features and the influence of appropriate sensor positioning for motion monitoring and classification.

Neuroscience in service research: an overview and discussion of its possibilities

Purpose

The purpose of this paper is to discuss recent developments in neuroscientific methods and demonstrate its potential for the service field. This work is a call to action for more service researchers to adopt promising and increasingly accessible neuro-tools that allow the service field to benefit from neuroscience theories and insights.

Design/methodology/approach

The paper synthesizes key literature from a variety of domains (e.g. neuroscience, consumer neuroscience and organizational neuroscience) to provide an in-depth background to start applying neuro-tools. Specifically, this paper outlines the most important neuro-tools today and discusses their theoretical and empirical value.

Findings

To date, the use of neuro-tools in the service field is limited. This is surprising given the great potential they hold to advance service research. To stimulate the use of neuro-tools in the service area, the authors provide a roadmap to enable neuroscientific service studies and conclude with a discussion on promising areas (e.g. service experience and servicescape) ripe for neuroscientific input.

Originality/value

The paper offers service researchers a starting point to understand the potential benefits of adopting the neuroscientific method and shows their complementarity with traditional service research methods like surveys, experiments and qualitative research. In addition, this paper may also help reviewers and editors to better assess the quality of neuro-studies in service.

3D Convolutional Neural Networks for Remote Pulse Rate Measurement and Mapping from Facial Video

Remote pulse rate measurement from facial video has gained particular attention over the last few years. Research exhibits significant advancements and demonstrates that common video cameras correspond to reliable devices that can be employed to measure a large set of biomedical parameters without any contact with the subject. A new framework for measuring and mapping pulse rate from video is presented in this pilot study. The method, which relies on convolutional 3D networks, is fully automatic and does not require any special image preprocessing. In addition, the network ensures concurrent mapping by producing a prediction for each local group of pixels. A particular training procedure that employs only synthetic data is proposed. Preliminary results demonstrate that this convolutional 3D network can effectively extract pulse rate from video without the need for any processing of frames. The trained model was compared with other state-of-the-art methods on public data. Results exhibit significant agreement between estimated and ground-truth measurements: the root mean square error computed from pulse rate values assessed with the convolutional 3D network is equal to 8.64 bpm, which is superior to 10 bpm for the other state-of-the-art methods. The robustness of the method to natural motion and increases in performance correspond to the two main avenues that will be considered in future works.

Robust heart-rate estimation from facial videos using Project_ICA

OBJECTIVE

Remote photoplethysmography (rPPG) can achieve non-contact measurement of heart rate (HR) from a continuous video sequence by scanning the skin surface. However, practical applications are still limited by factors such as non-rigid facial motion and head movement. In this work, a detailed system framework for remotely estimating heart rate from facial videos under various movement conditions is described.

APPROACH

After the rPPG signal has been obtained from a defined region of the facial skin, a method, termed 'Project_ICA', based on a skin reflection model, is employed to extract the pulse signal from the original signal.

MAIN RESULTS

To evaluate the performance of the proposed algorithm, a dataset containing 112 videos including the challenges of various skin tones, body motion and HR recovery after exercise was created from 28 participants.

SIGNIFICANCE

The results show that Project_ICA, when evaluated by several criteria, provides a more accurate and robust estimate of HR than most existing methods, although problems remain in obtaining reliable measurements from dark-skinned subjects.

Game-Calibrated and User-Tailored Remote Detection of Stress and Boredom in Games

Emotion detection based on computer vision and remote extraction of user signals commonly rely on stimuli where users have a passive role with limited possibilities for interaction or emotional involvement, e.g., images and videos. Predictive models are also trained on a group level, which potentially excludes or dilutes key individualities of users. We present a non-obtrusive, multifactorial, user-tailored emotion detection method based on remotely estimated psychophysiological signals. A neural network learns the emotional profile of a user during the interaction with calibration games, a novel game-based emotion elicitation material designed to induce emotions while accounting for particularities of individuals. We evaluate our method in two experiments ( n = 20 and n = 62 ) with mean classification accuracy of 61.6%, which is statistically significantly better than chance-level classification. Our approach and its evaluation present unique circumstances: our model is trained on one dataset (calibration games) and tested on another (evaluation game), while preserving the natural behavior of subjects and using remote acquisition of signals. Results of this study suggest our method is feasible and an initiative to move away from questionnaires and physical sensors into a non-obtrusive, remote-based solution for detecting emotions in a context involving more naturalistic user behavior and games.

Cardiovascular assessment by imaging photoplethysmography - a review

Over the last few years, the contactless acquisition of cardiovascular parameters using cameras has gained immense attention. The technique provides an optical means to acquire cardiovascular information in a very convenient way. This review provides an overview on the technique's background and current realizations. Besides giving detailed information on the most widespread application of the technique, namely the contactless acquisition of heart rate, we outline further concepts and we critically discuss the current state.

Heart Rate Variability in Dental Science

Dentistry has made progress as a profession by integration with both medicine and other human sciences, especially when it uses empirical metrics to study process and outcome variables. Notably, progress in our understanding of genomic, biomic, and other molecular biological phenomena has been valuable. As has been identified by Drury (1, 2), it is proposed in this commentary that the inclusion of heart rate variability (HRV) as a biomarker of health may further this integrative progress. HRV is derived by various linear and non-linear statistical analyses of the R-R, beat-to-beat ECG interval in microseconds. Over twenty three thousand reports are identified in a recent PubMed search of the term heart rate variability, most of which demonstrate HRV's sensitivity to a wide diversity of physical and psychosocial pathologies. The small literature of dental use of HRV in both assessment and treatment will be selectively reviewed and relevant exemplars for other important health applications of HRV will be discussed. This will lead to a proposed agenda for researching HRV's value to professional dentistry as a human health and wellness profession.

A Maximum Likelihood Formulation To Exploit Heart Rate Variability for Robust Heart Rate Estimation From Facial Video

The problem of estimating the heart rate (HR) from a racial video is considered. A typical approach for this problem is to use independent component analysis (ICA) on the red, blue, green intensity prof iles averaged over the facial region. This provides estimates of the underlying source signals, whose spectral peaks are used to predict HR in every analysis window. In this work, we propose a maximum likelihood formulation to optimally select a source signal in each window such that the predicted HR trajectory not only corresponds to the most likely spectral peaks but also ensures a realistic HR variability (HRV) across analysis windows. The likelihood function is efficiently optimized using dynamic programming in a manner similar to Viterbi decoding. The proposed scheme for HR estimation is denoted by vICA. The performance of vICA is compared with a typical ICA approach as well as a recently proposed sparse spectral peak tracking (SSPT) technique that ensures that the predicted HR does not vary drastically across analysis windows. Experiments are performed in a five fold setup using racial videos of 15 subjects recorded using two types of smartphones (Samsung Galaxy and iPhone) at three different distances (6inches, lfoot, 2feet) between the phone camera and the subject. Mean absolute error (MAE) between the original and predicted HR reveals that the proposed vICA scheme performs better than the best of the baseline schemes, namely SSPT by -8.69%, 52.77% and 8.00% when Samsung Galaxy phone was used at a distance of 6inches, lfoot, and 2feet respectively. These improvements are 12.13%, 13.59% and 18.34% when iPhone was used. This, in turn, suggests that the HR predicted from a racial video becomes more accurate when the smoothness of HRV is utilized in predicting the HR trajectory as done in the proposed vICA.

Fast And Robust Heart Rate Estimation From Videos Through Dynamic Region Selection

Remote heart rate (HR) estimation from videos is useful because it facilitates monitoring ongoing health conditions without sensors that are often uncomfortable to wear. In the HR estimation from videos, choice of the image region, at which the HR is calculated, is critically important as it greatly affects the estimation accuracy. In this paper, a novel algorithm for HR estimation that uses dynamic region selection is proposed. The image regions that clearly contain pulse waveforms are quickly found by a region selector using a machine learning technique. In addition, the proposed method enhances the robustness of tracking the temporal change of the HR by using a particle filter. The experimental results show that the proposed method achieves the absolute average error less than 1.1BPM (Beats Per Minute) with the processing time less than 0.6s for a single HR estimation.

Remote Heart Rate Measurement from RGB-NIR Video Based on Spatial and Spectral Face Patch Selection

In this paper, we propose a novel heart rate (HR) estimation method using simultaneously recorded RGB and near-infrared (NIR) face videos. The key idea of our method is to automatically select suitable face patches for HR estimation in both spatial and spectral domains. The spatial and spectral face patch selection enables us to robustly estimate HR under various situations, including scenes under which existing RGB camera-based methods fail to accurately estimate HR. For a challenging scene in low light and with light fluctuations, our method can successfully estimate HR for all 20 subjects $( \pm 3$ beats per minute), while the RGB camera-based methods succeed only for 25% of the subjects.

Heart Rate and Heart Rate Variability From Single-Channel Video and ICA Integration of Multiple Signals

Unobtrusive monitoring of vital signs is relevant for both medical (patient monitoring) and non-medical applications (e.g., stress and fatigue monitoring). In this paper, we focus on the use of imaging photoplethysmography (iPPG). High frame rate videos were acquired by using a monochrome camera and an optical band-pass filter ([Formula: see text] nm). To enhance iPPG signal, we investigated the use of independent component analysis (ICA) pre-processing applied to iPPG signal from different regions of the face. Methodology was tested on [Formula: see text] healthy volunteers. Heart rate (HR) and standard time and frequency domain descriptors of heart rate variability (HRV), simultaneously extracted from videos and ECG data, were compared. A mean absolute error (MAE) about 3.812 ms was observed for normal-to-normal intervals with or without ICA pre-processing. Smaller MAE values of frequency domain descriptors were observed when ICA pre-processing was used. The impact of both video frame rate and video signal interval were also analyzed. All the results support the conclusion that proposed ICA pre-processing can effectively improve the HR and HRV assessment from iPPG.

Artificial intelligence and machine learning in emergency medicine

Interest in artificial intelligence (AI) research has grown rapidly over the past few years, in part thanks to the numerous successes of modern machine learning techniques such as deep learning, the availability of large datasets and improvements in computing power. AI is proving to be increasingly applicable to healthcare and there is a growing list of tasks where algorithms have matched or surpassed physician performance. Despite the successes there remain significant concerns and challenges surrounding algorithm opacity, trust and patient data security. Notwithstanding these challenges, AI technologies will likely become increasingly integrated into emergency medicine in the coming years. This perspective presents an overview of current AI research relevant to emergency medicine.

A study of color illumination effect on the SNR of rPPG signals

Remote photoplethysmography (rPPG) can be used to measure cardiac activity by detecting the subtle color variation of the human skin tissue using an RGB camera. Recent studies have presented the feasibility and proposed multiple methods to improve the motion robustness for the subject movements. However, enhancing the signal-to-noise ratio (SNR) of the rPPG signal is still an important issue for the contactless measurement. In this paper, we conducted an experiment to study the lighting effect on the SNR of rPPG signals. The results point out that different colors of light sources provide different SNR in each RGB channel. By providing the dedicated light sources (λ= 490-620) nm, the SNR of rPPG signals captured from the green color channel can be enhanced. Among the tested light sources, light green provides the most significant improvement from -11.09 to -6.6 dB compared with the fluorescent light.

A Comparative Survey of Methods for Remote Heart Rate Detection From Frontal Face Videos

Remotely measuring physiological activity can provide substantial benefits for both the medical and the affective computing applications. Recent research has proposed different methodologies for the unobtrusive detection of heart rate (HR) using human face recordings. These methods are based on subtle color changes or motions of the face due to cardiovascular activities, which are invisible to human eyes but can be captured by digital cameras. Several approaches have been proposed such as signal processing and machine learning. However, these methods are compared with different datasets, and there is consequently no consensus on method performance. In this article, we describe and evaluate several methods defined in literature, from 2008 until present day, for the remote detection of HR using human face recordings. The general HR processing pipeline is divided into three stages: face video processing, face blood volume pulse (BVP) signal extraction, and HR computation. Approaches presented in the paper are classified and grouped according to each stage. At each stage, algorithms are analyzed and compared based on their performance using the public database MAHNOB-HCI. Results found in this article are limited on MAHNOB-HCI dataset. Results show that extracted face skin area contains more BVP information. Blind source separation and peak detection methods are more robust with head motions for estimating HR.

Multiple Instance Dictionary Learning for Beat-to-Beat Heart Rate Monitoring From Ballistocardiograms

A multiple instance dictionary learning approach, dictionary learning using functions of multiple instances (DL-FUMI), is used to perform beat-to-beat heart rate estimation and to characterize heartbeat signatures from ballistocardiogram (BCG) signals collected with a hydraulic bed sensor. DL-FUMI estimates a "heartbeat concept" that represents an individual's personal ballistocardiogram heartbeat pattern. DL-FUMI formulates heartbeat detection and heartbeat characterization as a multiple instance learning problem to address the uncertainty inherent in aligning BCG signals with ground truth during training. Experimental results show that the estimated heartbeat concept obtained by DL-FUMI is an effective heartbeat prototype and achieves superior performance over comparison algorithms.

Video pulse rate variability analysis in stationary and motion conditions

BACKGROUND

In the last few years, some studies have measured heart rate (HR) or heart rate variability (HRV) parameters using a video camera. This technique focuses on the measurement of the small changes in skin colour caused by blood perfusion. To date, most of these works have obtained HRV parameters in stationary conditions, and there are practically no studies that obtain these parameters in motion scenarios and by conducting an in-depth statistical analysis.

METHODS

In this study, a video pulse rate variability (PRV) analysis is conducted by measuring the pulse-to-pulse (PP) intervals in stationary and motion conditions. Firstly, given the importance of the sampling rate in a PRV analysis and the low frame rate of commercial cameras, we carried out an analysis of two models to evaluate their performance in the measurements. We propose a selective tracking method using the Viola-Jones and KLT algorithms, with the aim of carrying out a robust video PRV analysis in stationary and motion conditions. Data and results of the proposed method are contrasted with those reported in the state of the art.

RESULTS

The webcam achieved better results in the performance analysis of video cameras. In stationary conditions, high correlation values were obtained in PRV parameters with results above 0.9. The PP time series achieved an RMSE (mean ± standard deviation) of 19.45 ± 5.52 ms (1.70 ± 0.75 bpm). In the motion analysis, most of the PRV parameters also achieved good correlation results, but with lower values as regards stationary conditions. The PP time series presented an RMSE of 21.56 ± 6.41 ms (1.79 ± 0.63 bpm).

CONCLUSIONS

The statistical analysis showed good agreement between the reference system and the proposed method. In stationary conditions, the results of PRV parameters were improved by our method in comparison with data reported in related works. An overall comparative analysis of PRV parameters in motion conditions was more limited due to the lack of studies or studies containing insufficient data analysis. Based on the results, the proposed method could provide a low-cost, contactless and reliable alternative for measuring HR or PRV parameters in non-clinical environments.

Novel health monitoring method using an RGB camera

In this paper we present a novel health monitoring method by estimating the heart rate and respiratory rate using an RGB camera. The heart rate and the respiratory rate are estimated from the photoplethysmography (PPG) and the respiratory motion. The method mainly operates by using the green spectrum of the RGB camera to generate a multivariate PPG signal to perform multivariate de-noising on the video signal to extract the resultant PPG signal. A periodicity based voting scheme (PVS) was used to measure the heart rate and respiratory rate from the estimated PPG signal. We evaluated our proposed method with a state of the art heart rate measuring method for two scenarios using the MAHNOB-HCI database and a self collected naturalistic environment database. The methods were furthermore evaluated for various scenarios at naturalistic environments such as a motion variance session and a skin tone variance session. Our proposed method operated robustly during the experiments and outperformed the state of the art heart rate measuring methods by compensating the effects of the naturalistic environment.

Heart Rate Detection Using Microsoft Kinect: Validation and Comparison to Wearable Devices

Contactless detection is one of the new frontiers of technological innovation in the field of healthcare, enabling unobtrusive measurements of biomedical parameters. Compared to conventional methods for Heart Rate (HR) detection that employ expensive and/or uncomfortable devices, such as the Electrocardiograph (ECG) or pulse oximeter, contactless HR detection offers fast and continuous monitoring of heart activities and provides support for clinical analysis without the need for the user to wear a device. This paper presents a validation study for a contactless HR estimation method exploiting RGB (Red, Green, Blue) data from a Microsoft Kinect v2 device. This method, based on Eulerian Video Magnification (EVM), Photoplethysmography (PPG) and Videoplethysmography (VPG), can achieve performance comparable to classical approaches exploiting wearable systems, under specific test conditions. The output given by a Holter, which represents the gold-standard device used in the test for ECG extraction, is considered as the ground-truth, while a comparison with a commercial smartwatch is also included. The validation process is conducted with two modalities that differ for the availability of a priori knowledge about the subjects’ normal HR. The two test modalities provide different results. In particular, the HR estimation differs from the ground-truth by 2% when the knowledge about the subject’s lifestyle and his/her HR is considered and by 3.4% if no information about the person is taken into account.

Blood perfusion display based on video pulse wave

In this study, an easy system for monitoring dynamic blood perfusion patterns and the pulse wave velocity (PWV) has been developed by processing video images of a human body to assess blood circulation for daily management of physical conditions or for detecting persons in poor physical condition in public places. The experiment suggested that this tool can be used to easily evaluate the PWV; however, the obtained value from the video image of the face was about 1/10 of the standard value calculated from thick vessels. This difference may be related to the difference between thick vessels and thin-branched arterioles.

Techniques for estimating blood pressure variation using video images

It is important to know about a sudden blood pressure change that occurs in everyday life and may pose a danger to human health. However, monitoring the blood pressure variation in daily life is difficult because a bulky and expensive sensor is needed to measure the blood pressure continuously. In this study, a new non-contact method is proposed to estimate the blood pressure variation using video images. In this method, the pulse propagation time difference or instantaneous phase difference is calculated between two pulse waves obtained from different parts of a subject's body captured by a video camera. The forehead, left cheek, and right hand are selected as regions to obtain pulse waves. Both the pulse propagation time difference and instantaneous phase difference were calculated from the video images of 20 healthy subjects performing the Valsalva maneuver. These indices are considered to have a negative correlation with the blood pressure variation because they approximate the pulse transit time obtained from a photoplethysmograph. However, the experimental results showed that the correlation coefficients between the blood pressure and the proposed indices were approximately 0.6 for the pulse wave obtained from the right hand. This result is considered to be due to the difference in the transmission depth into the skin between the green and infrared light used as light sources for the video image and conventional photoplethysmogram, respectively. In addition, the difference in the innervation of the face and hand may be related to the results.