Exploiting spatial redundancy of image sensor for motion robust rPPG

Privacy-Protected Contactless Sleep Parameters Measurement Using a Defocused Camera

Sleep monitoring plays a vital role in various scenarios such as hospitals and living-assisted homes, contributing to the prevention of sleep accidents as well as the assessment of sleep health. Contactless camera-based sleep monitoring is promising due to its user-friendly nature and rich visual semantics. However, the privacy concern of video cameras limits their applications in sleep monitoring. In this paper, we explored the opportunity of using a defocused camera that does not allow identification of the monitored subject when measuring sleep-related parameters, as face detection and recognition are impossible on optically blurred images. We proposed a novel privacy-protected sleep parameters measurement framework, including a physiological measurement branch and a semantic analysis branch based on ResNet-18. Four important sleep parameters are measured: heart rate (HR), respiration rate (RR), sleep posture, and movement. The results of HR, RR, and movement have strong correlations with the reference (HR: R = 0.9076; RR: R = 0.9734; Movement: R = 0.9946). The overall mean absolute errors (MAE) for HR and RR are 5.2 bpm and 1.5 bpm respectively. The measurement of HR and RR achieve reliable estimation coverage of 72.1% and 93.6%, respectively. The sleep posture detection achieves an overall accuracy of 94.5%. Experimental results show that the defocused camera is promising for sleep monitoring as it fundamentally eliminates the privacy issue while still allowing the measurement of multiple parameters that are essential for sleep health informatics.

ST-Phys: Unsupervised Spatio-Temporal Contrastive Remote Physiological Measurement

Remote photoplethysmography (rPPG) is a non-contact method that employs facial videos for measuring physiological parameters. Existing rPPG methods have achieved remarkable performance. However, the success mainly profits from supervised learning over massive labeled data. On the other hand, existing unsupervised rPPG methods fail to fully utilize spatio-temporal features and encounter challenges in low-light or noise environments. To address these problems, we propose an unsupervised contrast learning approach, ST-Phys. We incorporate a low-light enhancement module, a temporal dilated module, and a spatial enhanced module to better deal with long-term dependencies under the random low-light conditions. In addition, we design a circular margin loss, wherein rPPG signals originating from identical videos are attracted, while those from distinct videos are repelled. Our method is assessed on six openly accessible datasets, including RGB and NIR videos. Extensive experiments reveal the superior performance of our proposed ST-Phys over state-of-the-art unsupervised rPPG methods. Moreover, it offers advantages in parameter reduction and noise robustness.

Multispectral Depolarization Based Living-Skin Detection: A New Measurement Principle

Camera-based photoplethysmographic imaging enabled the segmentation of living-skin tissues in a video, but it has inherent limitations to be used in real-life applications such as video health monitoring and face anti-spoofing. Inspired by the use of polarization for improving vital signs monitoring (i.e. specular reflection removal), we observed that skin tissues have an attractive property of wavelength-dependent depolarization due to its multi-layer structure containing different absorbing chromophores, i.e. polarized light photons with longer wavelengths (R) have deeper skin penetrability and thus experience thorougher depolarization than those with shorter wavelengths (G and B). Thus we proposed a novel dual-polarization setup and an elegant algorithm (named "MSD") that exploits the nature of multispectral depolarization of skin tissues to detect living-skin pixels, which only requires two images sampled at the parallel and cross polarizations to estimate the characteristic chromaticity changes (R/G) caused by tissue depolarization. Our proposal was verified in both the laboratory and hospital settings (ICU and NICU) focused on anti-spoofing and patient skin segmentation. The clinical experiments in ICU also indicate the potential of MSD for skin perfusion analysis, which may lead to a new diagnostic imaging approach in the future.

HRUNet: Assessing Uncertainty in Heart Rates Measured From Facial Videos

Video-based Photoplethysmography (VPPG) offers the capability to measure heart rate (HR) from facial videos. However, the reliability of the HR values extracted through this method remains uncertain, especially when videos are affected by various disturbances. Confronted by this challenge, we introduce an innovative framework for VPPG-based HR measurements, with a focus on capturing diverse sources of uncertainty in the predicted HR values. In this context, a neural network named HRUNet is structured for HR extraction from input facial videos. Departing from the conventional training approach of learning specific weight (and bias) values, we leverage the Bayesian posterior estimation to derive weight distributions within HRUNet. These distributions allow for sampling to encode uncertainty stemming from HRUNet's limited performance. On this basis, we redefine HRUNet's output as a distribution of potential HR values, as opposed to the traditional emphasis on the single most probable HR value. The underlying goal is to discover the uncertainty arising from inherent noise in the input video. HRUNet is evaluated across 1,098 videos from seven datasets, spanning three scenarios: undisturbed, motion-disturbed, and light-disturbed. The ensuing test outcomes demonstrate that uncertainty in the HR measurements increases significantly in the scenarios marked by disturbances, compared to that in the undisturbed scenario. Moreover, HRUNet outperforms state-of-the-art methods in HR accuracy when excluding HR values with 0.4 uncertainty. This underscores that uncertainty emerges as an informative indicator of potentially erroneous HR measurements. With enhanced reliability affirmed, the VPPG technique holds the promise for applications in safety-critical domains.

DSE-NN: Deeply Supervised Efficient Neural Network for Real-Time Remote Photoplethysmography

Non-contact remote photoplethysmography can be used in a variety of medical and healthcare fields by measuring vital signs continuously and unobtrusively. Recently, end-to-end deep learning methods have been proposed to replace the existing handcrafted features. However, since the existing deep learning methods are known as black box models, the problem of interpretability has been raised, and the same problem exists in the remote photoplethysmography (rPPG) network. In this study, we propose a method to visualize temporal and spectral representations for hidden layers, deeply supervise the spectral representation of intermediate layers through the depth of networks and optimize it for a lightweight model. The optimized network improves performance and enables fast training and inference times. The proposed spectral deep supervision helps to achieve not only high performance but also fast convergence speed through the regularization of the intermediate layers. The effect of the proposed methods was confirmed through a thorough ablation study on public datasets. As a result, similar or outperforming results were obtained in comparison to state-of-the-art models. In particular, our model achieved an RMSE of 1 bpm on the PURE dataset, demonstrating its high accuracy. Moreover, it excelled on the V4V dataset with an impressive RMSE of 6.65 bpm, outperforming other methods. We observe that our model began converging from the very first epoch, a significant improvement over other models in terms of learning efficiency. Our approach is expected to be generally applicable to models that learn spectral domain information as well as to the applications of regression that require the representations of periodicity.

Non-contact high precision pulse-rate monitoring system for moving subjects in different motion states

Remote photoplethysmography (rPPG) enables contact-free monitoring of the pulse rate by using a color camera. The fundamental limitation is that motion artifacts and changes in ambient light conditions greatly affect the accuracy of pulse-rate monitoring. We propose use of a high-speed camera and a motion suppression algorithm with high computational efficiency. This system incorporates a number of major improvements including reproduction of pulse wave details, high-precision pulse-rate monitoring of moving subjects, and excellent scene scalability. A series of quantization methods were used to evaluate the effect of different frame rates and different algorithms in pulse-rate monitoring of moving subjects. The experimental results show that use of 180-fps video and a Plane-Orthogonal-to-Skin (POS) algorithm can produce high-precision pulse-rate monitoring results with mean absolute error can be less than 5 bpm and the relative accuracy reaching 94.5%. Thus, it has significant potential to improve personal health care and intelligent health monitoring.

It does not need two: Assessing physiological linkage from videos across the valence dimension

The phenomenon of physiological linkage describes similar fluctuations of two individuals' physiology, for example, the cardiac inter-beat interval (IBI). Physiological linkage is a well-documented occurrence in research settings of interacting dyads but the literature on non-interacting dyads, that is, someone watching a video of another person, is sparse. The current study investigated whether physiological linkage, based on IBI, occurs from watching videos where strangers report about personal (neutral, positive, negative non-traumatic, and negative traumatic) experiences. Videos were produced with six individuals and then presented to observers (N = 26). Time-frequency-domain cross-wavelet analyses supplemented by threshold-free cluster enhancement (TFCE; to account for multiple testing) showed significant physiological linkage between the IBI of observers and persons in the videos for 16 out of the 21 tested videos. Although significant physiological linkage also emerged for neutral videos and positive, negative valence videos led to such associations more reliably. This study shows that physiological linkage can be investigated in highly controlled conditions based on video stimuli paving the path for experimental manipulation in future research. Furthermore, due to the provision of information on time and frequency, the use of cross-wavelet analysis is encouraged to learn more about factors modulating physiological linkage. The current study presents the next step toward identifying psychophysiological causal and modulating factors of physiological linkage.

Feasibility of Exploiting Physiological and Motion Features for Camera-based Sleep Staging: A Clinical Study

Camera-based sleep monitoring is an emergent research topic in sleep medicine. The feasibility of using both the physiological features and motion features measured by a video camera for sleep staging was not thoroughly investigated. In this paper, we built a camera-based non-contact sleep monitoring setup in the Institute of Respiratory Diseases, Shenzhen People's Hospital, and created a clinical sleep dataset (nocturnal video data of 11 adults) including the expert-corrected PSG references synchronized with the video. The camera-based measurements have shown high correlations with the PSG. It obtains an overall Mean Absolute Error (MAE) of 1.5 bpm for heart-rate (HR), 0.7 bpm for breathing-rate (BR), 13.9 ms for heart-rate variability (HRV), and an accuracy of 93.5% for leg motion detection. The statistical analysis indicates that the averaged HR and variations of BR are distinct features for annotating four sleep stages (awake, REM, light sleep, and deep sleep). HRV parameter (SDNN) can clearly differentiate rapid-eye-movement (REM) and non-REM, while the leg movement is a distinctive feature for separating awake and sleep. The clinical trial demonstrated the feasibility of using physiological and motion features measured by a camera for joint sleep staging, and provides insights for sleep-related feature selection.

Non-contact heart rate estimation based on singular spectrum component reconstruction using low-rank matrix and autocorrelation

The remote photoplethysmography (rPPG) based on cameras, a technology for extracting pulse wave from videos, has been proved to be an effective heart rate (HR) monitoring method and has great potential in many fields; such as health monitoring. However, the change of facial color intensity caused by cardiovascular activities is weak. Environmental illumination changes and subjects' facial movements will produce irregular noise in rPPG signals, resulting in distortion of heart rate pulse signals and affecting the accuracy of heart rate measurement. Given the irregular noises such as motion artifacts and illumination changes in rPPG signals, this paper proposed a new method named LA-SSA. It combines low-rank sparse matrix decomposition and autocorrelation function with singular spectrum analysis (SSA). The low-rank sparse matrix decomposition is employed to globally optimize the components of the rPPG signal obtained by SSA, and some irregular noise is removed. Then, the autocorrelation function is used to optimize the global optimization results locally. The periodic components related to the heartbeat signal are selected, and the denoised rPPG signal is obtained by weighted reconstruction with a singular value ratio. The experiment using UBFC-RPPG and PURE database is performed to assess the performance of the method proposed in this paper. The average absolute error was 1.37 bpm, the 95% confidence interval was -7.56 bpm to 6.45 bpm, and the Pearson correlation coefficient was 98%, superior to most existing video-based heart rate extraction methods. Experimental results show that the proposed method can estimate HR effectively.

Deep learning-based remote-photoplethysmography measurement from short-time facial video

Objective. Efficient non-contact heart rate (HR) measurement from facial video has received much attention in health monitoring. Past methods relied on prior knowledge and an unproven hypothesis to extract remote photoplethysmography (rPPG) signals, e.g. manually designed regions of interest (ROIs) and the skin reflection model.Approach. This paper presents a short-time end to end HR estimation framework based on facial features and temporal relationships of video frames. In the proposed method, a deep 3D multi-scale network with cross-layer residual structure is designed to construct an autoencoder and extract robust rPPG features. Then, a spatial-temporal fusion mechanism is proposed to help the network focus on features related to rPPG signals. Both shallow and fused 3D spatial-temporal features are distilled to suppress redundant information in the complex environment. Finally, a data augmentation strategy is presented to solve the problem of uneven distribution of HR in existing datasets.Main results. The experimental results on four face-rPPG datasets show that our method overperforms the state-of-the-art methods and requires fewer video frames. Compared with the previous best results, the proposed method improves the root mean square error (RMSE) by 5.9%, 3.4% and 21.4% on the OBF dataset (intra-test), COHFACE dataset (intra-test) and UBFC dataset (cross-test), respectively.Significance. Our method achieves good results on diverse datasets (i.e. highly compressed video, low-resolution and illumination variation), demonstrating that our method can extract stable rPPG signals in short time.

Towards a Non-Contact Method for Identifying Stress Using Remote Photoplethysmography in Academic Environments

Stress has become a common condition and is one of the chief causes of university course disenrollment. Most of the studies and tests on academic stress have been conducted in research labs or controlled environments, but these tests can not be extended to a real academic environment due to their complexity. Academic stress presents different associated symptoms, anxiety being one of the most common. This study focuses on anxiety derived from academic activities. This study aims to validate the following hypothesis: by using a non-contact method based on the use of remote photoplethysmography (rPPG), it is possible to identify academic stress levels with an accuracy greater than or equal to that of previous works which used contact methods. rPPG signals from 56 first-year engineering undergraduate students were recorded during an experimental task. The results show that the rPPG signals combined with students' demographic data and psychological scales (the State-Trait Anxiety Inventory) improve the accuracy of different classification methods. Moreover, the results demonstrate that the proposed method provides 96% accuracy by using K-nearest neighbors, J48, and random forest classifiers. The performance metrics show better or equal accuracy compared to other contact methods. In general, this study demonstrates that it is possible to implement a low-cost method for identifying academic stress levels in educational environments.

Contactless facial video recording with deep learning models for the detection of atrial fibrillation

Atrial fibrillation (AF) is often asymptomatic and paroxysmal. Screening and monitoring are needed especially for people at high risk. This study sought to use camera-based remote photoplethysmography (rPPG) with a deep convolutional neural network (DCNN) learning model for AF detection. All participants were classified into groups of AF, normal sinus rhythm (NSR) and other abnormality based on 12-lead ECG. They then underwent facial video recording for 10 min with rPPG signals extracted and segmented into 30-s clips as inputs of the training of DCNN models. Using voting algorithm, the participant would be predicted as AF if > 50% of their rPPG segments were determined as AF rhythm by the model. Of the 453 participants (mean age, 69.3 ± 13.0 years, women, 46%), a total of 7320 segments (1969 AF, 1604 NSR & 3747others) were analyzed by DCNN models. The accuracy rate of rPPG with deep learning model for discriminating AF from NSR and other abnormalities was 90.0% and 97.1% in 30-s and 10-min recording, respectively. This contactless, camera-based rPPG technique with a deep-learning model achieved significantly high accuracy to discriminate AF from non-AF and may enable a feasible way for a large-scale screening or monitoring in the future.

Evaluation of Remote Photoplethysmography Measurement Conditions toward Telemedicine Applications

Camera-based remote photoplethysmography (rPPG) is a low-cost and casual non-contact heart rate measurement method suitable for telemedicine. Several factors affect the accuracy of measuring the heart rate and heart rate variability (HRV) using rPPG despite HRV being an important indicator for healthcare monitoring. This study aimed to investigate the appropriate setup for precise HRV measurements using rPPG while considering the effects of possible factors including illumination, direction of the light, frame rate of the camera, and body motion. In the lighting conditions experiment, the smallest mean absolute R–R interval (RRI) error was obtained when light greater than 500 lux was cast from the front (among the following conditions—illuminance: 100, 300, 500, and 700 lux; directions: front, top, and front and top). In addition, the RRI and HRV were measured with sufficient accuracy at frame rates above 30 fps. The accuracy of the HRV measurement was greatly reduced when the body motion was not constrained; thus, it is necessary to limit the body motion, especially the head motion, in an actual telemedicine situation. The results of this study can act as guidelines for setting up the shooting environment and camera settings for rPPG use in telemedicine.

Contactless Vital Sign Monitoring System for Heart and Respiratory Rate Measurements with Motion Compensation Using a Near-Infrared Time-of-Flight Camera

This study describes a contactless vital sign monitoring (CVSM) system capable of measuring heart rate (HR) and respiration rate (RR) using a low-power, indirect time-of-flight (ToF) camera. The system takes advantage of both the active infrared illumination as well as the additional depth information from the ToF camera to compensate for the motion-induced artifacts during the HR measurements. The depth information captures how the user is moving with respect to the camera and, therefore, can be used to differentiate where the intensity change in the raw signal is from the underlying heartbeat or motion. Moreover, from the depth information, the system can acquire respiration rate by directly measuring the motion of the chest wall during breathing. We also conducted a pilot human study using this system with 29 participants of different demographics such as age, gender, and skin color. Our study shows that with depth-based motion compensation, the success rate (system measurement within 10% of reference) of HR measurements increases to 75%, as compared to 35% when motion compensation is not used. The mean HR deviation from the reference also drops from 21 BPM to −6.25 BPM when we apply the depth-based motion compensation. In terms of the RR measurement, our system shows a mean deviation of 1.7 BPM from the reference measurement. The pilot human study shows the system performance is independent of skin color but weakly dependent on gender and age.

VidAF: A Motion-Robust Model for Screening Atrial Fibrillation from Facial Videos

Atrial fibrillation (AF) is the most common arrhythmia, but an estimated 30% of patients with AF are unaware of their conditions. The purpose of this work is to design a model for AF screening from facial videos, with a focus on addressing typical motion disturbances in our real life, such as head movements and expression changes. This model detects a pulse signal from the skin color changes in a facial video by a convolution neural network, incorporating a phase-driven attention mechanism to suppress motion signals in the space domain. It then encodes the pulse signal into discriminative features for AF classification by a coding neural network, using a de-noise coding strategy to improve the robustness of the features to motion signals in the time domain. The proposed model was tested on a dataset containing 1200 samples of 100 AF patients and 100 non-AF subjects. Experimental results demonstrated that VidAF had significant robustness to facial motions, predicting clean pulse signals with the mean absolute error of inter-pulse intervals less than 100 milliseconds. Besides, the model achieved promising performance in AF identification, showing an accuracy of more than 90% in multiple challenging scenarios. VidAF provides a more convenient and cost-effective approach for opportunistic AF screening in the community.

Modified Camera Setups for Day-and-Night Pulse-rate Monitoring

Camera systems have been studied as a means for ubiquitous remote photoplethysmography. It was first considered for daytime applications using ambient light. However, main applications for continuous monitoring are in dark or low-light conditions (e.g. sleep monitoring) and, more recently, suitable light sources and simple camera adaptations have been considered for infrared-based solutions. This paper explores suitable camera configurations for pulse-rate monitoring during both day and night (24/7). Various configurations differing in the recorded spectral range are defined, i.e. straight-forward adaptations of a standard RGB camera by choosing proper optical filters. These systems have been studied in a benchmark involving day and night monitoring with various degrees of motion disturbances. The results indicate that, for the 24/7 monitoring, it is best to deploy the full spectral band of an RGB camera, and this can be done without compromising the monitoring performance at night.

Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing

Synthetic data is a powerful tool in training data hungry deep learning algorithms. However, to date, camera-based physiological sensing has not taken full advantage of these techniques. In this work, we leverage a high-fidelity synthetics pipeline for generating videos of faces with faithful blood flow and breathing patterns. We present systematic experiments showing how physiologically-grounded synthetic data can be used in training camera-based multi-parameter cardiopulmonary sensing. We provide empirical evidence that heart and breathing rate measurement accuracy increases with the number of synthetic avatars in the training set. Furthermore, training with avatars with darker skin types leads to better overall performance than training with avatars with lighter skin types. Finally, we discuss the opportunities that synthetics present in the domain of camera-based physiological sensing and limitations that need to be overcome.

Examination of Potential of Thermopile-Based Contactless Respiratory Gating

To control the spread of coronavirus disease 2019 (COVID-19), it is effective to perform a fast screening of the respiratory rate of the subject at the gate before entering a space to assess the potential risks. In this paper, we examine the potential of a novel yet cost-effective solution, called thermopile-based respiratory gating, to contactlessly screen a subject by measuring their respiratory rate in the scenario with an entrance gate. Based on a customized thermopile array system, we investigate different image and signal processing methods that measure respiratory rate from low-resolution thermal videos, where an automatic region-of-interest selection-based approach obtains a mean absolute error (MAE) of 0.8 breaths per minute. We show the feasibility of thermopile-based respiratory gating and quantify its limitations and boundary conditions in a benchmark (e.g., appearance of face mask, measurement distance and screening time). The technical validation provided by this study is helpful for designing and implementing a respiratory gating solution toward the prevention of the spread of COVID-19 during the pandemic.

A Review of Deep Learning-Based Contactless Heart Rate Measurement Methods

The interest in contactless or remote heart rate measurement has been steadily growing in healthcare and sports applications. Contactless methods involve the utilization of a video camera and image processing algorithms. Recently, deep learning methods have been used to improve the performance of conventional contactless methods for heart rate measurement. After providing a review of the related literature, a comparison of the deep learning methods whose codes are publicly available is conducted in this paper. The public domain UBFC dataset is used to compare the performance of these deep learning methods for heart rate measurement. The results obtained show that the deep learning method PhysNet generates the best heart rate measurement outcome among these methods, with a mean absolute error value of 2.57 beats per minute and a mean square error value of 7.56 beats per minute.

Motion-Tolerant Non-Contact Heart-Rate Measurements from Radar Sensor Fusion

Microwave radar technology is very attractive for ubiquitous short-range health monitoring due to its non-contact, see-through, privacy-preserving and safe features compared to the competing remote technologies such as optics. The possibility of radar-based approaches for breathing and cardiac sensing was demonstrated a few decades ago. However, investigation regarding the robustness of radar-based vital-sign monitoring (VSM) is not available in the current radar literature. In this paper, we aim to close this gap by presenting an extensive experimental study of vital-sign radar approach. We consider diversity in test subjects, fitness levels, poses/postures, and, more importantly, random body movement (RBM) in the study. We discuss some new insights that lead to robust radar heart-rate (HR) measurements. A novel active motion cancellation signal-processing technique is introduced, exploiting dual ultra-wideband (UWB) radar system for motion-tolerant HR measurements. Additionally, we propose a spectral pruning routine to enhance HR estimation performance. We validate the proposed method theoretically and experimentally. Totally, we record and analyze about 3500 seconds of radar measurements from multiple human subjects.

Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions

Using a camera as an optical sensor to monitor physiological parameters has garnered considerable research interest in biomedical engineering in recent decades. Researchers have explored the use of a camera for monitoring a variety of physiological waveforms, together with the vital signs carried by these waveforms. Most of the obtained waveforms are related to the human respiratory and cardiovascular systems, and in addition of being indicative of overall health, they can also detect early signs of certain diseases. While using a camera for noncontact physiological signal monitoring offers the advantages of low cost and operational ease, it also has the disadvantages such as vulnerability to motion and lack of burden-free calibration solutions in some use cases. This study presents an overview of the existing camera-based methods that have been reported in recent years. It introduces the physiological principles behind these methods, signal acquisition approaches, various types of acquired signals, data processing algorithms, and application scenarios of these methods. It also discusses the technological gaps between the camera-based methods and traditional medical techniques, which are mostly contact-based. Furthermore, we present the manner in which noncontact physiological signal monitoring use has been extended, particularly over the recent years, to more day-to-day aspects of individuals' lives, so as to go beyond the more conventional use case scenarios. We also report on the development of novel approaches that facilitate easier measurement of less often monitored and recorded physiological signals. These have the potential of ushering a host of new medical and lifestyle applications. We hope this study can provide useful information to the researchers in the noncontact physiological signal measurement community.

Advancements in Methods and Camera-Based Sensors for the Quantification of Respiration

Assessment of respiratory function allows early detection of potential disorders in the respiratory system and provides useful information for medical management. There is a wide range of applications for breathing assessment, from measurement systems in a clinical environment to applications involving athletes. Many studies on pulmonary function testing systems and breath monitoring have been conducted over the past few decades, and their results have the potential to broadly impact clinical practice. However, most of these works require physical contact with the patient to produce accurate and reliable measures of the respiratory function. There is still a significant shortcoming of non-contact measuring systems in their ability to fit into the clinical environment. The purpose of this paper is to provide a review of the current advances and systems in respiratory function assessment, particularly camera-based systems. A classification of the applicable research works is presented according to their techniques and recorded/quantified respiration parameters. In addition, the current solutions are discussed with regards to their direct applicability in different settings, such as clinical or home settings, highlighting their specific strengths and limitations in the different environments.

Detecting Pulse Wave From Unstable Facial Videos Recorded From Consumer-Level Cameras: A Disturbance-Adaptive Orthogonal Matching Pursuit

OBJECTIVE

Modern consumer-level cameras can detect subtle changes in human facial skin color due to varying blood flow; they are beginning to be used as noncontact devices to detect pulse waves. Little, however, do we know about their capacity to perform pulse wave detection when the recorded faces are unstable.

METHODS

Here, we propose a novel method that can extract pulse waves from videos with drastic facial unsteadiness such as head twists and alternating expressions. The method first uses chrominance characteristics in multiple facial sub-regions to construct a raw pulse matrix. Subsequently, it employs a disturbance-adaptive orthogonal matching pursuit (DAOMP) algorithm to recover the underlying pulse matrix corrupted by facial unsteadiness.

RESULTS

To evaluate the efficacy of the method, we perform analyses on two datasets including 268 samples from 67 testing subjects. The results demonstrate that the proposed method outperforms state-of-the-art algorithms, especially in the terrain where drastic facial unsteadiness is present.

CONCLUSION

The proposed framework shows promise to achieve videos-based noncontact pulse wave detection from both steady and unsteady faces recorded by consumer-level cameras.

SIGNIFICANCE

By employing the proposed method, disturbance robustness in noncontact pulse wave detection can be significantly improved.

PPG3D: Does 3D head tracking improve camera-based PPG estimation?

Over the last few years, camera-based estimation of vital signs referred to as imaging photoplethysmography (iPPG) has garnered significant attention due to the relative simplicity, ease, unobtrusiveness and flexibility offered by such measurements. It is expected that iPPG may be integrated into a host of emerging applications in areas as diverse as autonomous cars, neonatal monitoring, and telemedicine. In spite of this potential, the primary challenge of non-contact camera-based measurements is the relative motion between the camera and the subjects. Current techniques employ 2D feature tracking to reduce the effect of subject and camera motion but they are limited to handling translational and in-plane motion. In this paper, we study, for the first-time, the utility of 3D face tracking to allow iPPG to retain robust performance even in presence of out-of-plane and large relative motions. We use a RGB-D camera to obtain 3D information from the subjects and use the spatial and depth information to fit a 3D face model and track the model over the video frames. This allows us to estimate correspondence over the entire video with pixel-level accuracy, even in the presence of out-of-plane or large motions. We then estimate iPPG from the warped video data that ensures per-pixel correspondence over the entire window-length used for estimation. Our experiments demonstrate improvement in robustness when head motion is large.

Identification and Tracking of Physiological Parameters from Skin using Video Photoplethysmography

In recent years, there has been a growing interest in video Photoplethysmography (vPPG), a technique able to estimate cardiovascular parameters from video recordings of the skin. Despite the growing interest in vPPG technology, there are still problems in extracting the correct waveform of blood volume pulse, mainly due to real world artifacts, such as changes in light condition and movement artifacts. Another important issue is the correct definition of skin against background. Therefore, we propose an algorithm of skin detection that is able to recognize skin pixels solid to variations of luminosity. We recorded the signals of interest during an experimental protocol designed to provide thermal stimulation and observe the resulting Autonomic Nervous System changes. Experimental data were gathered from 10 young healthy subjects (age: 21±2 years). Video recordings are processed using a band-pass filter and then an automatic algorithm of peak detection is applied to detect the pulse wave peaks, then used to estimate heart rate variability (HRV). The efficiency and stability of the algorithm are compared against finger-PPG waveforms. Preliminary results show an overall statistical agreement between time and frequency domain indexes. However, further efforts are required to improve the estimation of frequency components, particularly during rest.

Impact of makeup on remote-PPG monitoring

Camera-based remote photoplethysmography (remote-PPG) enables contactless measurement of blood volume pulse from the human skin. Skin visibility is essential to remote-PPG as the camera needs to capture the light reflected from the skin that penetrates deep into skin tissues and carries blood pulsation information. The use of facial makeup may jeopardize this measurement by reducing the amount of light penetrating into and reflecting from the skin. In this paper, we conduct an empirical study to thoroughly investigate the impact of makeup on remote-PPG monitoring, in both the visible (RGB) and invisible (Near Infrared, NIR) lighting conditions. The experiment shows that makeup has negative influence on remote-PPG, which reduces the relative PPG strength (AC/DC) at different wavelengths and changes the normalized PPG signature across multiple wavelengths. It makes (i) the pulse-rate extraction more difficult in both the RGB and NIR, although NIR is less affected than RGB, and (ii) the blood oxygen saturation extraction in NIR impossible. To the best of our knowledge, this is the first work that systematically investigate the impact of makeup on camera-based remote-PPG monitoring.

Analysis of CNN-based remote-PPG to understand limitations and sensitivities

Deep learning based on convolutional neural network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer four questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption variation to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. The availability of multiple convolutional kernels is necessary for CNN to arrive at a flexible channel combination through the spatial operation, but may not provide the same motion-robustness as a multi-site measurement using knowledge-based PPG extraction. We also find that the PPG-related prior knowledge may still be helpful for the CNN-based PPG extraction, and recommend further investigation of hybrid CNN-based methods that include prior knowledge in their design.

Modified RGB Cameras for Infrared Remote-PPG

Multi-wavelength cameras play an essential role in remote photoplethysmography (PPG). Whereas these are readily available for visible light, this is not the case for near infrared (NIR). We propose to modify existing RGB cameras to make them suited for NIR-PPG. In particular, we exploit the spectral leakage of the RGB channels in infrared in combination with a narrow dual-band optical filter. Such camera modification is simple, cost-effective, easy to implement, and it is shown to attain a pulse-rate extraction performance comparable to that of multiple narrow-band NIR cameras.

An open-source remote heart rate imaging method with practical apparatus and algorithms

Recent developments in computer science and digital image processing have enabled the extraction of an individual’s heart pulsations from pixel changes in recorded video images of human skin surfaces. This method is termed remote photoplethysmography (rPPG) and can be achieved with consumer-level cameras (e.g., a webcam or mobile camera). The goal of the present publication is two-fold. First, we aim to organize future rPPG software developments in a tractable and nontechnical manner, such that the public gains access to a basic open-source rPPG code, comes to understand its utility, and can follow its most recent progressions. The second goal is to investigate rPPG’s accuracy in detecting heart rates from the skin surfaces of several body parts after physical exercise and under ambient lighting conditions with a consumer-level camera. We report that rPPG is highly accurate when the camera is aimed at facial skin tissue, but that the heart rate recordings from wrist regions are less reliable, and recordings from the calves are unreliable. Facial rPPG remained accurate despite the high heart rates after exercise. The proposed research procedures and the experimental findings provide guidelines for future studies on rPPG.

Sleep/wake classification via remote PPG signals

This paper proposes a remote sleep/wake classification method by combining vision-based heart rate (HR) estimation and convolutional neural network (CNN). Instead of inputting the estimated HR with low temporal resolution, remote PPG (Photoplethysmogram) signals, which contain high-temporal-resolution HR information, are input into the CNN. To reduce noise in the remote PPG signals, we propose a dynamic HR filter. Evaluation results show that the dynamic HR filter works more effectively in comparison with the static filter, which helps improve the area under the ROC curve (AUC) to 0.70, which is almost as good as the reference 0.71 for HR from a wearable sensor.

Modelling and Validation of Computer Vision Techniques to Assess Heart Rate, Eye Temperature, Ear-Base Temperature and Respiration Rate in Cattle

Simple Summary

Animal monitoring normally requires procedures that are time- and labour-consuming. The implementation of novel non-invasive technologies could be a good approach to monitor animal health and welfare. This study aimed to evaluate the use of images and computer-based methods to track specific features of the face and to assess temperature; respiration rate and heart rate in cattle. The measurements were compared with measures obtained with conventional methods during the same time period. The data were collected from ten dairy cows that were recorded during six handling procedures across two consecutive days. The results from this study show over 92% of accuracy from the computer algorithm that was developed to track the areas selected on the videos collected. In addition, acceptable correlation was observed between the temperature calculated from thermal infrared images and temperature collected using intravaginal loggers. Moreover, there was acceptable correlation between the respiration rate calculated from infrared videos and from visual observation. Furthermore, a low to high relationship was found between the heart rate obtained from videos and from attached monitors. The study also showed that both the position of the cameras and the area analysed on the images are very important, as both had large impact on the accuracy of the methods. The positive outcomes and the limitations observed in this study suggest the need for further research

Abstract

Precision livestock farming has emerged with the aim of providing detailed information to detect and reduce problems related to animal management. This study aimed to develop and validate computer vision techniques to track required features of cattle face and to remotely assess eye temperature, ear-base temperature, respiration rate, and heart rate in cattle. Ten dairy cows were recorded during six handling procedures across two consecutive days using thermal infrared cameras and RGB (red, green, blue) video cameras. Simultaneously, core body temperature, respiration rate and heart rate were measured using more conventional ‘invasive’ methods to be compared with the data obtained with the proposed algorithms. The feature tracking algorithm, developed to improve image processing, showed an accuracy between 92% and 95% when tracking different areas of the face of cows. The results of this study also show correlation coefficients up to 0.99 between temperature measures obtained invasively and those obtained remotely, with the highest values achieved when the analysis was performed within individual cows. In the case of respiration rate, a positive correlation (r = 0.87) was found between visual observations and the analysis of non-radiometric infrared videos. Low to high correlation coefficients were found between the heart rates (0.09–0.99) obtained from attached monitors and from the proposed method. Furthermore, camera location and the area analysed appear to have a relevant impact on the performance of the proposed techniques. This study shows positive outcomes from the proposed computer vision techniques when measuring physiological parameters. Further research is needed to automate and improve these techniques to measure physiological changes in farm animals considering their individual characteristics.

Non-contact heart and respiratory rate monitoring of preterm infants based on a computer vision system: a method comparison study

BACKGROUND

Non-contact heart rate (HR) and respiratory rate (RR) monitoring is necessary for preterm infants due to the potential for the adhesive electrodes of conventional electrocardiogram (ECG) to cause damage to the epidermis. This study was performed to evaluate the agreement between HR and RR measurements of preterm infants using a non-contact computer vision system with comparison to measurements obtained by the ECG.

METHODS

A single-centre, cross-sectional observational study was conducted in a Neonatal Unit. Ten infants and their ECG monitors were videoed using two Nikon cameras for 10 min. HR and RR measurements obtained from the non-contact system were extracted using advanced signal processing techniques and later compared to the ECG readings using Bland-Altman analysis.

RESULTS

The non-contact system was able to detect an apnoea when the ECG determined movement as respirations. Although the mean bias between both methods was relatively low, the limits of agreement for HR were -8.3 to 17.4 beats per minute (b.p.m.) and for RR, -22 to 23.6 respirations per minute (r.p.m.).

CONCLUSIONS

This study provides necessary data for improving algorithms to address confounding variables common to the neonatal population. Further studies investigating the robustness of the proposed system for premature infants are therefore required.

Remote heart rate monitoring - Assessment of the Facereader rPPg by Noldus

Remote photoplethysmography (rPPG) allows contactless monitoring of human cardiac activity through a video camera. In this study, we assessed the accuracy and precision for heart rate measurements of the only consumer product available on the market, namely the FacereaderTM rPPG by Noldus, with respect to a gold standard electrocardiograph. Twenty-four healthy participants were asked to sit in front of a computer screen and alternate two periods of rest with two stress tests (i.e. Go/No-Go task), while their heart rate was simultaneously acquired for 20 minutes using the ECG criterion measure and the FacereaderTM rPPG. Results show that the FacereaderTM rPPG tends to overestimate lower heart rates and underestimate higher heart rates compared to the ECG. The Facereader™ rPPG revealed a mean bias of 9.8 bpm, the 95% limits of agreement (LoA) ranged from almost -30 up to +50 bpm. These results suggest that whilst the rPPG FacereaderTM technology has potential for contactless heart rate monitoring, its predictions are inaccurate for higher heart rates, with unacceptable precision across the entire range, rendering its estimates unreliable for monitoring individuals.

RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HRVIPL-HR is available at: ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.

Video-Based Contactless Heart-Rate Detection and Counting via Joint Blind Source Separation with Adaptive Noise Canceller

Driver assistance systems are a major focus of the automotive industry. Although technological functions that help drivers are improving, the monitoring of driver state functions receives less attention. In this respect, the human heart rate (HR) is one of the most important bio-signals, and it can be detected remotely using consumer-grade cameras. Based on this, a video-based driver state monitoring system using HR signals is proposed in this paper. In a practical automotive environment, monitoring the HR is very challenging due to changes in illumination, vibrations, and human motion. In order to overcome these problems, source separation strategies were employed using joint blind source separation, and feature combination was adopted to maximize HR variation. Noise-assisted data analysis was then adopted using ensemble empirical mode decomposition to extract the pure HR. Finally, power spectral density analysis was conducted in the frequency domain, and a post-processing smoothing filter was applied. The performance of the proposed approach was tested based on commonly employed metrics using the MAHNOB-HCI public dataset and compared with recently proposed competing methods. The experimental results proved that our method is robust for a variety of driving conditions based on testing using a driving dataset and static indoor environments.

Discriminative Signatures for Remote-PPG

Near-infrared (NIR) remote photoplethysmography (PPG) promises attractive applications in darkness, as it involves unobtrusive, invisible light. However, since the PPG strength (AC/DC) is much lower in the NIR spectrum than in the RGB spectrum, robust vital signs monitoring is more challenging. In this paper, we propose a new PPG-extraction method, DIScriminative signature based extraction (DIS), to significantly improve the pulse-rate measurement in NIR. Our core idea is to use both the color signals containing blood absorption variations and additional disturbance signals as input for PPG extraction. By defining a discriminative signature, we use one-step least-squares regression (joint optimization) to retrieve the pulsatile component from color signals and suppress disturbance signals simultaneously. A large-scale lab experiment, recorded in NIR with heavy body motions, shows the significant improvement of DIS over the state-of-the-art method, whereas its principle is simple and generally applicable.

Robust heart-rate estimation from facial videos using Project_ICA

OBJECTIVE

Remote photoplethysmography (rPPG) can achieve non-contact measurement of heart rate (HR) from a continuous video sequence by scanning the skin surface. However, practical applications are still limited by factors such as non-rigid facial motion and head movement. In this work, a detailed system framework for remotely estimating heart rate from facial videos under various movement conditions is described.

APPROACH

After the rPPG signal has been obtained from a defined region of the facial skin, a method, termed 'Project_ICA', based on a skin reflection model, is employed to extract the pulse signal from the original signal.

MAIN RESULTS

To evaluate the performance of the proposed algorithm, a dataset containing 112 videos including the challenges of various skin tones, body motion and HR recovery after exercise was created from 28 participants.

SIGNIFICANCE

The results show that Project_ICA, when evaluated by several criteria, provides a more accurate and robust estimate of HR than most existing methods, although problems remain in obtaining reliable measurements from dark-skinned subjects.

Frame Registration for Motion Compensation in Imaging Photoplethysmography

Imaging photoplethysmography (iPPG) is an emerging technology used to assess microcirculation and cardiovascular signs by collecting backscattered light from illuminated tissue using optical imaging sensors. An engineering approach is used to evaluate whether a silicone cast of a human palm might be effectively utilized to predict the results of image registration schemes for motion compensation prior to their application on live human tissue. This allows us to establish a performance baseline for each of the algorithms and to isolate performance and noise fluctuations due to the induced motion from the temporally changing physiological signs. A multi-stage evaluation model is developed to qualitatively assess the influence of the region of interest (ROI), system resolution and distance, reference frame selection, and signal normalization on extracted iPPG waveforms from live tissue. We conclude that the application of image registration is able to deliver up to 75% signal-to-noise (SNR) improvement (4.75 to 8.34) over an uncompensated iPPG signal by employing an intensity-based algorithm with a moving reference frame.

Single Element Remote-PPG

Camera-based remote photoplethysmography technology (remote-PPG) has shown great potential for contactless pulse-rate monitoring. However, remote-PPG systems typically analyze face images, which may restrict applications in view of privacy-preserving regulations such as the recently announced General Data Protection Regulation in the European Union. In this paper, we investigate the case of using single-element sensing as an input for remote-PPG extraction, which prohibits facial analysis and thus evades privacy issues. It also improves the efficiency of data storage and transmission. In contrast to known remote-PPG solutions using skin-selection techniques, the input signals in a single-element setup will contain a non-negligible degree of signal components associated with non-skin areas. Current remote-PPG extraction methods based on physiological and optical properties of skin reflections are therefore no longer valid. A new remote-PPG method, named Soft Signature based extraction (SoftSig), is proposed to deal with this situation by softening the dependence of pulse extraction on prior knowledge. A large scale experiment validates the concept of single-element remote-PPG monitoring and shows the improvement of SoftSig over general purpose solutions.

Remote Heart Rate Measurement from RGB-NIR Video Based on Spatial and Spectral Face Patch Selection

In this paper, we propose a novel heart rate (HR) estimation method using simultaneously recorded RGB and near-infrared (NIR) face videos. The key idea of our method is to automatically select suitable face patches for HR estimation in both spatial and spectral domains. The spatial and spectral face patch selection enables us to robustly estimate HR under various situations, including scenes under which existing RGB camera-based methods fail to accurately estimate HR. For a challenging scene in low light and with light fluctuations, our method can successfully estimate HR for all 20 subjects $( \pm 3$ beats per minute), while the RGB camera-based methods succeed only for 25% of the subjects.

Heart Rate and Heart Rate Variability From Single-Channel Video and ICA Integration of Multiple Signals

Unobtrusive monitoring of vital signs is relevant for both medical (patient monitoring) and non-medical applications (e.g., stress and fatigue monitoring). In this paper, we focus on the use of imaging photoplethysmography (iPPG). High frame rate videos were acquired by using a monochrome camera and an optical band-pass filter ([Formula: see text] nm). To enhance iPPG signal, we investigated the use of independent component analysis (ICA) pre-processing applied to iPPG signal from different regions of the face. Methodology was tested on [Formula: see text] healthy volunteers. Heart rate (HR) and standard time and frequency domain descriptors of heart rate variability (HRV), simultaneously extracted from videos and ECG data, were compared. A mean absolute error (MAE) about 3.812 ms was observed for normal-to-normal intervals with or without ICA pre-processing. Smaller MAE values of frequency domain descriptors were observed when ICA pre-processing was used. The impact of both video frame rate and video signal interval were also analyzed. All the results support the conclusion that proposed ICA pre-processing can effectively improve the HR and HRV assessment from iPPG.

Video-based heart rate monitoring across a range of skin pigmentations during an acute hypoxic challenge

The robust monitoring of heart rate from the video-photoplethysmogram (video-PPG) during challenging conditions requires new analysis techniques. The work reported here extends current research in this area by applying a motion tolerant algorithm to extract high quality video-PPGs from a cohort of subjects undergoing marked heart rate changes during a hypoxic challenge, and exhibiting a full range of skin pigmentation types. High uptimes in reported video-based heart rate (HRvid) were targeted, while retaining high accuracy in the results. Ten healthy volunteers were studied during a double desaturation hypoxic challenge. Video-PPGs were generated from the acquired video image stream and processed to generate heart rate. HRvid was compared to the pulse rate posted by a reference pulse oximeter device (HRp). Agreement between video-based heart rate and that provided by the pulse oximeter was as follows: Bias = - 0.21 bpm, RMSD = 2.15 bpm, least squares fit gradient = 1.00 (Pearson R = 0.99, p < 0.0001), with a 98.78% reporting uptime. The difference between the HRvid and HRp exceeded 5 and 10 bpm, for 3.59 and 0.35% of the reporting time respectively, and at no point did these differences exceed 25 bpm. Excellent agreement was found between the HRvid and HRp in a study covering the whole range of skin pigmentation types (Fitzpatrick scales I-VI), using standard room lighting and with moderate subject motion. Although promising, further work should include a larger cohort with multiple subjects per Fitzpatrick class combined with a more rigorous motion and lighting protocol.

Full video pulse extraction

This paper introduces a new method to automate heart-rate detection using remote photoplethysmography (rPPG). The method replaces the commonly used region of interest (RoI) detection and tracking, and does not require initialization. Instead, it combines a number of candidate pulse-signals computed in the parallel, each biased towards differently colored objects in the scene. The method is based on the observation that the temporally averaged colors of video objects (skin and background) are usually quite stable over time in typical application-driven scenarios, such as the monitoring of a subject sleeping in bed, or an infant in an incubator. The resulting system, called full video pulse extraction (FVP), allows the direct use of raw video streams for pulse extraction. Our benchmark set of diverse videos shows that FVP enables long-term sleep monitoring in visible light and in infrared, and works for adults and neonates. Although we only demonstrate the concept for heart-rate monitoring, we foresee the adaptation to a range of vital signs, thus benefiting the larger video health monitoring field.

A study of color illumination effect on the SNR of rPPG signals

Remote photoplethysmography (rPPG) can be used to measure cardiac activity by detecting the subtle color variation of the human skin tissue using an RGB camera. Recent studies have presented the feasibility and proposed multiple methods to improve the motion robustness for the subject movements. However, enhancing the signal-to-noise ratio (SNR) of the rPPG signal is still an important issue for the contactless measurement. In this paper, we conducted an experiment to study the lighting effect on the SNR of rPPG signals. The results point out that different colors of light sources provide different SNR in each RGB channel. By providing the dedicated light sources (λ= 490-620) nm, the SNR of rPPG signals captured from the green color channel can be enhanced. Among the tested light sources, light green provides the most significant improvement from -11.09 to -6.6 dB compared with the fluorescent light.

Accurate face alignment and adaptive patch selection for heart rate estimation from videos under realistic scenarios

Non-contact heart rate (HR) measurement from facial videos has attracted high interests due to its convenience and cost effectiveness. However, accurate and robust HR estimation under various realistic scenarios remain a very challenging problem. In this paper, we develop a novel system which can achieve a robust and accurate HR estimation under those challenging scenarios. First, to minimize tracking-artifacts arising from large head motions and facial expressions, we propose a joint face detection and alignment method which can produce alignment-friendly facial bounding boxes with reliable initial facial shapes, facilitating accurate and robust face alignment even in the presence of large pose variations and expressions. Second, different from most existing methods [1–5] which derive pulse signals from predetermined grid cells (i.e. local patches), our patches are varying-sized triangles generated adaptively to exclude negative effects from non-rigid facial motions. Third, we propose an adaptive patch selection method to choose patches which contain skin regions and are more likely to contain useful information, followed by an independent component analysis, for an accurate HR estimate. Extensive experiments on both public datasets and our own dataset demonstrated that, comparing with the state-of-the-art methods [1–3], our method reduces the root mean square error (RMSE) by a large margin, ranging from 12% to 63%, and can provide a robust and accurate estimation under various challenging scenarios.