Remote plethysmographic imaging using ambient light

Cuffless Blood Pressure in clinical practice: challenges, opportunities and current limits

Background: Cuffless blood pressure measurement technologies have attracted significant attention for their potential to transform cardiovascular monitoring.Methods: This updated narrative review thoroughly examines the challenges, opportunities, and limitations associated with the implementation of cuffless blood pressure monitoring systems.Results: Diverse technologies, including photoplethysmography, tonometry, and ECG analysis, enable cuffless blood pressure measurement and are integrated into devices like smartphones and smartwatches. Signal processing emerges as a critical aspect, dictating the accuracy and reliability of readings. Despite its potential, the integration of cuffless technologies into clinical practice faces obstacles, including the need to address concerns related to accuracy, calibration, and standardization across diverse devices and patient populations. The development of robust algorithms to mitigate artifacts and environmental disturbances is essential for extracting clear physiological signals. Based on extensive research, this review emphasizes the necessity for standardized protocols, validation studies, and regulatory frameworks to ensure the reliability and safety of cuffless blood pressure monitoring devices and their implementation in mainstream medical practice. Interdisciplinary collaborations between engineers, clinicians, and regulatory bodies are crucial to address technical, clinical, and regulatory complexities during implementation. In conclusion, while cuffless blood pressure monitoring holds immense potential to transform cardiovascular care. The resolution of existing challenges and the establishment of rigorous standards are imperative for its seamless incorporation into routine clinical practice.Conclusion: The emergence of these new technologies shifts the paradigm of cardiovascular health management, presenting a new possibility for non-invasive continuous and dynamic monitoring. The concept of cuffless blood pressure measurement is viable and more finely tuned devices are expected to enter the market, which could redefine our understanding of blood pressure and hypertension.

Remote photoplethysmography (rPPG) in the wild: Remote heart rate imaging via online webcams

Remote photoplethysmography (rPPG) is a low-cost technique to measure physiological parameters such as heart rate by analyzing videos of a person. There has been growing attention to this technique due to the increased possibilities and demand for running psychological experiments on online platforms. Technological advancements in commercially available cameras and video processing algorithms have led to significant progress in this field. However, despite these advancements, past research indicates that suboptimal video recording conditions can severely compromise the accuracy of rPPG. In this study, we aimed to develop an open-source rPPG methodology and test its performance on videos collected via an online platform, without control of the hardware of the participants and the contextual variables, such as illumination, distance, and motion. Across two experiments, we compared the results of the rPPG extraction methodology to a validated dataset used for rPPG testing. Furthermore, we then collected 231 online video recordings and compared the results of the rPPG extraction to finger pulse oximeter data acquired with a validated mobile heart rate application. Results indicated that the rPPG algorithm was highly accurate, showing a significant degree of convergence with both datasets thus providing an improved tool for recording and analyzing heart rate in online experiments.

Evaluation of video-based rPPG in challenging environments: Artifact mitigation and network resilience

Video-based remote photoplethysmography (rPPG) has emerged as a promising technology for non-contact vital sign monitoring, especially under controlled conditions. However, the accurate measurement of vital signs in real-world scenarios faces several challenges, including artifacts induced by videocodecs, low-light noise, degradation, low dynamic range, occlusions, and hardware and network constraints. In this article, a systematic and comprehensive investigation of these issues is conducted, measuring their detrimental effects on the quality of rPPG measurements. Additionally, practical strategies are proposed for mitigating these challenges to improve the dependability and resilience of video-based rPPG systems. Methods for effective biosignal recovery in the presence of network limitations are detailed, along with denoising and inpainting techniques aimed at preserving video frame integrity. Compared to previous studies, this paper addresses a broader range of variables and demonstrates improved accuracy across various rPPG methods, emphasizing generalizability for practical applications in diverse scenarios with varying data quality. Extensive evaluations and direct comparisons demonstrate the effectiveness of these approaches in enhancing rPPG measurements under challenging environments, contributing to the development of more reliable and effective remote vital sign monitoring technologies.

Innovative approaches in imaging photoplethysmography for remote blood oxygen monitoring

Peripheral Capillary Oxygen Saturation (SpO2) has received increasing attention during the COVID-19 pandemic. Clinical investigations have demonstrated that individuals afflicted with COVID-19 exhibit notably reduced levels of SpO2 before the deterioration of their health status. To cost-effectively enable individuals to monitor their SpO2, this paper proposes a novel neural network model named "ITSCAN" based on Temporal Shift Module. Benefiting from the widespread use of smartphones, this model can assess an individual's SpO2 in real time, utilizing standard facial video footage, with a temporal granularity of seconds. The model is interweaved by two distinct branches: the motion branch, responsible for extracting spatiotemporal data features and the appearance branch, focusing on the correlation between feature channels and the location information of feature map using coordinate attention mechanisms. Accordingly, the SpO2 estimator generates the corresponding SpO2 value. This paper summarizes for the first time 5 loss functions commonly used in the SpO2 estimation model. Subsequently, a novel loss function has been contributed through the examination of various combinations and careful selection of hyperparameters. Comprehensive ablation experiments analyze the independent impact of each module on the overall model performance. Finally, the experimental results based on the public dataset (VIPL-HR) show that our model has obvious advantages in MAE (1.10%) and RMSE (1.19%) compared with related work, which implies more accuracy of the proposed method to contribute to public health.

ST-Phys: Unsupervised Spatio-Temporal Contrastive Remote Physiological Measurement

Remote photoplethysmography (rPPG) is a non-contact method that employs facial videos for measuring physiological parameters. Existing rPPG methods have achieved remarkable performance. However, the success mainly profits from supervised learning over massive labeled data. On the other hand, existing unsupervised rPPG methods fail to fully utilize spatio-temporal features and encounter challenges in low-light or noise environments. To address these problems, we propose an unsupervised contrast learning approach, ST-Phys. We incorporate a low-light enhancement module, a temporal dilated module, and a spatial enhanced module to better deal with long-term dependencies under the random low-light conditions. In addition, we design a circular margin loss, wherein rPPG signals originating from identical videos are attracted, while those from distinct videos are repelled. Our method is assessed on six openly accessible datasets, including RGB and NIR videos. Extensive experiments reveal the superior performance of our proposed ST-Phys over state-of-the-art unsupervised rPPG methods. Moreover, it offers advantages in parameter reduction and noise robustness.

3D DenseNet with temporal transition layer for heart rate estimation from real-life RGB videos

BACKGROUND

Deep learning has demonstrated superior performance over traditional methods for the estimation of heart rates in controlled contexts. However, in less controlled scenarios this performance seems to vary based on the training dataset and the architecture of the deep learning models.

OBJECTIVES

In this paper, we develop a deep learning-based model leveraging the power of 3D convolutional neural networks (3DCNN) to extract temporal and spatial features that lead to an accurate heart rates estimation from RGB no pre-defined region of interest (ROI) videos.

METHODS

We propose a 3D DenseNet with a 3D temporal transition layer for the estimation of heart rates from a large-scale dataset of videos that appear more hospital-like and real-life than other existing facial video-based datasets.

RESULTS

Experimentally, our model was trained and tested on this less controlled dataset and showed heart rate estimation performance with root mean square error (RMSE) of 8.68 BPM and mean absolute error (MAE) of 3.34 BPM.

CONCLUSION

Moreover, we show that such a model can also achieve better results than the state-of-the-art models when tested on the VIPL-HR public dataset.

A Fast FPGA Hardware Accelerator for Remote Heart Rate Detection Based on RGB Vision

A fast hardware accelerator is created by this work via field programmable gate array (FPGA) to estimate heart rate (HR) through the video recorded by a RGB camera based on the technology of remote photoplethysmography (rPPG). The method of rPPG acquires physiological signals of a human body by analyzing the subtle color changes on the surface of the human skin. The hardware implementation of rPPG to estimate HR is proposed herein to aim for a much faster calculation speed than software for a number of applications, like heart failure pre-warning of an in-action athlete and drowsiness detection of a driver. In this accelerator, ICA (Independent Component Analysis) is used to recover the blood volume pulse from the raw signals of remote PPG, and then obtain the heart rate value. The architecture of the hardware circuit is described in Verilog HDL and verified by Quartus II, and also implemented in an Altera DE10-Standard FPGA board, which consists of image capture, heart rate algorithm and image display. A TRDB-D5M camera is utilized for image capture. Two experiments were conducted with image collecting duration of 16 seconds and 8 seconds respectively, and the commercial device Omron HEM-6111 was used as the golden value. The proposed system achieves an accuracy in (ME ± 1.96SD) of -0.76 ± 5.09 and -0.70 ± 8.71 bpm in the short periods of 16-second and 8-second versions, respectively, which outperforms all the reported prior works in combined computation time and accuracy.

An effective cross-scenario remote heart rate estimation network based on global-local information and video transformer

Remote photoplethysmography (rPPG) technology is a non-contact physiological signal measurement method, characterized by non-invasiveness and ease of use. It has broad application potential in medical health, human factors engineering, and other fields. However, current rPPG technology is highly susceptible to variations in lighting conditions, head pose changes, and partial occlusions, posing significant challenges for its widespread application. In order to improve the accuracy of remote heart rate estimation and enhance model generalization, we propose PulseFormer, a dual-path network based on transformer. By integrating local and global information and utilizing fast and slow paths, PulseFormer effectively captures the temporal variations of key regions and spatial variations of the global area, facilitating the extraction of rPPG feature information while mitigating the impact of background noise variations. Heart rate estimation results on the popular rPPG dataset show that PulseFormer achieves state-of-the-art performance on public datasets. Additionally, we establish a dataset containing facial expressions and synchronized physiological signals in driving scenarios and test the pre-trained model from the public dataset on this collected dataset. The results indicate that PulseFormer exhibits strong generalization capabilities across different data distributions in cross-scenario settings. Therefore, this model is applicable for heart rate estimation of individuals in various scenarios.

FM-FCN: A Neural Network with Filtering Modules for Accurate Vital Signs Extraction

Neural networks excel at capturing local spatial patterns through convolutional modules, but they may struggle to identify and effectively utilize the morphological and amplitude periodic nature of physiological signals. In this work, we propose a novel network named filtering module fully convolutional network (FM-FCN), which fuses traditional filtering techniques with neural networks to amplify physiological signals and suppress noise. First, instead of using a fully connected layer, we use an FCN to preserve the time-dimensional correlation information of physiological signals, enabling multiple cycles of signals in the network and providing a basis for signal processing. Second, we introduce the FM as a network module that adapts to eliminate unwanted interference, leveraging the structure of the filter. This approach builds a bridge between deep learning and signal processing methodologies. Finally, we evaluate the performance of FM-FCN using remote photoplethysmography. Experimental results demonstrate that FM-FCN outperforms the second-ranked method in terms of both blood volume pulse (BVP) signal and heart rate (HR) accuracy. It substantially improves the quality of BVP waveform reconstruction, with a decrease of 20.23% in mean absolute error (MAE) and an increase of 79.95% in signal-to-noise ratio (SNR). Regarding HR estimation accuracy, FM-FCN achieves a decrease of 35.85% in MAE, 29.65% in error standard deviation, and 32.88% decrease in 95% limits of agreement width, meeting clinical standards for HR accuracy requirements. The results highlight its potential in improving the accuracy and reliability of vital sign measurement through high-quality BVP signal extraction. The codes and datasets are available online at https://github.com/zhaoqi106/FM-FCN.

HRUNet: Assessing Uncertainty in Heart Rates Measured From Facial Videos

Video-based Photoplethysmography (VPPG) offers the capability to measure heart rate (HR) from facial videos. However, the reliability of the HR values extracted through this method remains uncertain, especially when videos are affected by various disturbances. Confronted by this challenge, we introduce an innovative framework for VPPG-based HR measurements, with a focus on capturing diverse sources of uncertainty in the predicted HR values. In this context, a neural network named HRUNet is structured for HR extraction from input facial videos. Departing from the conventional training approach of learning specific weight (and bias) values, we leverage the Bayesian posterior estimation to derive weight distributions within HRUNet. These distributions allow for sampling to encode uncertainty stemming from HRUNet's limited performance. On this basis, we redefine HRUNet's output as a distribution of potential HR values, as opposed to the traditional emphasis on the single most probable HR value. The underlying goal is to discover the uncertainty arising from inherent noise in the input video. HRUNet is evaluated across 1,098 videos from seven datasets, spanning three scenarios: undisturbed, motion-disturbed, and light-disturbed. The ensuing test outcomes demonstrate that uncertainty in the HR measurements increases significantly in the scenarios marked by disturbances, compared to that in the undisturbed scenario. Moreover, HRUNet outperforms state-of-the-art methods in HR accuracy when excluding HR values with 0.4 uncertainty. This underscores that uncertainty emerges as an informative indicator of potentially erroneous HR measurements. With enhanced reliability affirmed, the VPPG technique holds the promise for applications in safety-critical domains.

Remote photoplethysmography based on reflected light angle estimation

Objective. In previous studies, the factors affecting the accuracy of imaging photoplethysmography (iPPG) heart rate (HR) measurement have been focused on the light intensity, facial reflection angle, and motion artifacts. However, the factor of specularly reflected light has not been studied in detail. We explored the effect of specularly reflected light on the accuracy of HR estimation and proposed an estimation method for the direction of specularly radiated light.Approach. To study the HR measurement accuracy influenced by specularly reflected light, we control the component of specularly reflected light by controlling its angle. A total of 100 videos from four different reflected light angles were collected, and 25 subjects participated in the dataset collection. We extracted angles and illuminations for 71 facial regions, fitting sample points through interpolation, and selecting the angle corresponding to the maximum weight in the fitted curve as the estimated reflected angle.Main results. The experimental results show that higher specularly reflected light compromises HR estimation accuracy under the same value of light intensity. Notably, at a 60° angle, the HR accuracy (ACC) increased by 0.7%, while the signal-to-noise ratio and Pearson correlation coefficient increased by 0.8 dB and 0.035, respectively, compared to 0°. The overall root mean squared error, standard deviation, and mean error of our proposed reflected light angle estimation method on the illumination multi-angle incidence (IMAI) dataset are 1.173°, 0.978°, and 0.773°. The average Pearson value is 0.8 in the PURE rotation dataset. In addition, the average ACC of HR measurements in the PURE dataset is improved by 1.73% in our method compared to the state-of-the-art traditional methods.Significance. Our method has great potential for clinical applications, especially in bright light environments such as during surgery, to improve accuracy and monitor blood volume changes in blood vessels.

Assessing heart rate and blood pressure estimation from image photoplethysmography using a digital blood pressure meter

This study presents a non-contact approach to measuring heart rate and blood pressure using an image photoplethysmography (iPPG) signal, and compares the results to those from an oscillometric blood pressure meter. Facial videos of 100 subjects were recorded via a webcam under ambient lighting conditions to extract iPPG signals. The results revealed a strong correlation between the heart rate derived from iPPG and that obtained from an oscillometric blood pressure meter. In addition, a continuous wavelet transform images with a 6-s duration were used as input for a custom convolutional neural network model, providing the most accurate blood pressure estimation. The proposed method received a grade A for diastolic and grade B for systolic blood pressure based on the British Hypertension Society's criteria. It also met the standards set by the Association for the Advancement of Medical Instrumentation. This non-contact framework shows promising potential for efficient screening purposes.

Validity of resting heart rate derived from contact-based smartphone photoplethysmography compared with electrocardiography: a scoping review and checklist for optimal acquisition and reporting

Background

With the rise of smartphone ownership and increasing evidence to support the suitability of smartphone usage in healthcare, the light source and smartphone camera could be utilized to perform photoplethysmography (PPG) for the assessment of vital signs, such as heart rate (HR). However, until rigorous validity assessment has been conducted, PPG will have limited use in clinical settings.

Objective

We aimed to conduct a scoping review assessing the validity of resting heart rate (RHR) acquisition from PPG utilizing contact-based smartphone devices. Our four specific objectives of this scoping review were to (1) conduct a systematic search of the published literature concerning contact-based smartphone device-derived PPG, (2) map study characteristics and methodologies, (3) identify if methodological and technological advancements have been made, and (4) provide recommendations for the advancement of the investigative area.

Methods

ScienceDirect, PubMed and SPORTDiscus were searched for relevant studies between January 1st, 2007, and November 6th, 2022. Filters were applied to ensure only literature written in English were included. Reference lists of included studies were manually searched for additional eligible studies.

Results

In total 10 articles were included. Articles varied in terms of methodology including study characteristics, index measurement characteristics, criterion measurement characteristics, and experimental procedure. Additionally, there were variations in reporting details including primary outcome measure and measure of validity. However, all studies reached the same conclusion, with agreement ranging between good to very strong and correlations ranging from r = .98 to 1.

Conclusions

Smartphone applications measuring RHR derived from contact-based smartphone PPG appear to agree with gold standard electrocardiography (ECG) in healthy subjects. However, agreement was established under highly controlled conditions. Future research could investigate their validity and consider effective approaches that transfer these methods from laboratory conditions into the "real-world", in both healthy and clinical populations.

Non-invasive methods for heart rate measurement in fish based on photoplethysmography

Heart rate is a crucial physiological indicator for fish, but current measurement methods are often invasive or require delicate manipulation. In this study, we introduced two non-invasive and easy-to-operate methods based on photoplethysmography, namely reflectance-type photoplethysmography (PPG) and remote photoplethysmography (rPPG), which we applied to the large yellow croaker (Larimichthys crocea). PPG showed perfect synchronization with electrocardiogram (ECG), with a Pearson's correlation coefficient of 0.99999. For rPPG, the results showed good agreement with ECG. Under active provision of green light, the Pearson's correlation coefficient was 0.966, surpassing the value of 0.947 under natural light. Additionally, the root mean square error was 0.810, which was lower than the value of 1.30 under natural light, indicating not only that the rPPG method had relatively high accuracy but also that green light may have the potential to further improve its accuracy.

Conventional and deep learning methods in heart rate estimation from RGB face videos

Contactless vital signs monitoring is a fast-advancing scientific field that aims to employ monitoring methods that do not necessitate the use of leads or physical attachments to the patient in order to overcome the shortcomings and limits of traditional monitoring systems. Several traditional methods have been applied to extract the heart rate (HR) signal from the face. Moreover, machine learning has recently contributed majorly to the development of such a field in which deep networks and other deep learning methods are employed to extract the HR signal from RGB face videos. In this paper, we evaluate the state-of-the-art conventional and deep learning methods for HR estimates, focusing on the limits of deep learning methods and the availability of less-controlled face video datasets. We aim to present an extensive review that helps the various approaches of remote photoplethysmography extraction and HR estimation to be understood, in addition to their drawbacks and benefits.

cbPPGGAN: A Generic Enhancement Framework for Unpaired Pulse Waveforms in Camera-Based Photoplethysmography

Camera-based photoplethysmography (cbP PG) is a non-contact technique that measures cardiac-related blood volume alterations in skin surface vessels through the analysis of facial videos. While traditional approaches can estimate heart rate (HR) under different illuminations, their accuracy can be affected by motion artifacts, leading to poor waveform fidelity and hindering further analysis of heart rate variability (HRV); deep learning-based approaches reconstruct high-quality pulse waveform, yet their performance significantly degrades under illumination variations. In this work, we aim to leverage the strength of these two methods and propose a framework that possesses favorable generalization capabilities while maintaining waveform fidelity. For this purpose, we propose the cbPPGGAN, an enhancement framework for cbPPG that enables the flexible incorporation of both unpaired and paired data sources in the training process. Based on the waveforms extracted by traditional approaches, the cbPPGGAN reconstructs high-quality waveforms that enable accurate HR estimation and HRV analysis. In addition, to address the lack of paired training data problems in real-world applications, we propose a cycle consistency loss that guarantees the time-frequency consistency before/after mapping. The method enhances the waveform quality of traditional POS approaches in different illumination tests (BH-rPPG) and cross-datasets (UBFC-rPPG) with mean absolute error (MAE) values of 1.34 bpm and 1.65 bpm, and average beat-to-beat (AVBB) values of 27.46 ms and 45.28 ms, respectively. Experimental results demonstrate that the cbPPGGAN enhances cbPPG signal quality and outperforms the state-of-the-art approaches in HR estimation and HRV analysis. The proposed framework opens a new pathway toward accurate HR estimation in an unconstrained environment.

Remote Heart Rate Estimation Based on Transformer with Multi-Skip Connection Decoder: Method and Evaluation in the Wild

Heart rate is an essential vital sign to evaluate human health. Remote heart monitoring using cheaply available devices has become a necessity in the twenty-first century to prevent any unfortunate situation caused by the hectic pace of life. In this paper, we propose a new method based on the transformer architecture with a multi-skip connection biLSTM decoder to estimate heart rate remotely from videos. Our method is based on the skin color variation caused by the change in blood volume in its surface. The presented heart rate estimation framework consists of three main steps: (1) the segmentation of the facial region of interest (ROI) based on the landmarks obtained by 3DDFA; (2) the extraction of the spatial and global features; and (3) the estimation of the heart rate value from the obtained features based on the proposed method. This paper investigates which feature extractor performs better by captioning the change in skin color related to the heart rate as well as the optimal number of frames needed to achieve better accuracy. Experiments were conducted using two publicly available datasets (LGI-PPGI and Vision for Vitals) and our own in-the-wild dataset (12 videos collected by four drivers). The experiments showed that our approach achieved better results than the previously published methods, making it the new state of the art on these datasets.

Simulating cardiac signals on 3D human models for photoplethysmography development

Introduction: Image-based heart rate estimation technology offers a contactless approach to healthcare monitoring that could improve the lives of millions of people. In order to comprehensively test or optimize image-based heart rate extraction methods, the dataset should contain a large number of factors such as body motion, lighting conditions, and physiological states. However, collecting high-quality datasets with complete parameters is a huge challenge. Methods: In this paper, we introduce a bionic human model based on a three-dimensional (3D) representation of the human body. By integrating synthetic cardiac signal and body involuntary motion into the 3D model, five well-known traditional and four deep learning iPPG (imaging photoplethysmography) extraction methods are used to test the rendered videos. Results: To compare with different situations in the real world, four common scenarios (stillness, expression/talking, light source changes, and physical activity) are created on each 3D human. The 3D human can be built with any appearance and different skin tones. A high degree of agreement is achieved between the signals extracted from videos with the synthetic human and videos with a real human-the performance advantages and disadvantages of the selected iPPG methods are consistent for both real and 3D humans. Discussion: This technology has the capability to generate synthetic humans within various scenarios, utilizing precisely controlled parameters and disturbances. Furthermore, it holds considerable potential for testing and optimizing image-based vital signs methods in challenging situations where real people with reliable ground truth measurements are difficult to obtain, such as in drone rescue.

Evaluating RGB channels in remote photoplethysmography: a comparative study with contact-based PPG

Remote photoplethysmography (rPPG) provides a non-contact method for measuring blood volume changes. In this study, we compared rPPG signals obtained from video cameras with traditional contact-based photoplethysmography (cPPG) to assess the effectiveness of different RGB channels in cardiac signal extraction. Our objective was to determine the most effective RGB channel for detecting blood volume changes and estimating heart rate. We employed dynamic time warping, Pearson's correlation coefficient, root-mean-square error, and Beats-per-minute Difference to evaluate the performance of each RGB channel relative to cPPG. The results revealed that the green channel was superior, outperforming the blue and red channels in detecting volumetric changes and accurately estimating heart rate across various activities. We also observed that the reliability of RGB signals varied based on recording conditions and subject activity. This finding underscores the importance of understanding the performance nuances of RGB inputs, crucial for constructing rPPG signals in algorithms. Our study is significant in advancing rPPG research, offering insights that could benefit clinical applications by improving non-contact methods for blood volume assessment.

DSE-NN: Deeply Supervised Efficient Neural Network for Real-Time Remote Photoplethysmography

Non-contact remote photoplethysmography can be used in a variety of medical and healthcare fields by measuring vital signs continuously and unobtrusively. Recently, end-to-end deep learning methods have been proposed to replace the existing handcrafted features. However, since the existing deep learning methods are known as black box models, the problem of interpretability has been raised, and the same problem exists in the remote photoplethysmography (rPPG) network. In this study, we propose a method to visualize temporal and spectral representations for hidden layers, deeply supervise the spectral representation of intermediate layers through the depth of networks and optimize it for a lightweight model. The optimized network improves performance and enables fast training and inference times. The proposed spectral deep supervision helps to achieve not only high performance but also fast convergence speed through the regularization of the intermediate layers. The effect of the proposed methods was confirmed through a thorough ablation study on public datasets. As a result, similar or outperforming results were obtained in comparison to state-of-the-art models. In particular, our model achieved an RMSE of 1 bpm on the PURE dataset, demonstrating its high accuracy. Moreover, it excelled on the V4V dataset with an impressive RMSE of 6.65 bpm, outperforming other methods. We observe that our model began converging from the very first epoch, a significant improvement over other models in terms of learning efficiency. Our approach is expected to be generally applicable to models that learn spectral domain information as well as to the applications of regression that require the representations of periodicity.

Challenges and prospects of visual contactless physiological monitoring in clinical study

The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.

Capturing the pulse: a state-of-the-art review on camera-based jugular vein assessment

Heart failure is associated with a rehospitalisation rate of up to 50% within six months. Elevated central venous pressure may serve as an early warning sign. While invasive procedures are used to measure central venous pressure for guiding treatment in hospital, this becomes impractical upon discharge. A non-invasive estimation technique exists, where the clinician visually inspects the pulsation of the jugular veins in the neck, but it is less reliable due to human limitations. Video and signal processing technologies may offer a high-fidelity alternative. This state-of-the-art review analyses existing literature on camera-based methods for jugular vein assessment. We summarize key design considerations and suggest avenues for future research. Our review highlights the neck as a rich imaging target beyond the jugular veins, capturing comprehensive cardiac signals, and outlines factors affecting signal quality and measurement accuracy. Addressing an often quoted limitation in the field, we also propose minimum reporting standards for future studies.

Video-based sympathetic arousal assessment via peripheral blood flow estimation

Electrodermal activity (EDA) is considered a standard marker of sympathetic activity. However, traditional EDA measurement requires electrodes in steady contact with the skin. Can sympathetic arousal be measured using only an optical sensor, such as an RGB camera? This paper presents a novel approach to infer sympathetic arousal by measuring the peripheral blood flow on the face or hand optically. We contribute a self-recorded dataset of 21 participants, comprising synchronized videos of participants' faces and palms and gold-standard EDA and photoplethysmography (PPG) signals. Our results show that we can measure peripheral sympathetic responses that closely correlate with the ground truth EDA. We obtain median correlations of 0.57 to 0.63 between our inferred signals and the ground truth EDA using only videos of the participants' palms or foreheads or PPG signals from the foreheads or fingers. We also show that sympathetic arousal is best inferred from the forehead, finger, or palm.

The 2023 wearable photoplethysmography roadmap

Photoplethysmography is a key sensing technology which is used in wearable devices such as smartwatches and fitness trackers. Currently, photoplethysmography sensors are used to monitor physiological parameters including heart rate and heart rhythm, and to track activities like sleep and exercise. Yet, wearable photoplethysmography has potential to provide much more information on health and wellbeing, which could inform clinical decision making. This Roadmap outlines directions for research and development to realise the full potential of wearable photoplethysmography. Experts discuss key topics within the areas of sensor design, signal processing, clinical applications, and research directions. Their perspectives provide valuable guidance to researchers developing wearable photoplethysmography technology.

Face2PPG: An Unsupervised Pipeline for Blood Volume Pulse Extraction From Faces

Photoplethysmography (PPG) signals have become a key technology in many fields, such as medicine, well-being, or sports. Our work proposes a set of pipelines to extract remote PPG signals (rPPG) from the face robustly, reliably, and configurably. We identify and evaluate the possible choices in the critical steps of unsupervised rPPG methodologies. We assess a state-of-the-art processing pipeline in six different datasets, incorporating important corrections in the methodology that ensure reproducible and fair comparisons. In addition, we extend the pipeline by proposing three novel ideas; 1) a new method to stabilize the detected face based on a rigid mesh normalization; 2) a new method to dynamically select the different regions in the face that provide the best raw signals, and 3) a new RGB to rPPG transformation method, called Orthogonal Matrix Image Transformation (OMIT) based on QR decomposition, that increases robustness against compression artifacts. We show that all three changes introduce noticeable improvements in retrieving rPPG signals from faces, obtaining state-of-the-art results compared with unsupervised, non-learning-based methodologies and, in some databases, very close to supervised, learning-based methods. We perform a comparative study to quantify the contribution of each proposed idea. In addition, we depict a series of observations that could help in future implementations.

Estimation of vital signs from facial videos via video magnification and deep learning

The continuous monitoring of vital signs is one of the hottest topics in healthcare. Recent technological advances in sensors, signal processing, and image processing spawned the development of no-contact techniques such as remote photoplethysmography (rPPG). To solve the common problems of rPPG including weak extracted signals, body movements, and generalization with limited data resources, we proposed a dual-path estimation method based on video magnification and deep learning. First, image processes are applied to detect, track, and magnificate facial ROIs automatically. Then, the steady part of the wave of each processed ROI is used for the extraction of features including heart rate, PTT, and features of pulse wave waveform. The blood pressures are estimated from the features via a small CNN. Results comply with the current standard and promise potential clinical applications in the future.

Non-contact high precision pulse-rate monitoring system for moving subjects in different motion states

Remote photoplethysmography (rPPG) enables contact-free monitoring of the pulse rate by using a color camera. The fundamental limitation is that motion artifacts and changes in ambient light conditions greatly affect the accuracy of pulse-rate monitoring. We propose use of a high-speed camera and a motion suppression algorithm with high computational efficiency. This system incorporates a number of major improvements including reproduction of pulse wave details, high-precision pulse-rate monitoring of moving subjects, and excellent scene scalability. A series of quantization methods were used to evaluate the effect of different frame rates and different algorithms in pulse-rate monitoring of moving subjects. The experimental results show that use of 180-fps video and a Plane-Orthogonal-to-Skin (POS) algorithm can produce high-precision pulse-rate monitoring results with mean absolute error can be less than 5 bpm and the relative accuracy reaching 94.5%. Thus, it has significant potential to improve personal health care and intelligent health monitoring.

Noise-Robust Pulse Wave Estimation from Near-Infrared Face Video Images Using the Wiener Estimation Method

In this paper, we propose a noise-robust pulse wave estimation method from near-infrared face video images. Pulse wave estimation in a near-infrared environment is expected to be applied to non-contact monitoring in dark areas. The conventional method cannot consider noise when performing estimation. As a result, the accuracy of pulse wave estimation in noisy environments is not very high. This may adversely affect the accuracy of heart rate data and other data obtained from pulse wave signals. Therefore, the objective of this study is to perform pulse wave estimation robust to noise. The Wiener estimation method, which is a simple linear computation that can consider noise, was used in this study. Experimental results showed that the combination of the proposed method and signal processing (detrending and bandpass filtering) increased the SNR (signal to noise ratio) by more than 2.5 dB compared to the conventional method and signal processing. The correlation coefficient between the pulse wave signal measured using a pulse wave meter and the estimated pulse wave signal was 0.30 larger on average for the proposed method. Furthermore, the AER (absolute error rate) between the heart rate measured with the pulse wave meter was 0.82% on average for the proposed method, which was lower than the value of the conventional method (12.53% on average). These results show that the proposed method is more robust to noise than the conventional method for pulse wave estimation.

Non-Contact Measurement of Blood Oxygen Saturation Using Facial Video Without Reference Values

The continuous measurement of percutaneous oxygen saturation (SpO2) enables diseases that cause hypoxemia to be detected early and patients' conditions to be monitored. Currently, SpO2 is mainly measured using a pulse oximeter, which, owing to its simplicity, can be used in clinical settings and at home. However, the pulse oximeter requires a sensor to be in contact with the skin; therefore, prolonged use of the pulse oximeter for neonates or patients with sensitive skin may cause local inflammation or stress due to restricted movement. In addition, owing to COVID-19, there has been a growing demand for the contactless measurement of SpO2. Several studies on measuring SpO2 without contact used skin video images have been conducted. However, in these studies, the SpO2 values were estimated using a linear regression model or a look-up table that required reference values obtained using a contact-type pulse oximeter. In this study, we propose a new technique for the contactless measurement of SpO2 that does not require reference values. Specifically, we used certain approaches that reduced the influence of non-pulsating components and utilized different light wavelengths of video images that penetrated subcutaneously to different depths. We experimentally investigated the accuracy of SpO2 measurements using the proposed methods. The results indicate that the proposed methods were more accurate than the conventional method.

Multi-Level Constrained Intra and Inter Subject Feature Representation for Facial Video Based BVP Signal Measurement

Facial video-based blood volume pulse (BVP) signal measurement holds great potential for remote health monitoring, while existing methods have issues with convolutional kernel perceptual field constraints. This article proposes an end-to-end multi-level constrained spatiotemporal representation structure for facial video-based BVP signal measurement. First, an intra- and inter-subject feature representation is proposed to strengthen the BVP-related features generation at high, semantic, and shallow levels, respectively. Second, the global-local association is presented to enhance BVP signal period pattern learning, and the global temporal features are introduced into the local spatial convolution of each frame by adaptive kernel weights. Finally, the multi-dimensional fused features are mapped to one-dimensional BVP signals by the task-oriented signal estimator. The experimental results on the publicly available MMSE-HR dataset demonstrate that the proposed structure overperforms state-of-the-art methods (e.g., AutoHR) in BVP signal measurement, with a 20% and 40% reduction in mean absolute error and root mean squared error, respectively. The proposed structure would be a powerful tool for telemedical and non-contact heart health monitoring.

MMPD: Multi-Domain Mobile Video Physiology Dataset

Remote photoplethysmography (rPPG) is an attractive method for noninvasive, convenient and concomitant measurement of physiological vital signals. Public benchmark datasets have served a valuable role in the development of this technology and improvements in accuracy over recent years. However, there remain gaps in the public datasets. First, despite the ubiquity of cameras on mobile devices, there are few datasets recorded specifically with mobile phone cameras. Second, most datasets are relatively small and therefore are limited in diversity, both in appearance (e.g., skin tone), behaviors (e.g., motion) and environment (e.g., lighting conditions). In an effort to help the field advance, we present the Multi-domain Mobile Video Physiology Dataset (MMPD), comprising 11 hours of recordings from mobile phones of 33 subjects. The dataset is designed to capture videos with greater representation across skin tone, body motion, and lighting conditions. MMPD is comprehensive with eight descriptive labels and can be used in conjunction with the rPPG-toolbox [1]. The reliability of the dataset is verified by mainstream unsupervised methods and neural methods. The GitHub repository of our dataset: https://github.com/THU-CS-PI/MMPD_rPPG_dataset.

Camera-based Cardiovascular Screening based on Heart Rate and Its Variability In Pre- and Post-Exercise Conditions

Analysis of heart rate (HR) and heart rate variability (HRV) in pre- and post-exercise conditions can provide useful information about the health condition of cardiovascular system. Remote photoplethysmography (rPPG) that uses a contactless camera to measure vital signs from the human face might allow for ubiquitous applications in the fitness scenario. This paper benchmarks the accuracy of HR and HRV measured by ECG, PPG and rPPG, and investigates the difference of HR and HRV parameters between individuals with/without the exercise habit in pre- and post-exercise phases. We built a fitness benchmark consisting of 14 healthy subjects to perform running exercise on a treadmill, with video and reference data recorded simultaneously. The results show that rPPG has a similar performance as PPG in both the estimation of HR and HRV. The HRV parameters (Mean IBI, SDNN, LF, VLF, and SD2) of rPPG/PPG show good agreements with the ECG reference. Subjects with a regular exercise habit show lower resting HR and smaller values of HRV parameters in pre-exercise phase, fewer changes in HR and HRV at the same exercise intensity, and faster HR recovery after exercising, as compared with those without an exercise habit. The preliminary results suggest that rPPG is a promising surrogate of PPG for screening HR and HRV before and after exercise, to indicate the performance of cardiac training.

Enhanced Contactless Heart Rate Monitoring Using Camera with Motion Artifact Removal During Physical Activities

Contactless monitoring of heart rate (HR) can improve passive and continuous tracking of cardiovascular activities and overall people's health. Remote photoplethysmography (rPPG) using a camera eliminates the need for a wearable device. rPPG-based HR has shown promising results to be accurate and comparable to conventional methods such as contact PPG. Most experiments use stationary subjects while motion is known to affect the accuracy of remote PPG. In this paper, a novel methodology is introduced to enhance the accuracy and reliability of HR monitoring based on rPPG in the presence of physical activities like Yoga. This method quickly and accurately tracks HR and analyzes head motion to exclude unreliable data within short windows of rPPG signals. The method was tested with smartphone video data collected from 60 subjects when they are doing activities with varying levels of movement. Results show that our method without motion removal improves the accuracy of the HR readings by 0.7 bpm, reaching 3.57 bpm on average for a 30-sec-window. The accuracy is further improved by another 1.3 bpm after removing the motion artifacts, and reaches 2.29 bpm.Clinical relevance- The enhancement of HR readings from shorter rPPG signal with motion tolerance during physical activities can ultimately help with a more reliable HR tracking of people in uncontrolled settings like home which is a critical step towards remote health-care or wellness tracking.

Photoplethysmography upon cold stress-impact of measurement site and acquisition mode

Photoplethysmography (PPG) allows various statements about the physiological state. It supports multiple recording setups, i.e., application to various body sites and different acquisition modes, rendering the technique a versatile tool for various situations. Owing to anatomical, physiological and metrological factors, PPG signals differ with the actual setup. Research on such differences can deepen the understanding of prevailing physiological mechanisms and path the way towards improved or novel methods for PPG analysis. The presented work systematically investigates the impact of the cold pressor test (CPT), i.e., a painful stimulus, on the morphology of PPG signals considering different recording setups. Our investigation compares contact PPG recorded at the finger, contact PPG recorded at the earlobe and imaging PPG (iPPG), i.e., non-contact PPG, recorded at the face. The study bases on own experimental data from 39 healthy volunteers. We derived for each recording setup four common morphological PPG features from three intervals around CPT. For the same intervals, we derived blood pressure and heart rate as reference. To assess differences between the intervals, we used repeated measures ANOVA together with paired t-tests for each feature and we calculated Hedges' g to quantify effect sizes. Our analyses show a distinct impact of CPT. As expected, blood pressure shows a highly significant and persistent increase. Independently of the recording setup, all PPG features show significant changes upon CPT as well. However, there are marked differences between recording setups. Effect sizes generally differ with the finger PPG showing the strongest response. Moreover, one feature (pulse width at half amplitude) shows an inverse behavior in finger PPG and head PPG (earlobe PPG and iPPG). In addition, iPPG features behave partially different from contact PPG features as they tend to return to baseline values while contact PPG features remain altered. Our findings underline the importance of recording setup and physiological as well as metrological differences that relate to the setups. The actual setup must be considered in order to properly interpret features and use PPG. The existence of differences between recording setups and a deepened knowledge on such differences might open up novel diagnostic methods in the future.

Contactless Technologies, Sensors, and Systems for Cardiac and Respiratory Measurement during Sleep: A Systematic Review

Sleep is essential to physical and mental health. However, the traditional approach to sleep analysis-polysomnography (PSG)-is intrusive and expensive. Therefore, there is great interest in the development of non-contact, non-invasive, and non-intrusive sleep monitoring systems and technologies that can reliably and accurately measure cardiorespiratory parameters with minimal impact on the patient. This has led to the development of other relevant approaches, which are characterised, for example, by the fact that they allow greater freedom of movement and do not require direct contact with the body, i.e., they are non-contact. This systematic review discusses the relevant methods and technologies for non-contact monitoring of cardiorespiratory activity during sleep. Taking into account the current state of the art in non-intrusive technologies, we can identify the methods of non-intrusive monitoring of cardiac and respiratory activity, the technologies and types of sensors used, and the possible physiological parameters available for analysis. To do this, we conducted a literature review and summarised current research on the use of non-contact technologies for non-intrusive monitoring of cardiac and respiratory activity. The inclusion and exclusion criteria for the selection of publications were established prior to the start of the search. Publications were assessed using one main question and several specific questions. We obtained 3774 unique articles from four literature databases (Web of Science, IEEE Xplore, PubMed, and Scopus) and checked them for relevance, resulting in 54 articles that were analysed in a structured way using terminology. The result was 15 different types of sensors and devices (e.g., radar, temperature sensors, motion sensors, cameras) that can be installed in hospital wards and departments or in the environment. The ability to detect heart rate, respiratory rate, and sleep disorders such as apnoea was among the characteristics examined to investigate the overall effectiveness of the systems and technologies considered for cardiorespiratory monitoring. In addition, the advantages and disadvantages of the considered systems and technologies were identified by answering the identified research questions. The results obtained allow us to determine the current trends and the vector of development of medical technologies in sleep medicine for future researchers and research.

Contactless Camera-Based Heart Rate and Respiratory Rate Monitoring Using AI on Hardware

Detecting vital signs by using a contactless camera-based approach can provide several advantages over traditional clinical methods, such as lower financial costs, reduced visit times, increased comfort, and enhanced safety for healthcare professionals. Specifically, Eulerian Video Magnification (EVM) or Remote Photoplethysmography (rPPG) methods can be utilised to remotely estimate heart rate and respiratory rate biomarkers. In this paper two contactless camera-based health monitoring architectures are developed using EVM and rPPG, respectively; to this end, two different CNNs, (Mediapipe's BlazeFace and FaceMesh) are used to extract suitable regions of interest from incoming video frames. These two methods are implemented and deployed on four off-the-shelf edge devices as well as on a PC and evaluated in terms of latency (in each stage of the application's pipeline), throughput (FPS), power consumption (Watt), efficiency (throughput/Watt), and value (throughput/cost). This work provides important insights about the computational costs and bottlenecks of each method on each hardware platform, as well as which platform to use depending on the target metric. One of our insights shows that the Jetson Xavier NX platform is the best platform in terms of throughput and efficiency, while Raspberry Pi 4 8 GB is the best platform in terms of value.

Monitoring Pulse Rate in the Background Using Front Facing Cameras of Mobile Devices

We propose a novel framework to passively monitor pulse rate during the time spent by users on their personal mobile devices. Our framework is based on passively capturing the user's pulse signal using the front-facing camera. Signal capture is performed in the background, while the user is interacting with the device as he/she normally would, e.g., watch movies, read emails, text, and play games. The framework does not require subject participation with the monitoring procedure, thereby addressing the well-known problem of low adherence with such procedures. We investigate various techniques to suppress the impact of spontaneous user motion and fluctuations in ambient light conditions expected in non-participatory environments. Techniques include traditional signal processing, machine learning classifiers, and deep learning methods. Our performance evaluation is based on a clinical study encompassing 113 patients with a history of atrial fibrillation (Afib) who are passively monitored at home using a tablet for a period of two weeks. Our results show that the proposed framework accurately monitors pulse rate, thereby providing a gateway for long-term monitoring without relying on subject participation or the use of a dedicated wearable device.

[A non-contact continuous blood pressure measurement method based on video stream]

Hypertension is the primary disease that endangers human health. A convenient and accurate blood pressure measurement method can help to prevent the hypertension. This paper proposed a continuous blood pressure measurement method based on facial video signal. Firstly, color distortion filtering and independent component analysis were used to extract the video pulse wave of the region of interest in the facial video signal, and the multi-dimensional feature extraction of the pulse wave was preformed based on the time-frequency domain and physiological principles; Secondly, an integrated feature selection method was designed to extract the universal optimal feature subset; After that, we compared the single person blood pressure measurement models established by Elman neural network based on particle swarm optimization, support vector machine (SVM) and deep belief network; Finally, we used SVM algorithm to build a general blood pressure prediction model, which was compared and evaluated with the real blood pressure value. The experimental results showed that the blood pressure measurement results based on facial video were in good agreement with the standard blood pressure values. Comparing the estimated blood pressure from the video with standard blood pressure value, the mean absolute error (MAE) of systolic blood pressure was 4.9 mm Hg with a standard deviation (STD) of 5.9 mm Hg, and the MAE of diastolic blood pressure was 4.6 mm Hg with a STD of 5.0 mm Hg, which met the AAMI standards. The non-contact blood pressure measurement method based on video stream proposed in this paper can be used for blood pressure measurement.

Arterial pulsation modulates the optical attenuation coefficient of skin

Photoplethysmographic (PPG) signals arise from the modulation of light reflectivity on the skin due to changes of physiological origin. Imaging plethysmography (iPPG) is a video-based PPG method that can remotely monitor vital signs in a non-invasive manner. iPPG signals result from skin reflectivity modulation. The origin of such reflectivity modulation is still a subject of debate. Here, we have used optical coherence tomography (OCT) imaging to find whether iPPG signals may result from skin optical properties being directly or indirectly modulated by arterial transmural pressure propagation. The light intensity across the tissue was modeled through a simple exponential decay (Beer-Lambert law) to analyze in vivo the modulation of the optical attenuation coefficient of the skin by arterial pulsation. The OCT transversal images were acquired from a forearm of three subjects in a pilot study. The results show that the optical attenuation coefficient of skin changes at the same frequency as the arterial pulsation due to transmural pressure propagation (local ballistographic effect), but we cannot discard the contribution of global ballistographic effects.

Local attention and long-distance interaction of rPPG for deepfake detection

With the development of generative models, abused Deepfakes have aroused public concerns. As a defense mechanism, face forgery detection methods have been intensively studied. Remote photoplethysmography (rPPG) technology extract heartbeat signal from recorded videos by examining the subtle changes in skin color caused by cardiac activity. Since the face forgery process inevitably disrupts the periodic changes in facial color, rPPG signal proves to be a powerful biological indicator for Deepfake detection. Motivated by the key observation that rPPG signals produce unique rhythmic patterns in terms of different manipulation methods, we regard Deepfake detection also as a source detection task. The Multi-scale Spatial-Temporal PPG map is adopted to further exploit heartbeat signal from multiple facial regions. Moreover, to capture both spatial and temporal inconsistencies, we propose a two-stage network consisting of a Mask-Guided Local Attention module (MLA) to capture unique local patterns of PPG maps, and a Temporal Transformer to interact features of adjacent PPG maps in long distance. Abundant experiments on FaceForensics + + and Celeb-DF datasets prove the superiority of our method over all other rPPG-based approaches. Visualization also demonstrates the effectiveness of the proposed method.

Benchmarking of Contactless Heart Rate Measurement Systems in ARM-Based Embedded Platforms

Heart rate monitoring is especially important for aging individuals because it is associated with longevity and cardiovascular risk. Typically, this vital parameter can be measured using wearable sensors, which are widely available commercially. However, wearable sensors have some disadvantages in terms of acceptability, especially when used by elderly people. Thus, contactless solutions have increasingly attracted the scientific community in recent years. Camera-based photoplethysmography (also known as remote photoplethysmography) is an emerging method of contactless heart rate monitoring that uses a camera and a processing unit on the hardware side, and appropriate image processing methodologies on the software side. This paper describes the design and implementation of a novel pipeline for heart rate estimation using a commercial and low-cost camera as the input device. The pipeline's performance was tested and compared on a desktop PC, a laptop, and three different ARM-based embedded platforms (Raspberry Pi 4, Odroid N2+, and Jetson Nano). The results showed that the designed and implemented pipeline achieved an average accuracy of about 96.7% for heart rate estimation, with very low variance (between 1.5% and 2.5%) across processing platforms, user distances from the camera, and frame resolutions. Furthermore, benchmark analysis showed that the Odroid N2+ platform was the most convenient in terms of CPU load, RAM usage, and average execution time of the algorithmic pipeline.

For Heart Rate Assessments from Drone Footage in Disaster Scenarios

The ability to use drones to obtain important vital signs could be very valuable for emergency personnel during mass-casualty incidents. The rapid and robust remote assessment of heart rates could serve as a life-saving decision aid for first-responders. With the flight sensor data of a specialized drone, a pipeline was developed to achieve a robust, non-contact assessment of heart rates through remote photoplethysmography (rPPG). This robust assessment was achieved through adaptive face-aware exposure and comprehensive de-noising of a large number of predicted noise sources. In addition, we performed a proof-of-concept study that involved 18 stationary subjects with clean skin and 36 recordings of their vital signs, using the developed pipeline in outdoor conditions. In this study, we could achieve a single-value heart-rate assessment with an overall root-mean-squared error of 14.3 beats-per-minute, demonstrating the basic feasibility of our approach. However, further research is needed to verify the applicability of our approach in actual disaster situations, where remote photoplethysmography readings could be impacted by other factors, such as blood, dirt, and body positioning.

X-iPPGNet: A novel one stage deep learning architecture based on depthwise separable convolutions for video-based pulse rate estimation

Pulse rate (PR) is one of the most important markers for assessing a person's health. With the increasing demand for long-term health monitoring, much attention is being paid to contactless PR estimation using imaging photoplethysmography (iPPG). This non-invasive technique is based on the analysis of subtle changes in skin color. Despite efforts to improve iPPG, the existing algorithms are vulnerable to less-constrained scenarios (i.e., head movements, facial expressions, and environmental conditions). In this article, we propose a novel end-to-end spatio-temporal network, namely X-iPPGNet, for instantaneous PR estimation directly from facial video recordings. Unlike most existing systems, our model learns the iPPG concept from scratch without incorporating any prior knowledge or going through the extraction of blood volume pulse signals. Inspired by the Xception network architecture, color channel decoupling is used to learn additional photoplethysmographic information and to effectively reduce the computational cost and memory requirements. Moreover, X-iPPGNet predicts the pulse rate from a short time window (2 s), which has advantages with high and sharply fluctuating pulse rates. The experimental results revealed high performance under all conditions including head motions, facial expressions, and skin tone. Our approach significantly outperforms all current state-of-the-art methods on three benchmark datasets: MMSE-HR (MAE = 4.10 ; RMSE = 5.32 ; r = 0.85), UBFC-rPPG (MAE = 4.99 ; RMSE = 6.26 ; r = 0.67), MAHNOB-HCI (MAE = 3.17 ; RMSE = 3.93 ; r = 0.88).

Emotion Recognition Using Different Sensors, Emotion Models, Methods and Datasets: A Comprehensive Review

In recent years, the rapid development of sensors and information technology has made it possible for machines to recognize and analyze human emotions. Emotion recognition is an important research direction in various fields. Human emotions have many manifestations. Therefore, emotion recognition can be realized by analyzing facial expressions, speech, behavior, or physiological signals. These signals are collected by different sensors. Correct recognition of human emotions can promote the development of affective computing. Most existing emotion recognition surveys only focus on a single sensor. Therefore, it is more important to compare different sensors or unimodality and multimodality. In this survey, we collect and review more than 200 papers on emotion recognition by literature research methods. We categorize these papers according to different innovations. These articles mainly focus on the methods and datasets used for emotion recognition with different sensors. This survey also provides application examples and developments in emotion recognition. Furthermore, this survey compares the advantages and disadvantages of different sensors for emotion recognition. The proposed survey can help researchers gain a better understanding of existing emotion recognition systems, thus facilitating the selection of suitable sensors, algorithms, and datasets.

PhysFormer++: Facial Video-Based Physiological Measurement with SlowFast Temporal Difference Transformer

Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.

GRGB rPPG: An Efficient Low-Complexity Remote Photoplethysmography-Based Algorithm for Heart Rate Estimation

Remote photoplethysmography (rPPG) is a promising contactless technology that uses videos of faces to extract health parameters, such as heart rate. Several methods for transforming red, green, and blue (RGB) video signals into rPPG signals have been introduced in the existing literature. The RGB signals represent variations in the reflected luminance from the skin surface of an individual over a given period of time. These methods attempt to find the best combination of color channels to reconstruct an rPPG signal. Usually, rPPG methods use a combination of prepossessed color channels to convert the three RGB signals to one rPPG signal that is most influenced by blood volume changes. This study examined simple yet effective methods to convert the RGB to rPPG, relying only on RGB signals without applying complex mathematical models or machine learning algorithms. A new method, GRGB rPPG, was proposed that outperformed most machine-learning-based rPPG methods and was robust to indoor lighting and participant motion. Moreover, the proposed method estimated the heart rate better than well-established rPPG methods. This paper also discusses the results and provides recommendations for further research.

Neural Network Model Combination for Video-Based Blood Pressure Estimation: New Approach and Evaluation

One of the most effective vital signs of health conditions is blood pressure. It has such an impact that changes your state from completely relaxed to extremely unpleasant, which makes the task of blood pressure monitoring a main procedure that almost everyone undergoes whenever there is something wrong or suspicious with his/her health condition. The most popular and accurate ways to measure blood pressure are cuff-based, inconvenient, and pricey, but on the bright side, many experimental studies prove that changes in the color intensities of the RGB channels represent variation in the blood that flows beneath the skin, which is strongly related to blood pressure; hence, we present a novel approach to blood pressure estimation based on the analysis of human face video using hybrid deep learning models. We deeply analyzed proposed approaches and methods to develop combinations of state-of-the-art models that were validated by their testing results on the Vision for Vitals (V4V) dataset compared to the performance of other available proposed models. Additionally, we came up with a new metric to evaluate the performance of our models using Pearson's correlation coefficient between the predicted blood pressure of the subjects and their respiratory rate at each minute, which is provided by our own dataset that includes 60 videos of operators working on personal computers for almost 20 min in each video. Our method provides a cuff-less, fast, and comfortable way to estimate blood pressure with no need for any equipment except the camera of your smartphone.

Contactless Cardiovascular Assessment by Imaging Photoplethysmography: A Comparison with Wearable Monitoring

Despite the notable recent developments in the field of remote photoplethysmography (rPPG), extracting a reliable pulse rate variability (PRV) signal still remains a challenge. In this study, eight image-based photoplethysmography (iPPG) extraction methods (GRD, AGRD, PCA, ICA, LE, SPE, CHROM, and POS) were compared in terms of pulse rate (PR) and PRV features. The algorithms were made robust for motion and illumination artifacts by using ad hoc pre- and postprocessing steps. Then, they were systematically tested on the public dataset UBFC-RPPG, containing data from 42 subjects sitting in front of a webcam (30 fps) while playing a time-sensitive mathematical game. The performances of the algorithms were evaluated by statistically comparing iPPG-based and finger-PPG-based PR and PRV features in terms of Spearman's correlation coefficient, normalized root mean square error (NRMSE), and Bland-Altman analysis. The study revealed POS and CHROM techniques to be the most robust for PR estimation and the assessment of overall autonomic nervous system (ANS) dynamics by using PRV features in time and frequency domains. Furthermore, we demonstrated that a reliable characterization of the vagal tone is made possible by computing the Poincaré map of PRV series derived from the POS and CHROM methods. This study supports the use of iPPG systems as promising tools to obtain clinically useful and specific information about ANS dynamics.