Contactless Vital Signs Monitoring From Videos Recorded With Digital Cameras: An Overview

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Contactless Blood Oxygen Saturation Estimation from Facial Videos Using Deep Learning

Blood oxygen saturation (SpO2) is an essential physiological parameter for evaluating a person's health. While conventional SpO2 measurement devices like pulse oximeters require skin contact, advanced computer vision technology can enable remote SpO2 monitoring through a regular camera without skin contact. In this paper, we propose novel deep learning models to measure SpO2 remotely from facial videos and evaluate them using a public benchmark database, VIPL-HR. We utilize a spatial-temporal representation to encode SpO2 information recorded by conventional RGB cameras and directly pass it into selected convolutional neural networks to predict SpO2. The best deep learning model achieves 1.274% in mean absolute error and 1.71% in root mean squared error, which exceed the international standard of 4% for an approved pulse oximeter. Our results significantly outperform the conventional analytical Ratio of Ratios model for contactless SpO2 measurement. Results of sensitivity analyses of the influence of spatial-temporal representation color spaces, subject scenarios, acquisition devices, and SpO2 ranges on the model performance are reported with explainability analyses to provide more insights for this emerging research field.

Challenges and prospects of visual contactless physiological monitoring in clinical study

The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.

Capturing the pulse: a state-of-the-art review on camera-based jugular vein assessment

Heart failure is associated with a rehospitalisation rate of up to 50% within six months. Elevated central venous pressure may serve as an early warning sign. While invasive procedures are used to measure central venous pressure for guiding treatment in hospital, this becomes impractical upon discharge. A non-invasive estimation technique exists, where the clinician visually inspects the pulsation of the jugular veins in the neck, but it is less reliable due to human limitations. Video and signal processing technologies may offer a high-fidelity alternative. This state-of-the-art review analyses existing literature on camera-based methods for jugular vein assessment. We summarize key design considerations and suggest avenues for future research. Our review highlights the neck as a rich imaging target beyond the jugular veins, capturing comprehensive cardiac signals, and outlines factors affecting signal quality and measurement accuracy. Addressing an often quoted limitation in the field, we also propose minimum reporting standards for future studies.

Thermal Image Processing for Respiratory Estimation from Cubical Data with Expandable Depth

As healthcare costs continue to rise, finding affordable and non-invasive ways to monitor vital signs is increasingly important. One of the key metrics for assessing overall health and identifying potential issues early on is respiratory rate (RR). Most of the existing methods require multiple steps that consist of image and signal processing. This might be difficult to deploy on edge devices that often do not have specialized digital signal processors (DSP). Therefore, the goal of this study is to develop a single neural network realizing the entire process of RR estimation in a single forward pass. The proposed solution builds on recent advances in video recognition, capturing both spatial and temporal information in a multi-path network. Both paths process the data at different sampling rates to capture rapid and slow changes that are associated with differences in the temperature of the nostril area during the breathing episodes. The preliminary results show that the introduced end-to-end solution achieves better performance compared to state-of-the-art methods, without requiring additional pre/post-processing steps and signal-processing techniques. In addition, the presented results demonstrate its robustness on low-resolution thermal video sequences that are often used at the embedded edge due to the size and power constraints of such systems. Taking that into account, the proposed approach has the potential for efficient and convenient respiratory rate estimation across various markets in solutions deployed locally, close to end users.

Photoplethysmography upon cold stress-impact of measurement site and acquisition mode

Photoplethysmography (PPG) allows various statements about the physiological state. It supports multiple recording setups, i.e., application to various body sites and different acquisition modes, rendering the technique a versatile tool for various situations. Owing to anatomical, physiological and metrological factors, PPG signals differ with the actual setup. Research on such differences can deepen the understanding of prevailing physiological mechanisms and path the way towards improved or novel methods for PPG analysis. The presented work systematically investigates the impact of the cold pressor test (CPT), i.e., a painful stimulus, on the morphology of PPG signals considering different recording setups. Our investigation compares contact PPG recorded at the finger, contact PPG recorded at the earlobe and imaging PPG (iPPG), i.e., non-contact PPG, recorded at the face. The study bases on own experimental data from 39 healthy volunteers. We derived for each recording setup four common morphological PPG features from three intervals around CPT. For the same intervals, we derived blood pressure and heart rate as reference. To assess differences between the intervals, we used repeated measures ANOVA together with paired t-tests for each feature and we calculated Hedges' g to quantify effect sizes. Our analyses show a distinct impact of CPT. As expected, blood pressure shows a highly significant and persistent increase. Independently of the recording setup, all PPG features show significant changes upon CPT as well. However, there are marked differences between recording setups. Effect sizes generally differ with the finger PPG showing the strongest response. Moreover, one feature (pulse width at half amplitude) shows an inverse behavior in finger PPG and head PPG (earlobe PPG and iPPG). In addition, iPPG features behave partially different from contact PPG features as they tend to return to baseline values while contact PPG features remain altered. Our findings underline the importance of recording setup and physiological as well as metrological differences that relate to the setups. The actual setup must be considered in order to properly interpret features and use PPG. The existence of differences between recording setups and a deepened knowledge on such differences might open up novel diagnostic methods in the future.

Monitoring Pulse Rate in the Background Using Front Facing Cameras of Mobile Devices

We propose a novel framework to passively monitor pulse rate during the time spent by users on their personal mobile devices. Our framework is based on passively capturing the user's pulse signal using the front-facing camera. Signal capture is performed in the background, while the user is interacting with the device as he/she normally would, e.g., watch movies, read emails, text, and play games. The framework does not require subject participation with the monitoring procedure, thereby addressing the well-known problem of low adherence with such procedures. We investigate various techniques to suppress the impact of spontaneous user motion and fluctuations in ambient light conditions expected in non-participatory environments. Techniques include traditional signal processing, machine learning classifiers, and deep learning methods. Our performance evaluation is based on a clinical study encompassing 113 patients with a history of atrial fibrillation (Afib) who are passively monitored at home using a tablet for a period of two weeks. Our results show that the proposed framework accurately monitors pulse rate, thereby providing a gateway for long-term monitoring without relying on subject participation or the use of a dedicated wearable device.

Radar-based remote physiological sensing: Progress, challenges, and opportunities

Modern microwave Doppler radar-based physiological sensing is playing an important role in healthcare applications and during the last decade, there has been a significant advancement in this non-contact respiration sensing technology. The advantages of contactless, unobtrusive respiration monitoring have drawn interest in various medical applications such as sleep apnea, sudden infant death syndromes (SIDS), remote respiratory monitoring of burn victims, and COVID patients. This paper provides a perspective on recent advances in biomedical and healthcare applications of Doppler radar that can detect the tiny movement of the chest surfaces to extract heartbeat and respiration and its associated different vital signs parameters (tidal volume, heart rate variability (HRV), and so on) of the human subject. Additionally, it also highlights the challenges, and opportunities of this remote physiological sensing technology and several future research directions will be laid out to deploy this sensor technology in our day-to-day life.

Assessment of facial video-based detection of atrial fibrillation across human complexion

Background

Early self-detection of atrial fibrillation (AF) can help delay and/or prevent significant associated complications, including embolic stroke and heart failure. We developed a facial video technology, videoplethysmography (VPG), to detect AF based on the analysis of facial pulsatile signals.

Objective

The purpose of this study was to evaluate the accuracy of a video-based technology to detect AF on a smartphone and to test the performance of the technology in AF patients across the whole spectrum of skin complexion and under various recording conditions.

Methods

The performance of video-based monitoring depends on a set of factors such as the angle and the distance between the camera and the patient's face, the strength of illumination, and the patient's skin tone. We conducted a clinical study involving 60 subjects with a confirmed diagnosis of AF. A continuous electrocardiogram was used as the gold standard for cardiac rhythm annotation. The VPG technology was fine-tuned on a smartphone for the first 15 subjects. Validation recordings were then done using 7053 measurements collected from the remaining 45 subjects.

Results

The VPG technology detected the presence of AF using the video camera from a common smartphone with sensitivity and specificity ≥90%. The ambient level of illumination needs to be ≥100 lux for the technology to deliver consistent performance across all skin tones.

Conclusion

We demonstrated that facial video-based detection of AF provides accurate outpatient cardiac monitoring including high pulse rate accuracy and medical-grade performance for AF detection.

Multi-ROI Spectral Approach for the Continuous Remote Cardio-Respiratory Monitoring from Mobile Device Built-In Cameras

Heart rate (HR) and respiratory rate (f_R) can be estimated by processing videos framing the upper body and face regions without any physical contact with the subject. This paper proposed a technique for continuously monitoring HR and f_R via a multi-ROI approach based on the spectral analysis of RGB video frames recorded with a mobile device (i.e., a smartphone's camera). The respiratory signal was estimated by the motion of the chest, whereas the cardiac signal was retrieved from the pulsatile activity at the level of right and left cheeks and forehead. Videos were recorded from 18 healthy volunteers in four sessions with different user-camera distances (i.e., 0.5 m and 1.0 m) and illumination conditions (i.e., natural and artificial light). For HR estimation, three approaches were investigated based on single or multi-ROI approaches. A commercially available multiparametric device was used to record reference respiratory signals and electrocardiogram (ECG). The results demonstrated that the multi-ROI approach outperforms the single-ROI approach providing temporal trends of both the vital parameters comparable to those provided by the reference, with a mean absolute error (MAE) consistently below 1 breaths·min^-1 for f_R in all the scenarios, and a MAE between 0.7 bpm and 6 bpm for HR estimation, whose values increase at higher distances.