Improvements in remote cardiopulmonary measurement using a five band digital camera

A Comparative Study of Principled rPPG-Based Pulse Rate Tracking Algorithms for Fitness Activities

Performance improvements obtained by recent principled approaches for pulse rate (PR) estimation from face videos have typically been achieved by adding or modifying certain modules within a reconfigurable system. Yet, evaluations of such remote photoplethysmography (rPPG) are usually performed only at the system level. To better understand each module's contribution and facilitate future research in explainable learning and artificial intelligence for physiological monitoring, this paper conducts a comparative study of video-based, principled PR tracking algorithms, with a focus on challenging fitness scenarios. A review of the progress achieved over the last decade and a half in this field is utilized to construct the major processing modules of a reconfigurable remote pulse rate sensing system. Experiments are conducted on two challenging datasets-an internal collection of 25 videos of two Asian males exercising on stationary-bike, elliptical, and treadmill machines and 34 videos from a public ECG fitness database of 14 men and 3 women exercising on elliptical and stationary-bike machines. The signal-to-noise ratio (SNR), Pearson's correlation coefficient, error count ratio, error rate, and root mean squared error are used for performance evaluation. The top-performing configuration produces respective values of 0.8 dB, 0.86, 9%, 1.7%, and 3.3 beats per minute (bpm) for the internal dataset and 1.3 dB, 0.77, 28.6%, 6.0%, and 8.1 bpm for the ECG Fitness dataset, achieving significant improvements over alternative configurations. Our results suggest a synergistic effect between pulse color mapping and adaptive motion filtering, as well as the importance of a robust frequency tracking algorithm for PR estimation in low SNR settings.

Deep learning and remote photoplethysmography powered advancements in contactless physiological measurement

In recent decades, there has been ongoing development in the application of computer vision (CV) in the medical field. As conventional contact-based physiological measurement techniques often restrict a patient's mobility in the clinical environment, the ability to achieve continuous, comfortable and convenient monitoring is thus a topic of interest to researchers. One type of CV application is remote imaging photoplethysmography (rPPG), which can predict vital signs using a video or image. While contactless physiological measurement techniques have an excellent application prospect, the lack of uniformity or standardization of contactless vital monitoring methods limits their application in remote healthcare/telehealth settings. Several methods have been developed to improve this limitation and solve the heterogeneity of video signals caused by movement, lighting, and equipment. The fundamental algorithms include traditional algorithms with optimization and developing deep learning (DL) algorithms. This article aims to provide an in-depth review of current Artificial Intelligence (AI) methods using CV and DL in contactless physiological measurement and a comprehensive summary of the latest development of contactless measurement techniques for skin perfusion, respiratory rate, blood oxygen saturation, heart rate, heart rate variability, and blood pressure.

Capturing the pulse: a state-of-the-art review on camera-based jugular vein assessment

Heart failure is associated with a rehospitalisation rate of up to 50% within six months. Elevated central venous pressure may serve as an early warning sign. While invasive procedures are used to measure central venous pressure for guiding treatment in hospital, this becomes impractical upon discharge. A non-invasive estimation technique exists, where the clinician visually inspects the pulsation of the jugular veins in the neck, but it is less reliable due to human limitations. Video and signal processing technologies may offer a high-fidelity alternative. This state-of-the-art review analyses existing literature on camera-based methods for jugular vein assessment. We summarize key design considerations and suggest avenues for future research. Our review highlights the neck as a rich imaging target beyond the jugular veins, capturing comprehensive cardiac signals, and outlines factors affecting signal quality and measurement accuracy. Addressing an often quoted limitation in the field, we also propose minimum reporting standards for future studies.

The Evaluation of Remote Monitoring Technology Across Participants With Different Skin Tones

BACKGROUND

Many research studies seek to improve vital sign monitoring to enhance the conditions under which doctors and caregivers track patients' health. Non-invasive and contactless monitoring has emerged as an optimal solution for this problem, with telemedicine, self-monitoring, and well-being tools being the next generation of technology in the biomedical field. However, there is worldwide concern about the general purpose and bias toward a certain demographic group of these techniques. In particular, skin tone and the accuracy of monitoring dark skin tone groups have been key questions among researchers, with the lack of results and studies contributing to this uncertainty.

METHODS

This paper proposes a benchmark for remote monitoring solutions against a medical device across different skin tone people. Around 330 videos from 90 patients were analyzed, and heart rate (HR) and heart rate variability (HRV) were compared across different subgroups. The Fitzpatrick scale (1-6) was used to classify participants into three skin tone groups: 1 and 2, 3 and 4, and 5 and 6.

RESULTS

The results showed that our proposed methodology could estimate heart rate with a mean absolute error of 3 bpm across all samples and subgroups. Moreover, for heart rate variability (HRV) metrics, we achieved the following results: in terms of mobility assistive equipment (MAE), the HRV-inter-beat interval (IBI) was 10 ms, the HRV-standard deviation of normal to normal heartbeats (SDNN) was 14 ms, and the HRV-root mean square of successive differences (RMSSD) between normal heartbeats was 22 ms. No significant performance decrease was found for any skin tone group, and there was no error trend toward a certain group.

CONCLUSIONS

The study showed that our methodology meets acceptable agreement levels for the proposed metrics. Furthermore, the experiments showed that skin tone did not impact the results, which remained within the same range across all groups. Moreover, it enables the end users to understand their general well-being and improve their overall health.

X-iPPGNet: A novel one stage deep learning architecture based on depthwise separable convolutions for video-based pulse rate estimation

Pulse rate (PR) is one of the most important markers for assessing a person's health. With the increasing demand for long-term health monitoring, much attention is being paid to contactless PR estimation using imaging photoplethysmography (iPPG). This non-invasive technique is based on the analysis of subtle changes in skin color. Despite efforts to improve iPPG, the existing algorithms are vulnerable to less-constrained scenarios (i.e., head movements, facial expressions, and environmental conditions). In this article, we propose a novel end-to-end spatio-temporal network, namely X-iPPGNet, for instantaneous PR estimation directly from facial video recordings. Unlike most existing systems, our model learns the iPPG concept from scratch without incorporating any prior knowledge or going through the extraction of blood volume pulse signals. Inspired by the Xception network architecture, color channel decoupling is used to learn additional photoplethysmographic information and to effectively reduce the computational cost and memory requirements. Moreover, X-iPPGNet predicts the pulse rate from a short time window (2 s), which has advantages with high and sharply fluctuating pulse rates. The experimental results revealed high performance under all conditions including head motions, facial expressions, and skin tone. Our approach significantly outperforms all current state-of-the-art methods on three benchmark datasets: MMSE-HR (MAE = 4.10 ; RMSE = 5.32 ; r = 0.85), UBFC-rPPG (MAE = 4.99 ; RMSE = 6.26 ; r = 0.67), MAHNOB-HCI (MAE = 3.17 ; RMSE = 3.93 ; r = 0.88).

A Feasibility Study of Remote Non-Contact Vital Signs (NCVS) Monitoring in a Clinic Using a Novel Sensor Realized by Software-Defined Radio (SDR)

The COVID-19 outbreak has caused panic around the world as it is highly infectious and has caused about 5 million deaths globally. A robust wireless non-contact vital signs (NCVS) sensor system that can continuously monitor the respiration rate (RR) and heart rate (HR) of patients clinically and remotely with high accuracy can be very attractive to healthcare workers (HCWs), as such a system can not only avoid HCWs' close contact with people with COVID-19 to reduce the infection rate, but also be used on patients quarantined at home for telemedicine and wireless acute-care. Therefore, we developed a custom Doppler-based NCVS radar sensor system operating at 2.4 GHz using a software-defined radio (SDR) technology, and the novel biosensor system has achieved impressive real-time RR/HR monitoring accuracies within approximately 0.5/3 breath/beat per minute (BPM) on student volunteers tested in our engineering labs. To further test the sensor system's feasibility for clinical use, we applied and obtained an Internal Review Board (IRB) approval from Texas Tech University Health Sciences Center (TTUHSC) and have used this NCVS monitoring system in a doctor's clinic at TTUHSC; following testing on 20 actual patients for a small-scale clinical trial, we have found that the system was still able to achieve good NCVS monitoring accuracies within ~0.5/10 BPM across 20 patients of various weight, height and age. These results suggest our custom-designed NCVS monitoring system may be feasible for future clinical use to help combatting COVID-19 and other infectious diseases.

Optimal digital filter selection for remote photoplethysmography (rPPG) signal conditioning

Remote photoplethysmography (rPPG) using camera-based imaging has shown excellent potential recently in vital signs monitoring due to its contactless nature. However, the optimum filter selection for pre-processing rPPG data in signal conditioning is still not straightforward. The best algorithm selection improves the signal-to-noise ratio (SNR) and therefore improves the accuracy of the recognition and classification of vital signs. We recorded more than 300 temporal rPPG signals where the noise was not motion-induced. Then, we investigated the best digital filter in pre-processing temporal rPPG data and compared the performances of 10 filters with 10 orders each (i.e., a total of 100 filters). The performances are assessed using a signal quality metric on three levels. The quality of the raw signals was classified under three categories; Q1 being the best and Q3 being the worst. The results are presented in SNR scores, which show that the Chebyshev II orders of 2nd, 4th, and 6th perform the best for denoising rPPG signals.

Photoplethysmographic imaging and analysis of pulsatile pressure wave in palmar artery at 10 wavelengths

SIGNIFICANCE

As a noncontact method, imaging photoplethysmography (iPPG) may provide a powerful tool to measure pulsatile pressure wave (PPW) in superficial arteries and extract biomarkers for monitoring of artery wall stiffness.

AIM

We intend to develop a approach for extraction of the very weak cardiac component from iPPG data by identifying locations of strong PPW signals with optimized illumination wavelength and determining pulse wave velocity (PWV).

APPROACH

Monochromatic in vivo iPPG datasets have been acquired from left hands to investigate various algorithms for retrieval of PPW signals, distribution maps and waveforms, and their dependence on arterial location and wavelength.

RESULTS

A robust algorithm of pixelated independent component analysis (pICA) has been developed and combined with spatiotemporal filtering to retrieve PPW signals. Spatial distributions of PPW signals have been mapped in 10 wavelength bands from 445 to 940 nm and waveforms were analyzed at multiple locations near the palmar artery tree. At the wavelength of 850 nm selected for timing analysis, we determined PWV values from 12 healthy volunteers in a range of 0.5 to 5.8 m/s across the hand region from wrist to midpalm and fingertip.

CONCLUSIONS

These results demonstrate the potentials of the iPPG method based on pICA algorithm for translation into a monitoring tool to characterize wall stiffness of superficial artery by rapid and noncontact measurement of PWV and other biomarkers within 10 s.

Intelligent Remote Photoplethysmography-Based Methods for Heart Rate Estimation from Face Videos: A Survey

Over the last few years, a rich amount of research has been conducted on remote vital sign monitoring of the human body. Remote photoplethysmography (rPPG) is a camera-based, unobtrusive technology that allows continuous monitoring of changes in vital signs and thereby helps to diagnose and treat diseases earlier in an effective manner. Recent advances in computer vision and its extensive applications have led to rPPG being in high demand. This paper specifically presents a survey on different remote photoplethysmography methods and investigates all facets of heart rate analysis. We explore the investigation of the challenges of the video-based rPPG method and extend it to the recent advancements in the literature. We discuss the gap within the literature and suggestions for future directions.

Evaluating Visual Photoplethysmography Method

Regular monitoring of common physiological signs, including heart rate, blood pressure, and oxygen saturation, can be an effective way to either prevent or detect many kinds of chronic conditions. In particular, cardiovascular diseases (CVDs) are a worldwide concern. According to the World Health Organization, 32% of all deaths worldwide are from CVDs. In addition, stress-related illnesses cost $190 billion in healthcare costs per year. Currently, contact devices are required to extract most of an individual’s physiological information, which can be uncomfortable for users and can cause discomfort. However, in recent years, remote photoplethysmography (rPPG) technology is gaining interest, which enables contactless monitoring of the blood volume pulse signal using a regular camera, and ultimately can provide the same physiological information as a contact device. In this paper, we propose a benchmark comparison using a new multimodal database consisting of 56 subjects where each subject was submitted to three different tasks. Each subject wore a wearable device capable of extracting photoplethysmography signals and was filmed to allow simultaneous rPPG signal extraction. Several experiments were conducted, including a comparison between information from contact and remote signals and stress state recognition. Results have shown that in this dataset, rPPG signals were capable of dealing with motion artifacts better than contact PPG sensors and overall had better quality if compared to the signals from the contact sensor. Moreover, the statistical analysis of the variance method had shown that at least two heart-rate variability (HRV) features, NNi 20 and SAMPEN, were capable of differentiating between stress and non-stress states. In addition, three features, inter-beat interval (IBI), NNi 20, and SAMPEN, were capable of differentiating between tasks relating to different levels of difficulty. Furthermore, using machine learning to classify a "stressed" or "unstressed" state, the models were able to achieve an accuracy score of 83.11%.

Contactless Vital Signs Acquisition Using Video Photoplethysmography, Motion Analysis and Passive Infrared Thermography Devices During Emergency Department Walk-In Triage in Pandemic Conditions

BACKGROUND

Contactless vital signs (VS) measurement with video photoplethysmography (vPPG), motion analysis (MA), and passive infrared thermometry (pIR) has shown promise.

OBJECTIVES

To compare conventional (contact-based) and experimental contactless VS measurement approaches for emergency department (ED) walk-in triage in pandemic conditions.

METHODS

Patients' heart rates (HR), respiratory rates (RR), and temperatures were measured with cardiorespiratory monitor and vPPG, manual count and MA, and contact thermometers and pIR, respectively.

RESULTS

There were 475 walk-in ED patients studied (95% of eligible). Subjects were 35.2 ± 20.8 years old (range 4 days‒95 years); 52% female, 0.2% transgender; had Fitzpatrick skin type of 2.3 ± 1.4 (range 1‒6), Emergency Severity Index of 3.0 ± 0.6 (range 2‒5), and contact temperature of 36.83°C (range 35.89-39.4°C) (98.3°F [96.6‒103°F]). Pediatric HR and RR data were excluded from analysis due to research challenges associated with pandemic workflow. For a 30-s, unprimed "Triage" window in 377 adult patients, vPPG-MA acquired 377 (100%) HR measurements featuring a mean difference with cardiorespiratory monitor HR of 5.9 ± 12.8 beats/min (R = 0.6833) and 252 (66.8%) RR measurements featuring a mean difference with manual RR of -0.4 ± 2.6 beats/min (R = 0.8128). Subjects' Emergency Severity Index components based on conventional VS and contactless VS matched for 83.8% (HR) and 89.3% (RR). Filtering out vPPG-MA measurements with low algorithmic confidence reduced VS acquired while improving correlation with conventional measurements. The mean difference between contact and pIR temperatures was 0.83 ± 0.67°C (range -1.16-3.5°C) (1.5 ± 1.2°F [range -2.1-6.3°F]); pIR fever detection improved with post hoc adjustment for mean bias.

CONCLUSION

Contactless VS acquisition demonstrated good agreement with contact methods during adult walk-in ED patient triage in pandemic conditions; clinical applications will need further study.

PCG/PCGML evaluations: Introducing panda evaluation using the soft launch

This study takes a new perspective on the procedural content generation (PCG) evaluation problem, extracts current PCG evaluation methods from previous works, and presents a novel classification of these methods while showing each method’s capabilities. Also, the present study introduces a novel concept called Panda Evaluation. Additionally, the soft and hard launches were presented as two evaluation methods and possible building blocks of PE. A group of papers was analyzed to understand previous works and find new opportunities. In doing so, some missing PCG evaluation areas were found, and some new methods were proposed for future PCG evaluations. To the best of our knowledge, this is the first time these concepts have been presented in PCG evaluation.

An Overview of the Sensors for Heart Rate Monitoring Used in Extramural Applications

This work presents an overview of the main strategies that have been proposed for non-invasive monitoring of heart rate (HR) in extramural and home settings. We discuss three categories of sensing according to what physiological effect is used to measure the pulsatile activity of the heart, and we focus on an illustrative sensing modality for each of them. Therefore, electrocardiography, photoplethysmography, and mechanocardiography are presented as illustrative modalities to sense electrical activity, mechanical activity, and the peripheral effect of heart activity. In this paper, we describe the physical principles underlying the three categories and the characteristics of the different types of sensors that belong to each class, and we touch upon the most used software strategies that are currently adopted to effectively and reliably extract HR. In addition, we investigate the strengths and weaknesses of each category linked to the different applications in order to provide the reader with guidelines for selecting the most suitable solution according to the requirements and constraints of the application.

Diagnostic Features and Potential Applications of PPG Signal in Healthcare: A Systematic Review

Recent research indicates that Photoplethysmography (PPG) signals carry more information than oxygen saturation level (SpO2) and can be utilized for affordable, fast, and noninvasive healthcare applications. All these encourage the researchers to estimate its feasibility as an alternative to many expansive, time-wasting, and invasive methods. This systematic review discusses the current literature on diagnostic features of PPG signal and their applications that might present a potential venue to be adapted into many health and fitness aspects of human life. The research methodology is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines 2020. To this aim, papers from 1981 to date are reviewed and categorized in terms of the healthcare application domain. Along with consolidated research areas, recent topics that are growing in popularity are also discovered. We also highlight the potential impact of using PPG signals on an individual's quality of life and public health. The state-of-the-art studies suggest that in the years to come PPG wearables will become pervasive in many fields of medical practices, and the main domains include cardiology, respiratory, neurology, and fitness. Main operation challenges, including performance and robustness obstacles, are identified.

Using High-Fidelity Avatars to Advance Camera-based Cardiac Pulse Measurement

Non-contact physiological measurement has the potential to provide low-cost, non-invasive health monitoring. However, machine vision approaches are often limited by the availability and diversity of annotated video datasets resulting in poor generalization to complex real-life conditions. To address these challenges, this work proposes the use of synthetic avatars that display facial blood flow changes and allow for systematic generation of samples under a wide variety of conditions. Our results show that training on both simulated and real video data can lead to performance gains under challenging conditions. We show strong performance on three large benchmark datasets and improved robustness to skin type and motion. These results highlight the promise of synthetic data for training camera-based pulse measurement; however, further research and validation is needed to establish whether synthetic data alone could be sufficient for training models.

Comparative study on the effect of color spaces and color formats on heart rate measurement using the imaging photoplethysmography (IPPG) method

BACKGROUND

The imaging photoplethysmography (IPPG) technology has been demonstrated to be an effective method for heart rate (HR) monitoring. However, some interference caused by the ambient illumination variation and facial motion severely influences the accuracy of the HR measurement. Some color spaces and color formats are assumed to reduce the interference, and enhance the accuracy of HR estimation.

OBJECTIVE

The aim is to identify the optimal color space and format for IPPG based HR measurement.

METHODS

Six color spaces and 3 color formats are compared in this study, based on an IPPG based HR measurement system. 424 pieces of videos captured by the system are used for the selection of the optimal color channel and color space; while 10 pieces of videos are for the identification of the optimal color format.

RESULTS

The results shows that the green channel of RGB space is the optimal color channel, and RGB is the optimal color space, in respect of the mean squared error of HR estimation. BayerBG 8bit is found to be the optimal color format for video recording, which can significantly reduce the HR estimation error.

CONCLUSIONS

BayerBG 8bit color format for video recording, and RGB color space for video analysis is suggested for the IPPG based HR measurement system. The suitable configuration of color space and format could enhance the accuracy of HR measurement.

Non-Contact Measurement of Heart Rate Variability in Frail Geriatric Patients: Response to Early Geriatric Rehabilitation and Comparison with Healthy Old Community-Dwelling Individuals - A Pilot Study

INTRODUCTION

Frailty is a central geriatric syndrome characterized by a state of increased physiological vulnerability. As the key components of frailty are difficult to capture in their entirety, easily measurable and reliable surrogate parameters are desirable. Since frailty influences heart rate variability (HRV), HRV may be such a surrogate parameter. HRV is typically acquired by an ECG, which, however, may not be tolerated by all patients; in some, it may even trigger delirium. Therefore, we sought to measure HRV in a non-contact and unobtrusive way through photoplethysmography imaging (PPGI). Using our previously presented HRV estimation algorithm for PPGI, we investigated whether PPGI could reveal (1) HRV differences between frail and non-frail individuals and (2) the influences of early geriatric rehabilitation on HRV.

METHODS

The study involved 10 frail geriatric inpatients undergoing early geriatric rehabilitation and 10 healthy community-dwelling older adults. All participants underwent a comprehensive geriatric assessment. HRV measurements using a PPGI system and a reference ECG were made at the beginning and the end of the rehabilitation. HRV in terms of LF/HF ratio was analysed for both intra-individual changes during the geriatric rehabilitation and differences between frail geriatric patients and healthy community-dwelling individuals.

RESULTS

Across all geriatric patients, the median LF/HF ratio obtained with PPGI was found to be reduced by 0.178 (24.8%) during early geriatric rehabilitation. The assessment at the end of the rehabilitation revealed a simultaneous improvement of the functional state. Moreover, frail geriatric patients had a higher LF/HF ratio than their community-dwelling counterparts. Both observations in PPGI-based HRV were confirmed by the reference. The capability of PPGI to track intra-individual HRV changes was also analysed; a Spearman correlation of ρ = 1.0 between PPGI-based HRV and reference was achieved for 58.8% of the participants.

CONCLUSION

Early geriatric rehabilitation improves the functional state, which is associated with an increased HRV. PPGI is capable of detecting HRV changes/trends in that age group. While the tracking of intra-individual HRV changes is also possible, its reliability needs improvement. Nevertheless, the capabilities demonstrated in our study and the non-contact measurement principle of PPGI emphasize its potential for application in geriatric medicine.

Pulse Rate Variability Analysis Using Remote Photoplethysmography Signals

Pulse rate variability (PRV) refers to the change in the interval between pulses in the blood volume pulse (BVP) signal acquired using photoplethysmography (PPG). PRV is an indicator of the health status of an individual’s autonomic nervous system. A representative method for measuring BVP is contact PPG (CPPG). CPPG may cause discomfort to a user, because the sensor is attached to the finger for measurements. In contrast, noncontact remote PPG (RPPG) extracts BVP signals from face data using a camera without the need for a sensor. However, because the existing RPPG is a technology that extracts a single pulse rate rather than a continuous BVP signal, it is difficult to extract additional health status indicators. Therefore, in this study, PRV analysis is performed using lab-based RPPG technology that can yield continuous BVP signals. In addition, we intended to confirm that the analysis of PRV via RPPG can be performed with the same quality as analysis via CPPG. The experimental results confirmed that the temporal and frequency parameters of PRV extracted from RPPG and CPPG were similar. In terms of correlation, the PRVs of RPPG and CPPG yielded correlation coefficients between 0.98 and 1.0.

Noncontact Monitoring of Heart Rate and Heart Rate Variability in Geriatric Patients Using Photoplethysmography Imaging

OBJECTIVE

Geriatric patients, especially those with dementia or in a delirious state, do not accept conventional contact-based monitoring. Therefore, we propose to measure heart rate (HR) and heart rate variability (HRV) of geriatric patients in a noncontact and unobtrusive way using photoplethysmography imaging (PPGI).

METHODS

PPGI video sequences were recorded from 10 geriatric patients and 10 healthy elderly people using a monochrome camera operating in the near-infrared spectrum and a colour camera operating in the visible spectrum. PPGI waveforms were extracted from both cameras using superpixel-based regions of interests (ROI). A classifier based on bagged trees was trained to automatically select artefact-free ROIs for HR estimation. HRV was calculated in the time-domain and frequency-domain.

RESULTS

an RMSE of 1.03 bpm and a correlation of 0.8 with the reference was achieved using the NIR camera for HR estimation. Using the RGB camera, RMSE and correlation improved to 0.48 bpm and 0.95, respectively. Correlation for HRV in the frequency-domain (LF/HF-ratio) was 0.50 using the NIR camera and 0.70 using the RGB camera.

CONCLUSION

We were able to demonstrate that PPGI is very suitable to measure HR and HRV in geriatric patients. We strongly believe that PPGI will become clinically relevant in monitoring of geriatric patients.

SIGNIFICANCE

we are the first group to measure both HR and HRV in awake geriatric patients using PPGI. Moreover, we systematically evaluate the effects of the spectrum (near-infrared vs. visible), ROI, and additional motion artefact reduction algorithms on the accuracy of estimated HR and HRV.

Computer Vision for Brain Disorders Based Primarily on Ocular Responses

Real-time ocular responses are tightly associated with emotional and cognitive processing within the central nervous system. Patterns seen in saccades, pupillary responses, and spontaneous blinking, as well as retinal microvasculature and morphology visualized via office-based ophthalmic imaging, are potential biomarkers for the screening and evaluation of cognitive and psychiatric disorders. In this review, we outline multiple techniques in which ocular assessments may serve as a non-invasive approach for the early detections of various brain disorders, such as autism spectrum disorder (ASD), Alzheimer's disease (AD), schizophrenia (SZ), and major depressive disorder (MDD). In addition, rapid advances in artificial intelligence (AI) present a growing opportunity to use machine learning-based AI, especially computer vision (CV) with deep-learning neural networks, to shed new light on the field of cognitive neuroscience, which is most likely to lead to novel evaluations and interventions for brain disorders. Hence, we highlight the potential of using AI to evaluate brain disorders based primarily on ocular features.

Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions

Using a camera as an optical sensor to monitor physiological parameters has garnered considerable research interest in biomedical engineering in recent decades. Researchers have explored the use of a camera for monitoring a variety of physiological waveforms, together with the vital signs carried by these waveforms. Most of the obtained waveforms are related to the human respiratory and cardiovascular systems, and in addition of being indicative of overall health, they can also detect early signs of certain diseases. While using a camera for noncontact physiological signal monitoring offers the advantages of low cost and operational ease, it also has the disadvantages such as vulnerability to motion and lack of burden-free calibration solutions in some use cases. This study presents an overview of the existing camera-based methods that have been reported in recent years. It introduces the physiological principles behind these methods, signal acquisition approaches, various types of acquired signals, data processing algorithms, and application scenarios of these methods. It also discusses the technological gaps between the camera-based methods and traditional medical techniques, which are mostly contact-based. Furthermore, we present the manner in which noncontact physiological signal monitoring use has been extended, particularly over the recent years, to more day-to-day aspects of individuals' lives, so as to go beyond the more conventional use case scenarios. We also report on the development of novel approaches that facilitate easier measurement of less often monitored and recorded physiological signals. These have the potential of ushering a host of new medical and lifestyle applications. We hope this study can provide useful information to the researchers in the noncontact physiological signal measurement community.

Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation

Remote photo-plethysmography (rPPG) uses a camera to estimate a person’s heart rate (HR). Similar to how heart rate can provide useful information about a person’s vital signs, insights about the underlying physio/psychological conditions can be obtained from heart rate variability (HRV). HRV is a measure of the fine fluctuations in the intervals between heart beats. However, this measure requires temporally locating heart beats with a high degree of precision. We introduce a refined and efficient real-time rPPG pipeline with novel filtering and motion suppression that not only estimates heart rates, but also extracts the pulse waveform to time heart beats and measure heart rate variability. This unsupervised method requires no rPPG specific training and is able to operate in real-time. We also introduce a new multi-modal video dataset, VicarPPG 2, specifically designed to evaluate rPPG algorithms on HR and HRV estimation. We validate and study our method under various conditions on a comprehensive range of public and self-recorded datasets, showing state-of-the-art results and providing useful insights into some unique aspects. Lastly, we make available CleanerPPG, a collection of human-verified ground truth peak/heart-beat annotations for existing rPPG datasets. These verified annotations should make future evaluations and benchmarking of rPPG algorithms more accurate, standardized and fair.

Heart Rate Variability in the Perinatal Period: A Critical and Conceptual Review

Neonatal intensive care units (NICUs) greatly expand the use of technology. There is a need to accurately diagnose discomfort, pain, and complications, such as sepsis, mainly before they occur. While specific treatments are possible, they are often time-consuming, invasive, or painful, with detrimental effects for the development of the infant. In the last 40 years, heart rate variability (HRV) has emerged as a non-invasive measurement to monitor newborns and infants, but it still is underused. Hence, the present paper aims to review the utility of HRV in neonatology and the instruments available to assess it, showing how HRV could be an innovative tool in the years to come. When continuously monitored, HRV could help assess the baby’s overall wellbeing and neurological development to detect stress-/pain-related behaviors or pathological conditions, such as respiratory distress syndrome and hyperbilirubinemia, to address when to perform procedures to reduce the baby’s stress/pain and interventions, such as therapeutic hypothermia, and to avoid severe complications, such as sepsis and necrotizing enterocolitis, thus reducing mortality. Based on literature and previous experiences, the first step to efficiently introduce HRV in the NICUs could consist in a monitoring system that uses photoplethysmography, which is low-cost and non-invasive, and displays one or a few metrics with good clinical utility. However, to fully harness HRV clinical potential and to greatly improve neonatal care, the monitoring systems will have to rely on modern bioinformatics (machine learning and artificial intelligence algorithms), which could easily integrate infant’s HRV metrics, vital signs, and especially past history, thus elaborating models capable to efficiently monitor and predict the infant’s clinical conditions. For this reason, hospitals and institutions will have to establish tight collaborations between the obstetric, neonatal, and pediatric departments: this way, healthcare would truly improve in every stage of the perinatal period (from conception to the first years of life), since information about patients’ health would flow freely among different professionals, and high-quality research could be performed integrating the data recorded in those departments.

Non-contact imaging of peripheral hemodynamics during cognitive and psychological stressors

Peripheral hemodynamics, measured via the blood volume pulse and vasomotion, provide a valuable way of monitoring physiological state. Camera imaging-based systems can be used to measure these peripheral signals without contact with the body, at distances of multiple meters. While researchers have paid attention to non-contact imaging photoplethysmography, the study of peripheral hemodynamics and the effect of autonomic nervous system activity on these signals has received less attention. Using a method, based on a tissue-like model of the skin, we extract melanin [Formula: see text] and hemoglobin [Formula: see text] concentrations from videos of the hand and face and show that significant decreases in peripheral pulse signal power (by 36% ± 29%) and vasomotion signal power (by 50% ± 26%) occur during periods of cognitive and psychological stress. Via three experiments we show that similar results are achieved across different stimuli and regions of skin (face and hand). While changes in peripheral pulse and vasomotion power were significant the changes in pulse rate variability were less consistent across subjects and tasks.

StressFoot: Uncovering the Potential of the Foot for Acute Stress Sensing in Sitting Posture

Stress is a naturally occurring psychological response and identifiable by several body signs. We propose a novel way to discriminate acute stress and relaxation, using movement and posture characteristics of the foot. Based on data collected from 23 participants performing tasks that induced stress and relaxation, we developed several machine learning models to construct the validity of our method. We tested our models in another study with 11 additional participants. The results demonstrated replicability with an overall accuracy of 87%. To also demonstrate external validity, we conducted a field study with 10 participants, performing their usual everyday office tasks over a working day. The results showed substantial robustness. We describe ten significant features in detail to enable an easy replication of our models.

Video-Based Pulse Rate Variability Measurement Using Periodic Variance Maximization and Adaptive Two-Window Peak Detection

Many previous studies have shown that the remote photoplethysmography (rPPG) can measure the Heart Rate (HR) signal with very high accuracy. The remote measurement of the Pulse Rate Variability (PRV) signal is also possible, but this is much more complicated because it is then necessary to detect the peaks on the temporal rPPG signal, which is usually quite noisy and has a lower temporal resolution than PPG signals obtained by contact equipment. Since the PRV signal is vital for various applications such as remote recognition of stress and emotion, the improvement of PRV measurement by rPPG is a critical task. Contact based PRV measurement has already been investigated, but the research on remotely measured PRV is very limited. In this paper, we propose to use the Periodic Variance Maximization (PVM) method to extract the rPPG signal and event-related Two-Window algorithm to improve the peak detection for PRV measurement. We have made several contributions. Firstly, we show that the newly proposed PVM method and Two-Window algorithm can be used for PRV measurement in the non-contact scenario. Secondly, we propose a method to adaptively determine the parameters of the Two-Window method. Thirdly, we compare the algorithm with other attempts for improving the non-contact PRV measurement such as the Slope Sum Function (SSF) method and the Local Maximum method. We calculated several features and compared the accuracy based on the ground truth provided by contact equipment. Our experiments showed that this algorithm performed the best of all the algorithms.

Validation of novel automatic ultra-wideband radar for sleep apnea detection

Background

To validate the accuracy of ultra-wideband (UWB) wireless radar for the screening diagnosis of sleep apnea.

Methods

One hundred and seventy-six qualified participants were successfully recruited. Apnea-hypopnea index (AHI) results from polysomnography (PSG) were reviewed by physicians, while the radar device automatically calculated AHI values with an embedded chip. All results were statistically analyzed.

Results

A UWB radar-based AHI algorithm was successfully developed according to respiratory movement and body motion signals. Of all 176 participants, 63 exhibited normal results (AHI <5/hr) and the remaining 113 were diagnosed with obstructive sleep apnea. Significant correlation was detected between radar AHI and PSG AHI (Intraclass correlation coefficient 0.98, P<0.001). Receiver operating characteristic curve (ROC) analysis revealed high sensitivity and specificity. High concordance in participants with varying gender, age, BMI, and PSG AHI was reached.

Conclusions

The UWB radar may be a portable, convenient, and reliable device for obstructive sleep apnea screening.

Using Non-Contact Imaging Photoplethysmography to Recover Diurnal Patterns in Heart Rate

Daily patterns in cardiovascular signals can reveal important information about physiological processes, health and well-being. Traditionally, contact sensors have been used to collect longitudinal data of this kind. However, recent advances in non-contact imaging techniques have led to algorithms that can be used to measure vital signs unobtrusively. Imaging methods are highly scalable due to the availability of webcams and computing devices making them attractive for longitudinal, in-situ measurement. Using a software tool we captured over 1,000 hours of non-contact heart rate measurements, via imaging photoplethysmography. Using these data we were able to recover diurnal patterns in heart rate during the working day. Non-contact sensing techniques hold much promise but also raise ethical issues that need to be addressed seriously within the biomedical engineering community.

Visualizing Blood Flow of Palm in Different Muscle Tense State Using High-Speed Video Camera

In this paper, we propose a method to visualize blood flow of palm in different muscle tense state using RGB high-speed camera. Recently, new modalities are needed to develop a more accurate system to non-contact multi-modal affect analysis. Then, we focus on muscle tense. The muscle tense is caused by stress. Hence, the muscle tense is one of the effective modalities for non-contact multi-modal affect analysis. However, it is very difficult to measure muscle tense in the real environment because it requires a contact-type sensor. Therefore, we use iPPG to visualize the pulse wave during muscle tense from the skin video taken with the RGB video camera. As a result of this experiment, we found that it was possible to recognize the difference in pulse wave during muscle tense from the video that visualized the pulse wave. From this result, the realization of non-contact measurement of muscle tense can be expected.

RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HRVIPL-HR is available at: ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.

Remote Monitoring of Vital Signs in Diverse Non-Clinical and Clinical Scenarios Using Computer Vision Systems: A Review

Techniques for noncontact measurement of vital signs using camera imaging technologies have been attracting increasing attention. For noncontact physiological assessments, computer vision-based methods appear to be an advantageous approach that could be robust, hygienic, reliable, safe, cost effective and suitable for long distance and long-term monitoring. In addition, video techniques allow measurements from multiple individuals opportunistically and simultaneously in groups. This paper aims to explore the progress of the technology from controlled clinical scenarios with fixed monitoring installations and controlled lighting, towards uncontrolled environments, crowds and moving sensor platforms. We focus on the diversity of applications and scenarios being studied in this topic. From this review it emerges that automatic multiple regions of interest (ROIs) selection, removal of noise artefacts caused by both illumination variations and motion artefacts, simultaneous multiple person monitoring, long distance detection, multi-camera fusion and accepted publicly available datasets are topics that still require research to enable the technology to mature into many real-world applications.

3D Convolutional Neural Networks for Remote Pulse Rate Measurement and Mapping from Facial Video

Remote pulse rate measurement from facial video has gained particular attention over the last few years. Research exhibits significant advancements and demonstrates that common video cameras correspond to reliable devices that can be employed to measure a large set of biomedical parameters without any contact with the subject. A new framework for measuring and mapping pulse rate from video is presented in this pilot study. The method, which relies on convolutional 3D networks, is fully automatic and does not require any special image preprocessing. In addition, the network ensures concurrent mapping by producing a prediction for each local group of pixels. A particular training procedure that employs only synthetic data is proposed. Preliminary results demonstrate that this convolutional 3D network can effectively extract pulse rate from video without the need for any processing of frames. The trained model was compared with other state-of-the-art methods on public data. Results exhibit significant agreement between estimated and ground-truth measurements: the root mean square error computed from pulse rate values assessed with the convolutional 3D network is equal to 8.64 bpm, which is superior to 10 bpm for the other state-of-the-art methods. The robustness of the method to natural motion and increases in performance correspond to the two main avenues that will be considered in future works.

Closed-loop solution method of active vision reconstruction via a 3D reference and an external camera

Laser-plane-based vision reconstruction is an important approach for 3D profile measurement and optical inspection. This study develops a 3D reconstruction method to measure the object with a flexible method. The active vision reconstruction includes an external camera and a 3D reference with an internal camera and laser projector. The 3D calibration object, the external camera, and the 3D reference constitute a closed loop. The internal camera, the 3D calibration object, and the 3D reference compose the other closed loop. The intrinsic invariants of the 3D reference with an internal camera and laser projector are determined by the two loops above. The benefit of the method is to cancel the position constraint for the active vision system with the cubic LED reference and the monocular external camera. Furthermore, as the external camera captures only the images of the 3D reference, the reconstruction system of the projector and the camera is flexible to observe the occlusion location on the measured object. The methods, including the calibration model and reconstruction model of the measurement system, are verified by experiments to demonstrate the potential applications.

Wearables Meet IoT: Synergistic Personal Area Networks (SPANs)

Wearable monitoring and mobile health (mHealth) revolutionized healthcare diagnostics and delivery, while the exponential increase of deployed "things" in the Internet of things (IoT) transforms our homes and industries. "Things" with embedded activity and vital sign sensors that we refer to as "smart stuff" can interact with wearable and ambient sensors. A dynamic, ad-hoc personal area network can span multiple domains and facilitate processing in synergistic personal area networks-SPANs. The synergy of information from multiple sensors can provide: (a) New information that cannot be generated from existing data alone, (b) user identification, (c) more robust assessment of physiological signals, and (d) automatic annotation of events/records. In this paper, we present possible new applications of SPANs and results of feasibility studies. Preliminary tests indicate that users interact with smart stuff-in our case, a smart water bottle-dozens of times a day and sufficiently long to collect vital signs of the users. Synergistic processing of sensors from the smartwatch and objects of everyday use may provide user identification and assessment of new parameters that individual sensors could not generate, such as pulse wave velocity (PWV) and blood pressure. As a result, SPANs facilitate seamless monitoring and annotation of vital signs dozens of times per day, every day, every time the smart object is used, without additional setup of sensors and initiation of measurements. SPANs creates a dynamic "opportunistic bubble" for ad-hoc integration with other sensors of interest around the user, wherever they go. Continuous long-term monitoring of user's activity and vital signs can provide better diagnostic procedures and personalized feedback to motivate a proactive approach to health and wellbeing.

Robust heart-rate estimation from facial videos using Project_ICA

OBJECTIVE

Remote photoplethysmography (rPPG) can achieve non-contact measurement of heart rate (HR) from a continuous video sequence by scanning the skin surface. However, practical applications are still limited by factors such as non-rigid facial motion and head movement. In this work, a detailed system framework for remotely estimating heart rate from facial videos under various movement conditions is described.

APPROACH

After the rPPG signal has been obtained from a defined region of the facial skin, a method, termed 'Project_ICA', based on a skin reflection model, is employed to extract the pulse signal from the original signal.

MAIN RESULTS

To evaluate the performance of the proposed algorithm, a dataset containing 112 videos including the challenges of various skin tones, body motion and HR recovery after exercise was created from 28 participants.

SIGNIFICANCE

The results show that Project_ICA, when evaluated by several criteria, provides a more accurate and robust estimate of HR than most existing methods, although problems remain in obtaining reliable measurements from dark-skinned subjects.

Availability and performance of image-based, non-contact methods of monitoring heart rate, blood pressure, respiratory rate, and oxygen saturation: a systematic review

OBJECTIVE

Over the last 15 years, developments in camera technology have coincided with increased availability and affordability. This has led to an increasing interest in using these technologies in healthcare settings. Image-based monitoring methods potentially allow multiple vital signs to be measured concurrently using a non-contact sensor. We have undertaken a systematic review of the current availability and performance of these monitoring methods.

APPROACH

A multiple database search was conducted using MEDLINE, Embase, CINAHL, Cochrane Library, OpenGrey, IEEE Xplore Library and ACM Digital Library to July 2018. We included studies comparing image-based heart rate, respiratory rate, oxygen saturation and blood pressure monitoring methods against one or more validated reference device(s). Each included study was assessed using the modified GRRAS criteria for reporting bias.

MAIN RESULTS

Of 30 279 identified studies, 161 were included in the final analysis. Twenty studies (20/161, 12%) were carried out on patients in clinical settings, while the remainder were conducted in academic settings using healthy volunteer populations. The 18-40 age group was best represented across the identified studies. One hundred and twenty studies (120/161, 75%) estimated heart rate, followed by 62 studies (62/161, 39%) estimating respiratory rate. Fewer studies focused on oxygen saturation (11/161, 7%) or blood pressure (6/161, 4%) estimation. Fifty-one heart rate studies (51/120, 43%) and 24 respiratory rate studies (24/62, 39%) used Bland-Altman analysis to report their results. Of the heart rate studies, 28 studies (28/51, 55%) showed agreement within industry standards of [Formula: see text]5 beats per minute. Only two studies achieved this within clinical settings. Of the respiratory rate studies, 13 studies (13/24, 54%) showed agreement within industry standards of [Formula: see text]3 breaths per minute, but only one study achieved this in a clinical setting. Statistical analysis was heterogeneous across studies with frequent inappropriate use of correlation. The majority of studies (99/161, 61%) monitored subjects for under 5 min. Three studies (3/161, 2%) monitored subjects for over 60 min, all of which were conducted in hospital settings.

SIGNIFICANCE

Heart rate and respiratory rate monitoring using video images is currently possible and performs within clinically acceptable limits under experimental conditions. Camera-derived estimates were less accurate in the proportion of studies conducted in clinical settings. We would encourage thorough reporting of the population studied, details of clinically relevant aspects of methodology, and the use of appropriate statistical methods in future studies. Systematic review registration: PROSPERO CRD42016029167 Protocol: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-017-0615-3.

iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox

Imaging-based, non-contact measurement of physiology (including imaging photoplethysmography and imaging ballistocardiography) is a growing field of research. There are several strengths of imaging methods that make them attractive. They remove the need for uncomfortable contact sensors and can enable spatial and concomitant measurement from a single sensor. Furthermore, cameras are ubiquitous and often low-cost solutions for sensing. Open source toolboxes help accelerate the progress of research by providing a means to compare new approaches against standard implementations of the state-of-the-art. We present an open source imaging-based physiological measurement toolbox with implementations of many of the most frequently employed computational methods. We hope that this toolbox will contribute to the advancement of noncontact physiological sensing methods. Code: https://github.com/danmcduff/iphys-toolbox.

Game-Calibrated and User-Tailored Remote Detection of Stress and Boredom in Games

Emotion detection based on computer vision and remote extraction of user signals commonly rely on stimuli where users have a passive role with limited possibilities for interaction or emotional involvement, e.g., images and videos. Predictive models are also trained on a group level, which potentially excludes or dilutes key individualities of users. We present a non-obtrusive, multifactorial, user-tailored emotion detection method based on remotely estimated psychophysiological signals. A neural network learns the emotional profile of a user during the interaction with calibration games, a novel game-based emotion elicitation material designed to induce emotions while accounting for particularities of individuals. We evaluate our method in two experiments ( n = 20 and n = 62 ) with mean classification accuracy of 61.6%, which is statistically significantly better than chance-level classification. Our approach and its evaluation present unique circumstances: our model is trained on one dataset (calibration games) and tested on another (evaluation game), while preserving the natural behavior of subjects and using remote acquisition of signals. Results of this study suggest our method is feasible and an initiative to move away from questionnaires and physical sensors into a non-obtrusive, remote-based solution for detecting emotions in a context involving more naturalistic user behavior and games.

Influence of acquisition frame-rate and video compression techniques on pulse-rate variability estimation from vPPG signal

In this paper, common time- and frequency-domain variability indexes obtained by pulse rate variability (PRV) series extracted from video-photoplethysmographic signal (vPPG) were compared with heart rate variability (HRV) parameters calculated from synchronized ECG signals. The dual focus of this study was to analyze the effect of different video acquisition frame-rates starting from 60 frames-per-second (fps) down to 7.5 fps and different video compression techniques using both lossless and lossy codecs on PRV parameters estimation. Video recordings were acquired through an off-the-shelf GigE Sony XCG-C30C camera on 60 young, healthy subjects (age 23±4 years) in the supine position. A fully automated, signal extraction method based on the Kanade-Lucas-Tomasi (KLT) algorithm for regions of interest (ROI) detection and tracking, in combination with a zero-phase principal component analysis (ZCA) signal separation technique was employed to convert the video frames sequence to a pulsatile signal. The frame-rate degradation was simulated on video recordings by directly sub-sampling the ROI tracking and signal extraction modules, to correctly mimic videos recorded at a lower speed. The compression of the videos was configured to avoid any frame rejection caused by codec quality leveling, FFV1 codec was used for lossless compression and H.264 with variable quality parameter as lossy codec. The results showed that a reduced frame-rate leads to inaccurate tracking of ROIs, increased time-jitter in the signals dynamics and local peak displacements, which degrades the performances in all the PRV parameters. The root mean square of successive differences (RMSSD) and the proportion of successive differences greater than 50 ms (PNN50) indexes in time-domain and the low frequency (LF) and high frequency (HF) power in frequency domain were the parameters which highly degraded with frame-rate reduction. Such a degradation can be partially mitigated by up-sampling the measured signal at a higher frequency (namely 60 Hz). Concerning the video compression, the results showed that compression techniques are suitable for the storage of vPPG recordings, although lossless or intra-frame compression are to be preferred over inter-frame compression methods. FFV1 performances are very close to the uncompressed (UNC) version with less than 45% disk size. H.264 showed a degradation of the PRV estimation directly correlated with the increase of the compression ratio.

Cardiovascular assessment by imaging photoplethysmography - a review

Over the last few years, the contactless acquisition of cardiovascular parameters using cameras has gained immense attention. The technique provides an optical means to acquire cardiovascular information in a very convenient way. This review provides an overview on the technique's background and current realizations. Besides giving detailed information on the most widespread application of the technique, namely the contactless acquisition of heart rate, we outline further concepts and we critically discuss the current state.

Using Wearable ECG/PPG Sensors for Driver Drowsiness Detection Based on Distinguishable Pattern of Recurrence Plots

This paper aims to investigate the robust and distinguishable pattern of heart rate variability (HRV) signals, acquired from wearable electrocardiogram (ECG) or photoplethysmogram (PPG) sensors, for driver drowsiness detection. As wearable sensors are so vulnerable to slight movement, they often produce more noise in signals. Thus, from noisy HRV signals, we need to find good traits that differentiate well between drowsy and awake states. To this end, we explored three types of recurrence plots (RPs) generated from the R–R intervals (RRIs) of heartbeats: Bin-RP, Cont-RP, and ReLU-RP. Here Bin-RP is a binary recurrence plot, Cont-RP is a continuous recurrence plot, and ReLU-RP is a thresholded recurrence plot obtained by filtering Cont-RP with a modified rectified linear unit (ReLU) function. By utilizing each of these RPs as input features to a convolutional neural network (CNN), we examined their usefulness for drowsy/awake classification. For experiments, we collected RRIs at drowsy and awake conditions with an ECG sensor of the Polar H7 strap and a PPG sensor of the Microsoft (MS) band 2 in a virtual driving environment. The results showed that ReLU-RP is the most distinct and reliable pattern for drowsiness detection, regardless of sensor types (i.e., ECG or PPG). In particular, the ReLU-RP based CNN models showed their superiority to other conventional models, providing approximately 6–17% better accuracy for ECG and 4–14% for PPG in drowsy/awake classification.