HRVCam: robust camera-based measurement of heart rate variability

Deep learning and remote photoplethysmography powered advancements in contactless physiological measurement

In recent decades, there has been ongoing development in the application of computer vision (CV) in the medical field. As conventional contact-based physiological measurement techniques often restrict a patient's mobility in the clinical environment, the ability to achieve continuous, comfortable and convenient monitoring is thus a topic of interest to researchers. One type of CV application is remote imaging photoplethysmography (rPPG), which can predict vital signs using a video or image. While contactless physiological measurement techniques have an excellent application prospect, the lack of uniformity or standardization of contactless vital monitoring methods limits their application in remote healthcare/telehealth settings. Several methods have been developed to improve this limitation and solve the heterogeneity of video signals caused by movement, lighting, and equipment. The fundamental algorithms include traditional algorithms with optimization and developing deep learning (DL) algorithms. This article aims to provide an in-depth review of current Artificial Intelligence (AI) methods using CV and DL in contactless physiological measurement and a comprehensive summary of the latest development of contactless measurement techniques for skin perfusion, respiratory rate, blood oxygen saturation, heart rate, heart rate variability, and blood pressure.

Light and Displacement Compensation-Based iPPG for Heart-Rate Measurement in Complex Detection Conditions

A light and displacement-compensation-based iPPG algorithm is proposed in this paper for heart-rate measurement in complex detection conditions. Two compensation sub-algorithms, including light compensation and displacement compensation, are designed and integrated into the iPPG algorithm for more accurate heart-rate measurement. In the light-compensation sub-algorithm, the measurement deviation caused by the ambient light change is compensated by the mean filter-based light adjustment strategy. In the displacement-compensation sub-algorithm, the measurement deviation caused by the subject motion is compensated by the optical flow-based displacement calculation strategy. A series of heart-rate measurement experiments are conducted to verify the effectiveness of the proposed method. Compared with conventional iPPG, the average measurement accuracy increases by 3.8% under different detection distances and 5.0% under different light intensities.

Video-based sympathetic arousal assessment via peripheral blood flow estimation

Electrodermal activity (EDA) is considered a standard marker of sympathetic activity. However, traditional EDA measurement requires electrodes in steady contact with the skin. Can sympathetic arousal be measured using only an optical sensor, such as an RGB camera? This paper presents a novel approach to infer sympathetic arousal by measuring the peripheral blood flow on the face or hand optically. We contribute a self-recorded dataset of 21 participants, comprising synchronized videos of participants' faces and palms and gold-standard EDA and photoplethysmography (PPG) signals. Our results show that we can measure peripheral sympathetic responses that closely correlate with the ground truth EDA. We obtain median correlations of 0.57 to 0.63 between our inferred signals and the ground truth EDA using only videos of the participants' palms or foreheads or PPG signals from the foreheads or fingers. We also show that sympathetic arousal is best inferred from the forehead, finger, or palm.

Robust Heart Rate Variability Measurement from Facial Videos

Remote Photoplethysmography (rPPG) is a contactless method that enables the detection of various physiological signals from facial videos. rPPG utilizes a digital camera to detect subtle changes in skin color to measure vital signs such as heart rate variability (HRV), an important biomarker related to the autonomous nervous system. This paper presents a novel contactless HRV extraction algorithm, WaveHRV, based on the Wavelet Scattering Transform technique, followed by adaptive bandpass filtering and inter-beat-interval (IBI) analysis. Furthermore, a novel method is introduced to preprocess noisy contact-based PPG signals. WaveHRV is bench-marked against existing algorithms and public datasets. Our results show that WaveHRV is promising and achieves the lowest mean absolute error (MAE) of 10.5 ms and 6.15 ms for RMSSD and SDNN on the UBFCrPPG dataset.

Are infrared thermography, feeding behavior, and heart rate variability measures capable of characterizing group-housed sow social hierarchies?

Group gestation housing is quickly becoming standard practice in commercial swine production. However, poor performance and welfare in group housed sows may result from the formation and maintenance of the social hierarchy within the pen. In the future, the ability to quickly characterize the social hierarchy via precision technologies could be beneficial to producers for identifying animals at risk of poor welfare outcomes. Therefore, the objective of this study was to investigate the use of infrared thermography (IRT), automated electronic sow feeding systems, and heart rate monitors as potential technologies for detecting the social hierarchy within five groups of sows. Behavioral data collection occurred for 12 h after introducing five sow groups (1-5; n = 14, 12, 15, 15, and 17, respectively) to group gestation housing to determine the social hierarchy and allocate individual sows to 1 of 4 rank quartiles (RQ 1-4). Sows within RQ1 were ranked highest while RQ4 sows were ranked lowest within the hierarchy. Infrared thermal images were taken behind the neck at the base of the ear of each sow on days 3, 15, 30, 45, 60, 75, 90, and 105 of the experiment. Two electronic sow feeders tracked feeding behavior throughout the gestation period. Heart rate monitors were worn by 10 randomly selected sows per repetition for 1 h prior to and 4 h after reintroduction to group gestation housing to collect heart rate variability (HRV). No differences were found between RQ for any IRT characteristic. Sows within RQ3 and RQ4 had the greatest number of visits to the electronic sow feeders overall (P < 0.04) but spent shorter time per visit in feeders (P < 0.05) than RQ1 and RQ2 sows. There was an interaction of RQ with hour for feed offered (P = 0.0003), with differences between RQ occurring in hour 0, 1, 2, and 8. Higher-ranked sows (RQ1 and RQ2) occupied the feeder for longer during the first hour than lower ranking sows (RQ3 and RQ4; P < 0.04), while RQ3 sows occupied the feeder longer than RQ1 sows during hour 6, 7, and 8 (P < 0.02). Heart beat interval (RR) collected prior to group housing introduction differed between RQ (P < 0.02 for all), with RQ3 sows exhibiting the lowest RR, followed by RQ4, RQ1, and RQ2. Rank quartile also affected standard deviation of RR (P = 0.0043), with RQ4 sows having the lowest, followed by RQ1, RQ3, and RQ2 sows. Overall, these results indicate that feeding behavior and HRV measures may be capable of characterizing social hierarchy in a group housing system.

RobustPPG: camera-based robust heart rate estimation using motion cancellation

Camera-based heart rate measurement is becoming an attractive option as a non-contact modality for continuous remote health and engagement monitoring. However, reliable heart rate extraction from camera-based measurement is challenging in realistic scenarios, especially when the subject is moving. In this work, we develop a motion-robust algorithm, labeled RobustPPG, for extracting photoplethysmography signals (PPG) from face video and estimating the heart rate. Our key innovation is to explicitly model and generate motion distortions due to the movements of the person's face. We use inverse rendering to obtain the 3D shape and albedo of the face and environment lighting from video frames and then render the human face for each frame. The rendered face is similar to the original face but does not contain the heart rate signal; facial movements alone cause pixel intensity variation in the generated video frames. Finally, we use the generated motion distortion to filter the motion-induced measurements. We demonstrate that our approach performs better than the state-of-the-art methods in extracting a clean blood volume signal with over 2 dB signal quality improvement and 30% improvement in RMSE of estimated heart rate in intense motion scenarios.

Remote Photoplethysmography Is an Accurate Method to Remotely Measure Respiratory Rate: A Hospital-Based Trial

Remote photoplethysmography imaging (rPPG) is a new solution proposed to measure vital signs, such as respiratory rate (RR) in teleconsultation, by using a webcam. The results, presented here, aim at evaluating the accuracy of such remote measurement methods, compared with existing measurement methods, in a real-life clinical setting. For each patient, measurement of RR, using the standard system (control), has been carried out concomitantly with the experimental system. A 60-s time frame was used for the measurements made by our rPPG system. Age, gender, BMI, and skin phototype were collected. We performed the intraclass correlation coefficient and Bland-Altman plot to analyze the accuracy and precision of the rPPG algorithm readings. Measurements of RR, using the two methods, have been realized on 963 patients. Comparison of the two techniques showed excellent agreement (96.0%), with most of the patients (n = 924-standard patients) being in the confidence interval of 95% in Bland-Altman plotting. There were no significant differences between standard patients and outlier patients for demographic and clinical characteristics. This study indicates a good agreement between the rPPG system and the control, thus allowing clinical use of this remote assessment of the respiratory rate.

Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review

In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO2). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO2, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO2 are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO2 is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO2 monitoring.

Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature

Estimating heart rate is important for monitoring users in various situations. Estimates based on facial videos are increasingly being researched because they allow the monitoring of cardiac information in a non-invasive way and because the devices are simpler, as they require only cameras that capture the user’s face. From these videos of the user’s face, machine learning can estimate heart rate. This study investigates the benefits and challenges of using machine learning models to estimate heart rate from facial videos through patents, datasets, and article review. We have searched the Derwent Innovation, IEEE Xplore, Scopus, and Web of Science knowledge bases and identified seven patent filings, eleven datasets, and twenty articles on heart rate, photoplethysmography, or electrocardiogram data. In terms of patents, we note the advantages of inventions related to heart rate estimation, as described by the authors. In terms of datasets, we have discovered that most of them are for academic purposes and with different signs and annotations that allow coverage for subjects other than heartbeat estimation. In terms of articles, we have discovered techniques, such as extracting regions of interest for heart rate reading and using video magnification for small motion extraction, and models, such as EVM-CNN and VGG-16, that extract the observed individual’s heart rate, the best regions of interest for signal extraction, and ways to process them.

Real-time realizable mobile imaging photoplethysmography

Photoplethysmography imaging (PPGI) sensors have attracted a significant amount of attention as they enable the remote monitoring of heart rates (HRs) and thus do not require any additional devices to be worn on fingers or wrists. In this study, we mounted PPGI sensors on a robot for active and autonomous HR (R-AAH) estimation. We proposed an algorithm that provides accurate HR estimation, which can be performed in real time using vision and robot manipulation algorithms. By simplifying the extraction of facial skin images using saturation (S) values in the HSV color space, and selecting pixels based on the most frequent S value within the face image, we achieved a reliable HR assessment. The results of the proposed algorithm using the R-AAH method were evaluated by rigorous comparison with the results of existing algorithms on the UBFC-RPPG dataset (n = 42). The proposed algorithm yielded an average absolute error (AAE) of 0.71 beats per minute (bpm). The developed algorithm is simple, with a processing time of less than 1 s (275 ms for an 8-s window). The algorithm was further validated on our own dataset (BAMI-RPPG dataset [n = 14]) with an AAE of 0.82 bpm.

Forecasting Seizure Likelihood With Wearable Technology

The unpredictability of epileptic seizures exposes people with epilepsy to potential physical harm, restricts day-to-day activities, and impacts mental well-being. Accurate seizure forecasters would reduce the uncertainty associated with seizures but need to be feasible and accessible in the long-term. Wearable devices are perfect candidates to develop non-invasive, accessible forecasts but are yet to be investigated in long-term studies. We hypothesized that machine learning models could utilize heart rate as a biomarker for well-established cycles of seizures and epileptic activity, in addition to other wearable signals, to forecast high and low risk seizure periods. This feasibility study tracked participants' (n = 11) heart rates, sleep, and step counts using wearable smartwatches and seizure occurrence using smartphone seizure diaries for at least 6 months (mean = 14.6 months, SD = 3.8 months). Eligible participants had a diagnosis of refractory epilepsy and reported at least 20 seizures (mean = 135, SD = 123) during the recording period. An ensembled machine learning and neural network model estimated seizure risk either daily or hourly, with retraining occurring on a weekly basis as additional data was collected. Performance was evaluated retrospectively against a rate-matched random forecast using the area under the receiver operating curve. A pseudo-prospective evaluation was also conducted on a held-out dataset. Of the 11 participants, seizures were predicted above chance in all (100%) participants using an hourly forecast and in ten (91%) participants using a daily forecast. The average time spent in high risk (prediction time) before a seizure occurred was 37 min in the hourly forecast and 3 days in the daily forecast. Cyclic features added the most predictive value to the forecasts, particularly circadian and multiday heart rate cycles. Wearable devices can be used to produce patient-specific seizure forecasts, particularly when biomarkers of seizure and epileptic activity cycles are utilized.

Smartphone-based imaging systems for medical applications: a critical review

SIGNIFICANCE

Smartphones come with an enormous array of functionality and are being more widely utilized with specialized attachments in a range of healthcare applications. A review of key developments and uses, with an assessment of strengths/limitations in various clinical workflows, was completed.

AIM

Our review studies how smartphone-based imaging (SBI) systems are designed and tested for specialized applications in medicine and healthcare. An evaluation of current research studies is used to provide guidelines for improving the impact of these research advances.

APPROACH

First, the established and emerging smartphone capabilities that can be leveraged for biomedical imaging are detailed. Then, methods and materials for fabrication of optical, mechanical, and electrical interface components are summarized. Recent systems were categorized into four groups based on their intended application and clinical workflow: ex vivo diagnostic, in vivo diagnostic, monitoring, and treatment guidance. Lastly, strengths and limitations of current SBI systems within these various applications are discussed.

RESULTS

The native smartphone capabilities for biomedical imaging applications include cameras, touchscreens, networking, computation, 3D sensing, audio, and motion, in addition to commercial wearable peripheral devices. Through user-centered design of custom hardware and software interfaces, these capabilities have the potential to enable portable, easy-to-use, point-of-care biomedical imaging systems. However, due to barriers in programming of custom software and on-board image analysis pipelines, many research prototypes fail to achieve a prospective clinical evaluation as intended. Effective clinical use cases appear to be those in which handheld, noninvasive image guidance is needed and accommodated by the clinical workflow. Handheld systems for in vivo, multispectral, and quantitative fluorescence imaging are a promising development for diagnostic and treatment guidance applications.

CONCLUSIONS

A holistic assessment of SBI systems must include interpretation of their value for intended clinical settings and how their implementations enable better workflow. A set of six guidelines are proposed to evaluate appropriateness of smartphone utilization in terms of clinical context, completeness, compactness, connectivity, cost, and claims. Ongoing work should prioritize realistic clinical assessments with quantitative and qualitative comparison to non-smartphone systems to clearly demonstrate the value of smartphone-based systems. Improved hardware design to accommodate the rapidly changing smartphone ecosystem, creation of open-source image acquisition and analysis pipelines, and adoption of robust calibration techniques to address phone-to-phone variability are three high priority areas to move SBI research forward.