Multispectral Video Fusion for Non-Contact Monitoring of Respiratory Rate and Apnea

Integrating RGB-thermal image sensors for non-contact automatic respiration rate monitoring

Respiration rate (RR) holds significance as a human health indicator. Presently, the conventional RR monitoring system requires direct physical contact, which may cause discomfort and pain. Therefore, this paper proposes a non-contact RR monitoring system integrating RGB and thermal imaging through RGB-thermal image alignment. The proposed method employs an advanced image processing algorithm for automatic region of interest (ROI) selection. The experimental results demonstrated a close correlation and a lower error rate between measured thermal, measured RGB, and reference data. In summary, the proposed non-contact system emerges as a promising alternative to conventional contact-based approaches without the associated discomfort and pain.

Challenges and prospects of visual contactless physiological monitoring in clinical study

The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.

Contactless Monitoring of Respiratory Rate and Breathing Absence from Head Movements Using an RGB Camera

Contactless vital sign monitoring is more demanding for long-term, continuous, and unobtrusive measurements. Camera-based respiratory monitoring is receiving growing interest with advanced video technologies and computational power. The volume variations of the lungs for airflow changes create a periodic movement of the torso, but identifying the torso is more challenging than face detection in a video. In this paper, we present a unique approach to monitoring respiratory rate (RR) and breathing absence by leveraging head movements alone from an RGB video because respiratory motion also influences the head. Besides our novel RR estimation, an independent algorithm for breathing absence detection using signal feature extraction and machine learning techniques identifies an apnea event and improves overall RR estimation accuracy. The proposed approach was evaluated using videos from 30 healthy subjects who performed various breathing tasks. The breathing absence detector had 0.87 F1 score, 0.9 sensitivity, and 0.85 specificity. The accuracy of spontaneous breathing rate estimation increased from 2.46 to 1.91 bpm MAE and 3.54 to 2.7 bpm RMSE when combining the breathing absence result with the estimated RR.Clinical relevance- Our contactless respiratory monitoring can utilize a consumer RGB camera to offer a significant benefit in continuous monitoring of neonatal monitoring, sleep monitoring, telemedicine or telehealth, home fitness with mild physical movement, and emotion detection in the clinic and remote locations.

Geriatric Care Management System Powered by the IoT and Computer Vision Techniques

The digitalisation of geriatric care refers to the use of emerging technologies to manage and provide person-centered care to the elderly by collecting patients' data electronically and using them to streamline the care process, which improves the overall quality, accuracy, and efficiency of healthcare. In many countries, healthcare providers still rely on the manual measurement of bioparameters, inconsistent monitoring, and paper-based care plans to manage and deliver care to elderly patients. This can lead to a number of problems, including incomplete and inaccurate record-keeping, errors, and delays in identifying and resolving health problems. The purpose of this study is to develop a geriatric care management system that combines signals from various wearable sensors, noncontact measurement devices, and image recognition techniques to monitor and detect changes in the health status of a person. The system relies on deep learning algorithms and the Internet of Things (IoT) to identify the patient and their six most pertinent poses. In addition, the algorithm has been developed to monitor changes in the patient's position over a longer period of time, which could be important for detecting health problems in a timely manner and taking appropriate measures. Finally, based on expert knowledge and a priori rules integrated in a decision tree-based model, the automated final decision on the status of nursing care plan is generated to support nursing staff.

Multi-Person Breathing Detection With Switching Antenna Array Based on WiFi Signal

WiFi sensing, an emerging sensing technology, has been widely used in vital sign monitoring. However, most respiration monitoring studies have focused on single-person tasks. In this paper, we propose a multi-person breathing sensing system based on WiFi signals. Specifically, we use radio frequency (RF) switch to extend the antennas to form switching antenna array. A reference channel is introduced in the receiver, which is connected to the transmitter by cable and attenuator. The phase offset introduced by asynchronous transceiver devices can be eliminated by using the ratio of the channel frequency response (CFR) between the antenna array and the reference channel. In order to realize multi-person breathing perception, we use beamforming technology to conduct two-dimensional scanning of the whole scene. After eliminating static clutter, we combine frequency domain and angle of arrival (AOA) domain analysis to construct the AOA and frequency (AOA-FREQ) spectrogram. Finally, the respiratory frequency and position of each target are obtained by clustering. Experimental results show that the proposed system can not only estimate the direction and respiration rate of multi-person, but also monitor abnormal respiration in multi-person scenarios. The proposed low-cost, non-contact, rapid multi-person respiratory detection technology can meet the requirements of long-term home health monitoring.

Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review

In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO₂). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO₂, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO₂ are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO₂ is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO₂ monitoring.

Multi-ROI Spectral Approach for the Continuous Remote Cardio-Respiratory Monitoring from Mobile Device Built-In Cameras

Heart rate (HR) and respiratory rate (f_R) can be estimated by processing videos framing the upper body and face regions without any physical contact with the subject. This paper proposed a technique for continuously monitoring HR and f_R via a multi-ROI approach based on the spectral analysis of RGB video frames recorded with a mobile device (i.e., a smartphone's camera). The respiratory signal was estimated by the motion of the chest, whereas the cardiac signal was retrieved from the pulsatile activity at the level of right and left cheeks and forehead. Videos were recorded from 18 healthy volunteers in four sessions with different user-camera distances (i.e., 0.5 m and 1.0 m) and illumination conditions (i.e., natural and artificial light). For HR estimation, three approaches were investigated based on single or multi-ROI approaches. A commercially available multiparametric device was used to record reference respiratory signals and electrocardiogram (ECG). The results demonstrated that the multi-ROI approach outperforms the single-ROI approach providing temporal trends of both the vital parameters comparable to those provided by the reference, with a mean absolute error (MAE) consistently below 1 breaths·min^-1 for f_R in all the scenarios, and a MAE between 0.7 bpm and 6 bpm for HR estimation, whose values increase at higher distances.

Vital Signs Identification System With Doppler Radars and Thermal Camera

During the global epidemic, non-contact methods for monitoring the vital signs of several people have become particularly important. Advanced signal processing techniques have recently been demonstrated to separate and track the vital signs of multiple people. In this paper, we further develop the multi-person vital signs identification (VSign-ID) system to make non-contact detection available in public places. VSign-ID not only extracts multi-person vital signs but also states from whom these vital signs are collected. We utilize multiple doppler radars to expand the effective range of the measurement area and propose a space and time matching mechanism for vital signs identification. We use a thermal camera to detect the number of people and their movements. VSign-ID efficiently coordinates these two types of sensors (i.e., the doppler radars and the thermal camera) to track and identify the respiration rates and heartbeat rates of multiple people. A series of experiments and simulations are conducted to measure the efficiency of VSign-ID. In the case of five people sitting closely, the estimation errors for respiration and heartbeat rates are -4.85 dB and -2.36 dB lower than the standard resolution of the system, respectively, despite using only two independent radars.

Contactless Measurement of Vital Signs Using Thermal and RGB Cameras: A Study of COVID 19-Related Health Monitoring

In this study, a contactless vital signs monitoring system was proposed, which can measure body temperature (BT), heart rate (HR) and respiration rate (RR) for people with and without face masks using a thermal and an RGB camera. The convolution neural network (CNN) based face detector was applied and three regions of interest (ROIs) were located based on facial landmarks for vital sign estimation. Ten healthy subjects from a variety of ethnic backgrounds with skin colors from pale white to darker brown participated in several different experiments. The absolute error (AE) between the estimated HR using the proposed method and the reference HR from all experiments is 2.70±2.28 beats/min (mean ± std), and the AE between the estimated RR and the reference RR from all experiments is 1.47±1.33 breaths/min (mean ± std) at a distance of 0.6-1.2 m.

Implementation of Thermal Camera for Non-Contact Physiological Measurement: A Systematic Review

Non-contact physiological measurements based on image sensors have developed rapidly in recent years. Among them, thermal cameras have the advantage of measuring temperature in the environment without light and have potential to develop physiological measurement applications. Various studies have used thermal camera to measure the physiological signals such as respiratory rate, heart rate, and body temperature. In this paper, we provided a general overview of the existing studies by examining the physiological signals of measurement, the used platforms, the thermal camera models and specifications, the use of camera fusion, the image and signal processing step (including the algorithms and tools used), and the performance evaluation. The advantages and challenges of thermal camera-based physiological measurement were also discussed. Several suggestions and prospects such as healthcare applications, machine learning, multi-parameter, and image fusion, have been proposed to improve the physiological measurement of thermal camera in the future.

Towards Accurate, Cost-Effective, Ultra-Low-Power and Non-Invasive Respiration Monitoring: A Reusable Wireless Wearable Sensor for an Off-the-Shelf KN95 Mask

Respiratory rate is a critical vital sign that indicates health condition, sleep quality, and exercise intensity. This paper presents a non-invasive, ultra-low-power, and cost-effective wireless wearable sensor, which is installed on an off-the-shelf KN95 mask to facilitate respiration monitoring. The sensing principle is based on the periodic airflow temperature variations caused by exhaled hot air and inhaled cool air in respiratory cycles. By measuring the periodic temperature variations at the exhalation valve of mask, the respiratory parameters can be accurately and reliably detected, regardless of body movements and breathing pathways through nose or mouth. Specifically, we propose a voltage divider with controllable resistors and corresponding selection criteria to improve the sensitivity of temperature measurement, a peak detection algorithm with spline interpolation to increase sampling period without reducing the detection accuracy, and effective low-power optimization measures to prolong the battery life. The experimental results have demonstrated the effectiveness of the proposed sensor, showing a small mean absolute error (MAE) of 0.449 bpm and a very low power consumption of 131.4 μW. As a high accuracy, low cost, low power, and reusable miniature wearing device for convenient respiration monitoring in daily life, the proposed sensor holds promise in real-world feasibility.

Automatic Separation of Respiratory Flow from Motion in Thermal Videos for Infant Apnea Detection

Both Respiratory Flow (RF) and Respiratory Motion (RM) are visible in thermal recordings of infants. Monitoring these two signals usually requires landmark detection for the selection of a region of interest. Other approaches combine respiratory signals coming from both RF and RM, obtaining a Mixed Respiratory (MR) signal. The detection and classification of apneas, particularly common in preterm infants with low birth weight, would benefit from monitoring both RF and RM, or MR, signals. Therefore, we propose in this work an automatic RF pixel detector not based on facial/body landmarks. The method is based on the property of RF pixels in thermal videos, which are in areas with a smooth circular gradient. We defined 5 features combined with the use of a bank of Gabor filters that together allow selection of the RF pixels. The algorithm was tested on thermal recordings of 9 infants amounting to a total of 132 min acquired in a neonatal ward. On average the percentage of correctly identified RF pixels was 84%. Obstructive Apneas (OAs) were simulated as a proof of concept to prove the advantage in monitoring the RF signal compared to the MR signal. The sensitivity in the simulated OA detection improved for the RF signal reaching 73% against the 23% of the MR signal. Overall, the method yielded promising results, although the positioning and number of cameras used could be further optimized for optimal RF visibility.

Examination of Potential of Thermopile-Based Contactless Respiratory Gating

To control the spread of coronavirus disease 2019 (COVID-19), it is effective to perform a fast screening of the respiratory rate of the subject at the gate before entering a space to assess the potential risks. In this paper, we examine the potential of a novel yet cost-effective solution, called thermopile-based respiratory gating, to contactlessly screen a subject by measuring their respiratory rate in the scenario with an entrance gate. Based on a customized thermopile array system, we investigate different image and signal processing methods that measure respiratory rate from low-resolution thermal videos, where an automatic region-of-interest selection-based approach obtains a mean absolute error (MAE) of 0.8 breaths per minute. We show the feasibility of thermopile-based respiratory gating and quantify its limitations and boundary conditions in a benchmark (e.g., appearance of face mask, measurement distance and screening time). The technical validation provided by this study is helpful for designing and implementing a respiratory gating solution toward the prevention of the spread of COVID-19 during the pandemic.

Estimation of Motion and Respiratory Characteristics during the Meditation Practice Based on Video Analysis

Meditation practice is mental health training. It helps people to reduce stress and suppress negative thoughts. In this paper, we propose a camera-based meditation evaluation system, that helps meditators to improve their performance. We rely on two main criteria to measure the focus: the breathing characteristics (respiratory rate, breathing rhythmicity and stability), and the body movement. We introduce a contactless sensor to measure the respiratory rate based on a smartphone camera by detecting the chest keypoint at each frame, using an optical flow based algorithm to calculate the displacement between frames, filtering and de-noising the chest movement signal, and calculating the number of real peaks in this signal. We also present an approach to detecting the movement of different body parts (head, thorax, shoulders, elbows, wrists, stomach and knees). We have collected a non-annotated dataset for meditation practice videos consists of ninety videos and the annotated dataset consists of eight videos. The non-annotated dataset was categorized into beginner and professional meditators and was used for the development of the algorithm and for tuning the parameters. The annotated dataset was used for evaluation and showed that human activity during meditation practice could be correctly estimated by the presented approach and that the mean absolute error for the respiratory rate is around 1.75 BPM, which can be considered tolerable for the meditation application.

Towards Continuous Camera-Based Respiration Monitoring in Infants

Aiming at continuous unobtrusive respiration monitoring, motion robustness is paramount. However, some types of motion can completely hide the respiration information and the detection of these events is required to avoid incorrect rate estimations. Therefore, this work proposes a motion detector optimized to specifically detect severe motion of infants combined with a respiration rate detection strategy based on automatic pixels selection, which proved to be robust to motion of the infants involving head and limbs. A dataset including both thermal and RGB (Red Green Blue) videos was used amounting to a total of 43 h acquired on 17 infants. The method was successfully applied to both RGB and thermal videos and compared to the chest impedance signal. The Mean Absolute Error (MAE) in segments where some motion is present was 1.16 and 1.97 breaths/min higher than the MAE in the ideal moments where the infants were still for testing and validation set, respectively. Overall, the average MAE on the testing and validation set are 3.31 breaths/min and 5.36 breaths/min, using 64.00% and 69.65% of the included video segments (segments containing events such as interventions were excluded based on a manual annotation), respectively. Moreover, we highlight challenges that need to be overcome for continuous camera-based respiration monitoring. The method can be applied to different camera modalities, does not require skin visibility, and is robust to some motion of the infants.