A Vision-Based Respiration Monitoring System for Passive Airway Resistance Estimation

Camera-Based Short Physical Performance Battery and Timed Up and Go Assessment for Older Adults With Cancer

This paper presents an automatic camera-based device to monitor and evaluate the gait speed, standing balance, and 5 times sit-stand (5TSS) tests of the Short Physical Performance Battery (SPPB) and the Timed Up and Go (TUG) test. The proposed design measures and calculates the parameters of the SPPB tests automatically. The SPPB data can be used for physical performance assessment of older patients under cancer treatment. This stand-alone device has a Raspberry Pi (RPi) computer, three cameras, and two DC motors. The left and right cameras are used for gait speed tests. The center camera is used for standing balance, 5TSS, and TUG tests and for angle positioning of the camera platform toward the subject using DC motors by turning the camera left/right and tilting it up/down. The key algorithm for operating the proposed system is developed using Channel and Spatial Reliability Tracking in the cv2 module in Python. Graphical User Interfaces (GUIs) in the RPi are developed to run tests and adjust cameras, controlled remotely via smartphone and its Wi-Fi hotspot. We have tested the implemented camera setup prototype and extracted all SPPB and TUG parameters by conducting several experiments on a human subject population of 8 volunteers (male and female, light and dark complexions) in 69 test runs. The measured data and calculated outputs of the system consist of tests of gait speed (0.041 to 1.92 m/s with average accuracy of >95%), and standing balance, 5TSS, TUG, all with average time accuracy of >97%.

Pressure eye: In-bed contact pressure estimation via contact-less imaging

Computer vision has achieved great success in interpreting semantic meanings from images, yet estimating underlying (non-visual) physical properties of an object is often limited to their bulk values rather than reconstructing a dense map. In this work, we present our pressure eye (PEye) approach to estimate contact pressure between a human body and the surface she is lying on with high resolution from vision signals directly. PEye approach could ultimately enable the prediction and early detection of pressure ulcers in bed-bound patients, that currently depends on the use of expensive pressure mats. Our PEye network is configured in a dual encoding shared decoding form to fuse visual cues and some relevant physical parameters in order to reconstruct high resolution pressure maps (PMs). We also present a pixel-wise resampling approach based on Naive Bayes assumption to further enhance the PM regression performance. A percentage of correct sensing (PCS) tailored for sensing estimation accuracy evaluation is also proposed which provides another perspective for performance evaluation under varying error tolerances. We tested our approach via a series of extensive experiments using multimodal sensing technologies to collect data from 102 subjects while lying on a bed. The individual's high resolution contact pressure data could be estimated from their RGB or long wavelength infrared (LWIR) images with 91.8% and 91.2% estimation accuracies in PCS_efs0.1 criteria, superior to state-of-the-art methods in the related image regression/translation tasks.

The use of time-of-flight camera to assess respiratory rates and thoracoabdominal depths in patients with chronic respiratory disease

INTRODUCTION

Over the last 5 years, the analysis of respiratory patterns presents a growing usage in clinical and research purposes, but there is still currently a lack of easy-to-use and affordable devices to perform such kind of evaluation.

OBJECTIVES

The aim of this study is to validate a new specifically developed method, based on Kinect sensor, to assess respiratory patterns against spirometry under various conditions.

METHODS

One hundred and one participants took parts in one of the three validations studies. Twenty-five chronic respiratory disease patients (14 with chronic obstructive pulmonary disease (COPD) [65 ± 10 years old, FEV₁  = 37 (15% predicted value), VC = 62 (20% predicted value)], and 11 with lung fibrosis (LF) [64 ± 14 years old, FEV₁  = 55 (19% predicted value), VC = 62 (20% predicted value)]) and 76 healthy controls (HC) were recruited. The correlations between the signal of the Kinect (depth and respiratory rate) and the spirometer (tidal volume and respiratory rate) were computed in part 1. We then included 66 HC to test the ability of the system to detect modifications of respiratory patterns induced by various conditions known to modify respiratory pattern (cognitive load, inspiratory load and combination) in parts 2 and 3.

RESULTS

There is a strong correlation between the depth recorded by the Kinect and the tidal volume recorded by the spirometer: r = 0.973 for COPD patients, r = 0.989 for LF patients and r = 0.984 for HC. The Kinect is able to detect changes in breathing patterns induced by different respiratory disturbance conditions, gender and oral task.

CONCLUSIONS

Measurements performed with the Kinect sensors are highly correlated with the spirometer in HC and patients with COPD and LF. Kinect is also able to assess respiratory patterns under various loads and disturbances. This method is affordable, easy to use, fully automated and could be used in the current clinical context. Respiratory patterns are important to assess in daily clinics. However, there is currently no affordable and easy-to-use tool to evaluate these parameters in clinics. We validated a new system to assess respiratory patterns using the Kinect sensor in patients with chronic respiratory diseases.

Depth-Based Measurement of Respiratory Volumes: A Review

Depth-based plethysmography (DPG) for the measurement of respiratory parameters is a mobile and cost-effective alternative to spirometry and body plethysmography. In addition, natural breathing can be measured without a mouthpiece, and breathing mechanics can be visualized. This paper aims at showing further improvements for DPG by analyzing recent developments regarding the individual components of a DPG measurement. Starting from the advantages and application scenarios, measurement scenarios and recording devices, selection algorithms and location of a region of interest (ROI) on the upper body, signal processing steps, models for error minimization with a reference measurement device, and final evaluation procedures are presented and discussed. It is shown that ROI selection has an impact on signal quality. Adaptive methods and dynamic referencing of body points to select the ROI can allow more accurate placement and thus lead to better signal quality. Multiple different ROIs can be used to assess breathing mechanics and distinguish patient groups. Signal acquisition can be performed quickly using arithmetic calculations and is not inferior to complex 3D reconstruction algorithms. It is shown that linear models provide a good approximation of the signal. However, further dependencies, such as personal characteristics, may lead to non-linear models in the future. Finally, it is pointed out to focus developments with respect to single-camera systems and to focus on independence from an individual calibration in the evaluation.

Algorithmic insights of camera-based respiratory motion extraction

OBJECTIVE

Measuring the respiratory signal from a video based on body motion has been proposed and recently matured in products for contactless health monitoring. The core algorithm for this application is the measurement of tiny chest/abdominal motions induced by respiration (i.e. capturing sub-pixel displacement caused by subtle motion between subsequent video frames), and the fundamental challenge is motion sensitivity. Though prior art reported on the validation with real human subjects, there is no thorough or rigorous benchmark to quantify the sensitivities and boundary conditions of motion-based core respiratory algorithms.

APPROACH

A set-up was designed with a fully-controllable physical phantom to investigate the essence of core algorithms, together with a mathematical model incorporating two motion estimation strategies and three spatial representations, leading to six algorithmic combinations for respiratory signal extraction. Their promises and limitations are discussed and clarified through the phantom benchmark.

MAIN RESULTS

With the variation of phantom motion intensity between 0.5 mm and 8 mm, the recommended approach obtains an average precision, recall, coverage and MAE of 88.1%, 91.8%, 95.5% and 2.1bpm in the day-light condition, and 81.7%, 90.0%, 93.9% and 4.4 bpm in the night condition.

SIGNIFICANCE

The insights gained in this paper are intended to improve the understanding and applications of camera-based respiration measurement in health monitoring. The limitations of this study stem from the used physical phantom that does not consider human factors like body shape, sleeping posture, respiratory diseases, etc., and the investigated scenario is focused on sleep monitoring, not including scenarios with a sitting or standing patient like in clinical ward and triage.

Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions

Using a camera as an optical sensor to monitor physiological parameters has garnered considerable research interest in biomedical engineering in recent decades. Researchers have explored the use of a camera for monitoring a variety of physiological waveforms, together with the vital signs carried by these waveforms. Most of the obtained waveforms are related to the human respiratory and cardiovascular systems, and in addition of being indicative of overall health, they can also detect early signs of certain diseases. While using a camera for noncontact physiological signal monitoring offers the advantages of low cost and operational ease, it also has the disadvantages such as vulnerability to motion and lack of burden-free calibration solutions in some use cases. This study presents an overview of the existing camera-based methods that have been reported in recent years. It introduces the physiological principles behind these methods, signal acquisition approaches, various types of acquired signals, data processing algorithms, and application scenarios of these methods. It also discusses the technological gaps between the camera-based methods and traditional medical techniques, which are mostly contact-based. Furthermore, we present the manner in which noncontact physiological signal monitoring use has been extended, particularly over the recent years, to more day-to-day aspects of individuals' lives, so as to go beyond the more conventional use case scenarios. We also report on the development of novel approaches that facilitate easier measurement of less often monitored and recorded physiological signals. These have the potential of ushering a host of new medical and lifestyle applications. We hope this study can provide useful information to the researchers in the noncontact physiological signal measurement community.

Quantitative Assessment of the Airway Response to Bronchial Tests Based on a Spirometric Curve Shift

OBJECTIVE

Although spirometry is the most common pulmonary function test, there is no method to quantitatively infer about airway resistance or other properties from the flow-volume curves. Recently, an identifiable inverse model for forced expiration was proposed, as well as the idea to deduce changes in airway resistances and compliances from spirometric curve evolution. The aim of this work was to combine the above advances in a method for assessing the airway response to bronchial tests from a spirometric curve shift.

METHODS

The approach is based on the differential measurement of the degree, site of maximal effect and width of changes, further recalculated into relative changes in the distribution of airway resistances (δR_g) and compliances (δC_g) along the bronchial tree. To this end, appropriate models were identified using the pre- and post-test spirometry data. The accuracy was validated using sets of data simulated by the anatomy and physiology based models. Finally, the method was used to analyze the bronchodilation tests of three asthmatic subjects.

RESULTS

The expected errors in assessing the degree, site and width of changes in the zone of conducting airways were 6.3%, 2.4 generations and 22%, respectively, and for δR_g and δC_g were 5-10% and 13-16%, respectively. The analyses of clinical data indicated a significant reduction in resistances and an increase in compliances of airway generations 8-12, consistent with clinical knowledge.

CONCLUSION

An unprecedented method to plausibly transforming the spirometry data into the site and degree of changes in airway properties has been proposed.

SIGNIFICANCE

The method can be used to deduce about the effects of bronchial tests, as well as to monitor changes in the airways between visits or to investigate how inhaled pharmaceuticals affect the bronchi.

Advancements in Methods and Camera-Based Sensors for the Quantification of Respiration

Assessment of respiratory function allows early detection of potential disorders in the respiratory system and provides useful information for medical management. There is a wide range of applications for breathing assessment, from measurement systems in a clinical environment to applications involving athletes. Many studies on pulmonary function testing systems and breath monitoring have been conducted over the past few decades, and their results have the potential to broadly impact clinical practice. However, most of these works require physical contact with the patient to produce accurate and reliable measures of the respiratory function. There is still a significant shortcoming of non-contact measuring systems in their ability to fit into the clinical environment. The purpose of this paper is to provide a review of the current advances and systems in respiratory function assessment, particularly camera-based systems. A classification of the applicable research works is presented according to their techniques and recorded/quantified respiration parameters. In addition, the current solutions are discussed with regards to their direct applicability in different settings, such as clinical or home settings, highlighting their specific strengths and limitations in the different environments.

Development and Clinical Application of a Novel Non-contact Early Airflow Limitation Screening System Using an Infrared Time-of-Flight Depth Image Sensor

Obstructive pulmonary diseases, such as diffuse panbronchiolitis (DPB), asthma, chronic obstructive pulmonary disease (COPD), and asthma COPD overlap syndrome (ACOS) trigger a severe reaction at some situations. Detecting early airflow limitation caused by diseases above is critical to stop the progression. Thus, there is a need for tools to enable self-screening of early airflow limitation at home. Here, we developed a novel non-contact early airflow limitation screening system (EAFL-SS) that does not require calibration to the individual by a spirometer. The system is based on an infrared time-of-flight (ToF) depth image sensor, which is integrated into several smartphones for photography focusing or augmented reality. The EAFL-SS comprised an 850 nm infrared ToF depth image sensor (224 × 171 pixels) and custom-built data processing algorithms to visualize anterior-thorax three-dimensional motions in real-time. Multiple linear regression analysis was used to determine the amount of air compulsorily exhaled after maximal inspiration (referred to as the forced vital capacity, FVC EAFL -SS) from the ToF-derived anterior-thorax forced vital capacity (FVC), height, and body mass index as explanatory variables and spirometer-derived FVC as the objective variable. The non-contact measurement is automatically started when an examinee is sitting 35 cm away from the EAFL-SS. A clinical test was conducted with 32 COPD patients (27/5 M/F, 67-93 years) as typical airflow limitation cases recruited at St. Marianna University Hospital and 21 healthy volunteers (10/11 M/F, 23-79 years). The EAFL-SS was used to monitor the respiration of examinees during forced exhalation while sitting still, and a spirometer was used simultaneously as a reference. The forced expiratory volume in 1 s (FEV1% EAFL -SS) was evaluated as a percentage of the FVC EAFL -SS, where values less than 70% indicated suspected airflow limitation. Leave-one-out cross-validation analysis revealed that this system provided 81% sensitivity and 90% specificity. Further, the FEV1 EAFL -SS values were closely correlated with that measured using a spirometer (r = 0.85, p < 0.0001). Hence, EAFL-SS appears promising for early airflow limitation screening at home.

Sustainable Smartphone-Based Healthcare Systems: A Systems Engineering Approach to Assess the Efficacy of Respiratory Monitoring Apps

Recent technological developments along with advances in smart healthcare have been rapidly changing the healthcare industry and improving outcomes for patients. To ensure reliable smartphone-based healthcare interfaces with high levels of efficacy, a system dynamics model with sustainability indicators is proposed. The focus of this paper is smartphone-based breathing monitoring systems that could possibly use breathing sounds as the data acquisition input. This can especially be useful for the self-testing procedure of the ongoing global COVID-19 crisis in which the lungs are attacked and breathing is affected. The method of investigation is based on a systems engineering approach using system dynamics modeling. In this paper, first, a causal model for a smartphone-based respiratory function monitoring is introduced. Then, a systems thinking approach is applied to propose a system dynamics model of the smartphone-based respiratory function monitoring system. The system dynamics model investigates the level of efficacy and sustainability of the system by studying the behavior of various factors of the system including patient wellbeing and care, cost, convenience, user friendliness, in addition to other embedded software and hardware breathing monitoring system design and performance metrics (e.g., accuracy, real-time response, etc.). The sustainability level is also studied through introducing various indicators that directly relate to the three pillars of sustainability. Various scenarios have been applied and tested on the proposed model. The results depict the dynamics of the model for the efficacy and sustainability of smartphone-based breathing monitoring systems. The proposed ideas provide a clear insight to envision sustainable and effective smartphone-based healthcare monitoring systems.

Depth-Based Whole Body Photoplethysmography in Remote Pulmonary Function Testing

OBJECTIVE

We propose a novel depth-based photoplethysmography (dPPG) approach to reduce motion artifacts in respiratory volume-time data and improve the accuracy of remote pulmonary function testing (PFT) measures.

METHOD

Following spatial and temporal calibration of two opposing RGB-D sensors, a dynamic three-dimensional model of the subject performing PFT is reconstructed and used to decouple trunk movements from respiratory motions. Depth-based volume-time data is then retrieved, calibrated, and used to compute 11 clinical PFT measures for forced vital capacity and slow vital capacity spirometry tests.

RESULTS

A dataset of 35 subjects (298 sequences) was collected and used to evaluate the proposed dPPG method by comparing depth-based PFT measures to the measures provided by a spirometer. Other comparative experiments between the dPPG and the single Kinect approach, such as Bland-Altman analysis, similarity measures performance, intra-subject error analysis, and statistical analysis of tidal volume and main effort scaling factors, all show the superior accuracy of the dPPG approach.

CONCLUSION

We introduce a depth-based whole body photoplethysmography approach, which reduces motion artifacts in depth-based volume-time data and highly improves the accuracy of depth-based computed measures.

SIGNIFICANCE

The proposed dPPG method remarkably drops the error mean and standard deviation of FEF , FEF , FEF, IC , and ERV measures by half, compared to the single Kinect approach. These significant improvements establish the potential for unconstrained remote respiratory monitoring and diagnosis.

Toward Respiratory Assessment Using Depth Measurements from a Time-of-Flight Sensor

Introduction: There is increasing interest in technologies that may enable remote monitoring of respiratory disease. Traditional methods for assessing respiratory function such as spirometry can be expensive and require specialist training to perform and interpret. Remote, non-contact tracking of chest wall movement has been explored in the past using structured light, accelerometers and impedance pneumography, but these have often been costly and clinical utility remains to be defined. We present data from a 3-Dimensional time-of-flight camera (found in gaming consoles) used to estimate chest volume during routine spirometry maneuvres.

Methods: Patients were recruited from a general respiratory physiology laboratory. Spirometry was performed according to international standards using an unmodified spirometer. A Microsoft Kinect V2 time-of-flight depth sensor was used to reconstruct 3-dimensional models of the subject's thorax to estimate volume-time and flow-time curves following the introduction of a scaling factor to transform measurements to volume estimates. The Bland-Altman method was used to assess agreement of model estimation with simultaneous recordings from the spirometer. Patient characteristics were used to assess predictors of error using regression analysis and to further explore the scaling factors.

Results: The chest volume change estimated by the Kinect camera during spirometry tracked respiratory rate accurately and estimated forced vital capacity (FVC) and vital capacity to within ± <1%. Forced expiratory volume estimation did not demonstrate acceptable limits of agreement, with 61.9% of readings showing >150 ml difference. Linear regression including age, gender, height, weight, and pack years of smoking explained 37.0% of the variance in the scaling factor for volume estimation. This technique had a positive predictive value of 0.833 to detect obstructive spirometry.

Conclusion: These data illustrate the potential of 3D time-of-flight cameras to remotely monitor respiratory rate. This is not a replacement for conventional spirometry and needs further refinement. Further algorithms are being developed to allow its independence from spirometry. Benefits include simplicity of set-up, no specialist training, and cost. This technique warrants further refinement and validation in larger cohorts.

Remote, Depth-Based Lung Function Assessment

OBJECTIVE

We propose a remote, noninvasive approach to develop pulmonary function testing (PFT) using a depth sensor.

METHOD

After generating a point cloud from scene depth values, we construct a three-dimensional model of the subject's chest. Then, by estimating the chest volume variation throughout a sequence, we generate volume-time and flow-time data for two prevalent spirometry tests: forced vital capacity (FVC) and slow vital capacity (SVC). Tidal volume and main effort sections of volume-time data are analyzed and calibrated separately to remove the effects of a subject's torso motion. After automatic extraction of keypoints from the volume-time and flow-time curves, seven FVC ( FVC, FEV1, PEF, FEF _25%, FEF _50%, FEF _75%, and FEF [Formula: see text]) and four SVC measures ( VC, IC, TV, and ERV) are computed and then validated against measures from a spirometer. A dataset of 85 patients (529 sequences in total), attending respiratory outpatient service for spirometry, was collected and used to evaluate the proposed method.

RESULTS

High correlation for FVC and SVC measures on intra-test and intra-subject measures between the proposed method and the spirometer.

CONCLUSION

Our proposed depth-based approach is able to remotely compute eleven clinical PFT measures, which gives highly accurate results when evaluated against a spirometer on a dataset comprising 85 patients.

SIGNIFICANCE

Experimental results computed over an unprecedented number of clinical patients confirm that chest surface motion is linearly related to the changes in volume of lungs, which establishes the potential toward an accurate, low-cost, and remote alternative to traditional cumbersome methods, such as spirometry.