Non-Contact Automatic Vital Signs Monitoring of Infants in a Neonatal Intensive Care Unit Based on Neural Networks

Integrating RGB-thermal image sensors for non-contact automatic respiration rate monitoring

Respiration rate (RR) holds significance as a human health indicator. Presently, the conventional RR monitoring system requires direct physical contact, which may cause discomfort and pain. Therefore, this paper proposes a non-contact RR monitoring system integrating RGB and thermal imaging through RGB-thermal image alignment. The proposed method employs an advanced image processing algorithm for automatic region of interest (ROI) selection. The experimental results demonstrated a close correlation and a lower error rate between measured thermal, measured RGB, and reference data. In summary, the proposed non-contact system emerges as a promising alternative to conventional contact-based approaches without the associated discomfort and pain.

Contactless assessment of heart rate in neonates within a clinical environment using imaging photoplethysmography

Introduction

In neonatology, the accurate determination of vital parameters plays a pivotal role in monitoring critically ill newborns and premature infants, as well as aiding in disease diagnosis. In response to the limitations associated with contact-based measurement methods, substantial efforts have been directed toward developing contactless measurement techniques, particularly over the past decade.

Methods

Building upon the insights gained from our pilot study, we realized a new investigation to assess the precision of our imaging photoplethysmography-based system within a clinical environment of the neonatal intermediate care unit. We conducted measurements in 20 preterm infants or newborns requiring therapeutic interventions. As a point of reference, we employed a conventional pulse oximeter. To analytically predict measurement artifacts, we analyzed the potential influence of confounding factors, such as motion artifacts, illumination fluctuations (under- and overexposure), and loss of region of interest prior to heart rate evaluation. This reduced the amount of data we evaluated for heart rate to 56.1% of its original volume.

Results

In artifact-free time segments, the mean difference between the pulse oximetry and the imaging photoplethysmography-based system for 1 s sampling intervals resulted in -0.2 bpm (95% CI -0.8 to 0.4, LOA ± 12.2). For the clinical standard of 8 s averaging time, the mean difference resulted in -0.09 bpm (95% CI -0.7 to 0.6, LOA ± 10.1). These results match the medical standards.

Discussion

While further research is needed to increase the range of measurable vital parameters and more diverse patient collectives need to be considered in the future, we could demonstrate very high accuracy for non-contact heart rate measurement in newborn infants in the clinical setting, provided artifacts are excluded. In particular, performing a priori signal assessment helps make clinical measurements safer by identifying unreliable readings.

Challenges and prospects of visual contactless physiological monitoring in clinical study

The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.

E-Health: A Game Changer in Fetal and Neonatal Cardiology?

Technological advancements have greatly impacted the healthcare industry, including the integration of e-health in pediatric cardiology. The use of telemedicine, mobile health applications, and electronic health records have demonstrated a significant potential to improve patient outcomes, reduce healthcare costs, and enhance the quality of care. Telemedicine provides a useful tool for remote clinics, follow-up visits, and monitoring for infants with congenital heart disease, while mobile health applications enhance patient and parents' education, medication compliance, and in some instances, remote monitoring of vital signs. Despite the benefits of e-health, there are potential limitations and challenges, such as issues related to availability, cost-effectiveness, data privacy and security, and the potential ethical, legal, and social implications of e-health interventions. In this review, we aim to highlight the current application and perspectives of e-health in the field of fetal and neonatal cardiology, including expert parents' opinions.

Thermal Image Processing for Respiratory Estimation from Cubical Data with Expandable Depth

As healthcare costs continue to rise, finding affordable and non-invasive ways to monitor vital signs is increasingly important. One of the key metrics for assessing overall health and identifying potential issues early on is respiratory rate (RR). Most of the existing methods require multiple steps that consist of image and signal processing. This might be difficult to deploy on edge devices that often do not have specialized digital signal processors (DSP). Therefore, the goal of this study is to develop a single neural network realizing the entire process of RR estimation in a single forward pass. The proposed solution builds on recent advances in video recognition, capturing both spatial and temporal information in a multi-path network. Both paths process the data at different sampling rates to capture rapid and slow changes that are associated with differences in the temperature of the nostril area during the breathing episodes. The preliminary results show that the introduced end-to-end solution achieves better performance compared to state-of-the-art methods, without requiring additional pre/post-processing steps and signal-processing techniques. In addition, the presented results demonstrate its robustness on low-resolution thermal video sequences that are often used at the embedded edge due to the size and power constraints of such systems. Taking that into account, the proposed approach has the potential for efficient and convenient respiratory rate estimation across various markets in solutions deployed locally, close to end users.

Neonatal Face and Facial Landmark Detection from Video Recordings

This paper explores automated face and facial landmark detection of neonates, which is an important first step in many video-based neonatal health applications, such as vital sign estimation, pain assessment, sleep-wake classification, and jaundice detection. Utilising three publicly available datasets of neonates in the clinical environment, 366 images (258 subjects) and 89 (66 subjects) were annotated for training and testing, respectively. Transfer learning was applied to two YOLO-based models, with input training images augmented with random horizontal flipping, photo-metric colour distortion, translation and scaling during each training epoch. Additionally, the re-orientation of input images and fusion of trained deep learning models was explored. Our proposed model based on YOLOv7Face outperformed existing methods with a mean average precision of 84.8% for face detection, and a normalised mean error of 0.072 for facial landmark detection. Overall, this will assist in the development of fully automated neonatal health assessment algorithms.Clinical relevance- Accurate face and facial landmark detection provides an automated and non-contact option to assist in video-based neonatal health applications.

Sensor Fusion for the Robust Detection of Facial Regions of Neonates Using Neural Networks

The monitoring of vital signs and increasing patient comfort are cornerstones of modern neonatal intensive care. Commonly used monitoring methods are based on skin contact which can cause irritations and discomfort in preterm neonates. Therefore, non-contact approaches are the subject of current research aiming to resolve this dichotomy. Robust neonatal face detection is essential for the reliable detection of heart rate, respiratory rate and body temperature. While solutions for adult face detection are established, the unique neonatal proportions require a tailored approach. Additionally, sufficient open-source data of neonates on the NICU is lacking. We set out to train neural networks with the thermal-RGB-fusion data of neonates. We propose a novel indirect fusion approach including the sensor fusion of a thermal and RGB camera based on a 3D time-of-flight (ToF) camera. Unlike other approaches, this method is tailored for close distances encountered in neonatal incubators. Two neural networks were used with the fusion data and compared to RGB and thermal networks. For the class "head" we reached average precision values of 0.9958 (RetinaNet) and 0.9455 (YOLOv3) for the fusion data. Compared with the literature, similar precision was achieved, but we are the first to train a neural network with fusion data of neonates. The advantage of this approach is in calculating the detection area directly from the fusion image for the RGB and thermal modality. This increases data efficiency by 66%. Our results will facilitate the future development of non-contact monitoring to further improve the standard of care for preterm neonates.

Device for remote and realtime monitoring of neonatal vital signs in neonatal intensive care unit using internet of things: proof-of-concept study

BACKGROUND

Realtime and remote monitoring of neonatal vital signs is a crucial part of providing appropriate care in neonatal intensive care units (NICU) to reduce mortality and morbidity of newborns. In this study, a new approach, a device for remote and real-time monitoring of neonatal vital signs (DRRMNVS) in the neonatal intensive care unit using the internet of things (IoT), was proposed. The system integrates four vital signs: oxygen saturation, pulse rate, body temperature and respiration rate for continuous monitoring using the Blynk app and ThingSpeak IoT platforms.

METHODS

The Wemos D1 mini, a Wi-Fi microcontroller, was used to acquire the four biological biomarkers from sensors, process them and display the result on an OLED display for point of care monitoring and on the Blynk app and ThingSpeak for remote and continuous monitoring of vital signs. The Bland-Altman test was employed to test the agreement of DRRMNVS measurement with reference standards by taking measurements from ten healthy adults.

RESULTS

The prototype of the proposed device was successfully developed and tested. Bias [limits of agreement] were: Oxygen saturation (SpO2): -0.1 [- 1.546 to + 1.346] %; pulse rate: -0.3 [- 2.159 to + 1.559] bpm; respiratory rate: -0.7 [- 0.247 to + 1.647] breaths/min; temperature: 0.21 [+ 0.015˚C to + 0.405˚C] ˚C. The proof-of-concept prototype was developed for $33.19.

CONCLUSION

The developed DRRMNVS device was cheap and had acceptable measurement accuracy of vital signs in a controlled environment. The system has the potential to advance healthcare service delivery for neonates with further development from this proof-of-concept level.

Non-contact Vital Signs Monitoring in Paediatric Anaesthesia - Current Challenges and Future Direction

Non-contact vital sign monitoring is an area of increasing interest in the clinical scenario since it offers advantages over traditional monitoring using leads and wires. These advantages include reduction in transmission of infection and more freedom of movement. Yet there is a paucity of studies available in the clinical setting particularly in paediatric anaesthesia. This scoping review aims to investigate why contactless monitoring, specifically with red-green-blue cameras, is not implemented in mainstream practise. The challenges, drawbacks and limitations of non-contact vital sign monitoring, will be outlined, together with future direction on how it can potentially be implemented in the setting of paediatric anaesthesia, and in the critical care scenario.

Non-contact respiratory rate monitoring using thermal and visible imaging: a pilot study on neonates

Respiratory rate (RR) monitoring is essential in neonatal intensive care units. Despite its importance, RR is still monitored intermittently by manual counting instead of continuous monitoring due to the risk of skin damage with prolonged use of contact electrodes in preterm neonates and false signals due to displacement of electrodes. Thermal imaging has recently gained significance as a non-contact method for RR detection because of its many advantages. However, due to the lack of information in thermal images, the selection and tracking of the region of interest (ROI) in thermal images for neonates are challenging. This paper presents the integration of visible (RGB) and thermal (T) image sequences for the selection and tracking of ROI for breathing rate extraction. The deep-learning based tracking-by-detection approach is employed to detect the ROI in the RGB images, and it is mapped to the thermal images using the RGB-T image registration. The mapped ROI in thermal spectrum sequences gives the respiratory rate. The study was conducted first on healthy adults in different modes, including steady, motion, talking, and variable respiratory order. Subsequently, the method is tested on neonates in a clinical settings. The findings have been validated with a contact-based reference method.The average absolute error between the proposed and belt-based contact method in healthy adults reached 0.1 bpm and for more challenging conditions was approximately 1.5 bpm and 1.8 bpm, respectively. In the case of neonates, the average error is 1.5 bpm, which are promising results. The Bland–Altman analysis showed a good agreement of estimated RR with the reference method RR and this pilot study provided the evidence of using the proposed approach as a contactless method for the respiratory rate detection of neonates in clinical settings.

Deep learning based non-contact physiological monitoring in Neonatal Intensive Care Unit

Preterm babies in the Neonatal Intensive Care Unit (NICU) have to undergo continuous monitoring of their cardiac health. Conventional monitoring approaches are contact-based, making the neonates prone to various nosocomial infections. Video-based monitoring approaches have opened up potential avenues for contactless measurement. This work presents a pipeline for remote estimation of cardiopulmonary signals from videos in NICU setup. We have proposed an end-to-end deep learning (DL) model that integrates a non-learning-based approach to generate surrogate ground truth (SGT) labels for supervision, thus refraining from direct dependency on true ground truth labels. We have performed an extended qualitative and quantitative analysis to examine the efficacy of our proposed DL-based pipeline and achieved an overall average mean absolute error of 4.6 beats per minute (bpm) and root mean square error of 6.2 bpm in the estimated heart rate.

Camera fusion for real-time temperature monitoring of neonates using deep learning

Abstract

The continuous monitoring of vital signs is a crucial aspect of medical care in neonatal intensive care units. Since cable-based sensors pose a potential risk for the immature skin of preterm infants, unobtrusive monitoring techniques using camera systems are increasingly investigated. The combination of deep learning–based algorithms and camera modalities such as RGB and infrared thermography can improve the development of cable-free methods for the extraction of vital parameters. In this study, a real-time approach for local extraction of temperatures on the body surface of neonates using a multi-modal clinical dataset was implemented. Therefore, a trained deep learning–based keypoint detector was used for body landmark prediction in RGB. Image registration was conducted to transfer the RGB points to the corresponding thermographic recordings. These landmarks were used to extract the body surface temperature in various regions to determine the central-peripheral temperature difference. A validation of the keypoint detector showed a mean average precision of 0.82. The registration resulted in mean absolute errors of 16.4 px (8.2 mm) for x and 22.4 px (11.2 mm) for y. The evaluation of the temperature extraction revealed a mean absolute error of 0.55 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\circ }$$\end{document}∘C. A final performance of 31 fps was observed on the NVIDIA Jetson Xavier NX module, which proves real-time capability on an embedded GPU system. As a result, the approach can perform real-time temperature extraction on a low-cost GPU module.

Graphical abstract

Skin Pigmentation Influence on Pulse Oximetry Accuracy: A Systematic Review and Bibliometric Analysis

Nowadays, pulse oximetry has become the standard in primary and intensive care units, especially as a triage tool during the current COVID-19 pandemic. Hence, a deeper understanding of the measurement errors that can affect precise readings is a key element in clinical decision-making. Several factors may influence the accuracy of pulse oximetry, such as skin color, body temperature, altitude, or patient movement. The skin pigmentation effect on pulse oximetry accuracy has long been studied reporting some contradictory conclusions. Recent studies have shown a positive bias in oxygen saturation measurements in patients with darkly pigmented skin, particularly under low saturation conditions. This review aims to study the literature that assesses the influence of skin pigmentation on the accuracy of these devices. We employed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement to conduct a systematic review retrospectively since February 2022 using WOS, PubMed, and Scopus databases. We found 99 unique references, of which only 41 satisfied the established inclusion criteria. A bibliometric and scientometrics approach was performed to examine the outcomes of an exhaustive survey of the thematic content and trending topics.

High-Speed Continuous Wavelet Transform Processor for Vital Signal Measurement Using Frequency-Modulated Continuous Wave Radar

This paper proposes a high-speed continuous wavelet transform (CWT) processor to analyze vital signals extracted from a frequency-modulated continuous wave (FMCW) radar sensor. The proposed CWT processor consists of a fast Fourier transform (FFT) module, complex multiplier module, and inverse FFT (IFFT) module. For high-throughput processing, the FFT and IFFT modules are designed with the pipeline FFT architecture of radix-2 single-path delay feedback (R2SDF) and mixed-radix multipath delay commutator (MRMDC) architecture, respectively. In addition, the IFFT module and the complex multiplier module perform a four-channel operation to reduce the processing time from repeated operations. Simultaneously, the MRMDC IFFT module minimizes the circuit area by reducing the number of non-trivial multipliers by using a mixed-radix algorithm. In addition, the proposed CWT processor can support variable lengths of 8, 16, 32, 64, 128, 256, 512, and 1024 to analyze various vital signals. The proposed CWT processor was implemented in a field-programmable gate array (FPGA) device and verified through the measurement of heartbeat and respiration from an FMCW radar sensor. Experimental results showed that the proposed CWT processor can reduce the processing time by 48.4-fold and 40.7-fold compared to MATLAB software with Intel i7 CPU. Moreover, it can be confirmed that the proposed CWT processor can reduce the processing time by 73.3% compared to previous FPGA-based implementations.

A Setup for Camera-Based Detection of Simulated Pathological States Using a Neonatal Phantom

Premature infants are among the most vulnerable patients in a hospital. Due to numerous complications associated with immaturity, a continuous monitoring of vital signs with a high sensitivity and accuracy is required. Today, wired sensors are attached to the patient's skin. However, adhesive electrodes can be potentially harmful as they can damage the very thin immature skin. Although unobtrusive monitoring systems using cameras show the potential to replace cable-based techniques, advanced image processing algorithms are data-driven and, therefore, need much data to be trained. Due to the low availability of public neonatal image data, a patient phantom could help to implement algorithms for the robust extraction of vital signs from video recordings. In this work, a camera-based system is presented and validated using a neonatal phantom, which enabled a simulation of common neonatal pathologies such as hypo-/hyperthermia and brady-/tachycardia. The implemented algorithm was able to continuously measure and analyze the heart rate via photoplethysmography imaging with a mean absolute error of 0.91 bpm, as well as the distribution of a neonate's skin temperature with a mean absolute error of less than 0.55 °C. For accurate measurements, a temperature gain offset correction on the registered image from two infrared thermography cameras was performed. A deep learning-based keypoint detector was applied for temperature mapping and guidance for the feature extraction. The presented setup successfully detected several levels of hypo- and hyperthermia, an increased central-peripheral temperature difference, tachycardia and bradycardia.

Camera-based heart rate estimation for hospitalized newborns in the presence of motion artifacts

Background

Heart rate (HR) is an important vital sign for evaluating the physiological condition of a newborn infant. Recently, for measuring HR, novel RGB camera-based non-contact techniques have demonstrated their specific superiority compared with other techniques, such as dopplers and thermal cameras. However, they still suffered poor robustness in infants’ HR measurements due to frequent body movement.

Methods

This paper introduces a framework to improve the robustness of infants’ HR measurements by solving motion artifact problems. Our solution is based on the following steps: morphology-based filtering, region-of-interest (ROI) dividing, Eulerian video magnification and majority voting. In particular, ROI dividing improves ROI information utilization. The majority voting scheme improves the statistical robustness by choosing the HR with the highest probability. Additionally, we determined the dividing parameter that leads to the most accurate HR measurements. In order to examine the performance of the proposed method, we collected 4 hours of videos and recorded the corresponding electrocardiogram (ECG) of 9 hospitalized neonates under two different conditions—rest still and visible movements.

Results

Experimental results indicate a promising performance: the mean absolute error during rest still and visible movements are 3.39 beats per minute (BPM) and 4.34 BPM, respectively, which improves at least 2.00 and 1.88 BPM compared with previous works. The Bland-Altman plots also show the remarkable consistency of our results and the HR derived from the ground-truth ECG.

Conclusions

To the best of our knowledge, this is the first study aimed at improving the robustness of neonatal HR measurement under motion artifacts using an RGB camera. The preliminary results have shown the promising prospects of the proposed method, which hopefully reduce neonatal mortality in hospitals.