1
|
Anil Jalaja A, Kavitha M. Contactless face video based vital signs detection framework for continuous health monitoring using feature optimization and hybrid neural network. Biotechnol Bioeng 2024; 121:1191-1215. [PMID: 38221763 DOI: 10.1002/bit.28644] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Accepted: 12/18/2023] [Indexed: 01/16/2024]
Abstract
Continuous monitoring of vital signs such as respiration and heart rate is essential to detect and predict conditions that may affect the patient's well-being. To detect these vital signs most medical systems use contact sensors. They are not feasible for long term monitoring and are not repeatable. Vital signs using facial video-noncontact monitoring are becoming increasingly important. Researchers in the last few years although considerable progress has been made, challenging datasets absence timing of assessment process and the technology still has some limitations such as time consuming nature and lack of computer portability. To solve those problems, we propose a contactless video based vital signs detection framework for continuous health monitoring using feature optimization and hybrid neural network. In the proposed technique, modified war strategy optimization algorithm is proposed to segment the face portion from the input video frames. Then, we utilize the known data acquisition models to extract vital signs from the segmented face portions are heart rate, blood pressure, respiratory rate and oxygen saturation. An improved neural network structure (Lifting Net) is further used to achieve the adaptive extraction of deep hidden features for specific signs, for realizing the high precision of human health monitoring. The Hughes effect or dimensionality issue affects detection accuracy in sign classification when there are fewer training instances relative to the number of spectral features. The problem can be overcome through feature optimization here Northern goshawk optimization algorithm is used to select optimal best features which reduces the data dimensionality issue. Furthermore, hybrid deep ensemble reinforcement learning classifier is proposed for the human vital sign detection and classification which ensures the early detection of patient abnormality. Finally, we validate our framework using benchmark video datasets such as TokyoTechrPPG, PURE and COHFACE. To proves the effectiveness of proposed technique using simulation results and comparative analysis.
Collapse
Affiliation(s)
- Anju Anil Jalaja
- Department of Electronics and Communication Engineering, K. Ramakrishnan College of Technology, Trichy Affiliated to Anna University, Chennai, India
| | - Maruthai Kavitha
- Department of Electronics and Communication Engineering, K. Ramakrishnan College of Technology, Trichy, India
| |
Collapse
|
2
|
Solon IG, Santos WS, Jesus AA, Garcia FS, Nascimento GC, Cárnio EC, Branco LGS, Santos BM. Non-invasive evaluation of vascular permeability in formalin-induced orofacial pain model using infrared thermography. J Therm Biol 2024; 119:103782. [PMID: 38176292 DOI: 10.1016/j.jtherbio.2023.103782] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 12/13/2023] [Accepted: 12/19/2023] [Indexed: 01/06/2024]
Abstract
Enhanced vascular permeability at the site of injury is a prominent feature in acute inflammatory pain models, commonly assessed through the Evans Blue test. However, this invasive test requires euthanasia, thereby precluding further investigations on the same animal. Due to these limitations, the integration of non-invasive tools such as IRT has been sought. Here, we aimed to evaluate the use of thermography in a common orofacial pain model that employs formalin as a chemical irritant to induce local orofacial inflammation. Male Hannover rats (290-300 g, N = 43) were used. In the first approach, radiometric images were taken before and after formalin administration, assessing temperature changes and extravasated Evans Blue. The second approach included capturing pre- and post-formalin test radiometric images, followed by cytokine measurements in excised vibrissae tissue. Rats were anesthetized for vibrissae tissue collection, allowing correlations between thermographic patterns, nocifensive behavior duration, and cytokine levels in this area. Our findings revealed a positive correlation between local temperature, measured via thermography, and vascular permeability in the contralateral (r2 = 0.3483) and ipsilateral (r2 = 0.4502) side, measured using spectrophotometry. The obtained data supports the notion that thermography-based temperature assessment can effectively evaluate vascular permeability in the orofacial region.
Collapse
Affiliation(s)
- Isabelly G Solon
- Department of Basic and Oral Biology, Dental School of Ribeirão Preto, University of São Paulo, Ribeirão Preto, SP, Brazil
| | - Wanderson S Santos
- Department of Basic and Oral Biology, Dental School of Ribeirão Preto, University of São Paulo, Ribeirão Preto, SP, Brazil
| | - Aline A Jesus
- Department of General and Specialized Nursing Ribeirão Preto, College of Nursing - University of São Paulo, Ribeirão Preto, SP, Brazil
| | - Felipe S Garcia
- Department of Basic and Oral Biology, Dental School of Ribeirão Preto, University of São Paulo, Ribeirão Preto, SP, Brazil
| | - Glauce C Nascimento
- Department of Basic and Oral Biology, Dental School of Ribeirão Preto, University of São Paulo, Ribeirão Preto, SP, Brazil
| | - Evelin C Cárnio
- Department of General and Specialized Nursing Ribeirão Preto, College of Nursing - University of São Paulo, Ribeirão Preto, SP, Brazil
| | - Luiz G S Branco
- Department of Basic and Oral Biology, Dental School of Ribeirão Preto, University of São Paulo, Ribeirão Preto, SP, Brazil.
| | - Bruna M Santos
- Department of Basic and Oral Biology, Dental School of Ribeirão Preto, University of São Paulo, Ribeirão Preto, SP, Brazil; Department of General and Specialized Nursing Ribeirão Preto, College of Nursing - University of São Paulo, Ribeirão Preto, SP, Brazil.
| |
Collapse
|
3
|
Huang B, Hu S, Liu Z, Lin CL, Su J, Zhao C, Wang L, Wang W. Challenges and prospects of visual contactless physiological monitoring in clinical study. NPJ Digit Med 2023; 6:231. [PMID: 38097771 PMCID: PMC10721846 DOI: 10.1038/s41746-023-00973-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Accepted: 11/21/2023] [Indexed: 12/17/2023] Open
Abstract
The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.
Collapse
Affiliation(s)
- Bin Huang
- AI Research Center, Hangzhou Innovation Institute, Beihang University, 99 Juhang Rd., Binjiang Dist., Hangzhou, Zhejiang, China.
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China.
| | - Shen Hu
- Department of Obstetrics, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
- Department of Epidemiology, The Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Zimeng Liu
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China
| | - Chun-Liang Lin
- College of Electrical Engineering and Computer Science, National Chung Hsing University, 145 Xingda Rd., South Dist., Taichung, Taiwan.
| | - Junfeng Su
- Department of General Intensive Care Unit, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
- Key Laboratory of Early Warning and Intervention of Multiple Organ Failure, China National Ministry of Education, Hangzhou, Zhejiang, China
| | - Changchen Zhao
- AI Research Center, Hangzhou Innovation Institute, Beihang University, 99 Juhang Rd., Binjiang Dist., Hangzhou, Zhejiang, China
| | - Li Wang
- Department of Rehabilitation Medicine, The First Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
| | - Wenjin Wang
- Department of Biomedical Engineering, Southern University of Science and Technology, 1088 Xueyuan Ave, Nanshan Dist., Shenzhen, Guangdong, China.
| |
Collapse
|
4
|
Mumtaz H, Saqib M, Jabeen S, Muneeb M, Mughal W, Sohail H, Safdar M, Mehmood Q, Khan MA, Ismail SM. Exploring alternative approaches to precision medicine through genomics and artificial intelligence - a systematic review. Front Med (Lausanne) 2023; 10:1227168. [PMID: 37849490 PMCID: PMC10577305 DOI: 10.3389/fmed.2023.1227168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Accepted: 09/20/2023] [Indexed: 10/19/2023] Open
Abstract
The core idea behind precision medicine is to pinpoint the subpopulations that differ from one another in terms of disease risk, drug responsiveness, and treatment outcomes due to differences in biology and other traits. Biomarkers are found through genomic sequencing. Multi-dimensional clinical and biological data are created using these biomarkers. Better analytic methods are needed for these multidimensional data, which can be accomplished by using artificial intelligence (AI). An updated review of 80 latest original publications is presented on four main fronts-preventive medicine, medication development, treatment outcomes, and diagnostic medicine-All these studies effectively illustrated the significance of AI in precision medicine. Artificial intelligence (AI) has revolutionized precision medicine by swiftly analyzing vast amounts of data to provide tailored treatments and predictive diagnostics. Through machine learning algorithms and high-resolution imaging, AI assists in precise diagnoses and early disease detection. AI's ability to decode complex biological factors aids in identifying novel therapeutic targets, allowing personalized interventions and optimizing treatment outcomes. Furthermore, AI accelerates drug discovery by navigating chemical structures and predicting drug-target interactions, expediting the development of life-saving medications. With its unrivaled capacity to comprehend and interpret data, AI stands as an invaluable tool in the pursuit of enhanced patient care and improved health outcomes. It's evident that AI can open a new horizon for precision medicine by translating complex data into actionable information. To get better results in this regard and to fully exploit the great potential of AI, further research is required on this pressing subject.
Collapse
Affiliation(s)
| | | | | | - Muhammad Muneeb
- Department of Medicine, Dow University of Health Sciences, Karachi, Pakistan
| | - Wajiha Mughal
- Department of Medicine, Dow University of Health Sciences, Karachi, Pakistan
| | - Hassan Sohail
- Department of Medicine, Dow University of Health Sciences, Karachi, Pakistan
| | - Myra Safdar
- Armed Forces Institute of Cardiology and National Institute of Heart Diseases (AFIC-NIHD), Rawalpindi, Pakistan
| | - Qasim Mehmood
- Department of Medicine, King Edward Medical University, Lahore, Pakistan
| | - Muhammad Ahsan Khan
- Department of Medicine, Dow University of Health Sciences, Karachi, Pakistan
| | | |
Collapse
|
5
|
Zhang N, Wood O, Yang Z, Xie J. AI-Guided Computing Insights into a Thermostat Monitoring Neonatal Intensive Care Unit (NICU). SENSORS (BASEL, SWITZERLAND) 2023; 23:s23094492. [PMID: 37177696 PMCID: PMC10181714 DOI: 10.3390/s23094492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 04/19/2023] [Accepted: 05/03/2023] [Indexed: 05/15/2023]
Abstract
In any healthcare setting, it is important to monitor and control airflow and ventilation with a thermostat. Computational fluid dynamics (CFD) simulations can be carried out to investigate the airflow and heat transfer taking place inside a neonatal intensive care unit (NICU). In this present study, the NICU is modeled based on the realistic dimensions of a single-patient room in compliance with the appropriate square footage allocated per incubator. The physics of flow in NICU is predicted based on the Navier-Stokes conservation equations for an incompressible flow, according to suitable thermophysical characteristics of the climate. The results show sensible flow structures and heat transfer as expected from any indoor climate with this configuration. Furthermore, machine learning (ML) in an artificial intelligence (AI) model has been adopted to take the important geometric parameter values as input from our CFD settings. The model provides accurate predictions of the thermal performance (i.e., temperature evaluation) associated with that design in real time. Besides the geometric parameters, there are three thermophysical variables of interest: the mass flow rate (i.e., inlet velocity), the heat flux of the radiator (i.e., heat source), and the temperature gradient caused by the convection. These thermophysical variables have significantly recovered the physics of convective flows and enhanced the heat transfer throughout the incubator. Importantly, the AI model is not only trained to improve the turbulence modeling but also to capture the large temperature gradient occurring between the infant and surrounding air. These physics-informed (Pi) computing insights make the AI model more general by reproducing the flow of fluid and heat transfer with high levels of numerical accuracy. It can be concluded that AI can aid in dealing with large datasets such as those produced in NICU, and in turn, ML can identify patterns in data and help with the sensor readings in health care.
Collapse
Affiliation(s)
- Ning Zhang
- Faculty of Arts and Sciences, Beijing Normal University at Zhuhai, Zhuhai 519087, China
| | - Olivia Wood
- Galliford Try, Staffordshire Technology Park, Stafford ST18 0GP, UK
| | - Zhiyin Yang
- School of Computing and Engineering, University of Derby, Derby DE22 3AW, UK
| | - Jianfei Xie
- School of Computing and Engineering, University of Derby, Derby DE22 3AW, UK
| |
Collapse
|
6
|
Mustafa A, Ullah F, Rehman MU, Khan MB, Tanoli SAK, Ullah MK, Umar H, Chong KT. Non-intrusive RF sensing for early diagnosis of spinal curvature syndrome disorders. Comput Biol Med 2023; 155:106614. [PMID: 36780802 DOI: 10.1016/j.compbiomed.2023.106614] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2022] [Revised: 01/18/2023] [Accepted: 01/28/2023] [Indexed: 02/11/2023]
Abstract
The recent developments in communication and information ease people's lives to sit in one place and access any information from anywhere. However, the longevity of sitting and sitting in different postures raises the issues of spinal curvature. It necessitates a physical examination to identify the spinal illness in its early stages. This article aims to develop an intelligent monitoring framework for detecting and monitoring spinal curvature syndrome problems based on Software Defined Radio Frequency (SDRF) sensing and verify its feasibility for diagnosing actual patients. The proposed SDRF-based system identifies irregular spinal curvature syndrome and offers feedback signals when an incorrect posture is identified. We design the system using wireless university software-defined radio peripheral (USRP) kits to transmit and receive RF signals and record the wireless channel state information (WCSI) for kyphosis, Lordosis, and scoliosis spinal disorders. The statistical measures are extracted from the WCSI and apply machine learning algorithms to identify and classify the type of disorders. We record and test the system using 11 subjects with the spinal disorders kyphosis, Lordosis, and scoliosis. We acquire the WCSI, extract various statistical measures in terms of time and frequency domain features, and evaluate machine learning classifiers to identify and classify the spinal disorder. The performance comparison of the machine learning algorithms showed overall and each spinal curvature disorder recognition accuracy of more than 99%.
Collapse
Affiliation(s)
- Ali Mustafa
- Department of Electrical and Computer Engineering, COMSATS University Islamabad, Attock Campus, Punjab 43600, Pakistan.
| | - Farman Ullah
- Department of Electrical and Computer Engineering, COMSATS University Islamabad, Attock Campus, Punjab 43600, Pakistan; Division of Computer Science and Engineering, Jeonbuk National University, Jeonju, South Korea.
| | - Mobeen Ur Rehman
- Department of Electronics and Information Engineering, Jeonbuk National University, Jeonju 54896, South Korea.
| | - Muhammad Bilal Khan
- Department of Electrical and Computer Engineering, COMSATS University Islamabad, Attock Campus, Punjab 43600, Pakistan.
| | - Shujaat Ali Khan Tanoli
- Department of Electrical and Computer Engineering, COMSATS University Islamabad, Attock Campus, Punjab 43600, Pakistan.
| | - Muhammad Kaleem Ullah
- Department of Electrical and Computer Engineering, COMSATS University Islamabad, Attock Campus, Punjab 43600, Pakistan.
| | - Hamza Umar
- Department of Electrical and Computer Engineering, COMSATS University Islamabad, Attock Campus, Punjab 43600, Pakistan.
| | - Kil To Chong
- Department of Electronics and Information Engineering, Jeonbuk National University, Jeonju 54896, South Korea.
| |
Collapse
|
7
|
Jeong H, Kim D, Kim G, Kim S. VitRad: A low-cost continuous wave Doppler radar system with 3D-printed horn antennas for human vital sign detection. HARDWAREX 2022; 12:e00361. [PMID: 36188873 PMCID: PMC9515602 DOI: 10.1016/j.ohx.2022.e00361] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Revised: 09/01/2022] [Accepted: 09/18/2022] [Indexed: 06/12/2023]
Abstract
In this study, a low-cost continuous wave (CW) radar system with 3D-printed high-gain horn antennas called VitRad is proposed for human vital sign detection. The CW radar consists of 3D-printed high-gain horn antennas, commercially available low-cost surface-mounting devices, and monolithic ICs. The CW radar system operates at a frequency band of 5.8 GHz, and the backscattered I/Q data are collected using a digital storage oscilloscope (DSO). The data is processed on MATLAB to determine vital sign information such as respiratory and heartbeat rates. It is demonstrated that the proposed CW radar system for vital-sign monitoring can effectively measure respiratory and heartbeat rates.
Collapse
|
8
|
Yang X, Zhang Z, Huang Y, Zheng Y, Shen Y. Using a graph-based image segmentation algorithm for remote vital sign estimation and monitoring. Sci Rep 2022; 12:15197. [PMID: 36071124 PMCID: PMC9451121 DOI: 10.1038/s41598-022-19198-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Accepted: 08/25/2022] [Indexed: 12/02/2022] Open
Abstract
Reliable and contactless measurements of vital signs, such as respiration and heart rate, are still unmet needs in clinical and home settings. Mm-wave radar and video-based technologies are promising, but currently, the signal processing-based vital sign extraction methods are prone to body motion disruptions or illumination variations in the surrounding environment. Here we propose an image segmentation-based method to extract vital signs from the recorded video and mm-wave radar signals. The proposed method analyses time–frequency spectrograms obtained from Short-Time Fourier Transform rather than individual time-domain signals. This leads to much-improved robustness and accuracy of the heart rate and respiration rate extraction over existing methods. The experiments were conducted under pre- and post-exercise conditions and were repeated on multiple individuals. The results are evaluated by using four metrics against the gold standard contact-based measurements. Significant improvements were observed in terms of precision, accuracy, and stability. The performance was reflected by achieving an averaged Pearson correlation coefficient (PCC) of 93.8% on multiple subjects. We believe that the proposed estimation method will help address the needs for the increasingly popular remote cardiovascular sensing and diagnosing posed by Covid-19.
Collapse
Affiliation(s)
- Xingyu Yang
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
| | - Zijian Zhang
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
| | - Yi Huang
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
| | - Yalin Zheng
- Department of Eye and Vision Science, University of Liverpool, Liverpool, L7 8TX, UK
| | - Yaochun Shen
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK.
| |
Collapse
|
9
|
Salem M, Elkaseer A, El-Maddah IAM, Youssef KY, Scholz SG, Mohamed HK. Non-Invasive Data Acquisition and IoT Solution for Human Vital Signs Monitoring: Applications, Limitations and Future Prospects. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22176625. [PMID: 36081081 PMCID: PMC9460364 DOI: 10.3390/s22176625] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/26/2022] [Revised: 08/22/2022] [Accepted: 08/30/2022] [Indexed: 05/06/2023]
Abstract
The rapid development of technology has brought about a revolution in healthcare stimulating a wide range of smart and autonomous applications in homes, clinics, surgeries and hospitals. Smart healthcare opens the opportunity for a qualitative advance in the relations between healthcare providers and end-users for the provision of healthcare such as enabling doctors to diagnose remotely while optimizing the accuracy of the diagnosis and maximizing the benefits of treatment by enabling close patient monitoring. This paper presents a comprehensive review of non-invasive vital data acquisition and the Internet of Things in healthcare informatics and thus reports the challenges in healthcare informatics and suggests future work that would lead to solutions to address the open challenges in IoT and non-invasive vital data acquisition. In particular, the conducted review has revealed that there has been a daunting challenge in the development of multi-frequency vital IoT systems, and addressing this issue will help enable the vital IoT node to be reachable by the broker in multiple area ranges. Furthermore, the utilization of multi-camera systems has proven its high potential to increase the accuracy of vital data acquisition, but the implementation of such systems has not been fully developed with unfilled gaps to be bridged. Moreover, the application of deep learning to the real-time analysis of vital data on the node/edge side will enable optimal, instant offline decision making. Finally, the synergistic integration of reliable power management and energy harvesting systems into non-invasive data acquisition has been omitted so far, and the successful implementation of such systems will lead to a smart, robust, sustainable and self-powered healthcare system.
Collapse
Affiliation(s)
- Mahmoud Salem
- Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology, 76344 Eggenstein-Leopoldshafen, Germany
- Correspondence: ; Tel.: +49-0-721-608-25632
| | - Ahmed Elkaseer
- Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology, 76344 Eggenstein-Leopoldshafen, Germany
- Karlsruhe Nano Micro Facility, Karlsruhe Institute of Technology, 76344 Eggenstein-Leopoldshafen, Germany
- Faculty of Engineering, Port Said University, Port Said 42526, Egypt
| | | | - Khaled Y. Youssef
- Faculty of Navigation Science and Space Technology, Beni-Suef University, Beni-Suef 2731070, Egypt
| | - Steffen G. Scholz
- Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology, 76344 Eggenstein-Leopoldshafen, Germany
- Karlsruhe Nano Micro Facility, Karlsruhe Institute of Technology, 76344 Eggenstein-Leopoldshafen, Germany
- College of Engineering, Swansea University, Swansea SA2 8PP, UK
| | - Hoda K. Mohamed
- Faculty of Engineering, Ain Shams University, Cairo 11535, Egypt
| |
Collapse
|
10
|
Lyra S, Rixen J, Heimann K, Karthik S, Joseph J, Jayaraman K, Orlikowsky T, Sivaprakasam M, Leonhardt S, Hoog Antink C. Camera fusion for real-time temperature monitoring of neonates using deep learning. Med Biol Eng Comput 2022; 60:1787-1800. [PMID: 35505175 PMCID: PMC9079037 DOI: 10.1007/s11517-022-02561-9] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2021] [Accepted: 03/25/2022] [Indexed: 11/23/2022]
Abstract
Abstract The continuous monitoring of vital signs is a crucial aspect of medical care in neonatal intensive care units. Since cable-based sensors pose a potential risk for the immature skin of preterm infants, unobtrusive monitoring techniques using camera systems are increasingly investigated. The combination of deep learning–based algorithms and camera modalities such as RGB and infrared thermography can improve the development of cable-free methods for the extraction of vital parameters. In this study, a real-time approach for local extraction of temperatures on the body surface of neonates using a multi-modal clinical dataset was implemented. Therefore, a trained deep learning–based keypoint detector was used for body landmark prediction in RGB. Image registration was conducted to transfer the RGB points to the corresponding thermographic recordings. These landmarks were used to extract the body surface temperature in various regions to determine the central-peripheral temperature difference. A validation of the keypoint detector showed a mean average precision of 0.82. The registration resulted in mean absolute errors of 16.4 px (8.2 mm) for x and 22.4 px (11.2 mm) for y. The evaluation of the temperature extraction revealed a mean absolute error of 0.55 \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^{\circ }$$\end{document}∘C. A final performance of 31 fps was observed on the NVIDIA Jetson Xavier NX module, which proves real-time capability on an embedded GPU system. As a result, the approach can perform real-time temperature extraction on a low-cost GPU module. Graphical abstract ![]()
Collapse
|
11
|
Selvaraju V, Spicher N, Wang J, Ganapathy N, Warnecke JM, Leonhardt S, Swaminathan R, Deserno TM. Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review. SENSORS (BASEL, SWITZERLAND) 2022; 22:4097. [PMID: 35684717 PMCID: PMC9185528 DOI: 10.3390/s22114097] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Revised: 05/18/2022] [Accepted: 05/18/2022] [Indexed: 02/04/2023]
Abstract
In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO2). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO2, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO2 are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO2 is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO2 monitoring.
Collapse
Affiliation(s)
- Vinothini Selvaraju
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
- Non-Invasive Imaging and Diagnostic Laboratory, Biomedical Engineering, Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India;
| | - Nicolai Spicher
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Ju Wang
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Nagarajan Ganapathy
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Joana M. Warnecke
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Steffen Leonhardt
- Chair for Medical Information Technology, Helmholtz-Institute for Biomedical Engineering, RWTH Aachen University, D-52074 Aachen, Germany;
| | - Ramakrishnan Swaminathan
- Non-Invasive Imaging and Diagnostic Laboratory, Biomedical Engineering, Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India;
| | - Thomas M. Deserno
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| |
Collapse
|
12
|
Davoudi A, Shickel B, Tighe PJ, Bihorac A, Rashidi P. Potentials and Challenges of Pervasive Sensing in the Intensive Care Unit. Front Digit Health 2022; 4:773387. [PMID: 35656333 PMCID: PMC9152012 DOI: 10.3389/fdgth.2022.773387] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2021] [Accepted: 04/08/2022] [Indexed: 11/13/2022] Open
Abstract
Patients in critical care settings often require continuous and multifaceted monitoring. However, current clinical monitoring practices fail to capture important functional and behavioral indices such as mobility or agitation. Recent advances in non-invasive sensing technology, high throughput computing, and deep learning techniques are expected to transform the existing patient monitoring paradigm by enabling and streamlining granular and continuous monitoring of these crucial critical care measures. In this review, we highlight current approaches to pervasive sensing in critical care and identify limitations, future challenges, and opportunities in this emerging field.
Collapse
Affiliation(s)
- Anis Davoudi
- Department of Biomedical Engineering, University of Florida, Gainesville, FL, United States,*Correspondence: Anis Davoudi
| | - Benjamin Shickel
- Department of Medicine, University of Florida, Gainesville, FL, United States
| | - Patrick James Tighe
- Department of Anesthesiology, University of Florida, Gainesville, FL, United States
| | - Azra Bihorac
- Department of Medicine, University of Florida, Gainesville, FL, United States
| | - Parisa Rashidi
- Department of Biomedical Engineering, University of Florida, Gainesville, FL, United States
| |
Collapse
|
13
|
Wu CL, Liu SF, Yu TL, Shih SJ, Chang CH, Yang Mao SF, Li YS, Chen HJ, Chen CC, Chao WC. Deep Learning-Based Pain Classifier Based on the Facial Expression in Critically Ill Patients. Front Med (Lausanne) 2022; 9:851690. [PMID: 35372435 PMCID: PMC8968070 DOI: 10.3389/fmed.2022.851690] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Accepted: 02/17/2022] [Indexed: 11/13/2022] Open
Abstract
ObjectivePain assessment based on facial expressions is an essential issue in critically ill patients, but an automated assessment tool is still lacking. We conducted this prospective study to establish the deep learning-based pain classifier based on facial expressions.MethodsWe enrolled critically ill patients during 2020–2021 at a tertiary hospital in central Taiwan and recorded video clips with labeled pain scores based on facial expressions, such as relaxed (0), tense (1), and grimacing (2). We established both image- and video-based pain classifiers through using convolutional neural network (CNN) models, such as Resnet34, VGG16, and InceptionV1 and bidirectional long short-term memory networks (BiLSTM). The performance of classifiers in the test dataset was determined by accuracy, sensitivity, and F1-score.ResultsA total of 63 participants with 746 video clips were eligible for analysis. The accuracy of using Resnet34 in the polychromous image-based classifier for pain scores 0, 1, 2 was merely 0.5589, and the accuracy of dichotomous pain classifiers between 0 vs. 1/2 and 0 vs. 2 were 0.7668 and 0.8593, respectively. Similar accuracy of image-based pain classifier was found using VGG16 and InceptionV1. The accuracy of the video-based pain classifier to classify 0 vs. 1/2 and 0 vs. 2 was approximately 0.81 and 0.88, respectively. We further tested the performance of established classifiers without reference, mimicking clinical scenarios with a new patient, and found the performance remained high.ConclusionsThe present study demonstrates the practical application of deep learning-based automated pain assessment in critically ill patients, and more studies are warranted to validate our findings.
Collapse
Affiliation(s)
- Chieh-Liang Wu
- Department of Critical Care Medicine, Taichung Veterans General Hospital, Taichung, Taiwan
- Department of Industrial Engineering and Enterprise Information, Tunghai University, Taichung, Taiwan
- Artificial Intelligence Studio, Taichung Veterans General Hospital, Taichung, Taiwan
- Colledge of Medicine, National Chung Hsing University, Taichung, Taiwan
| | - Shu-Fang Liu
- Department of Nursing, Taichung Veterans General Hospital, Taichung, Taiwan
| | - Tian-Li Yu
- Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
| | - Sou-Jen Shih
- Department of Nursing, Taichung Veterans General Hospital, Taichung, Taiwan
| | - Chih-Hung Chang
- Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
| | - Shih-Fang Yang Mao
- Electronic and Optoelectronic System Research Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan
| | - Yueh-Se Li
- Electronic and Optoelectronic System Research Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan
| | - Hui-Jiun Chen
- Department of Nursing, Taichung Veterans General Hospital, Taichung, Taiwan
| | - Chia-Chen Chen
- Electronic and Optoelectronic System Research Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan
- *Correspondence: Chia-Chen Chen
| | - Wen-Cheng Chao
- Department of Critical Care Medicine, Taichung Veterans General Hospital, Taichung, Taiwan
- Colledge of Medicine, National Chung Hsing University, Taichung, Taiwan
- Department of Automatic Control Engineering, Feng Chia University, Taichung, Taiwan
- Big Data Center, National Chung Hsing University, Taichung, Taiwan
- Wen-Cheng Chao
| |
Collapse
|
14
|
A Setup for Camera-Based Detection of Simulated Pathological States Using a Neonatal Phantom. SENSORS 2022; 22:s22030957. [PMID: 35161702 PMCID: PMC8838518 DOI: 10.3390/s22030957] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 01/21/2022] [Accepted: 01/25/2022] [Indexed: 11/17/2022]
Abstract
Premature infants are among the most vulnerable patients in a hospital. Due to numerous complications associated with immaturity, a continuous monitoring of vital signs with a high sensitivity and accuracy is required. Today, wired sensors are attached to the patient's skin. However, adhesive electrodes can be potentially harmful as they can damage the very thin immature skin. Although unobtrusive monitoring systems using cameras show the potential to replace cable-based techniques, advanced image processing algorithms are data-driven and, therefore, need much data to be trained. Due to the low availability of public neonatal image data, a patient phantom could help to implement algorithms for the robust extraction of vital signs from video recordings. In this work, a camera-based system is presented and validated using a neonatal phantom, which enabled a simulation of common neonatal pathologies such as hypo-/hyperthermia and brady-/tachycardia. The implemented algorithm was able to continuously measure and analyze the heart rate via photoplethysmography imaging with a mean absolute error of 0.91 bpm, as well as the distribution of a neonate's skin temperature with a mean absolute error of less than 0.55 °C. For accurate measurements, a temperature gain offset correction on the registered image from two infrared thermography cameras was performed. A deep learning-based keypoint detector was applied for temperature mapping and guidance for the feature extraction. The presented setup successfully detected several levels of hypo- and hyperthermia, an increased central-peripheral temperature difference, tachycardia and bradycardia.
Collapse
|
15
|
Jung H, Kimball JP, Receveur T, Gazi AH, Agdeppa ED, Inan OT. Estimation of Tidal Volume Using Load Cells on a Hospital Bed. IEEE J Biomed Health Inform 2022; 26:3330-3341. [PMID: 34995200 DOI: 10.1109/jbhi.2022.3141209] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Although respiratory failure is one of the primary causes of admission to intensive care, the importance placed on measurement of respiratory parameters is commonly overshadowed compared to cardiac parameters. With the increased demand for unobtrusive yet quantifi- able respiratory monitoring, many technologies have been proposed recently. However, there are challenges to be addressed for such technologies to enable widespread use. In this work, we explore the feasibility of using load cell sensors embedded on a hospital bed for monitoring respi- ratory rate (RR) and tidal volume (TV). We propose a globalized machine learning (ML)-based algorithm for estimating TV without the requirement of subject-specific calibration or training. In a study of 15 healthy subjects performing respiratory tasks in four different postures, the outputs from four load cell channels and the reference spirometer were recorded simultaneously. A signal processing pipeline was implemented to extract features that capture respira- tory movement and the respiratory effects on the cardiac (i.e., ballistocardiogram, BCG) signals. The proposed RR estimation algorithm achieved a root mean square error (RMSE) of 0.6 breaths per minute (brpm) against the ground truth RR from the spirometer. The TV estimation results demonstrated that combining all three axes of the low- frequency force signals and the BCG heartbeat features best quantifies the respiratory effects of TV. The model resulted in a correlation and RMSE between the estimated and true TV values of 0.85 and 0.23 L, respectively, in the posture independent model without electrocardiogram (ECG) signals. This study suggests that load cell sensors already existing in certain hospital beds can be used for convenient and continuous respiratory monitoring in general care settings.
Collapse
|
16
|
Asano H, Hirakawa E, Hayashi H, Hamada K, Asayama Y, Oohashi M, Uchiyama A, Higashino T. A method for improving semantic segmentation using thermographic images in infants. BMC Med Imaging 2022; 22:1. [PMID: 34979965 PMCID: PMC8721998 DOI: 10.1186/s12880-021-00730-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2021] [Accepted: 12/23/2021] [Indexed: 12/20/2022] Open
Abstract
Background Regulation of temperature is clinically important in the care of neonates because it has a significant impact on prognosis. Although probes that make contact with the skin are widely used to monitor temperature and provide spot central and peripheral temperature information, they do not provide details of the temperature distribution around the body. Although it is possible to obtain detailed temperature distributions using multiple probes, this is not clinically practical. Thermographic techniques have been reported for measurement of temperature distribution in infants. However, as these methods require manual selection of the regions of interest (ROIs), they are not suitable for introduction into clinical settings in hospitals. Here, we describe a method for segmentation of thermal images that enables continuous quantitative contactless monitoring of the temperature distribution over the whole body of neonates. Methods The semantic segmentation method, U-Net, was applied to thermal images of infants. The optimal combination of Weight Normalization, Group Normalization, and Flexible Rectified Linear Unit (FReLU) was evaluated. U-Net Generative Adversarial Network (U-Net GAN) was applied to thermal images, and a Self-Attention (SA) module was finally applied to U-Net GAN (U-Net GAN + SA) to improve precision. The semantic segmentation performance of these methods was evaluated. Results The optimal semantic segmentation performance was obtained with application of FReLU and Group Normalization to U-Net, showing accuracy of 92.9% and Mean Intersection over Union (mIoU) of 64.5%. U-Net GAN improved the performance, yielding accuracy of 93.3% and mIoU of 66.9%, and U-Net GAN + SA showed further improvement with accuracy of 93.5% and mIoU of 70.4%. Conclusions FReLU and Group Normalization are appropriate semantic segmentation methods for application to neonatal thermal images. U-Net GAN and U-Net GAN + SA significantly improved the mIoU of segmentation.
Collapse
Affiliation(s)
- Hidetsugu Asano
- Technical Department, Atom Medical Corporation, 2-2-1, Dojo, Sakura-ku, Saitama city, Saitama, 338-0835, Japan.
| | - Eiji Hirakawa
- Department of Neonatology, Nagasaki Harbor Medical Center, 6-39, Shinchi-machi, Nagasaki City, Nagasaki, 850-8555, Japan.,Department of Neonatology, Kagoshima City Hospital, 37-1 Uearata-cho, Kagoshima City, Kagoshima, 890-8760, Japan
| | - Hayato Hayashi
- Technical Department, Atom Medical Corporation, 2-2-1, Dojo, Sakura-ku, Saitama city, Saitama, 338-0835, Japan
| | - Keisuke Hamada
- Department of Clinical Engineering, Nagasaki Harbor Medical Center, 6-39, Shinchi-machi, Nagasaki City, Nagasaki, 850-8555, Japan.,Department of Comprehensive Community Care Education, Nagasaki University Graduate School of Biomedical Sciences, 1-14, Bunkyo-machi, Nagasaki City, Nagasaki, 852-8521, Japan
| | - Yuto Asayama
- Technical Department, Atom Medical Corporation, 2-2-1, Dojo, Sakura-ku, Saitama city, Saitama, 338-0835, Japan
| | - Masaaki Oohashi
- Technical Department, Atom Medical Corporation, 2-2-1, Dojo, Sakura-ku, Saitama city, Saitama, 338-0835, Japan
| | - Akira Uchiyama
- Mobile Computing Laboratory, Graduate School of Information Science and Technology, Osaka University, 1-5, Yamadaoka, Suita, Osaka, 565-0871, Japan
| | - Teruo Higashino
- Mobile Computing Laboratory, Graduate School of Information Science and Technology, Osaka University, 1-5, Yamadaoka, Suita, Osaka, 565-0871, Japan
| |
Collapse
|
17
|
Manullang MCT, Lin YH, Lai SJ, Chou NK. Implementation of Thermal Camera for Non-Contact Physiological Measurement: A Systematic Review. SENSORS (BASEL, SWITZERLAND) 2021; 21:7777. [PMID: 34883780 PMCID: PMC8659982 DOI: 10.3390/s21237777] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Revised: 11/06/2021] [Accepted: 11/19/2021] [Indexed: 01/03/2023]
Abstract
Non-contact physiological measurements based on image sensors have developed rapidly in recent years. Among them, thermal cameras have the advantage of measuring temperature in the environment without light and have potential to develop physiological measurement applications. Various studies have used thermal camera to measure the physiological signals such as respiratory rate, heart rate, and body temperature. In this paper, we provided a general overview of the existing studies by examining the physiological signals of measurement, the used platforms, the thermal camera models and specifications, the use of camera fusion, the image and signal processing step (including the algorithms and tools used), and the performance evaluation. The advantages and challenges of thermal camera-based physiological measurement were also discussed. Several suggestions and prospects such as healthcare applications, machine learning, multi-parameter, and image fusion, have been proposed to improve the physiological measurement of thermal camera in the future.
Collapse
Affiliation(s)
- Martin Clinton Tosima Manullang
- Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei 10607, Taiwan; (M.C.T.M.); (S.-J.L.)
- Department of Informatics, Institut Teknologi Sumatera, South Lampung Regency 35365, Indonesia
| | - Yuan-Hsiang Lin
- Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei 10607, Taiwan; (M.C.T.M.); (S.-J.L.)
| | - Sheng-Jie Lai
- Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei 10607, Taiwan; (M.C.T.M.); (S.-J.L.)
| | - Nai-Kuan Chou
- Department of Cardiovascular Surgery, National Taiwan University Hospital, Taipei 10002, Taiwan
| |
Collapse
|
18
|
Ward RJ, Mark Jjunju FP, Kabenge I, Wanyenze R, Griffith EJ, Banadda N, Taylor S, Marshall A. FluNet: An AI-Enabled Influenza-Like Warning System. IEEE SENSORS JOURNAL 2021; 21:24740-24748. [PMID: 35582344 PMCID: PMC8864938 DOI: 10.1109/jsen.2021.3113467] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Revised: 09/05/2021] [Accepted: 09/09/2021] [Indexed: 06/15/2023]
Abstract
Influenza is an acute viral respiratory disease that is currently causing severe financial and resource strains worldwide. With the COVID-19 pandemic exceeding 153 million cases worldwide, there is a need for a low-cost and contactless surveillance system to detect symptomatic individuals. The objective of this study was to develop FluNet, a novel, proof-of-concept, low-cost and contactless device for the detection of high-risk individuals. The system conducts face detection in the LWIR with a precision rating of 0.98, a recall of 0.91, an F-score of 0.96, and a mean intersection over union of 0.74 while sequentially taking the temperature trend of faces with a thermal accuracy of ± 1 K. In parallel, determining if someone is coughing by using a custom lightweight deep convolutional neural network with a precision rating of 0.95, a recall of 0.92, an F-score of 0.94 and an AUC of 0.98. We concluded this study by testing the accuracy of the direction of arrival estimation for the cough detection revealing an error of ± 4.78°. If a subject is symptomatic, a photo is taken with a specified region of interest using a visible light camera. Two datasets have been constructed, one for face detection in the LWIR consisting of 250 images of 20 participants' faces at various rotations and coverings, including face masks. The other for the real-time detection of coughs comprised of 40,482 cough / not cough sounds. These findings could be helpful for future low-cost edge computing applications for influenza-like monitoring.
Collapse
Affiliation(s)
- Ryan J. Ward
- Department of Electrical Engineering and ElectronicsUniversity of LiverpoolLiverpoolL69 7ZXU.K.
| | - Fred Paul Mark Jjunju
- Department of Electrical Engineering and ElectronicsUniversity of LiverpoolLiverpoolL69 7ZXU.K.
| | - Isa Kabenge
- Department of Agricultural and Biosystems EngineeringMakerere UniversityKampalaUganda
| | | | - Elias J. Griffith
- Department of Electrical Engineering and ElectronicsUniversity of LiverpoolLiverpoolL69 7ZXU.K.
| | - Noble Banadda
- Department of Agricultural and Biosystems EngineeringMakerere UniversityKampalaUganda
| | - Stephen Taylor
- Department of Electrical Engineering and ElectronicsUniversity of LiverpoolLiverpoolL69 7ZXU.K.
| | - Alan Marshall
- Department of Electrical Engineering and ElectronicsUniversity of LiverpoolLiverpoolL69 7ZXU.K.
| |
Collapse
|
19
|
Lorato I, Stuijk S, Meftah M, Kommers D, Andriessen P, van Pul C, de Haan G. Automatic Separation of Respiratory Flow from Motion in Thermal Videos for Infant Apnea Detection. SENSORS (BASEL, SWITZERLAND) 2021; 21:6306. [PMID: 34577513 PMCID: PMC8472592 DOI: 10.3390/s21186306] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/28/2021] [Revised: 09/16/2021] [Accepted: 09/18/2021] [Indexed: 02/07/2023]
Abstract
Both Respiratory Flow (RF) and Respiratory Motion (RM) are visible in thermal recordings of infants. Monitoring these two signals usually requires landmark detection for the selection of a region of interest. Other approaches combine respiratory signals coming from both RF and RM, obtaining a Mixed Respiratory (MR) signal. The detection and classification of apneas, particularly common in preterm infants with low birth weight, would benefit from monitoring both RF and RM, or MR, signals. Therefore, we propose in this work an automatic RF pixel detector not based on facial/body landmarks. The method is based on the property of RF pixels in thermal videos, which are in areas with a smooth circular gradient. We defined 5 features combined with the use of a bank of Gabor filters that together allow selection of the RF pixels. The algorithm was tested on thermal recordings of 9 infants amounting to a total of 132 min acquired in a neonatal ward. On average the percentage of correctly identified RF pixels was 84%. Obstructive Apneas (OAs) were simulated as a proof of concept to prove the advantage in monitoring the RF signal compared to the MR signal. The sensitivity in the simulated OA detection improved for the RF signal reaching 73% against the 23% of the MR signal. Overall, the method yielded promising results, although the positioning and number of cameras used could be further optimized for optimal RF visibility.
Collapse
Affiliation(s)
- Ilde Lorato
- Department of Electrical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands; (S.S.); (G.d.H.)
| | - Sander Stuijk
- Department of Electrical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands; (S.S.); (G.d.H.)
| | - Mohammed Meftah
- Department of Family Care Solutions, Philips Research, 5656 AE Eindhoven, The Netherlands;
| | - Deedee Kommers
- Department of Neonatology, Máxima Medical Centre, 5504 DB Veldhoven, The Netherlands; (D.K.); (P.A.)
- Department of Applied Physics, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands;
| | - Peter Andriessen
- Department of Neonatology, Máxima Medical Centre, 5504 DB Veldhoven, The Netherlands; (D.K.); (P.A.)
- Department of Applied Physics, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands;
| | - Carola van Pul
- Department of Applied Physics, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands;
- Department of Clinical Physics, Máxima Medical Centre, 5504 DB Veldhoven, The Netherlands
| | - Gerard de Haan
- Department of Electrical Engineering, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands; (S.S.); (G.d.H.)
| |
Collapse
|
20
|
Estimation of Respiratory Rate from Thermography Using Respiratory Likelihood Index. SENSORS 2021; 21:s21134406. [PMID: 34199084 PMCID: PMC8271612 DOI: 10.3390/s21134406] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/31/2021] [Revised: 06/17/2021] [Accepted: 06/23/2021] [Indexed: 11/16/2022]
Abstract
Respiration is a key vital sign used to monitor human health status. Monitoring respiratory rate (RR) under non-contact is particularly important for providing appropriate pre-hospital care in emergencies. We propose an RR estimation system using thermal imaging cameras, which are increasingly being used in the medical field, such as recently during the COVID-19 pandemic. By measuring temperature changes during exhalation and inhalation, we aim to track the respiration of the subject in a supine or seated position in real-time without any physical contact. The proposed method automatically selects the respiration-related regions from the detected facial regions and estimates the respiration rate. Most existing methods rely on signals from nostrils and require close-up or high-resolution images, while our method only requires the facial region to be captured. Facial region is detected using YOLO v3, an object detection model based on deep learning. The detected facial region is divided into subregions. By calculating the respiratory likelihood of each segmented region using the newly proposed index, called the Respiratory Quality Index, the respiratory region is automatically selected and the RR is estimated. An evaluation of the proposed RR estimation method was conducted on seven subjects in their early twenties, with four 15 s measurements being taken. The results showed a mean absolute error of 0.66 bpm. The proposed method can be useful as an RR estimation method.
Collapse
|
21
|
Pérez I, Figueroa M. A Heterogeneous Hardware Accelerator for Image Classification in Embedded Systems. SENSORS 2021; 21:s21082637. [PMID: 33918668 PMCID: PMC8069940 DOI: 10.3390/s21082637] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/29/2021] [Revised: 04/05/2021] [Accepted: 04/05/2021] [Indexed: 11/16/2022]
Abstract
Convolutional neural networks (CNN) have been extensively employed for image classification due to their high accuracy. However, inference is a computationally-intensive process that often requires hardware acceleration to operate in real time. For mobile devices, the power consumption of graphics processors (GPUs) is frequently prohibitive, and field-programmable gate arrays (FPGA) become a solution to perform inference at high speed. Although previous works have implemented CNN inference on FPGAs, their high utilization of on-chip memory and arithmetic resources complicate their application on resource-constrained edge devices. In this paper, we present a scalable, low power, low resource-utilization accelerator architecture for inference on the MobileNet V2 CNN. The architecture uses a heterogeneous system with an embedded processor as the main controller, external memory to store network data, and dedicated hardware implemented on reconfigurable logic with a scalable number of processing elements (PE). Implemented on a XCZU7EV FPGA running at 200 MHz and using four PEs, the accelerator infers with 87% top-5 accuracy and processes an image of 224×224 pixels in 220 ms. It consumes 7.35 W of power and uses less than 30% of the logic and arithmetic resources used by other MobileNet FPGA accelerators.
Collapse
|