1
|
Warnecke JM, Lasenby J, Deserno TM. Robust in-vehicle heartbeat detection using multimodal signal fusion. Sci Rep 2023; 13:20864. [PMID: 38012195 PMCID: PMC10682004 DOI: 10.1038/s41598-023-47484-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2023] [Accepted: 11/14/2023] [Indexed: 11/29/2023] Open
Abstract
A medical check-up during driving enables the early detection of diseases. Heartbeat irregularities indicate possible cardiovascular diseases, which can be determined with continuous health monitoring. Therefore, we develop a redundant sensor system based on electrocardiography (ECG) and photoplethysmography (PPG) sensors attached to the steering wheel, a red, green, and blue (RGB) camera behind the steering wheel. For the video, we integrate the face recognition engine SeetaFace to detect landmarks of face segments continuously. Based on the green channel, we derive colour changes and, subsequently, the heartbeat. We record the ECG, PPG, video, and reference ECG with body electrodes of 19 volunteers during different driving scenarios, each lasting 15 min: city, highway, and countryside. We combine early, signal-based late, and sensor-based late fusion with a hybrid convolutional neural network (CNN) and integrated majority voting to deliver the final heartbeats that we compare to the reference ECG. Based on the measured and the reference heartbeat positions, the usable time was 51.75%, 58.62%, and 55.96% for the driving scenarios city, highway, and countryside, respectively, with the hybrid algorithm and combination of ECG and PPG. In conclusion, the findings suggest that approximately half the driving time can be utilised for in-vehicle heartbeat monitoring.
Collapse
Affiliation(s)
- Joana M Warnecke
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, 38106, Brunswick, Germany.
- Department of Engineering, University of Cambridge, Cambridge, CB2 1PZ, UK.
| | - Joan Lasenby
- Department of Engineering, University of Cambridge, Cambridge, CB2 1PZ, UK
| | - Thomas M Deserno
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, 38106, Brunswick, Germany
| |
Collapse
|
4
|
Mason AE, Hecht FM, Davis SK, Natale JL, Hartogensis W, Damaso N, Claypool KT, Dilchert S, Dasgupta S, Purawat S, Viswanath VK, Klein A, Chowdhary A, Fisher SM, Anglo C, Puldon KY, Veasna D, Prather JG, Pandya LS, Fox LM, Busch M, Giordano C, Mercado BK, Song J, Jaimes R, Baum BS, Telfer BA, Philipson CW, Collins PP, Rao AA, Wang EJ, Bandi RH, Choe BJ, Epel ES, Epstein SK, Krasnoff JB, Lee MB, Lee SW, Lopez GM, Mehta A, Melville LD, Moon TS, Mujica-Parodi LR, Noel KM, Orosco MA, Rideout JM, Robishaw JD, Rodriguez RM, Shah KH, Siegal JH, Gupta A, Altintas I, Smarr BL. Detection of COVID-19 using multimodal data from a wearable device: results from the first TemPredict Study. Sci Rep 2022; 12:3463. [PMID: 35236896 PMCID: PMC8891385 DOI: 10.1038/s41598-022-07314-0] [Citation(s) in RCA: 27] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2021] [Accepted: 02/14/2022] [Indexed: 12/23/2022] Open
Abstract
Early detection of diseases such as COVID-19 could be a critical tool in reducing disease transmission by helping individuals recognize when they should self-isolate, seek testing, and obtain early medical intervention. Consumer wearable devices that continuously measure physiological metrics hold promise as tools for early illness detection. We gathered daily questionnaire data and physiological data using a consumer wearable (Oura Ring) from 63,153 participants, of whom 704 self-reported possible COVID-19 disease. We selected 73 of these 704 participants with reliable confirmation of COVID-19 by PCR testing and high-quality physiological data for algorithm training to identify onset of COVID-19 using machine learning classification. The algorithm identified COVID-19 an average of 2.75 days before participants sought diagnostic testing with a sensitivity of 82% and specificity of 63%. The receiving operating characteristic (ROC) area under the curve (AUC) was 0.819 (95% CI [0.809, 0.830]). Including continuous temperature yielded an AUC 4.9% higher than without this feature. For further validation, we obtained SARS CoV-2 antibody in a subset of participants and identified 10 additional participants who self-reported COVID-19 disease with antibody confirmation. The algorithm had an overall ROC AUC of 0.819 (95% CI [0.809, 0.830]), with a sensitivity of 90% and specificity of 80% in these additional participants. Finally, we observed substantial variation in accuracy based on age and biological sex. Findings highlight the importance of including temperature assessment, using continuous physiological features for alignment, and including diverse populations in algorithm development to optimize accuracy in COVID-19 detection from wearables.
Collapse
Affiliation(s)
- Ashley E Mason
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA.
| | - Frederick M Hecht
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Shakti K Davis
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Joseph L Natale
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
| | - Wendy Hartogensis
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Natalie Damaso
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Kajal T Claypool
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Stephan Dilchert
- Department of Management, Zicklin School of Business, Baruch College, The City University of New York, New York, NY, USA
| | - Subhasis Dasgupta
- San Diego Supercomputer Center, University of California San Diego, San Diego, CA, USA
| | - Shweta Purawat
- San Diego Supercomputer Center, University of California San Diego, San Diego, CA, USA
| | - Varun K Viswanath
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA
| | - Amit Klein
- Department of Bioengineering: Bioinformatics, University of California San Diego, San Diego, CA, USA
| | - Anoushka Chowdhary
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Sarah M Fisher
- Department of Psychology, Drexel University, Pennsylvania, PA, USA
| | - Claudine Anglo
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Karena Y Puldon
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Danou Veasna
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Jenifer G Prather
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Leena S Pandya
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Lindsey M Fox
- Osher Center for Integrative Health, University of California San Francisco, San Francisco, CA, USA
| | - Michael Busch
- Vitalant Research Institute, University of California San Francisco, San Francisco, CA, USA
| | - Casey Giordano
- Department of Psychology, University of Minnesota - Twin Cities, Minneapolis, MN, USA
| | | | - Jining Song
- San Diego Supercomputer Center, University of California San Diego, San Diego, CA, USA
| | - Rafael Jaimes
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Brian S Baum
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Brian A Telfer
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Casandra W Philipson
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Paula P Collins
- MIT Lincoln Laboratory, Massachusetts Institute of Technology, Lexington, MA, USA
| | - Adam A Rao
- School of Medicine, University of California San Francisco, San Francisco, CA, USA
| | - Edward J Wang
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA
| | - Rachel H Bandi
- Department of Anesthesiology, Northwestern McGaw Medical Center, Feinberg School of Medicine, Chicago, IL, USA
| | - Bianca J Choe
- Department of Emergency Medicine, University of California Los Angeles Health, Los Angeles, CA, USA
| | - Elissa S Epel
- Center for Health and Community, University of California San Francisco, San Francisco, CA, USA
| | - Stephen K Epstein
- Department of Emergency Medicine, Beth Israel Deaconess Medical Center Boston, Boston, MA, USA
| | - Joanne B Krasnoff
- Schmidt College of Medicine, Florida Atlantic University, Boca Raton, FL, USA
| | - Marco B Lee
- Department of Neurosurgery, Santa Clara Valley Medical Center, Stanford University, San Jose, CA, USA
| | - Shi-Wen Lee
- Department of Emergency Medicine, Jamaica Hospital Medical Center, Jamaica, NY, USA
| | - Gina M Lopez
- Department of Emergency Medicine, Boston Medical Center, Boston, MA, USA
| | - Arpan Mehta
- Department of Anesthesiology: Pain Management and Perioperative Medicine, University of Miami, Miami, FL, USA
| | - Laura D Melville
- Department of Emergency Medicine, New York Presbyterian Brooklyn Methodist Hospital, Brooklyn, NY, USA
| | - Tiffany S Moon
- Department of Anesthesiology and Pain Management, University of Texas Southwestern, Dallas, TX, USA
| | - Lilianne R Mujica-Parodi
- Department of Biomedical Engineering, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY, USA
| | - Kimberly M Noel
- Stony Brook Medicine, Stony Brook University Renaissance School of Medicine, Stony Brook, NY, USA
| | - Michael A Orosco
- Department of Anesthesia: Perioperative and Pain Medicine, Kaiser Permanente San Diego, San Diego, CA, USA
| | - Jesse M Rideout
- Department of Emergency Medicine, Tufts Medical Center, Boston, MA, USA
| | - Janet D Robishaw
- Schmidt College of Medicine, Florida Atlantic University, Boca Raton, FL, USA
| | - Robert M Rodriguez
- Department of Emergency Medicine, University of California San Francisco, San Francisco, CA, USA
| | - Kaushal H Shah
- Weill Cornell Medical Center, Weill Cornell Medical School, New York, NY, USA
| | - Jonathan H Siegal
- New York Presbyterian Queens, Weill-Cornell Medical College, Queens, NY, USA
| | - Amarnath Gupta
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
- San Diego Supercomputer Center, University of California San Diego, San Diego, CA, USA
| | - Ilkay Altintas
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
- San Diego Supercomputer Center, University of California San Diego, San Diego, CA, USA
| | - Benjamin L Smarr
- Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, CA, USA
- Department of Bioengineering: Bioinformatics, University of California San Diego, San Diego, CA, USA
| |
Collapse
|
8
|
Zhang X, Li H, Lu Z, Yin G. Homology Characteristics of EEG and EMG for Lower Limb Voluntary Movement Intention. Front Neurorobot 2021; 15:642607. [PMID: 34220479 PMCID: PMC8249921 DOI: 10.3389/fnbot.2021.642607] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Accepted: 03/31/2021] [Indexed: 11/13/2022] Open
Abstract
In the field of lower limb exoskeletons, besides its electromechanical system design and control, attention has been paid to realizing the linkage of exoskeleton robots to humans via electroencephalography (EEG) and electromyography (EMG). However, even the state of the art performance of lower limb voluntary movement intention decoding still faces many obstacles. In the following work, focusing on the perspective of the inner mechanism, a homology characteristic of EEG and EMG for lower limb voluntary movement intention was conducted. A mathematical model of EEG and EMG was built based on its mechanism, which consists of a neural mass model (NMM), neuromuscular junction model, EMG generation model, decoding model, and musculoskeletal biomechanical model. The mechanism analysis and simulation results demonstrated that EEG and EMG signals were both excited by the same movement intention with a response time difference. To assess the efficiency of the proposed model, a synchronous acquisition system for EEG and EMG was constructed to analyze the homology and response time difference from EEG and EMG signals in the limb movement intention. An effective method of wavelet coherence was used to analyze the internal correlation between EEG and EMG signals in the same limb movement intention. To further prove the effectiveness of the hypothesis in this paper, six subjects were involved in the experiments. The experimental results demonstrated that there was a strong EEG-EMG coherence at 1 Hz around movement onset, and the phase of EEG was leading the EMG. Both the simulation and experimental results revealed that EEG and EMG are homologous, and the response time of the EEG signals are earlier than EMG signals during the limb movement intention. This work can provide a theoretical basis for the feasibility of EEG-based pre-perception and fusion perception of EEG and EMG in human movement detection.
Collapse
Affiliation(s)
- Xiaodong Zhang
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China.,Shaanxi Key Laboratory of Intelligent Robots, Xi'an Jiaotong University, Xi'an, China
| | - Hanzhe Li
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China
| | - Zhufeng Lu
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China
| | - Gui Yin
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China
| |
Collapse
|
9
|
Serhani MA, T. El Kassabi H, Ismail H, Nujum Navaz A. ECG Monitoring Systems: Review, Architecture, Processes, and Key Challenges. SENSORS (BASEL, SWITZERLAND) 2020; 20:E1796. [PMID: 32213969 PMCID: PMC7147367 DOI: 10.3390/s20061796] [Citation(s) in RCA: 63] [Impact Index Per Article: 15.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/29/2020] [Revised: 03/17/2020] [Accepted: 03/19/2020] [Indexed: 02/01/2023]
Abstract
Health monitoring and its related technologies is an attractive research area. The electrocardiogram (ECG) has always been a popular measurement scheme to assess and diagnose cardiovascular diseases (CVDs). The number of ECG monitoring systems in the literature is expanding exponentially. Hence, it is very hard for researchers and healthcare experts to choose, compare, and evaluate systems that serve their needs and fulfill the monitoring requirements. This accentuates the need for a verified reference guiding the design, classification, and analysis of ECG monitoring systems, serving both researchers and professionals in the field. In this paper, we propose a comprehensive, expert-verified taxonomy of ECG monitoring systems and conduct an extensive, systematic review of the literature. This provides evidence-based support for critically understanding ECG monitoring systems' components, contexts, features, and challenges. Hence, a generic architectural model for ECG monitoring systems is proposed, an extensive analysis of ECG monitoring systems' value chain is conducted, and a thorough review of the relevant literature, classified against the experts' taxonomy, is presented, highlighting challenges and current trends. Finally, we identify key challenges and emphasize the importance of smart monitoring systems that leverage new technologies, including deep learning, artificial intelligence (AI), Big Data and Internet of Things (IoT), to provide efficient, cost-aware, and fully connected monitoring systems.
Collapse
Affiliation(s)
- Mohamed Adel Serhani
- Department of Information Systems and Security, College of Information Technology, UAE University, Al Ain 15551, United Arab Emirates;
| | - Hadeel T. El Kassabi
- Department of Computer Science and Software Engineering, College of Information Technology, UAE University, Al Ain 15551, United Arab Emirates; (H.T.E.K.)
| | - Heba Ismail
- Department of Computer Science and Software Engineering, College of Information Technology, UAE University, Al Ain 15551, United Arab Emirates; (H.T.E.K.)
| | - Alramzana Nujum Navaz
- Department of Information Systems and Security, College of Information Technology, UAE University, Al Ain 15551, United Arab Emirates;
| |
Collapse
|
10
|
Wei HC, Ta N, Hu WR, Xiao MX, Tang XJ, Haryadi B, Liou JJ, Wu HT. Digital Volume Pulse Measured at the Fingertip as an Indicator of Diabetic Peripheral Neuropathy in the Aged and Diabetic. ENTROPY 2019; 21:1229. [PMCID: PMC7514575 DOI: 10.3390/e21121229] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
This study investigated the application of a modified percussion entropy index (PEIPPI) in assessing the complexity of baroreflex sensitivity (BRS) for diabetic peripheral neuropathy prognosis. The index was acquired by comparing the obedience of the fluctuation tendency in the change between the amplitudes of continuous digital volume pulse (DVP) and variations in the peak-to-peak interval (PPI) from a decomposed intrinsic mode function (i.e., IMF6) through ensemble empirical mode decomposition (EEMD). In total, 100 middle-aged subjects were split into 3 groups: healthy subjects (group 1, 48–89 years, n = 34), subjects with type 2 diabetes without peripheral neuropathy within 5 years (group 2, 42–86 years, n = 42, HbA1c ≥ 6.5%), and type 2 diabetic patients with peripheral neuropathy within 5 years (group 3, 37–75 years, n = 24). The results were also found to be very successful at discriminating between PEIPPI values among the three groups (p < 0.017), and indicated significant associations with the anthropometric (i.e., body weight and waist circumference) and serum biochemical (i.e., triglycerides, glycated hemoglobin, and fasting blood glucose) parameters in all subjects (p < 0.05). The present study, which utilized the DVP signals of aged, overweight subjects and diabetic patients, successfully determined the PPI intervals from IMF6 through EEMD. The PEIPPI can provide a prognosis of peripheral neuropathy from diabetic patients within 5 years after photoplethysmography (PPG) measurement.
Collapse
Affiliation(s)
- Hai-Cheng Wei
- School of Electrical and Information Engineering, North Minzu University, No. 204 North Wenchang Street, Yinchuan, Ningxia 750021, China; (H.-C.W.); (N.T.); (W.-R.H.); (M.-X.X.); (J.J.L.)
- Basic Experimental Teaching and Engineering Training Center, North Minzu University, No. 204 North Wenchang Street, Yinchuan, Ningxia 750021, China
| | - Na Ta
- School of Electrical and Information Engineering, North Minzu University, No. 204 North Wenchang Street, Yinchuan, Ningxia 750021, China; (H.-C.W.); (N.T.); (W.-R.H.); (M.-X.X.); (J.J.L.)
| | - Wen-Rui Hu
- School of Electrical and Information Engineering, North Minzu University, No. 204 North Wenchang Street, Yinchuan, Ningxia 750021, China; (H.-C.W.); (N.T.); (W.-R.H.); (M.-X.X.); (J.J.L.)
| | - Ming-Xia Xiao
- School of Electrical and Information Engineering, North Minzu University, No. 204 North Wenchang Street, Yinchuan, Ningxia 750021, China; (H.-C.W.); (N.T.); (W.-R.H.); (M.-X.X.); (J.J.L.)
| | - Xiao-Jing Tang
- School of Science, Ningxia Medical University, No. 1160 Shengli Street, Ningxia 750004, China;
| | - Bagus Haryadi
- Department of Physics, Universitas Ahmad Dahlan, Jendral A. Yani street, Kragilan, Tamanan, Kec. Banguntapan, Bantul, Daerah Istimewa, Yogyakarta 55191, Indonesia;
| | - Juin J. Liou
- School of Electrical and Information Engineering, North Minzu University, No. 204 North Wenchang Street, Yinchuan, Ningxia 750021, China; (H.-C.W.); (N.T.); (W.-R.H.); (M.-X.X.); (J.J.L.)
- College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, China
| | - Hsien-Tsai Wu
- Department of Electrical Engineering, Dong Hwa University, No. 1, Sec. 2, Da Hsueh Rd., Shoufeng, Hualien 97401, Taiwan, China
- Correspondence:
| |
Collapse
|