1
|
Nguyen N, Nguyen L, Li H, Bordallo López M, Álvarez Casado C. Evaluation of video-based rPPG in challenging environments: Artifact mitigation and network resilience. Comput Biol Med 2024; 179:108873. [PMID: 39053334 DOI: 10.1016/j.compbiomed.2024.108873] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2024] [Revised: 07/05/2024] [Accepted: 07/08/2024] [Indexed: 07/27/2024]
Abstract
Video-based remote photoplethysmography (rPPG) has emerged as a promising technology for non-contact vital sign monitoring, especially under controlled conditions. However, the accurate measurement of vital signs in real-world scenarios faces several challenges, including artifacts induced by videocodecs, low-light noise, degradation, low dynamic range, occlusions, and hardware and network constraints. In this article, a systematic and comprehensive investigation of these issues is conducted, measuring their detrimental effects on the quality of rPPG measurements. Additionally, practical strategies are proposed for mitigating these challenges to improve the dependability and resilience of video-based rPPG systems. Methods for effective biosignal recovery in the presence of network limitations are detailed, along with denoising and inpainting techniques aimed at preserving video frame integrity. Compared to previous studies, this paper addresses a broader range of variables and demonstrates improved accuracy across various rPPG methods, emphasizing generalizability for practical applications in diverse scenarios with varying data quality. Extensive evaluations and direct comparisons demonstrate the effectiveness of these approaches in enhancing rPPG measurements under challenging environments, contributing to the development of more reliable and effective remote vital sign monitoring technologies.
Collapse
Affiliation(s)
- Nhi Nguyen
- Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland.
| | - Le Nguyen
- Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland.
| | - Honghan Li
- Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland; Division of Bioengineering, Graduate School of Engineering Science, Osaka University, Osaka, Japan.
| | - Miguel Bordallo López
- Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland; VTT Technical Research Center of Finland Ltd., Oulu, Finland.
| | | |
Collapse
|
2
|
Plain B, Pielage H, Kramer SE, Richter M, Saunders GH, Versfeld NJ, Zekveld AA, Bhuiyan TA. Combining Cardiovascular and Pupil Features Using k-Nearest Neighbor Classifiers to Assess Task Demand, Social Context, and Sentence Accuracy During Listening. Trends Hear 2024; 28:23312165241232551. [PMID: 38549351 PMCID: PMC10981225 DOI: 10.1177/23312165241232551] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2023] [Revised: 01/04/2024] [Accepted: 01/25/2024] [Indexed: 04/01/2024] Open
Abstract
In daily life, both acoustic factors and social context can affect listening effort investment. In laboratory settings, information about listening effort has been deduced from pupil and cardiovascular responses independently. The extent to which these measures can jointly predict listening-related factors is unknown. Here we combined pupil and cardiovascular features to predict acoustic and contextual aspects of speech perception. Data were collected from 29 adults (mean = 64.6 years, SD = 9.2) with hearing loss. Participants performed a speech perception task at two individualized signal-to-noise ratios (corresponding to 50% and 80% of sentences correct) and in two social contexts (the presence and absence of two observers). Seven features were extracted per trial: baseline pupil size, peak pupil dilation, mean pupil dilation, interbeat interval, blood volume pulse amplitude, pre-ejection period and pulse arrival time. These features were used to train k-nearest neighbor classifiers to predict task demand, social context and sentence accuracy. The k-fold cross validation on the group-level data revealed above-chance classification accuracies: task demand, 64.4%; social context, 78.3%; and sentence accuracy, 55.1%. However, classification accuracies diminished when the classifiers were trained and tested on data from different participants. Individually trained classifiers (one per participant) performed better than group-level classifiers: 71.7% (SD = 10.2) for task demand, 88.0% (SD = 7.5) for social context, and 60.0% (SD = 13.1) for sentence accuracy. We demonstrated that classifiers trained on group-level physiological data to predict aspects of speech perception generalized poorly to novel participants. Individually calibrated classifiers hold more promise for future applications.
Collapse
Affiliation(s)
- Bethany Plain
- Otolaryngology Head and Neck Surgery, Ear & Hearing, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, the Netherlands
- Eriksholm Research Centre, Snekkersten, Denmark
| | - Hidde Pielage
- Otolaryngology Head and Neck Surgery, Ear & Hearing, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, the Netherlands
- Eriksholm Research Centre, Snekkersten, Denmark
| | - Sophia E. Kramer
- Otolaryngology Head and Neck Surgery, Ear & Hearing, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, the Netherlands
| | - Michael Richter
- School of Psychology, Liverpool John Moores University, Liverpool, UK
| | - Gabrielle H. Saunders
- Manchester Centre for Audiology and Deafness (ManCAD), University of Manchester, Manchester, UK
| | - Niek J. Versfeld
- Otolaryngology Head and Neck Surgery, Ear & Hearing, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, the Netherlands
| | - Adriana A. Zekveld
- Otolaryngology Head and Neck Surgery, Ear & Hearing, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam UMC, Amsterdam, the Netherlands
| | | |
Collapse
|
3
|
Van Wier MF, Urry E, Lissenberg-Witte BI, Kramer SE. User characteristics associated with use of wrist-worn wearables and physical activity apps by adults with and without impaired speech-in-noise recognition: a cross-sectional analysis. Int J Audiol 2024; 63:49-56. [PMID: 36373621 DOI: 10.1080/14992027.2022.2135031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2021] [Accepted: 10/07/2022] [Indexed: 11/16/2022]
Abstract
OBJECTIVE To study weekly use of smartwatches, fitness watches and physical activity apps among adults with and without impaired speech-in-noise (SIN) recognition, to identify subgroups of users. DESIGN Cross-sectional study. STUDY SAMPLE Adults (aged 28-80 years) with impaired (n = 384) and normal SIN recognition (n = 341) as measured with a web-based digits-in-noise test, from the Netherlands Longitudinal Study on Hearing. Multiple logistic regression analyses were used to study differences and to build an association model. RESULTS Employed adults in both groups are more likely to use each type of fitness technology (all ORs >3.4, all p-values < 0.004). Specific to fitness watch use, adults living with others use it more (OR 2.5, 95%CI 1.1;5.8, p = 0.033) whereas those abstaining from alcohol (OR 0.3, 95%CI 0.1;0.6) or consuming >2 glasses/week (OR 0.4, 95%CI 0.2;0.81, overall p = 0.006) and hearing aid users (OR 0.5, 95%CI 0.2;0.9, p = 0.024) make less use. CONCLUSIONS Subgroups of adults more and less likely to use fitness technology exist, but do not differ between adults with and without impaired SIN recognition. More research is needed to confirm these results and to develop interventions to increase physical activity levels among adults with hearing loss.
Collapse
Affiliation(s)
- Marieke F Van Wier
- Otolaryngology-Head and Neck Surgery, Section Ear & Hearing, Amsterdam UMC location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Quality of Care, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | - Emily Urry
- Research & Development, Sonova AG, Stäfa, Switzerland
| | - Birgit I Lissenberg-Witte
- Epidemiology and Data Science, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Sophia E Kramer
- Otolaryngology-Head and Neck Surgery, Section Ear & Hearing, Amsterdam UMC location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
- Quality of Care, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| |
Collapse
|
4
|
Aliakbaryhosseinabadi S, Keidser G, May T, Dau T, Wendt D, Rotger-Griful S. The Effects of Noise and Simulated Conductive Hearing Loss on Physiological Response Measures During Interactive Conversations. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2023; 66:4009-4024. [PMID: 37625145 DOI: 10.1044/2023_jslhr-23-00063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/27/2023]
Abstract
PURPOSE The purpose of this work was to study the effects of background noise and hearing attenuation associated with earplugs on three physiological measures, assumed to be markers of effort investment and arousal, during interactive communication. METHOD Twelve pairs of older people (average age of 63.2 years) with age-adjusted normal hearing took part in a face-to-face communication to solve a Diapix task. Communication was held in different levels of babble noise (0, 60, and 70 dBA) and with two levels of hearing attenuation (0 and 25 dB) in quiet. The physiological measures obtained included pupil size, heart rate variability, and skin conductance. In addition, subjective ratings of perceived communication success, frustration, and effort were obtained. RESULTS Ratings of perceived success, frustration, and effort confirmed that communication was more difficult in noise and with approximately 25-dB hearing attenuation and suggested that the implemented levels of noise and hearing attenuation resulted in comparable communication difficulties. Background noise at 70 dBA and hearing attenuation both led to an initial increase in pupil size (associated with effort), but only the effect of the background noise was sustained throughout the conversation. The 25-dB hearing attenuation led to a significant decrease of the high-frequency power of heart rate variability and a significant increase of skin conductance level, measured as the average z value of the electrodermal activity amplitude. CONCLUSION This study demonstrated that several physiological measures appear to be viable indicators of changing communication conditions, with pupillometry and cardiovascular as well as electrodermal measures potentially being markers of communication difficulty.
Collapse
Affiliation(s)
- Susan Aliakbaryhosseinabadi
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
- Hearing System Section, Department of Health Technology, Technical University of Denmark, Kongens Lyngby, Denmark
| | - Gitte Keidser
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
- Department of Behavioral Sciences and Learning, Linnaeus Center HEAD, Linköping University, Sweden
| | - Tobias May
- Hearing System Section, Department of Health Technology, Technical University of Denmark, Kongens Lyngby, Denmark
| | - Torsten Dau
- Hearing System Section, Department of Health Technology, Technical University of Denmark, Kongens Lyngby, Denmark
| | - Dorothea Wendt
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
- Hearing System Section, Department of Health Technology, Technical University of Denmark, Kongens Lyngby, Denmark
| | | |
Collapse
|
5
|
Keidser G, Naylor G, Brungart DS, Caduff A, Campos J, Carlile S, Carpenter MG, Grimm G, Hohmann V, Holube I, Launer S, Lunner T, Mehra R, Rapport F, Slaney M, Smeds K. The Quest for Ecological Validity in Hearing Science: What It Is, Why It Matters, and How to Advance It. Ear Hear 2021; 41 Suppl 1:5S-19S. [PMID: 33105255 PMCID: PMC7676618 DOI: 10.1097/aud.0000000000000944] [Citation(s) in RCA: 62] [Impact Index Per Article: 20.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2020] [Accepted: 07/10/2020] [Indexed: 12/03/2022]
Abstract
Ecological validity is a relatively new concept in hearing science. It has been cited as relevant with increasing frequency in publications over the past 20 years, but without any formal conceptual basis or clear motive. The sixth Eriksholm Workshop was convened to develop a deeper understanding of the concept for the purpose of applying it in hearing research in a consistent and productive manner. Inspired by relevant debate within the field of psychology, and taking into account the World Health Organization's International Classification of Functioning, Disability, and Health framework, the attendees at the workshop reached a consensus on the following definition: "In hearing science, ecological validity refers to the degree to which research findings reflect real-life hearing-related function, activity, or participation." Four broad purposes for striving for greater ecological validity in hearing research were determined: A (Understanding) better understanding the role of hearing in everyday life; B (Development) supporting the development of improved procedures and interventions; C (Assessment) facilitating improved methods for assessing and predicting ability to accomplish real-world tasks; and D (Integration and Individualization) enabling more integrated and individualized care. Discussions considered the effects of variables and phenomena commonly present in hearing-related research on the level of ecological validity of outcomes, supported by examples from a few selected outcome domains and for different types of studies. Illustrated with examples, potential strategies were offered for promoting a high level of ecological validity in a study and for how to evaluate the level of ecological validity of a study. Areas in particular that could benefit from more research to advance ecological validity in hearing science include: (1) understanding the processes of hearing and communication in everyday listening situations, and specifically the factors that make listening difficult in everyday situations; (2) developing new test paradigms that include more than one person (e.g., to encompass the interactive nature of everyday communication) and that are integrative of other factors that interact with hearing in real-life function; (3) integrating new and emerging technologies (e.g., virtual reality) with established test methods; and (4) identifying the key variables and phenomena affecting the level of ecological validity to develop verifiable ways to increase ecological validity and derive a set of benchmarks to strive for.
Collapse
Affiliation(s)
- Gitte Keidser
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
- Department of Behavioural Sciences and Learning, Linnaeus Centre HEAD, Linköping University, Linköping, Sweden
| | - Graham Naylor
- Hearing Sciences—Scottish Section, School of Medicine, University of Nottingham, Glasgow, United Kingdom
| | | | - Andreas Caduff
- Applied Physics Department and the Center for Electromagnetic Research and Characterization, The Hebrew University of Jerusalem, Jerusalem, Israel
| | - Jennifer Campos
- KITE—Toronto Rehabilitation Institute, University Health Network, Toronto, Canada
| | - Simon Carlile
- School of Medical Sciences, University of Sydney, Sydney, Australia
- X-The Moonshot Factory, Mountain View, California, USA
| | - Mark G. Carpenter
- School of Kinesiology, University of British Columbia, Vancouver, Canada
| | - Giso Grimm
- Auditory Signal Processing and Cluster of Excellence “Hearing4all”, Department of Medical Physics and Acoustics, University of Oldenburg, Oldenburg, Germany
| | - Volker Hohmann
- Auditory Signal Processing and Cluster of Excellence “Hearing4all”, Department of Medical Physics and Acoustics, University of Oldenburg, Oldenburg, Germany
| | - Inga Holube
- Institute of Hearing Technology and Audiology, Jade University of Applied Sciences, and Cluster of Excellence “Hearing4all”, Oldenburg, Germany
| | - Stefan Launer
- Department of Science and Technology, Sonova AG, Staefa, Switzerland
| | - Thomas Lunner
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
| | - Ravish Mehra
- Facebook Reality Labs Research, Redmond, Washington, DC, USA
| | - Frances Rapport
- Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Malcolm Slaney
- Machine Hearing Group, Google Research, Mountain View, California, USA
| | | |
Collapse
|
6
|
Saunders GH, Vercammen C, Timmer BHB, Singh G, Pelosi A, Meis M, Launer S, Kramer SE, Gagné JP, Bott A. Changing the narrative for hearing health in the broader context of healthy living: a call to action. Int J Audiol 2021; 60:86-91. [PMID: 33794720 DOI: 10.1080/14992027.2021.1905892] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/15/2023]
Abstract
OBJECTIVE To discuss the steps necessary to facilitate hearing health care in the context of well-being and healthy living. DESIGN Common themes among the articles in this special supplement of the International Journal of Audiology were used to identify issues that must be addressed if audiology is to move from being hearing-focussed to taking a holistic perspective of hearing care in the context of healthy aging. These are discussed within the context of other published literature. RESULTS AND CONCLUSIONS Three needs were identified: (i) Increased interdisciplinary education to raise awareness of the interplay between hearing and health. (ii) Increased emphasis on counselling education in audiology programs so that audiologists are equipped with the knowledge, competence and confidence to provide counselling and emotional support to their patients, beyond care. (iii) Redefinition of therapeutic goal setting and hearing outcomes to include aspects of well-being, so that audiologists can capture and patients realise that that good hearing outcomes can have a direct positive impact on a person's quality of life that extends beyond their improved ability to hear. It was emphasised that each of these needs to be considered within the context of the audiologists' scope of practice and audiologists' well-being.
Collapse
Affiliation(s)
- Gabrielle H Saunders
- Manchester Centre for Audiology and Deafness, University of Manchester, Manchester, UK
| | | | - Barbra H B Timmer
- Sonova AG, Stafa, Switzerland.,School of Health and Rehabilitation Sciences, The University of Queensland, St Lucia, Australia
| | - Gurjit Singh
- Phonak Canada, Mississauga, Canada.,Ryerson University, Toronto, Canada.,University of Toronto, Toronto, Canada
| | | | - Marcus Meis
- Hörzentrum Oldenburg gGmbH, Oldenburg, Germany.,Cluster of excellence Hearing4all, University of Oldenburg, Germany
| | - Stefan Launer
- Manchester Centre for Audiology and Deafness, University of Manchester, Manchester, UK.,Sonova AG, Stafa, Switzerland.,School of Health and Rehabilitation Sciences, The University of Queensland, St Lucia, Australia
| | - Sophia E Kramer
- Amsterdam UMC, Vrije Universiteit Amsterdam, Oldenburg, Netherlands.,Amsterdam Public Health Research Institute, Amsterdam, the Netherlands
| | - Jean-Pierre Gagné
- École d'orthophonie et d'audiologie, Université de Montréal, Montréal, Canada.,Fondation Caroline Durand en audition et vieillissement de l'Universtié de Montréal, Montréal, Canada.,Centre de recherche de l'Institut universitaire de gériatrie de Montréal, Montréal, Canada
| | - Anthea Bott
- School of Health and Rehabilitation Sciences, The University of Queensland, St Lucia, Australia.,GN Hearing A/S, Ballerup, Denmark
| |
Collapse
|
7
|
Christensen JH, Saunders GH, Porsbo M, Pontoppidan NH. The everyday acoustic environment and its association with human heart rate: evidence from real-world data logging with hearing aids and wearables. ROYAL SOCIETY OPEN SCIENCE 2021; 8:201345. [PMID: 33972852 PMCID: PMC8074664 DOI: 10.1098/rsos.201345] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/09/2023]
Abstract
We investigate the short-term association between multidimensional acoustic characteristics of everyday ambient sound and continuous mean heart rate. We used in-market data from hearing aid users who logged ambient acoustics via smartphone-connected hearing aids and continuous mean heart rate in 5 min intervals from their own wearables. We find that acoustic characteristics explain approximately 4% of the fluctuation in mean heart rate throughout the day. Specifically, increases in ambient sound pressure intensity are significantly related to increases in mean heart rate, corroborating prior laboratory and short-term real-world data. In addition, increases in ambient sound quality-that is, more favourable signal to noise ratios-are associated with decreases in mean heart rate. Our findings document a previously unrecognized mixed influence of everyday sounds on cardiovascular stress, and that the relationship is more complex than is seen from an examination of sound intensity alone. Thus, our findings highlight the relevance of ambient environmental sound in models of human ecophysiology.
Collapse
Affiliation(s)
| | - Gabrielle H. Saunders
- Manchester Centre for Audiology and Deafness, School of Health Sciences, University of Manchester, Manchester, UK
| | | | | |
Collapse
|
8
|
Editorial: Eriksholm Workshop on Ecologically Valid Assessments of Hearing and Hearing Devices. Ear Hear 2020; 41 Suppl 1:1S-4S. [PMID: 33105254 DOI: 10.1097/aud.0000000000000933] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
9
|
Ecological Momentary Assessment in Hearing Research: Current State, Challenges, and Future Directions. Ear Hear 2020; 41 Suppl 1:79S-90S. [DOI: 10.1097/aud.0000000000000934] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
|
10
|
|