1
|
Alsuhaibani M, Dodge HH, Mahoor MH. Mild cognitive impairment detection from facial video interviews by applying spatial-to-temporal attention module. EXPERT SYSTEMS WITH APPLICATIONS 2024; 252:124185. [PMID: 38881832 PMCID: PMC11174143 DOI: 10.1016/j.eswa.2024.124185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2024]
Abstract
Early detection of Mild Cognitive Impairment (MCI) leads to early interventions to slow the progression from MCI into dementia. Deep Learning (DL) algorithms could help achieve early non-invasive and low-cost detection of MCI. This paper presents the detection of MCI in older adults using DL models based only on facial features extracted from video-recorded conversations at home. We used the data collected from the I-CONECT behavioral intervention study (NCT02871921), where several sessions of semi-structured interviews between socially isolated older individuals and interviewers were video recorded. We develop a framework that extracts holistic spatial facial features using a convolutional autoencoder and temporal information using transformers. We proposed the Spatial-to-Temporal Attention Module (STAM) to detect the I-CONECT study participants' cognitive conditions (MCI vs. those with normal cognition (NC)) using facial and interaction features. The interaction features of the facial features improved the prediction performance compared with applying facial features solely. The detection accuracy using this combined method reached 88%, whereas the accuracy without applying the segments and sequences information of the facial features within a video on a certain theme was 84%. Overall, the results show that spatiotemporal facial features modeled using DL algorithms have a discriminating power for the detection of MCI.
Collapse
Affiliation(s)
- Muath Alsuhaibani
- Department of Electrical and Computer Engineering, University of Denver, Denver 80208, CO, United States
- Department of Electrical Engineering, Prince Sattam Bin Abdulaziz University, Al-Kharj 11942, Saudi Arabia
| | - Hiroko H. Dodge
- Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston 02114, MA, United States
| | - Mohammad H. Mahoor
- Department of Electrical and Computer Engineering, University of Denver, Denver 80208, CO, United States
| |
Collapse
|
2
|
Megari K, Frantzezou CK, Polyzopoulou ZA, Tzouni SK. Neurocognitive features in childhood & adulthood in autism spectrum disorder: A neurodiversity approach. Int J Dev Neurosci 2024; 84:471-499. [PMID: 38953464 DOI: 10.1002/jdn.10356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2024] [Revised: 05/30/2024] [Accepted: 06/10/2024] [Indexed: 07/04/2024] Open
Abstract
OBJECTIVES Autism spectrum disorder (ASD) is a neurodevelopmental disorder with a diverse profile of cognitive functions. Heterogeneity is observed among both baseline and comorbid features concerning the diversity of neuropathology in autism. Symptoms vary depending on the developmental stage, level of severity, or comorbidity with other medical or psychiatric diagnoses such as intellectual disability, epilepsy, and anxiety disorders. METHOD The neurodiversity movement does not face variations in neurological and cognitive development in ASD as deficits but as normal non-pathological human variations. Thus, ASD is not identified as a neurocognitive pathological disorder that deviates from the typical, but as a neuro-individuality, a normal manifestation of a neurobiological variation within the population. RESULTS In this light, neurodiversity is described as equivalent to any other human variation, such as ethnicity, gender, or sexual orientation. This review will provide insights about the neurodiversity approach in children and adults with ASD. Using a neurodiversity approach can be helpful when working with children who have autism spectrum disorder (ASD). DISCUSSION This method acknowledges and values the various ways that people with ASD interact with one another and experience the world in order to embrace the neurodiversity approach when working with children with ASD.
Collapse
Affiliation(s)
- Kalliopi Megari
- Department of Psychology, CITY College, University of York, Europe Campus, Thessaloniki, Greece
| | | | - Zoi A Polyzopoulou
- Department of Psychology, University of Western Macedonia, Florina, Greece
| | - Stella K Tzouni
- Department of Psychology, University of Western Macedonia, Florina, Greece
| |
Collapse
|
3
|
Jiang D, Yan L, Mayrand F. Emotion expressions and cognitive impairments in the elderly: review of the contactless detection approach. Front Digit Health 2024; 6:1335289. [PMID: 39040877 PMCID: PMC11260803 DOI: 10.3389/fdgth.2024.1335289] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Accepted: 06/20/2024] [Indexed: 07/24/2024] Open
Abstract
The aging population in Canada has been increasing continuously throughout the past decades. Amongst this demographic, around 11% suffer from some form of cognitive decline. While diagnosis through traditional means (i.e., Magnetic Resonance Imagings (MRIs), positron emission tomography (PET) scans, cognitive assessments, etc.) has been successful at detecting this decline, there remains unexplored measures of cognitive health that could reduce stress and cost for the elderly population, including approaches for early detection and preventive methods. Such efforts could additionally contribute to reducing the pressure and stress on the Canadian healthcare system, as well as improve the quality of life of the elderly population. Previous evidence has demonstrated emotional facial expressions being altered in individuals with various cognitive conditions such as dementias, mild cognitive impairment, and geriatric depression. This review highlights the commonalities among these cognitive health conditions, and research behind the contactless assessment methods to monitor the health and cognitive well-being of the elderly population through emotion expression. The contactless detection approach covered by this review includes automated facial expression analysis (AFEA), electroencephalogram (EEG) technologies and heart rate variability (HRV). In conclusion, a discussion of the potentials of the existing technologies and future direction of a novel assessment design through fusion of AFEA, EEG and HRV measures to increase detection of cognitive decline in a contactless and remote manner will be presented.
Collapse
Affiliation(s)
- Di Jiang
- Medical Devices Research Centre, National Research Council of Canada, Boucherville, QC, Canada
| | - Luowei Yan
- Department of Psychology, McGill University, Montreal, QC, Canada
| | - Florence Mayrand
- Department of Psychology, McGill University, Montreal, QC, Canada
| |
Collapse
|
4
|
Jiang Z, Seyedi S, Griner E, Abbasi A, Rad AB, Kwon H, Cotes RO, Clifford GD. Evaluating and mitigating unfairness in multimodal remote mental health assessments. PLOS DIGITAL HEALTH 2024; 3:e0000413. [PMID: 39046989 PMCID: PMC11268595 DOI: 10.1371/journal.pdig.0000413] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 06/13/2024] [Indexed: 07/27/2024]
Abstract
Research on automated mental health assessment tools has been growing in recent years, often aiming to address the subjectivity and bias that existed in the current clinical practice of the psychiatric evaluation process. Despite the substantial health and economic ramifications, the potential unfairness of those automated tools was understudied and required more attention. In this work, we systematically evaluated the fairness level in a multimodal remote mental health dataset and an assessment system, where we compared the fairness level in race, gender, education level, and age. Demographic parity ratio (DPR) and equalized odds ratio (EOR) of classifiers using different modalities were compared, along with the F1 scores in different demographic groups. Post-training classifier threshold optimization was employed to mitigate the unfairness. No statistically significant unfairness was found in the composition of the dataset. Varying degrees of unfairness were identified among modalities, with no single modality consistently demonstrating better fairness across all demographic variables. Post-training mitigation effectively improved both DPR and EOR metrics at the expense of a decrease in F1 scores. Addressing and mitigating unfairness in these automated tools are essential steps in fostering trust among clinicians, gaining deeper insights into their use cases, and facilitating their appropriate utilization.
Collapse
Affiliation(s)
- Zifan Jiang
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States of America
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia, United States of America
| | - Salman Seyedi
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States of America
| | - Emily Griner
- Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, Georgia, United States of America
| | - Ahmed Abbasi
- Department of IT, Analytics, and Operations, University of Notre Dame, Notre Dame, Indiana, United States of America
| | - Ali Bahrami Rad
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States of America
| | - Hyeokhyen Kwon
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States of America
| | - Robert O. Cotes
- Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, Georgia, United States of America
| | - Gari D. Clifford
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States of America
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia, United States of America
| |
Collapse
|
5
|
Pereira R, Mendes C, Ribeiro J, Ribeiro R, Miragaia R, Rodrigues N, Costa N, Pereira A. Systematic Review of Emotion Detection with Computer Vision and Deep Learning. SENSORS (BASEL, SWITZERLAND) 2024; 24:3484. [PMID: 38894274 PMCID: PMC11175284 DOI: 10.3390/s24113484] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/29/2024] [Revised: 05/20/2024] [Accepted: 05/24/2024] [Indexed: 06/21/2024]
Abstract
Emotion recognition has become increasingly important in the field of Deep Learning (DL) and computer vision due to its broad applicability by using human-computer interaction (HCI) in areas such as psychology, healthcare, and entertainment. In this paper, we conduct a systematic review of facial and pose emotion recognition using DL and computer vision, analyzing and evaluating 77 papers from different sources under Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our review covers several topics, including the scope and purpose of the studies, the methods employed, and the used datasets. The scope of this work is to conduct a systematic review of facial and pose emotion recognition using DL methods and computer vision. The studies were categorized based on a proposed taxonomy that describes the type of expressions used for emotion detection, the testing environment, the currently relevant DL methods, and the datasets used. The taxonomy of methods in our review includes Convolutional Neural Network (CNN), Faster Region-based Convolutional Neural Network (R-CNN), Vision Transformer (ViT), and "Other NNs", which are the most commonly used models in the analyzed studies, indicating their trendiness in the field. Hybrid and augmented models are not explicitly categorized within this taxonomy, but they are still important to the field. This review offers an understanding of state-of-the-art computer vision algorithms and datasets for emotion recognition through facial expressions and body poses, allowing researchers to understand its fundamental components and trends.
Collapse
Affiliation(s)
- Rafael Pereira
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Carla Mendes
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - José Ribeiro
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Roberto Ribeiro
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Rolando Miragaia
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Nuno Rodrigues
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Nuno Costa
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - António Pereira
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
- INOV INESC Inovação, Institute of New Technologies, Leiria Office, 2411-901 Leiria, Portugal
| |
Collapse
|
6
|
Sun J, Dodge HH, Mahoor MH. MC-ViViT: Multi-branch Classifier-ViViT to Detect Mild Cognitive Impairment in Older Adults Using Facial Videos. EXPERT SYSTEMS WITH APPLICATIONS 2024; 238:121929. [PMID: 39238945 PMCID: PMC11375964 DOI: 10.1016/j.eswa.2023.121929] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/07/2024]
Abstract
Deep machine learning models including Convolutional Neural Networks (CNN) have been successful in the detection of Mild Cognitive Impairment (MCI) using medical images, questionnaires, and videos. This paper proposes a novel Multi-branch Classifier-Video Vision Transformer (MC-ViViT) model to distinguish MCI from those with normal cognition by analyzing facial features. The data comes from the I-CONECT, a behavioral intervention trial aimed at improving cognitive function by providing frequent video chats. MC-ViViT extracts spatiotemporal features of videos in one branch and augments representations by the MC module. The I-CONECT dataset is challenging as the dataset is imbalanced containing Hard-Easy and Positive-Negative samples, which impedes the performance of MC-ViViT. We propose a loss function for Hard-Easy and Positive-Negative Samples (HP Loss) by combining Focal loss and AD-CORRE loss to address the imbalanced problem. Our experimental results on the I-CONECT dataset show the great potential of MC-ViViT in predicting MCI with a high accuracy of 90.63% accuracy on some of the interview videos.
Collapse
Affiliation(s)
- Jian Sun
- Department Of Computer Science, University of Denver, 2155 E Wesley Ave, Denver, Colorado, 80210, United States of America
| | - Hiroko H Dodge
- Department Of Neurology at Harvard Medical School, Harvard University, Massachusetts General Hospital, 55 Fruit St, Boston, Massachusetts, 02114, United States of America
| | - Mohammad H Mahoor
- Department Of Computer Engineering, University of Denver, 2155 E Wesley Ave, Denver, Colorado, 80210, United States of America
| |
Collapse
|
7
|
Jiang Z, Seyedi S, Griner E, Abbasi A, Rad AB, Kwon H, Cotes RO, Clifford GD. Multimodal Mental Health Digital Biomarker Analysis From Remote Interviews Using Facial, Vocal, Linguistic, and Cardiovascular Patterns. IEEE J Biomed Health Inform 2024; 28:1680-1691. [PMID: 38198249 PMCID: PMC10986761 DOI: 10.1109/jbhi.2024.3352075] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2024]
Abstract
OBJECTIVE Psychiatric evaluation suffers from subjectivity and bias, and is hard to scale due to intensive professional training requirements. In this work, we investigated whether behavioral and physiological signals, extracted from tele-video interviews, differ in individuals with psychiatric disorders. METHODS Temporal variations in facial expression, vocal expression, linguistic expression, and cardiovascular modulation were extracted from simultaneously recorded audio and video of remote interviews. Averages, standard deviations, and Markovian process-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination. RESULTS Statistically significant feature differences were found between psychiatric and control subjects. Correlations were found between features and self-rated depression and anxiety scores. Heart rate dynamics provided the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities provided AUROCs of 0.72-0.82. CONCLUSION Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status. SIGNIFICANCE The proposed multimodal approach has the potential to facilitate scalable, remote, and low-cost assessment for low-burden automated mental health services.
Collapse
|
8
|
Bianchini E, Rinaldi D, Alborghetti M, Simonelli M, D’Audino F, Onelli C, Pegolo E, Pontieri FE. The Story behind the Mask: A Narrative Review on Hypomimia in Parkinson's Disease. Brain Sci 2024; 14:109. [PMID: 38275529 PMCID: PMC10814039 DOI: 10.3390/brainsci14010109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2023] [Revised: 01/18/2024] [Accepted: 01/19/2024] [Indexed: 01/27/2024] Open
Abstract
Facial movements are crucial for social and emotional interaction and well-being. Reduced facial expressions (i.e., hypomimia) is a common feature in patients with Parkinson's disease (PD) and previous studies linked this manifestation to both motor symptoms of the disease and altered emotion recognition and processing. Nevertheless, research on facial motor impairment in PD has been rather scarce and only a limited number of clinical evaluation tools are available, often suffering from poor validation processes and high inter- and intra-rater variability. In recent years, the availability of technology-enhanced quantification methods of facial movements, such as automated video analysis and machine learning application, led to increasing interest in studying hypomimia in PD. In this narrative review, we summarize the current knowledge on pathophysiological hypotheses at the basis of hypomimia in PD, with particular focus on the association between reduced facial expressions and emotional processing and analyze the current evaluation tools and management strategies for this symptom, as well as future research perspectives.
Collapse
Affiliation(s)
- Edoardo Bianchini
- Department of Neuroscience, Mental Health and Sensory Organs (NESMOS), Sapienza University of Rome, 00189 Rome, Italy; (E.B.); (D.R.); (M.A.); (M.S.)
- AGEIS, Université Grenoble Alpes, 38000 Grenoble, France
- Sant’Andrea University Hospital, 00189 Rome, Italy;
| | - Domiziana Rinaldi
- Department of Neuroscience, Mental Health and Sensory Organs (NESMOS), Sapienza University of Rome, 00189 Rome, Italy; (E.B.); (D.R.); (M.A.); (M.S.)
- Sant’Andrea University Hospital, 00189 Rome, Italy;
| | - Marika Alborghetti
- Department of Neuroscience, Mental Health and Sensory Organs (NESMOS), Sapienza University of Rome, 00189 Rome, Italy; (E.B.); (D.R.); (M.A.); (M.S.)
- Sant’Andrea University Hospital, 00189 Rome, Italy;
| | - Marta Simonelli
- Department of Neuroscience, Mental Health and Sensory Organs (NESMOS), Sapienza University of Rome, 00189 Rome, Italy; (E.B.); (D.R.); (M.A.); (M.S.)
- Ospedale dei Castelli, ASL Rome 6, 00040 Ariccia, Italy
| | | | - Camilla Onelli
- Department of Molecular Medicine, University of Padova, 35121 Padova, Italy;
| | - Elena Pegolo
- Department of Information Engineering, University of Padova, 35131 Padova, Italy;
| | - Francesco E. Pontieri
- Department of Neuroscience, Mental Health and Sensory Organs (NESMOS), Sapienza University of Rome, 00189 Rome, Italy; (E.B.); (D.R.); (M.A.); (M.S.)
- Sant’Andrea University Hospital, 00189 Rome, Italy;
- Fondazione Santa Lucia IRCCS, 00179 Rome, Italy
| |
Collapse
|
9
|
Zhang C, Lei X, Ma W, Long J, Long S, Chen X, Luo J, Tao Q. Diagnosis Framework for Probable Alzheimer's Disease and Mild Cognitive Impairment Based on Multi-Dimensional Emotion Features. J Alzheimers Dis 2024; 97:1125-1137. [PMID: 38189751 DOI: 10.3233/jad-230703] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2024]
Abstract
BACKGROUND Emotion and cognition are intercorrelated. Impaired emotion is common in populations with Alzheimer's disease (AD) and mild cognitive impairment (MCI), showing promises as an early detection approach. OBJECTIVE We aim to develop a novel automatic classification tool based on emotion features and machine learning. METHODS Older adults aged 60 years or over were recruited among residents in the long-term care facilities and the community. Participants included healthy control participants with normal cognition (HC, n = 26), patients with MCI (n = 23), and patients with probable AD (n = 30). Participants watched emotional film clips while multi-dimensional emotion data were collected, including mental features of Self-Assessment Manikin (SAM), physiological features of electrodermal activity (EDA), and facial expressions. Emotional features of EDA and facial expression were abstracted by using continuous decomposition analysis and EomNet, respectively. Bidirectional long short-term memory (Bi-LSTM) was used to train classification model. Hybrid fusion was used, including early feature fusion and late decision fusion. Data from 79 participants were utilized into deep machine learning analysis and hybrid fusion method. RESULTS By combining multiple emotion features, the model's performance of AUC value was highest in classification between HC and probable AD (AUC = 0.92), intermediate between MCI and probable AD (AUC = 0.88), and lowest between HC and MCI (AUC = 0.82). CONCLUSIONS Our method demonstrated an excellent predictive power to differentiate HC/MCI/AD by fusion of multiple emotion features. The proposed model provides a cost-effective and automated method that can assist in detecting probable AD and MCI from normal aging.
Collapse
Affiliation(s)
- Chunchao Zhang
- Department of Public Health and Preventive Medicine, School of Medicine, Jinan University, Guangzhou, China
- Division of Medical Psychology and Behaviour Science, School of Medicine, Jinan University, Guangzhou, China
| | - Xiaolin Lei
- College of Information Science and Technology, Jinan University, Guangzhou, China
| | - Wenhao Ma
- Department of Public Health and Preventive Medicine, School of Medicine, Jinan University, Guangzhou, China
- Division of Medical Psychology and Behaviour Science, School of Medicine, Jinan University, Guangzhou, China
| | - Jinyi Long
- College of Information Science and Technology, Jinan University, Guangzhou, China
| | - Shun Long
- College of Information Science and Technology, Jinan University, Guangzhou, China
| | - Xiang Chen
- Rehabilitation Medicine, Second Affiliated Hospital of Nanchang University, Nanchang, China
| | - Jun Luo
- Rehabilitation Medicine, Second Affiliated Hospital of Nanchang University, Nanchang, China
| | - Qian Tao
- Department of Public Health and Preventive Medicine, School of Medicine, Jinan University, Guangzhou, China
- Division of Medical Psychology and Behaviour Science, School of Medicine, Jinan University, Guangzhou, China
- Neuroscience and Neurorehabilitation Institute, University of Health and Rehabilitation Science, Qingdao, China
| |
Collapse
|
10
|
Sjaelland NS, Gramkow MH, Hasselbalch SG, Frederiksen KS. Digital Biomarkers for the Assessment of Non-Cognitive Symptoms in Patients with Dementia with Lewy Bodies: A Systematic Review. J Alzheimers Dis 2024; 100:431-451. [PMID: 38943394 PMCID: PMC11307079 DOI: 10.3233/jad-240327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/14/2024] [Indexed: 07/01/2024]
Abstract
Background Portable digital health technologies (DHTs) could help evaluate non-cognitive symptoms, but evidence to support their use in patients with dementia with Lewy bodies (DLB) is uncertain. Objective 1) To describe portable or wearable DHTs used to obtain digital biomarkers in patients with DLB, 2) to assess the digital biomarkers' ability to evaluate non-cognitive symptoms, and 3) to assess the feasibility of applying digital biomarkers in patients with DLB. Methods We systematically searched databases MEDLINE, Embase, and Web of Science from inception through February 28, 2023. Studies assessing digital biomarkers obtained by portable or wearable DHTs and related to non-cognitive symptoms were eligible if including patients with DLB. The quality of studies was assessed using a modified check list based on the NIH Quality assessment tool for Observational Cohort and Cross-sectional Studies. A narrative synthesis of data was carried out. Results We screened 4,295 records and included 20 studies. Seventeen different DHTs were identified for assessment of most non-cognitive symptoms related to DLB. No thorough validation of digital biomarkers for measurement of non-cognitive symptoms in DLB was reported. Studies did not report on aspects of feasibility in a systematic way. Conclusions Knowledge about feasibility and validity of individual digital biomarkers remains extremely limited. Study heterogeneity is a barrier for establishing a broad evidence base for application of digital biomarkers in DLB. Researchers should conform to recommended standards for systematic evaluation of digital biomarkers.
Collapse
Affiliation(s)
- Nikolai S. Sjaelland
- Danish Dementia Research Centre, Department of Neurology, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
| | - Mathias H. Gramkow
- Danish Dementia Research Centre, Department of Neurology, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
| | - Steen G. Hasselbalch
- Danish Dementia Research Centre, Department of Neurology, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Kristian Steen Frederiksen
- Danish Dementia Research Centre, Department of Neurology, Copenhagen University Hospital - Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
11
|
Jiang Z, Seyedi S, Griner E, Abbasi A, Bahrami Rad A, Kwon H, Cotes RO, Clifford GD. Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.09.11.23295212. [PMID: 37745610 PMCID: PMC10516063 DOI: 10.1101/2023.09.11.23295212] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/26/2023]
Abstract
Objective The current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders. Methods Time series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination. Results Statistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models. Conclusion Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status. Significance The proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.
Collapse
|
12
|
Zheng C, Bouazizi M, Ohtsuki T, Kitazawa M, Horigome T, Kishimoto T. Detecting Dementia from Face-Related Features with Automated Computational Methods. Bioengineering (Basel) 2023; 10:862. [PMID: 37508889 PMCID: PMC10376259 DOI: 10.3390/bioengineering10070862] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Revised: 07/13/2023] [Accepted: 07/14/2023] [Indexed: 07/30/2023] Open
Abstract
Alzheimer's disease (AD) is a type of dementia that is more likely to occur as people age. It currently has no known cure. As the world's population is aging quickly, early screening for AD has become increasingly important. Traditional screening methods such as brain scans or psychiatric tests are stressful and costly. The patients are likely to feel reluctant to such screenings and fail to receive timely intervention. While researchers have been exploring the use of language in dementia detection, less attention has been given to face-related features. The paper focuses on investigating how face-related features can aid in detecting dementia by exploring the PROMPT dataset that contains video data collected from patients with dementia during interviews. In this work, we extracted three types of features from the videos, including face mesh, Histogram of Oriented Gradients (HOG) features, and Action Units (AU). We trained traditional machine learning models and deep learning models on the extracted features and investigated their effectiveness in dementia detection. Our experiments show that the use of HOG features achieved the highest accuracy of 79% in dementia detection, followed by AU features with 71% accuracy, and face mesh features with 66% accuracy. Our results show that face-related features have the potential to be a crucial indicator in automated computational dementia detection.
Collapse
Affiliation(s)
- Chuheng Zheng
- Graduate School of Science and Technology, Keio University, Yokohama 223-0061, Kanagawa, Japan
| | - Mondher Bouazizi
- Faculty of Science and Technology, Keio University, Yokohama 223-0061, Kanagawa, Japan
| | - Tomoaki Ohtsuki
- Faculty of Science and Technology, Keio University, Yokohama 223-0061, Kanagawa, Japan
| | - Momoko Kitazawa
- School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo 160-8582, Japan
| | - Toshiro Horigome
- School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo 160-8582, Japan
| | - Taishiro Kishimoto
- School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo 160-8582, Japan
| |
Collapse
|
13
|
Jiang Z, Seyedi S, Vickers KL, Manzanares CM, Lah JJ, Levey AI, Clifford GD. Disentangling visual exploration differences in cognitive impairment. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.05.17.23290054. [PMID: 37292683 PMCID: PMC10246124 DOI: 10.1101/2023.05.17.23290054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Objective Compared to individuals without cognitive impairment (CI), those with CI exhibit differences in both basic oculomotor functions and complex viewing behaviors. However, the characteristics of the differences and how those differences relate to various cognitive functions have not been widely explored. In this work we aimed to quantify those differences and assess general cognitive impairment and specific cognitive functions. Methods A validated passive viewing memory test with eyetracking was administered to 348 healthy controls and CI individuals. Spatial, temporal, semantic, and other composite features were extracted from the estimated eye-gaze locations on the corresponding pictures displayed during the test. These features were then used to characterize viewing patterns, classify cognitive impairment, and estimate scores in various neuropsychological tests using machine learning. Results Statistically significant differences in spatial, spatiotemporal, and semantic features were found between healthy controls and individuals with CI. CI group spent more time gazing at the center of the image, looked at more regions of interest (ROI), transitioned less often between ROI yet in a more unpredictable manner, and had different semantic preferences. A combination of these features achieved an area under the receiver-operator curve of 0.78 in differentiating CI individuals from controls. Statistically significant correlations were identified between actual and estimated MoCA scores and other neuropsychological tests. Conclusion Evaluating visual exploration behaviors provided quantitative and systematic evidence of differences in CI individuals, leading to an improved approach for passive cognitive impairment screening. Significance The proposed passive, accessible, and scalable approach could help with earlier detection and a better understanding of cognitive impairment.
Collapse
|
14
|
Sirilertmekasakul C, Rattanawong W, Gongvatana A, Srikiatkhachorn A. The current state of artificial intelligence-augmented digitized neurocognitive screening test. Front Hum Neurosci 2023; 17:1133632. [PMID: 37063100 PMCID: PMC10098088 DOI: 10.3389/fnhum.2023.1133632] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2022] [Accepted: 03/20/2023] [Indexed: 04/18/2023] Open
Abstract
The cognitive screening test is a brief cognitive examination that could be easily performed in a clinical setting. However, one of the main drawbacks of this test was that only a paper-based version was available, which restricts the test to be manually administered and graded by medical personnel at the health centers. The main solution to these problems was to develop a potential remote assessment for screening individuals with cognitive impairment. Currently, multiple studies have been adopting artificial intelligence (AI) technology into these tests, evolving the conventional paper-based neurocognitive test into a digitized AI-assisted neurocognitive test. These studies provided credible evidence of the potential of AI-augmented cognitive screening tests to be better and provided the framework for future studies to further improve the implementation of AI technology in the cognitive screening test. The objective of this review article is to discuss different types of AI used in digitized cognitive screening tests and their advantages and disadvantages.
Collapse
|
15
|
Cotes RO, Boazak M, Griner E, Jiang Z, Kim B, Bremer W, Seyedi S, Bahrami Rad A, Clifford GD. Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study. JMIR Res Protoc 2022; 11:e36417. [PMID: 35830230 PMCID: PMC9330209 DOI: 10.2196/36417] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Revised: 05/30/2022] [Accepted: 05/31/2022] [Indexed: 11/20/2022] Open
Abstract
Background Current standards of psychiatric assessment and diagnostic evaluation rely primarily on the clinical subjective interpretation of a patient’s outward manifestations of their internal state. While psychometric tools can help to evaluate these behaviors more systematically, the tools still rely on the clinician’s interpretation of what are frequently nuanced speech and behavior patterns. With advances in computing power, increased availability of clinical data, and improving resolution of recording and sensor hardware (including acoustic, video, accelerometer, infrared, and other modalities), researchers have begun to demonstrate the feasibility of cutting-edge technologies in aiding the assessment of psychiatric disorders. Objective We present a research protocol that utilizes facial expression, eye gaze, voice and speech, locomotor, heart rate, and electroencephalography monitoring to assess schizophrenia symptoms and to distinguish patients with schizophrenia from those with other psychiatric disorders and control subjects. Methods We plan to recruit three outpatient groups: (1) 50 patients with schizophrenia, (2) 50 patients with unipolar major depressive disorder, and (3) 50 individuals with no psychiatric history. Using an internally developed semistructured interview, psychometrically validated clinical outcome measures, and a multimodal sensing system utilizing video, acoustic, actigraphic, heart rate, and electroencephalographic sensors, we aim to evaluate the system’s capacity in classifying subjects (schizophrenia, depression, or control), to evaluate the system’s sensitivity to within-group symptom severity, and to determine if such a system can further classify variations in disorder subtypes. Results Data collection began in July 2020 and is expected to continue through December 2022. Conclusions If successful, this study will help advance current progress in developing state-of-the-art technology to aid clinical psychiatric assessment and treatment. If our findings suggest that these technologies are capable of resolving diagnoses and symptoms to the level of current psychometric testing and clinician judgment, we would be among the first to develop a system that can eventually be used by clinicians to more objectively diagnose and assess schizophrenia and depression with the possibility of less risk of bias. Such a tool has the potential to improve accessibility to care; to aid clinicians in objectively evaluating diagnoses, severity of symptoms, and treatment efficacy through time; and to reduce treatment-related morbidity. International Registered Report Identifier (IRRID) DERR1-10.2196/36417
Collapse
Affiliation(s)
- Robert O Cotes
- Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, GA, United States
| | - Mina Boazak
- Animo Sano Psychiatry, Durham, NC, United States
| | - Emily Griner
- Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, GA, United States
| | - Zifan Jiang
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States.,Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, United States
| | - Bona Kim
- Visual Medical Education, Emory School of Medicine, Atlanta, GA, United States
| | - Whitney Bremer
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States
| | - Salman Seyedi
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States
| | - Ali Bahrami Rad
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States
| | - Gari D Clifford
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States.,Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, United States
| |
Collapse
|
16
|
Abstract
Video-based dynamic facial emotion recognition (FER) is a challenging task, as one must capture and distinguish tiny facial movements representing emotional changes while ignoring the facial differences of different objects. Recent state-of-the-art studies have usually adopted more complex methods to solve this task, such as large-scale deep learning models or multimodal analysis with reference to multiple sub-models. According to the characteristics of the FER task and the shortcomings of existing methods, in this paper we propose a lightweight method and design three attention modules that can be flexibly inserted into the backbone network. The key information for the three dimensions of space, channel, and time is extracted by means of convolution layer, pooling layer, multi-layer perception (MLP), and other approaches, and attention weights are generated. By sharing parameters at the same level, the three modules do not add too many network parameters while enhancing the focus on specific areas of the face, effective feature information of static images, and key frames. The experimental results on CK+ and eNTERFACE’05 datasets show that this method can achieve higher accuracy.
Collapse
|
17
|
Jiang Z, Luskus M, Seyedi S, Griner EL, Rad AB, Clifford GD, Boazak M, Cotes RO. Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review. PLoS One 2022; 17:e0266828. [PMID: 35395049 PMCID: PMC8992987 DOI: 10.1371/journal.pone.0266828] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2021] [Accepted: 03/28/2022] [Indexed: 12/22/2022] Open
Abstract
Background Schizophrenia is a severe psychiatric disorder that causes significant social and functional impairment. Currently, the diagnosis of schizophrenia is based on information gleaned from the patient’s self-report, what the clinician observes directly, and what the clinician gathers from collateral informants, but these elements are prone to subjectivity. Utilizing computer vision to measure facial expressions is a promising approach to adding more objectivity in the evaluation and diagnosis of schizophrenia. Method We conducted a systematic review using PubMed and Google Scholar. Relevant publications published before (including) December 2021 were identified and evaluated for inclusion. The objective was to conduct a systematic review of computer vision for facial behavior analysis in schizophrenia studies, the clinical findings, and the corresponding data processing and machine learning methods. Results Seventeen studies published between 2007 to 2021 were included, with an increasing trend in the number of publications over time. Only 14 articles used interviews to collect data, of which different combinations of passive to evoked, unstructured to structured interviews were used. Various types of hardware were adopted and different types of visual data were collected. Commercial, open-access, and in-house developed models were used to recognize facial behaviors, where frame-level and subject-level features were extracted. Statistical tests and evaluation metrics varied across studies. The number of subjects ranged from 2-120, with an average of 38. Overall, facial behaviors appear to have a role in estimating diagnosis of schizophrenia and psychotic symptoms. When studies were evaluated with a quality assessment checklist, most had a low reporting quality. Conclusion Despite the rapid development of computer vision techniques, there are relatively few studies that have applied this technology to schizophrenia research. There was considerable variation in the clinical paradigm and analytic techniques used. Further research is needed to identify and develop standardized practices, which will help to promote further advances in the field.
Collapse
Affiliation(s)
- Zifan Jiang
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States of America
- Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, United States of America
- * E-mail:
| | - Mark Luskus
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, United States of America
| | - Salman Seyedi
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States of America
| | - Emily L. Griner
- Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, GA, United States of America
| | - Ali Bahrami Rad
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States of America
| | - Gari D. Clifford
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, United States of America
- Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, United States of America
| | - Mina Boazak
- Animo Sano Psychiatry, PLLC, Durham, NC, United States of America
| | - Robert O. Cotes
- Department of Psychiatry and Behavioral Sciences, Emory University School of Medicine, Atlanta, GA, United States of America
| |
Collapse
|