1
|
Cîrneanu AL, Popescu D, Iordache D. New Trends in Emotion Recognition Using Image Analysis by Neural Networks, A Systematic Review. SENSORS (BASEL, SWITZERLAND) 2023; 23:7092. [PMID: 37631629 PMCID: PMC10458371 DOI: 10.3390/s23167092] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Revised: 07/29/2023] [Accepted: 08/02/2023] [Indexed: 08/27/2023]
Abstract
Facial emotion recognition (FER) is a computer vision process aimed at detecting and classifying human emotional expressions. FER systems are currently used in a vast range of applications from areas such as education, healthcare, or public safety; therefore, detection and recognition accuracies are very important. Similar to any computer vision task based on image analyses, FER solutions are also suitable for integration with artificial intelligence solutions represented by different neural network varieties, especially deep neural networks that have shown great potential in the last years due to their feature extraction capabilities and computational efficiency over large datasets. In this context, this paper reviews the latest developments in the FER area, with a focus on recent neural network models that implement specific facial image analysis algorithms to detect and recognize facial emotions. This paper's scope is to present from historical and conceptual perspectives the evolution of the neural network architectures that proved significant results in the FER area. This paper endorses convolutional neural network (CNN)-based architectures against other neural network architectures, such as recurrent neural networks or generative adversarial networks, highlighting the key elements and performance of each architecture, and the advantages and limitations of the proposed models in the analyzed papers. Additionally, this paper presents the available datasets that are currently used for emotion recognition from facial expressions and micro-expressions. The usage of FER systems is also highlighted in various domains such as healthcare, education, security, or social IoT. Finally, open issues and future possible developments in the FER area are identified.
Collapse
Affiliation(s)
- Andrada-Livia Cîrneanu
- Faculty of Automatic Control and Computers, University Politehnica of Bucharest, 060042 Bucharest, Romania;
| | - Dan Popescu
- Faculty of Automatic Control and Computers, University Politehnica of Bucharest, 060042 Bucharest, Romania;
| | - Dragoș Iordache
- The National Institute for Research & Development in Informatics-ICI Bucharest, 011455 Bucharest, Romania;
| |
Collapse
|
2
|
Borgalli RA, Surve S. Review on learning framework for facial expression recognition. THE IMAGING SCIENCE JOURNAL 2023. [DOI: 10.1080/13682199.2023.2172526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/17/2023]
Affiliation(s)
- Rohan Appasaheb Borgalli
- Department of Electronics Engineering, Fr. Conceicao Rodrigues College of Engineering, Bandra, University of Mumbai, Mumbai, Maharashtra, India
| | - Sunil Surve
- Department of Computer Engineering, Fr. Conceicao Rodrigues College of Engineering, Bandra, University of Mumbai, Mumbai, Maharashtra, India
| |
Collapse
|
3
|
Sham AH, Khan A, Lamas D, Tikka P, Anbarjafari G. Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings. SENSORS (BASEL, SWITZERLAND) 2023; 23:458. [PMID: 36617055 PMCID: PMC9824663 DOI: 10.3390/s23010458] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Revised: 12/27/2022] [Accepted: 12/28/2022] [Indexed: 06/17/2023]
Abstract
Emotion recognition is a significant issue in many sectors that use human emotion reactions as communication for marketing, technological equipment, or human-robot interaction. The realistic facial behavior of social robots and artificial agents is still a challenge, limiting their emotional credibility in dyadic face-to-face situations with humans. One obstacle is the lack of appropriate training data on how humans typically interact in such settings. This article focused on collecting the facial behavior of 60 participants to create a new type of dyadic emotion reaction database. For this purpose, we propose a methodology that automatically captures the facial expressions of participants via webcam while they are engaged with other people (facial videos) in emotionally primed contexts. The data were then analyzed using three different Facial Expression Analysis (FEA) tools: iMotions, the Mini-Xception model, and the Py-Feat FEA toolkit. Although the emotion reactions were reported as genuine, the comparative analysis between the aforementioned models could not agree with a single emotion reaction prediction. Based on this result, a more-robust and -effective model for emotion reaction prediction is needed. The relevance of this work for human-computer interaction studies lies in its novel approach to developing adaptive behaviors for synthetic human-like beings (virtual or robotic), allowing them to simulate human facial interaction behavior in contextually varying dyadic situations with humans. This article should be useful for researchers using human emotion analysis while deciding on a suitable methodology to collect facial expression reactions in a dyadic setting.
Collapse
Affiliation(s)
- Abdallah Hussein Sham
- Digital Technology Institute, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia
- Enactive Virtuality Lab, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia
| | - Amna Khan
- Digital Technology Institute, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia
| | - David Lamas
- Digital Technology Institute, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia
| | - Pia Tikka
- Enactive Virtuality Lab, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia
| | - Gholamreza Anbarjafari
- iCV Lab, University of Tartu, Narva mnt 18, 51009 Tartu, Estonia
- iVCV OÜ, 51009 Tartu, Estonia
- PwC Advisory, 00180 Helsinki, Finland
- Institute of Higher Education, Yildiz Technical University, Beşiktaş, Istanbul 34349, Turkey
| |
Collapse
|
4
|
Aktas K, Kiisk M, Giammanco A, Anbarjafari G, Mägi M. A Comparison of Neural Networks and Center of Gravity in Muon Hit Position Estimation. ENTROPY (BASEL, SWITZERLAND) 2022; 24:1659. [PMID: 36421514 PMCID: PMC9689399 DOI: 10.3390/e24111659] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 11/08/2022] [Accepted: 11/13/2022] [Indexed: 06/16/2023]
Abstract
The performance of cosmic-ray tomography systems is largely determined by their tracking accuracy. With conventional scintillation detector technology, good precision can be achieved with a small pitch between the elements of the detector array. Improving the resolution implies increasing the number of read-out channels, which in turn increases the complexity and cost of the tracking detectors. As an alternative to that, a scintillation plate detector coupled with multiple silicon photomultipliers could be used as a technically simple solution. In this paper, we present a comparison between two deep-learning-based methods and a conventional Center of Gravity (CoG) algorithm, used to calculate cosmic-ray muon hit positions on the plate detector using the signals from the photomultipliers. In this study, we generated a dataset of muon hits on a detector plate using the Monte Carlo simulation toolkit GEANT4. We demonstrate that two deep-learning-based methods outperform the conventional CoG algorithm by a significant margin. Our proposed algorithm, Fully Connected Network, produces a 0.72 mm average error measured in Euclidean distance between the actual and predicted hit coordinates, showing great improvement in comparison with CoG, which yields 1.41 mm on the same dataset. Additionally, we investigated the effects of different sensor configurations on performance.
Collapse
Affiliation(s)
- Kadir Aktas
- iCV Research Lab., Institute of Technology, University of Tartu, 51009 Tartu, Estonia
| | - Madis Kiisk
- Institute of Physics, University of Tartu, 51009 Tartu, Estonia
- GScan Ltd., Mäealuse 2/1, 12618 Tallinn, Estonia
| | - Andrea Giammanco
- Centre for Cosmology, Particle Physics and Phenomenology (CP3), Université Catholique de Louvain, B-1348 Louvain la Neuve, Belgium
| | - Gholamreza Anbarjafari
- iCV Research Lab., Institute of Technology, University of Tartu, 51009 Tartu, Estonia
- Higher Education Institute, Yildiz Technical University, Istanbul 34349, Turkey
| | - Märt Mägi
- GScan Ltd., Mäealuse 2/1, 12618 Tallinn, Estonia
| |
Collapse
|
5
|
Aktas K, Ignjatovic V, Ilic D, Marjanovic M, Anbarjafari G. Deep convolutional neural networks for detection of abnormalities in chest X-rays trained on the very large dataset. SIGNAL, IMAGE AND VIDEO PROCESSING 2022; 17:1035-1041. [PMID: 35873389 PMCID: PMC9296894 DOI: 10.1007/s11760-022-02309-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/31/2021] [Revised: 04/10/2022] [Accepted: 06/28/2022] [Indexed: 06/15/2023]
Abstract
One of the main challenges in the current pandemic is the detection of coronavirus. Conventional techniques (PT-PCR) have their limitations such as long response time and limited accessibility. On the other hand, X-ray machines are widely available and they are already digitized in the health systems. Thus, their usage is faster and more available. Therefore, in this research, we evaluate how well deep CNNs do when it comes to classifying normal versus pathological chest X-rays. Compared to the previous research, we trained our network on the largest number of images, 103,468 in total, including 5 classes such as COPD signs, COVID, normal, others and Pneumonia. We achieved COVID accuracy of 97% and overall accuracy of 81%. Additionally, we achieved classification accuracy of 84% for categorization into normal (78%) and abnormal (88%).
Collapse
Affiliation(s)
- Kadir Aktas
- iCV Research Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia
- iVCV OÜ, 51011 Tartu, Estonia
| | | | - Dragan Ilic
- Singidunum University, Belgrade, 11010 Serbia
| | | | - Gholamreza Anbarjafari
- iCV Research Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia
- iVCV OÜ, 51011 Tartu, Estonia
- PwC Advisory, Helsinki, Finland
- Yildiz Technical University, Istanbul, Turkey
| |
Collapse
|
6
|
Sleiman E, Mutlu OC, Surabhi S, Husic A, Kline A, Washington P, Wall DP. Deep Learning-Based Autism Spectrum Disorder Detection Using Emotion Features From Video Recordings (Preprint). JMIR BIOMEDICAL ENGINEERING 2022. [DOI: 10.2196/39982] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
|
7
|
Estimation of Symmetry in the Recognition System with Adaptive Application of Filters. Symmetry (Basel) 2022. [DOI: 10.3390/sym14050903] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/04/2022] Open
Abstract
The aim of this work is to study the influence of lighting on different types of filters in order to create adaptive systems of perception in the visible spectrum. This problem is solved by estimating symmetry operations (operations responsible for image/image transformations). Namely, the authors are interested in an objective assessment of the possibility of reproducing the image of the object (objective symmetry of filters) after the application of filters. This paper investigates and shows the results of the most common edge detection filters depending on the light level; that is, the behavior of the system in a room with indirect natural and standard (according to the requirements of the educational process in Ukraine) electric lighting was studied. The methods of Sobel, Sobel x, Sobel y, Prewitt, Prewitt x, Prewitt y, and Canny were used and compared in experiments. The conclusions provide a subjective assessment of the performance of each of the filters in certain conditions. Dependencies are defined that allow giving priority to certain filters (from those studied) depending on the lighting.
Collapse
|
8
|
Vecvanags A, Aktas K, Pavlovs I, Avots E, Filipovs J, Brauns A, Done G, Jakovels D, Anbarjafari G. Ungulate Detection and Species Classification from Camera Trap Images Using RetinaNet and Faster R-CNN. ENTROPY 2022; 24:e24030353. [PMID: 35327863 PMCID: PMC8947003 DOI: 10.3390/e24030353] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Revised: 02/23/2022] [Accepted: 02/25/2022] [Indexed: 02/04/2023]
Abstract
Changes in the ungulate population density in the wild has impacts on both the wildlife and human society. In order to control the ungulate population movement, monitoring systems such as camera trap networks have been implemented in a non-invasive setup. However, such systems produce a large number of images as the output, hence making it very resource consuming to manually detect the animals. In this paper, we present a new dataset of wild ungulates which was collected in Latvia. Moreover, we demonstrate two methods, which use RetinaNet and Faster R-CNN as backbones, respectively, to detect the animals in the images. We discuss the optimization of training and impact of data augmentation on the performance. Finally, we show the result of aforementioned tune networks over the real world data collected in Latvia.
Collapse
Affiliation(s)
- Alekss Vecvanags
- Institute for Environmental Solutions, LV-4126 Cēsis, Latvia; (A.V.); (J.F.); (A.B.); (D.J.); (G.A.)
| | - Kadir Aktas
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (K.A.); (I.P.)
| | - Ilja Pavlovs
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (K.A.); (I.P.)
| | - Egils Avots
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (K.A.); (I.P.)
- Forest Owners Consulting Center LCC, LV-4101 Cēsis, Latvia
- Correspondence: ; Tel.: +372-737-4855
| | - Jevgenijs Filipovs
- Institute for Environmental Solutions, LV-4126 Cēsis, Latvia; (A.V.); (J.F.); (A.B.); (D.J.); (G.A.)
| | - Agris Brauns
- Institute for Environmental Solutions, LV-4126 Cēsis, Latvia; (A.V.); (J.F.); (A.B.); (D.J.); (G.A.)
| | - Gundega Done
- Latvian State Forest Research Institute “Silava”, LV-2169 Salaspils, Latvia;
| | - Dainis Jakovels
- Institute for Environmental Solutions, LV-4126 Cēsis, Latvia; (A.V.); (J.F.); (A.B.); (D.J.); (G.A.)
| | - Gholamreza Anbarjafari
- Institute for Environmental Solutions, LV-4126 Cēsis, Latvia; (A.V.); (J.F.); (A.B.); (D.J.); (G.A.)
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (K.A.); (I.P.)
- PwC Advisory, 00180 Helsinki, Finland
- Faculty of Engineering, Hasan Kalyoncu University, Gaziantep 27410, Turkey
| |
Collapse
|
9
|
Avots E, Jermakovs K, Bachmann M, Päeske L, Ozcinar C, Anbarjafari G. Ensemble Approach for Detection of Depression Using EEG Features. ENTROPY 2022; 24:e24020211. [PMID: 35205506 PMCID: PMC8871180 DOI: 10.3390/e24020211] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/18/2021] [Revised: 01/26/2022] [Accepted: 01/27/2022] [Indexed: 11/21/2022]
Abstract
Depression is a public health issue that severely affects one’s well being and can cause negative social and economic effects to society. To raise awareness of these problems, this research aims at determining whether the long-lasting effects of depression can be determined from electroencephalographic (EEG) signals. The article contains an accuracy comparison for SVM, LDA, NB, kNN, and D3 binary classifiers, which were trained using linear (relative band power, alpha power variability, spectral asymmetry index) and nonlinear (Higuchi fractal dimension, Lempel–Ziv complexity, detrended fluctuation analysis) EEG features. The age- and gender-matched dataset consisted of 10 healthy subjects and 10 subjects diagnosed with depression at some point in their lifetime. Most of the proposed feature selection and classifier combinations achieved accuracy in the range of 80% to 95%, and all the models were evaluated using a 10-fold cross-validation. The results showed that the motioned EEG features used in classifying ongoing depression also work for classifying the long-lasting effects of depression.
Collapse
Affiliation(s)
- Egils Avots
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (E.A.); (K.J.); (C.O.)
| | - Klāvs Jermakovs
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (E.A.); (K.J.); (C.O.)
| | - Maie Bachmann
- Biosignal Processing Laboratory, Tallinn University of Technology, 19086 Tallinn, Estonia; (M.B.); (L.P.)
| | - Laura Päeske
- Biosignal Processing Laboratory, Tallinn University of Technology, 19086 Tallinn, Estonia; (M.B.); (L.P.)
| | - Cagri Ozcinar
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (E.A.); (K.J.); (C.O.)
| | - Gholamreza Anbarjafari
- iCV Lab, Institute of Technology, University of Tartu, 51009 Tartu, Estonia; (E.A.); (K.J.); (C.O.)
- PwC Advisory, 00180 Helsinki, Finland
- Faculty of Egineering, Hasan Kalyoncu University, 27000 Gaziantep, Turkey
- Correspondence: ; Tel.: +90-372-737-4855
| |
Collapse
|