A Survey on Computer Vision for Assistive Medical Diagnosis From Faces

Artificial Intelligence-Based Facial Palsy Evaluation: A Survey

Facial palsy evaluation (FPE) aims to assess facial palsy severity of patients, which plays a vital role in facial functional treatment and rehabilitation. The traditional manners of FPE are based on subjective judgment by clinicians, which may ultimately depend on individual experience. Compared with subjective and manual evaluation, objective and automated evaluation using artificial intelligence (AI) has shown great promise in improving traditional manners and recently received significant attention. The motivation of this survey paper is mainly to provide a systemic review that would guide researchers in conducting their future research work and thus make automatic FPE applicable in real-life situations. In this survey, we comprehensively review the state-of-the-art development of AI-based FPE. First, we summarize the general pipeline of FPE systems with the related background introduction. Following this pipeline, we introduce the existing public databases and give the widely used objective evaluation metrics of FPE. In addition, the preprocessing methods in FPE are described. Then, we provide an overview of selected key publications from 2008 and summarize the state-of-the-art methods of FPE that are designed based on AI techniques. Finally, we extensively discuss the current research challenges faced by FPE and provide insights about potential future directions for advancing state-of-the-art research in this field.

Predicting Acute Exacerbation Phenotype in Chronic Obstructive Pulmonary Disease Patients Using VGG-16 Deep Learning

INTRODUCTION

Exacerbations of chronic obstructive pulmonary disease (COPD) have a significant impact on hospitalizations, morbidity, and mortality of patients. This study aimed to develop a model for predicting acute exacerbation in COPD patients (AECOPD) based on deep-learning (DL) features.

METHODS

We performed a retrospective study on 219 patients with COPD who underwent inspiratory and expiratory HRCT scans. By recording the acute respiratory events of the previous year, these patients were further divided into non-AECOPD group and AECOPD group according to the presence of acute exacerbation events. Sixty-nine quantitative CT (QCT) parameters of emphysema and airway were calculated by NeuLungCARE software, and 2,000 DL features were extracted by VGG-16 method. The logistic regression method was employed to identify AECOPD patients, and 29 patients of external validation cohort were used to access the robustness of the results.

RESULTS

The model 3-B achieved an area under the receiver operating characteristic curve (AUC) of 0.933 and 0.865 in the testing cohort and external validation cohort, respectively. Model 3-I obtained AUC of 0.895 in the testing cohort and AUC of 0.774 in the external validation cohort. Model 7-B combined clinical characteristics, QCT parameters, and DL features achieved the best performance with an AUC of 0.979 in the testing cohort and demonstrating robust predictability with an AUC of 0.932 in the external validation cohort. Likewise, model 7-I achieved an AUC of 0.938 and 0.872 in the testing cohort and external validation cohort, respectively.

CONCLUSIONS

DL features extracted from HRCT scans can effectively predict acute exacerbation phenotype in COPD patients.

Applied Artificial Intelligence in Healthcare: A Review of Computer Vision Technology Application in Hospital Settings

Computer vision (CV), a type of artificial intelligence (AI) that uses digital videos or a sequence of images to recognize content, has been used extensively across industries in recent years. However, in the healthcare industry, its applications are limited by factors like privacy, safety, and ethical concerns. Despite this, CV has the potential to improve patient monitoring, and system efficiencies, while reducing workload. In contrast to previous reviews, we focus on the end-user applications of CV. First, we briefly review and categorize CV applications in other industries (job enhancement, surveillance and monitoring, automation, and augmented reality). We then review the developments of CV in the hospital setting, outpatient, and community settings. The recent advances in monitoring delirium, pain and sedation, patient deterioration, mechanical ventilation, mobility, patient safety, surgical applications, quantification of workload in the hospital, and monitoring for patient events outside the hospital are highlighted. To identify opportunities for future applications, we also completed journey mapping at different system levels. Lastly, we discuss the privacy, safety, and ethical considerations associated with CV and outline processes in algorithm development and testing that limit CV expansion in healthcare. This comprehensive review highlights CV applications and ideas for its expanded use in healthcare.

Thermographic findings related to facial pain - a survey of 40 subjects

OBJECTIVE

The purpose of this study was to assess how thermography findings relate painful symptoms and signs of temporomandibular disorders (TMD).

METHODS

Thermography, combined with chewing of paraffin wax, was performed on 40 subjects. The results were analyzed according to gender and pain-related TMD symptoms and clinical signs.

RESULTS

The overall temperatures after chewing were higher in TMD patients than in controls. For females, the most significant findings were the thermal increase between the relaxed state and subjects' state after chewing in temporal and temporomandibular joint (TMJ) regions. For males, all calculated parameters demonstrated a poor ability to discriminate TMD from controls.

CONCLUSION

Thermography could be a potential tool in diagnostics of female TMD patients. The results suggest that the thermal information assessed in specific facial areas could help to discriminate TMD patients from non-TMD patients and could be used to quantify the pain associated with TMD.

A novel multi-feature learning model for disease diagnosis using face skin images

BACKGROUND

Facial skin characteristics can provide valuable information about a patient's underlying health conditions.

OBJECTIVE

In practice, there are often samples with divergent characteristics (commonly known as divergent samples) that can be attributed to environmental factors, living conditions, or genetic elements. These divergent samples significantly degrade the accuracy of diagnoses.

METHODOLOGY

To tackle this problem, we propose a novel multi-feature learning method called Multi-Feature Learning with Centroid Matrix (MFLCM), which aims to mitigate the influence of divergent samples on the accurate classification of samples located on the boundary. In this approach, we introduce a novel discriminator that incorporates a centroid matrix strategy and simultaneously adapt it to a classifier in a unified model. We effectively apply the centroid matrix to the embedding feature spaces, which are transformed from the multi-feature observation space, by calculating a relaxed Hamming distance. The purpose of the centroid vectors for each category is to act as anchors, ensuring that samples from the same class are positioned close to their corresponding centroid vector while being pushed further away from the remaining centroids.

RESULTS

Validation of the proposed method with clinical facial skin dataset showed that the proposed method achieved F1 scores of 92.59%, 83.35%, 82.84% and 85.46%, respectively for the detection the Healthy, Diabetes Mellitus (DM), Fatty Liver (FL) and Chronic Renal Failure (CRF).

CONCLUSION

Experimental results demonstrate the superiority of the proposed method compared with typical classifiers single-view-based and state-of-the-art multi-feature approaches. To the best of our knowledge, this study represents the first to demonstrate concept of multi-feature learning using only facial skin images as an effective non-invasive approach for simultaneously identifying DM, FL and CRF in Han Chinese, the largest ethnic group in the world.

Comparing 2D and 3D representations for face-based genetic syndrome diagnosis

Human genetic syndromes are often challenging to diagnose clinically. Facial phenotype is a key diagnostic indicator for hundreds of genetic syndromes and computer-assisted facial phenotyping is a promising approach to assist diagnosis. Most previous approaches to automated face-based syndrome diagnosis have analyzed different datasets of either 2D images or surface mesh-based 3D facial representations, making direct comparisons of performance challenging. In this work, we developed a set of subject-matched 2D and 3D facial representations, which we then analyzed with the aim of comparing the performance of 2D and 3D image-based approaches to computer-assisted syndrome diagnosis. This work represents the most comprehensive subject-matched analyses to date on this topic. In our analyses of 1907 subject faces representing 43 different genetic syndromes, 3D surface-based syndrome classification models significantly outperformed 2D image-based models trained and evaluated on the same subject faces. These results suggest that the clinical adoption of 3D facial scanning technology and continued collection of syndromic 3D facial scan data may substantially improve face-based syndrome diagnosis.

Application of a novel deep learning technique using CT images for COVID-19 diagnosis on embedded systems

Problem

A novel coronavirus (COVID-19) has created a worldwide pneumonia epidemic, and it's important to make a computer-aided way for doctors to use computed tomography (CT) images to find people with COVID-19 as soon as possible. Aim: A fully automated, novel deep-learning method for diagnosis and prognostic analysis of COVID-19 on the embedded system is presented.

Methods

In this study, CT scans are utilized to identify individuals with COVID-19, pneumonia, or normal class. To achieve classification two pre-trained CNN models, namely ResNet50 and MobileNetv2, which are commonly used for image classification tasks. Additionally, a novel CNN architecture called CovidxNet-CT is introduced specifically designed for COVID-19 diagnosis using three classes of CT scans. To evaluate the effectiveness of the proposed method, k-fold cross-validation is employed, which is a common approach to estimate the performance of deep learning. The study is also evaluated the proposed method on two embedded system platforms, Jetson Nano and Tx2, to demonstrate its feasibility for deployment in resource-constrained environments.

Results

With an average accuracy of %98.83 and an AUC of 0.988, the system is trained and verified using a 4 fold cross-validation approach.

Conclusion

The optimistic outcomes from the investigation propose that CovidxNet-CT has the capacity to support radiologists and contribute towards the efforts to combat COVID-19. This study proposes a fully automated, deep-learning-based method for COVID-19 diagnosis and prognostic analysis that is specifically designed for use on embedded systems.

A New COVID-19 Detection Method Based on CSK/QAM Visible Light Communication and Machine Learning

This article proposes a novel method for detecting coronavirus disease 2019 (COVID-19) in an underground channel using visible light communication (VLC) and machine learning (ML). We present mathematical models of COVID-19 Deoxyribose Nucleic Acid (DNA) gene transfer in regular square constellations using a CSK/QAM-based VLC system. ML algorithms are used to classify the bands present in each electrophoresis sample according to whether the band corresponds to a positive, negative, or ladder sample during the search for the optimal model. Complexity studies reveal that the square constellation N=22i×22i,(i=3) yields a greater profit. Performance studies indicate that, for BER = 10-3, there are gains of -10 [dB], -3 [dB], 3 [dB], and 5 [dB] for N=22i×22i,(i=0,1,2,3), respectively. Based on a total of 630 COVID-19 samples, the best model is shown to be XGBoots, which demonstrated an accuracy of 96.03%, greater than that of the other models, and a recall of 99% for positive values.

An automatic facial landmarking for children with rare diseases

Two to three thousand syndromes modify facial features: their screening requires the eye of an expert in dysmorphology. A widely used tool in shape characterization is geometric morphometrics based on landmarks, which are precise and reproducible anatomical points. Landmark positioning is user dependent and time consuming. Many automatic landmarking tools are currently available but do not work for children, because they have mainly been trained using photographic databases of healthy adults. Here, we developed a method for building an automatic landmarking pipeline for frontal and lateral facial photographs as well as photographs of external ears. We evaluated the algorithm on patients diagnosed with Treacher Collins (TC) syndrome as it is the most frequent mandibulofacial dysostosis in humans and is clinically recognizable although highly variable in severity. We extracted photographs from the photographic database of the maxillofacial surgery and plastic surgery department of Hôpital Necker-Enfants Malades in Paris, France with the diagnosis of TC syndrome. The control group was built from children admitted for craniofacial trauma or skin lesions. After testing two methods of object detection by bounding boxes, a Haar Cascade-based tool and a Faster Region-based Convolutional Neural Network (Faster R-CNN)-based tool, we evaluated three different automatic annotation algorithms: the patch-based active appearance model (AAM), the holistic AAM, and the constrained local model (CLM). The final error corresponding to the distance between the points placed by automatic annotation and those placed by manual annotation was reported. We included, respectively, 1664, 2044, and 1375 manually annotated frontal, profile, and ear photographs. Object recognition was optimized with the Faster R-CNN-based detector. The best annotation model was the patch-based AAM (p < 0.001 for frontal faces, p = 0.082 for profile faces and p < 0.001 for ears). This automatic annotation model resulted in the same classification performance as manually annotated data. Pretraining on public photographs did not improve the performance of the model. We defined a pipeline to create automatic annotation models adapted to faces with congenital anomalies, an essential prerequisite for research in dysmorphology.

Can Hierarchical Transformers Learn Facial Geometry?

Human faces are a core part of our identity and expression, and thus, understanding facial geometry is key to capturing this information. Automated systems that seek to make use of this information must have a way of modeling facial features in a way that makes them accessible. Hierarchical, multi-level architectures have the capability of capturing the different resolutions of representation involved. In this work, we propose using a hierarchical transformer architecture as a means of capturing a robust representation of facial geometry. We further demonstrate the versatility of our approach by using this transformer as a backbone to support three facial representation problems: face anti-spoofing, facial expression representation, and deepfake detection. The combination of effective fine-grained details alongside global attention representations makes this architecture an excellent candidate for these facial representation problems. We conduct numerous experiments first showcasing the ability of our approach to address common issues in facial modeling (pose, occlusions, and background variation) and capture facial symmetry, then demonstrating its effectiveness on three supplemental tasks.

COVID-19 Detection: A Systematic Review of Machine and Deep Learning-Based Approaches Utilizing Chest X-Rays and CT Scans

This review study presents the state-of-the-art machine and deep learning-based COVID-19 detection approaches utilizing the chest X-rays or computed tomography (CT) scans. This study aims to systematically scrutinize as well as to discourse challenges and limitations of the existing state-of-the-art research published in this domain from March 2020 to August 2021. This study also presents a comparative analysis of the performance of four majorly used deep transfer learning (DTL) models like VGG16, VGG19, ResNet50, and DenseNet over the COVID-19 local CT scans dataset and global chest X-ray dataset. A brief illustration of the majorly used chest X-ray and CT scan datasets of COVID-19 patients utilized in state-of-the-art COVID-19 detection approaches are also presented for future research. The research databases like IEEE Xplore, PubMed, and Web of Science are searched exhaustively for carrying out this survey. For the comparison analysis, four deep transfer learning models like VGG16, VGG19, ResNet50, and DenseNet are initially fine-tuned and trained using the augmented local CT scans and global chest X-ray dataset in order to observe their performance. This review study summarizes major findings like AI technique employed, type of classification performed, used datasets, results in terms of accuracy, specificity, sensitivity, F1 score, etc., along with the limitations, and future work for COVID-19 detection in tabular manner for conciseness. The performance analysis of the four majorly used deep transfer learning models affirms that Visual Geometry Group 19 (VGG19) model delivered the best performance over both COVID-19 local CT scans dataset and global chest X-ray dataset.

Detecting 3D syndromic faces as outliers using unsupervised normalizing flow models

Many genetic syndromes are associated with distinctive facial features. Several computer-assisted methods have been proposed that make use of facial features for syndrome diagnosis. Training supervised classifiers, the most common approach for this purpose, requires large, comprehensive, and difficult to collect databases of syndromic facial images. In this work, we use unsupervised, normalizing flow-based manifold and density estimation models trained entirely on unaffected subjects to detect syndromic 3D faces as statistical outliers. Furthermore, we demonstrate a general, user-friendly, gradient-based interpretability mechanism that enables clinicians and patients to understand model inferences. 3D facial surface scans of 2471 unaffected subjects and 1629 syndromic subjects representing 262 different genetic syndromes were used to train and evaluate the models. The flow-based models outperformed unsupervised comparison methods, with the best model achieving an ROC-AUC of 86.3% on a challenging, age and sex diverse data set. In addition to highlighting the viability of outlier-based syndrome screening tools, our methods generalize and extend previously proposed outlier scores for 3D face-based syndrome detection, resulting in improved performance for unsupervised syndrome detection.

E-Prevention: Advanced Support System for Monitoring and Relapse Prevention in Patients with Psychotic Disorders Analyzing Long-Term Multimodal Data from Wearables and Video Captures

Wearable technologies and digital phenotyping foster unique opportunities for designing novel intelligent electronic services that can address various well-being issues in patients with mental disorders (i.e., schizophrenia and bipolar disorder), thus having the potential to revolutionize psychiatry and its clinical practice. In this paper, we present e-Prevention, an innovative integrated system for medical support that facilitates effective monitoring and relapse prevention in patients with mental disorders. The technologies offered through e-Prevention include: (i) long-term continuous recording of biometric and behavioral indices through a smartwatch; (ii) video recordings of patients while being interviewed by a clinician, using a tablet; (iii) automatic and systematic storage of these data in a dedicated Cloud server and; (iv) the ability of relapse detection and prediction. This paper focuses on the description of the e-Prevention system and the methodologies developed for the identification of feature representations that correlate with and can predict psychopathology and relapses in patients with mental disorders. Specifically, we tackle the problem of relapse detection and prediction using Machine and Deep Learning techniques on all collected data. The results are promising, indicating that such predictions could be made and leading eventually to the prediction of psychopathology and the prevention of relapses.

Breaking CAPTCHA with Capsule Networks

Convolutional Neural Networks have achieved state-of-the-art performance in image classification. Their lack of ability to recognise the spatial relationship between features, however, leads to misclassification of the variants of the same image. Capsule Networks were introduced to address this issue by incorporating the spatial information of image features into neural networks. In this paper, we are interested in showcasing the digit recognition task on CAPTCHA images, widely considered a challenge for computers in relation to human capabilities. Our intention is to provide a rigorous empirical regime in which we can compare the competitive performance of Capsule Networks against the Convolutional Neural Networks. Indeed since CAPTCHA distorts images, by adjusting the spatial positioning of features, we aim to demonstrate the advantages and limitations of Capsule Networks architecture. We train the Capsule Networks with Dynamic Routing version and the convolutional-neural-network-based deep-CAPTCHA baseline model to predict the digit sequences on numerical CAPTCHAs, investigate the performance results and propose two improvements to the Capsule Networks model.

Automatic Facial Palsy Diagnosis as a Classification Problem Using Regional Information Extracted from a Photograph

The incapability to move the facial muscles is known as facial palsy, and it affects various abilities of the patient, for example, performing facial expressions. Recently, automatic approaches aiming to diagnose facial palsy using images and machine learning algorithms have emerged, focusing on providing an objective evaluation of the paralysis severity. This research proposes an approach to analyze and assess the lesion severity as a classification problem with three levels: healthy, slight, and strong palsy. The method explores the use of regional information, meaning that only certain areas of the face are of interest. Experiments carrying on multi-class classification tasks are performed using four different classifiers to validate a set of proposed hand-crafted features. After a set of experiments using this methodology on available image databases, great results are revealed (up to 95.61% of correct detection of palsy patients and 95.58% of correct assessment of the severity level). This perspective leads us to believe that the analysis of facial paralysis is possible with partial occlusions if face detection is accomplished and facial features are obtained adequately. The results also show that our methodology is suited to operate with other databases while attaining high performance, even though the image conditions are different and the participants do not perform equivalent facial expressions.

6G and Artificial Intelligence Technologies for Dementia Care: Literature Review and Practical Analysis

Background

The dementia epidemic is progressing fast. As the world’s older population keeps skyrocketing, the traditional incompetent, time-consuming, and laborious interventions are becoming increasingly insufficient to address dementia patients’ health care needs. This is particularly true amid COVID-19. Instead, efficient, cost-effective, and technology-based strategies, such as sixth-generation communication solutions (6G) and artificial intelligence (AI)-empowered health solutions, might be the key to successfully managing the dementia epidemic until a cure becomes available. However, while 6G and AI technologies hold great promise, no research has examined how 6G and AI applications can effectively and efficiently address dementia patients’ health care needs and improve their quality of life.

Objective

This study aims to investigate ways in which 6G and AI technologies could elevate dementia care to address this study gap.

Methods

A literature review was conducted in databases such as PubMed, Scopus, and PsycINFO. The search focused on three themes: dementia, 6G, and AI technologies. The initial search was conducted on April 25, 2021, complemented by relevant articles identified via a follow-up search on November 11, 2021, and Google Scholar alerts.

Results

The findings of the study were analyzed in terms of the interplay between people with dementia’s unique health challenges and the promising capabilities of health technologies, with in-depth and comprehensive analyses of advanced technology-based solutions that could address key dementia care needs, ranging from impairments in memory (eg, Egocentric Live 4D Perception), speech (eg, Project Relate), motor (eg, Avatar Robot Café), cognitive (eg, Affectiva), to social interactions (eg, social robots).

Conclusions

To live is to grow old. Yet dementia is neither a proper way to live nor a natural aging process. By identifying advanced health solutions powered by 6G and AI opportunities, our study sheds light on the imperative of leveraging the potential of advanced technologies to elevate dementia patients’ will to live, enrich their daily activities, and help them engage in societies across shapes and forms.

Static and Motion Facial Analysis for Craniofacial Assessment and Diagnosing Diseases

Deviation from a normal facial shape and symmetry can arise from numerous sources, including physical injury and congenital birth defects. Such abnormalities can have important aesthetic and functional consequences. Furthermore, in clinical genetics distinctive facial appearances are often associated with clinical or genetic diagnoses; the recognition of a characteristic facial appearance can substantially narrow the search space of potential diagnoses for the clinician. Unusual patterns of facial movement and expression can indicate disturbances to normal mechanical functioning or emotional affect. Computational analyses of static and moving 2D and 3D images can serve clinicians and researchers by detecting and describing facial structural, mechanical, and affective abnormalities objectively. In this review we survey traditional and emerging methods of facial analysis, including statistical shape modeling, syndrome classification, modeling clinical face phenotype spaces, and analysis of facial motion and affect. Expected final online publication date for the Annual Review of Biomedical Data Science, Volume 5 is August 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review

Background

Schizophrenia is a severe psychiatric disorder that causes significant social and functional impairment. Currently, the diagnosis of schizophrenia is based on information gleaned from the patient’s self-report, what the clinician observes directly, and what the clinician gathers from collateral informants, but these elements are prone to subjectivity. Utilizing computer vision to measure facial expressions is a promising approach to adding more objectivity in the evaluation and diagnosis of schizophrenia.

Method

We conducted a systematic review using PubMed and Google Scholar. Relevant publications published before (including) December 2021 were identified and evaluated for inclusion. The objective was to conduct a systematic review of computer vision for facial behavior analysis in schizophrenia studies, the clinical findings, and the corresponding data processing and machine learning methods.

Results

Seventeen studies published between 2007 to 2021 were included, with an increasing trend in the number of publications over time. Only 14 articles used interviews to collect data, of which different combinations of passive to evoked, unstructured to structured interviews were used. Various types of hardware were adopted and different types of visual data were collected. Commercial, open-access, and in-house developed models were used to recognize facial behaviors, where frame-level and subject-level features were extracted. Statistical tests and evaluation metrics varied across studies. The number of subjects ranged from 2-120, with an average of 38. Overall, facial behaviors appear to have a role in estimating diagnosis of schizophrenia and psychotic symptoms. When studies were evaluated with a quality assessment checklist, most had a low reporting quality.

Conclusion

Despite the rapid development of computer vision techniques, there are relatively few studies that have applied this technology to schizophrenia research. There was considerable variation in the clinical paradigm and analytic techniques used. Further research is needed to identify and develop standardized practices, which will help to promote further advances in the field.

A Deep Invertible 3D Facial Shape Model For Interpretable Genetic Syndrome Diagnosis

One of the primary difficulties in treating patients with genetic syndromes is diagnosing their condition. Many syndromes are associated with characteristic facial features that can be imaged and utilized by computer-assisted diagnosis systems. In this work, we develop a novel 3D facial surface modeling approach with the objective of maximizing diagnostic model interpretability within a flexible deep learning framework. Therefore, an invertible normalizing flow architecture is introduced to enable both inferential and generative tasks in a unified and efficient manner. The proposed model can be used (1) to infer syndrome diagnosis and other demographic variables given a 3D facial surface scan and (2) to explain model inferences to non-technical users via multiple interpretability mechanisms. The model was trained and evaluated on more than 4700 facial surface scans from subjects with 47 different syndromes. For the challenging task of predicting syndrome diagnosis given a new 3D facial surface scan, age, and sex of a subject, the model achieves a competitive overall top-1 accuracy of 71%, and a mean sensitivity of 43% across all syndrome classes. We believe that invertible models such as the one presented in this work can achieve competitive inferential performance while greatly increasing model interpretability in the domain of medical diagnosis.

Automatic Detection of Human Maxillofacial Tumors by Using Thermal Imaging: A Preliminary Study

Traditional computed tomography (CT) delivers a relatively high dose of radiation to the patient and cannot be used as a method for screening of pathologies. Instead, infrared thermography (IRT) might help in the detection of pathologies, but interpreting thermal imaging (TI) is difficult even for the expert. The main objective of this work is to present a new, automated IRT method capable to discern the absence or presence of tumor in the orofacial/maxillofacial region of patients. We evaluated the use of a special feature vector extracted from face and mouth cavity thermograms in classifying TIs against the absence/presence of tumor (n = 23 patients per group). Eight statistical features extracted from TI were used in a k-nearest neighbor (kNN) classifier. Classification accuracy of kNN was evaluated by CT, and by creating a vector with the true class labels for TIs. The presented algorithm, constructed from a training data set, gives good results of classification accuracy of kNN: sensitivity of 77.9%, specificity of 94.9%, and accuracy of 94.1%. The new algorithm exhibited almost the same accuracy in detecting the absence/presence of tumor as CT, and is a proof-of-principle that IRT could be useful as an additional reliable screening tool for detecting orofacial/maxillofacial tumors.

Similarities and disparities between visual analysis and high-resolution electromyography of facial expressions

Computer vision (CV) is widely used in the investigation of facial expressions. Applications range from psychological evaluation to neurology, to name just two examples. CV for identifying facial expressions may suffer from several shortcomings: CV provides indirect information about muscle activation, it is insensitive to activations that do not involve visible deformations, such as jaw clenching. Moreover, it relies on high-resolution and unobstructed visuals. High density surface electromyography (sEMG) recordings with soft electrode array is an alternative approach which provides direct information about muscle activation, even from freely behaving humans. In this investigation, we compare CV and sEMG analysis of facial muscle activation. We used independent component analysis (ICA) and multiple linear regression (MLR) to quantify the similarity and disparity between the two approaches for posed muscle activations. The comparison reveals similarity in event detection, but discrepancies and inconsistencies in source identification. Specifically, the correspondence between sEMG and action unit (AU)-based analyses, the most widely used basis of CV muscle activation prediction, appears to vary between participants and sessions. We also show a comparison between AU and sEMG data of spontaneous smiles, highlighting the differences between the two approaches. The data presented in this paper suggests that the use of AU-based analysis should consider its limited ability to reliably compare between different sessions and individuals and highlight the advantages of high-resolution sEMG for facial expression analysis.

Predicting neuropsychiatric symptoms of persons with dementia in a day care center using a facial expression recognition system

BACKGROUND

Behavioral and psychological symptoms of dementia (BPSD) affect 90% of persons with dementia (PwD), resulting in various adverse outcomes and aggravating care burdens among their caretakers. This study aimed to explore the potential of artificial intelligence-based facial expression recognition systems (FERS) in predicting BPSDs among PwD.

METHODS

A hybrid of human labeling and a preconstructed deep learning model was used to differentiate basic facial expressions of individuals to predict the results of Neuropsychiatric Inventory (NPI) assessments by stepwise linear regression (LR), random forest (RF) with importance ranking, and ensemble method (EM) of equal importance, while the accuracy was determined by mean absolute error (MAE) and root-mean-square error (RMSE) methods.

RESULTS

Twenty-three PwD from an adult day care center were enrolled with ≥ 11,500 FERS data series and 38 comparative NPI scores. The overall accuracy was 86% on facial expression recognition. Negative facial expressions and variance in emotional switches were important features of BPSDs. A strong positive correlation was identified in each model (EM: r = 0.834, LR: r = 0.821, RF: r = 0.798 by the patientwise method; EM: r = 0.891, LR: r = 0.870, RF: r = 0.886 by the MinimPy method), and EM exhibited the lowest MAE and RMSE.

CONCLUSIONS

FERS successfully predicted the BPSD of PwD by negative emotions and the variance in emotional switches. This finding enables early detection and management of BPSDs, thus improving the quality of dementia care.

Do People Trust in Robot-Assisted Surgery? Evidence from Europe

(1) Background: The goal of the paper was to establish the factors that influence how people feel about having a medical operation performed on them by a robot. (2) Methods: Data were obtained from a 2017 Flash Eurobarometer (number 460) of the European Commission with 27,901 citizens aged 15 years and over in the 28 countries of the European Union. Logistic regression (odds ratios, OR) to model the predictors of trust in robot-assisted surgery was calculated through motivational factors, using experience and sociodemographic independent variables. (3) Results: The results obtained indicate that, as the experience of using robots increases, the predictive coefficients related to information, attitude, and perception of robots become more negative. Furthermore, sociodemographic variables played an important predictive role. The effect of experience on trust in robots for surgical interventions was greater among men, people between 40 and 54 years old, and those with higher educational levels. (4) Conclusions: The results show that trust in robots goes beyond rational decision-making, since the final decision about whether it should be a robot that performs a complex procedure like a surgical intervention depends almost exclusively on the patient’s wishes.

A CNN-LSTM network with multi-level feature extraction-based approach for automated detection of coronavirus from CT scan and X-ray images

Auto-detection of diseases has become a prime issue in medical sciences as population density is fast growing. An intelligent framework for disease detection helps physicians identify illnesses, give reliable and consistent results, and reduce death rates. Coronavirus (Covid-19) has recently been one of the most severe and acute diseases in the world. An automatic detection framework should therefore be introduced as the fastest diagnostic alternative to avoid Covid-19 spread. In this paper, an automatic Covid-19 identification in the CT scan and chest X-ray is obtained with the help of a combined deep learning and multi-level feature extraction methodology. In this method, the multi-level feature extraction approach comprises GIST, Scale Invariant Feature Transform (SIFT), and Convolutional Neural Network (CNN) extract features from CT scans and chest X-rays. The objective of multi-level feature extraction is to reduce the training complexity of CNN network, which significantly assists in accurate and robust Covid-19 identification. Finally, Long Short-Term Memory (LSTM) along the CNN network is used to detect the extracted Covid-19 features. The Kaggle SARS-CoV-2 CT scan dataset and the Italian SIRM Covid-19 CT scan and chest X-ray dataset were employed for testing purposes. Experimental outcomes show that proposed approach obtained 98.94% accuracy with the SARS-CoV-2 CT scan dataset and 83.03% accuracy with the SIRM Covid-19 CT scan and chest X-ray dataset. The proposed approach helps radiologists and practitioners to detect and treat Covid-19 cases effectively over the pandemic.

Lightweight Face Anti-Spoofing Network for Telehealth Applications

Online healthcare applications have grown more popular over the years. For instance,telehealth is an online healthcare application that allows patients and doctors to schedule consultations,prescribe medication,share medical documents,and monitor health conditions conveniently. Apart from this,telehealth can also be used to store a patients personal and medical information. Given the amount of sensitive data it stores,security measures are necessary. With its rise in usage due to COVID-19,its usefulness may be undermined if security issues are not addressed. A simple way of making these applications more secure is through user authentication. One of the most common and often used authentications is face recognition. It is convenient and easy to use. However,face recognition systems are not foolproof. They are prone to malicious attacks like printed photos,paper cutouts,re-played videos,and 3D masks. In order to counter this,multiple face anti-spoofing methods have been proposed. The goal of face anti-spoofing is to differentiate real users (live) from attackers (spoof). Although effective in terms of performance,existing methods use a significant amount of parameters,making them resource-heavy and unsuitable for handheld devices. Apart from this,they fail to generalize well to new environments like changes in lighting or background. This paper proposes a lightweight face anti-spoofing framework that does not compromise on performance. A lightweight model is critical for applications like telehealth that run on handheld devices. Our proposed method achieves good performance with the help of an ArcFace Classifier (AC). The AC encourages differentiation between spoof and live samples by making clear boundaries between them. With clear boundaries,classification becomes more accurate. We further demonstrate our models capabilities by comparing the number of parameters,FLOPS,and performance with other state-of-the-art methods.

Computational diagnostic methods on 2D photographs: A review of the literature

Here we provide a literature review of all the methods reported to date for analyzing 2D pictures for diagnostic purposes. Pubmed was used to screen the MEDLINE database using MeSH (Medical Subject Heading) terms and keyworks. The different recognition steps and the main results were reported. All human studies involving 2D facial photographs used to diagnose one or several conditions in healthy populations or in patients were included. We included 1515 articles and 27 publications were finally retained. 67% of the articles aimed at diagnosing one particular syndrome versus healthy controls and 33% aimed at performing multi-class syndrome recognition. Data volume varied from 15 to 17,106 patient pictures. Manual or automatic landmarks were one of the most commonly used tools in order to extract morphological information from images, in 22/27 (81%) publications. Geometrical features were extracted from landmarks based on Procrustes superimposition in 4/27 (15%). Textural features were extracted in 19/27 (70%) publications. Features were then classified using machine learning methods in 89% of publications, while deep learning methods were used in 11%. Facial recognition tools were generally successful in identifying rare conditions in dysmorphic patients, with comparable or higher recognition accuracy than clinical experts.

A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders

We present the first public dataset with videos of oro-facial gestures performed by individuals with oro-facial impairment due to neurological disorders, such as amyotrophic lateral sclerosis (ALS) and stroke. Perceptual clinical scores from trained clinicians are provided as metadata. Manual annotation of facial landmarks is also provided for a subset of over 3300 frames. Through extensive experiments with multiple facial landmark detection algorithms, including state-of-the-art convolutional neural network (CNN) models, we demonstrated the presence of bias in the landmark localization accuracy of pre-trained face alignment approaches in our participant groups. The pre-trained models produced higher errors in the two clinical groups compared to age-matched healthy control subjects. We also investigated how this bias changes when the existing models are fine-tuned using data from the target population. The release of this dataset aims to propel the development of face alignment algorithms robust to the presence of oro-facial impairment, support the automatic analysis and recognition of oro-facial gestures, enhance the automatic identification of neurological diseases, as well as the estimation of disease severity from videos and images.

Facial Paralysis Detection on Images Using Key Point Analysis

The inability to move the muscles of the face on one or both sides is known as facial paralysis, which may affect the ability of the patient to speak, blink, swallow saliva, eat, or communicate through natural facial expressions. The well-being of the patient could also be negatively affected. Computer-based systems as a means to detect facial paralysis are important in the development of standardized tools for medical assessment, treatment, and monitoring; additionally, they are expected to provide user-friendly tools for patient monitoring at home. In this work, a methodology to detect facial paralysis in a face photograph is proposed. A system consisting of three modules—facial landmark extraction, facial measure computation, and facial paralysis classification—was designed. Our facial measures aim to identify asymmetry levels within the face elements using facial landmarks, and a binary classifier based on a multi-layer perceptron approach provides an output label. The Weka suite was selected to design the classifier and implement the learning algorithm. Tests on publicly available databases reveal outstanding classification results on images, showing that our methodology that was used to design a binary classifier can be expanded to other databases with great results, even if the participants do not execute similar facial expressions.

Are We Ready for Video Recognition and Computer Vision in the Intensive Care Unit? A Survey

OBJECTIVE

Video recording and video recognition (VR) with computer vision have become widely used in many aspects of modern life. Hospitals have employed VR technology for security purposes, however, despite the growing number of studies showing the feasibility of VR software for physiologic monitoring or detection of patient movement, its use in the intensive care unit (ICU) in real-time is sparse and the perception of this novel technology is unknown. The objective of this study is to understand the attitudes of providers, patients, and patient's families toward using VR in the ICU.

DESIGN

A 10-question survey instrument was used and distributed into two groups of participants: clinicians (MDs, advance practice providers, registered nurses), patients and families (adult patients and patients' relatives). Questions were specifically worded and section for free text-comments created to elicit respondents' thoughts and attitudes on potential issues and barriers toward implementation of VR in the ICU.

SETTING

The survey was conducted at Mayo Clinic in Minnesota and Florida.

RESULTS

A total of 233 clinicians' and 50 patients' surveys were collected. Both cohorts favored VR under specific circumstances (e.g., invasive intervention and diagnostic manipulation). Acceptable reasons for VR usage according to clinicians were anticipated positive impact on patient safety (70%), and diagnostic suggestions and decision support (51%). A minority of providers was concerned that artificial intelligence (AI) would replace their job (14%) or erode professional skills (28%). The potential use of VR in lawsuits (81% clinicians) and privacy breaches (59% patients) were major areas of concern. Further identified barriers were lack of trust for AI, deterioration of the patient-clinician rapport. Patients agreed with VR unless it does not reduce nursing care or record sensitive scenarios.

CONCLUSION

The survey provides valuable information on the acceptance of VR cameras in the critical care setting including an overview of real concerns and attitudes toward the use of VR technology in the ICU.

Computer Vision-Based Assessment of Motor Functioning in Schizophrenia: Use of Smartphones for Remote Measurement of Schizophrenia Symptomatology

Introduction

Motor abnormalities have been shown to be a distinct component of schizophrenia symptomatology. However, objective and scalable methods for assessment of motor functioning in schizophrenia are lacking. Advancements in machine learning-based digital tools have allowed for automated and remote "digital phenotyping" of disease symptomatology. Here, we assess the performance of a computer vision-based assessment of motor functioning as a characteristic of schizophrenia using video data collected remotely through smartphones.

Methods

Eighteen patients with schizophrenia and 9 healthy controls were asked to remotely participate in smartphone-based assessments daily for 14 days. Video recorded from the smartphone front-facing camera during these assessments was used to quantify the Euclidean distance of head movement between frames through a pretrained computer vision model. The ability of head movement measurements to distinguish between patients and healthy controls as well as their relationship to schizophrenia symptom severity as measured through traditional clinical scores was assessed.

Results

The rate of head movement in participants with schizophrenia (1.48 mm/frame) and those without differed significantly (2.50 mm/frame; p = 0.01), and a logistic regression demonstrated that head movement was a significant predictor of schizophrenia diagnosis (p = 0.02). Linear regression between head movement and clinical scores of schizophrenia showed that head movement has a negative relationship with schizophrenia symptom severity (p = 0.04), primarily with negative symptoms of schizophrenia.

Conclusions

Remote, smartphone-based assessments were able to capture meaningful visual behavior for computer vision-based objective measurement of head movement. The measurements of head movement acquired were able to accurately classify schizophrenia diagnosis and quantify symptom severity in patients with schizophrenia.

Video-Based Facial Weakness Analysis

OBJECTIVE

Facial weakness is a common sign of neurological diseases such as Bell's palsy and stroke. However, recognizing facial weakness still remains as a challenge, because it requires experience and neurological training.

METHODS

We propose a framework for facial weakness detection, which models the temporal dynamics of both shape and appearance-based features of each target frame through a bi-directional long short-term memory network (Bi-LSTM). The system is evaluated on a "in-the-wild"video dataset that is verified by three board-certified neurologists. In addition, three emergency medical services (EMS) personnel and three upper level residents rated the dataset. We compare the evaluation of the proposed algorithm with other comparison methods as well as the human raters.

RESULTS

Experimental evaluation demonstrates that: (1) the proposed algorithm achieves the accuracy, sensitivity, and specificity of 94.3%, 91.4%, and 95.7%, which outperforms other comparison methods and achieves the equal performance to paramedics; (2) the framework can provide visualizable and interpretable results that increases model transparency and interpretability; (3) a prototype is implemented as a proof-of-concept showcase to show the feasibility of an inexpensive solution for facial weakness detection.

CONCLUSION

The experiment results suggest that the proposed framework can identify facial weakness effectively.

SIGNIFICANCE

We provide a proof-of-concept study, showing that such technology could be used by non-neurologists to more readily identify facial weakness in the field, leading to increasing coverage and earlier treatment.

Automatic Identification of High-Risk Autism Spectrum Disorder: A Feasibility Study Using Video and Audio Data Under the Still-Face Paradigm

It is reported that the symptoms of autism spectrum disorder (ASD) could be improved by effective early interventions, which arouses an urgent need for large-scale early identification of ASD. Until now, the screening of ASD has relied on the child psychiatrist to collect medical history and conduct behavioral observations with the help of psychological assessment tools. Such screening measures inevitably have some disadvantages, including strong subjectivity, relying on experts and low-efficiency. With the development of computer science, it is possible to realize a computer-aided screening for ASD and alleviate the disadvantages of manual evaluation. In this study, we propose a behavior-based automated screening method to identify high-risk ASD (HR-ASD) for babies aged 8-24 months. The still-face paradigm (SFP) was used to elicit baby's spontaneous social behavior through a face-to-face interaction, in which a mother was required to maintain a normal interaction to amuse her baby for 2 minutes (a baseline episode) and then suddenly change to the no-reaction and no-expression status with 1 minute (a still-face episode). Here, multiple cues derived from baby's social stress response behavior during the latter episode, including head-movements, facial expressions and vocal characteristics, were statistically analyzed between HR-ASD and typical developmental (TD) groups. An automated identification model of HR-ASD was constructed based on these multi-cue features and the support vector machine (SVM) classifier; moreover, its screening performance was satisfied, for all the accuracy, specificity and sensitivity exceeded 90% on the cases included in this study. The experimental results suggest its feasibility in the early screening of HR-ASD.

Computer vision in autism spectrum disorder research: a systematic review of published studies from 2009 to 2019

The current state of computer vision methods applied to autism spectrum disorder (ASD) research has not been well established. Increasing evidence suggests that computer vision techniques have a strong impact on autism research. The primary objective of this systematic review is to examine how computer vision analysis has been useful in ASD diagnosis, therapy and autism research in general. A systematic review of publications indexed on PubMed, IEEE Xplore and ACM Digital Library was conducted from 2009 to 2019. Search terms included ['autis*' AND ('computer vision' OR 'behavio* imaging' OR 'behavio* analysis' OR 'affective computing')]. Results are reported according to PRISMA statement. A total of 94 studies are included in the analysis. Eligible papers are categorised based on the potential biological/behavioural markers quantified in each study. Then, different computer vision approaches that were employed in the included papers are described. Different publicly available datasets are also reviewed in order to rapidly familiarise researchers with datasets applicable to their field and to accelerate both new behavioural and technological work on autism research. Finally, future research directions are outlined. The findings in this review suggest that computer vision analysis is useful for the quantification of behavioural/biological markers which can further lead to a more objective analysis in autism research.

Pain intensity estimation based on a spatial transformation and attention CNN

Models designed to detect abnormalities that reflect disease from facial structures are an emerging area of research for automated facial analysis, which has important potential value in smart healthcare applications. However, most of the proposed models directly analyze the whole face image containing the background information, and rarely consider the effects of the background and different face regions on the analysis results. Therefore, in view of these effects, we propose an end-to-end attention network with spatial transformation to estimate different pain intensities. In the proposed method, the face image is first provided as input to a spatial transformation network for solving the problem of background interference; then, the attention mechanism is used to adaptively adjust the weights of different face regions of the transformed face image; finally, a convolutional neural network (CNN) containing a Softmax function is utilized to classify the pain levels. The extensive experiments and analysis are conducted on the benchmarking and publicly available database, namely the UNBC-McMaster shoulder pain. More specifically, in order to verify the superiority of our proposed method, the comparisons with the basic CNNs and the-state-of-the-arts are performed, respectively. The experiments show that the introduced spatial transformation and attention mechanism in our method can significantly improve the estimation performances and outperform the-state-of-the-arts.

A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images

Nowadays, automatic disease detection has become a crucial issue in medical science due to rapid population growth. An automatic disease detection framework assists doctors in the diagnosis of disease and provides exact, consistent, and fast results and reduces the death rate. Coronavirus (COVID-19) has become one of the most severe and acute diseases in recent times and has spread globally. Therefore, an automated detection system, as the fastest diagnostic option, should be implemented to impede COVID-19 from spreading. This paper aims to introduce a deep learning technique based on the combination of a convolutional neural network (CNN) and long short-term memory (LSTM) to diagnose COVID-19 automatically from X-ray images. In this system, CNN is used for deep feature extraction and LSTM is used for detection using the extracted feature. A collection of 4575 X-ray images, including 1525 images of COVID-19, were used as a dataset in this system. The experimental results show that our proposed system achieved an accuracy of 99.4%, AUC of 99.9%, specificity of 99.2%, sensitivity of 99.3%, and F1-score of 98.9%. The system achieved desired results on the currently available dataset, which can be further improved when more COVID-19 images become available. The proposed system can help doctors to diagnose and treat COVID-19 patients easily.

A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images

Nowadays, automatic disease detection has become a crucial issue in medical science due to rapid population growth. An automatic disease detection framework assists doctors in the diagnosis of disease and provides exact, consistent, and fast results and reduces the death rate. Coronavirus (COVID-19) has become one of the most severe and acute diseases in recent times and has spread globally. Therefore, an automated detection system, as the fastest diagnostic option, should be implemented to impede COVID-19 from spreading. This paper aims to introduce a deep learning technique based on the combination of a convolutional neural network (CNN) and long short-term memory (LSTM) to diagnose COVID-19 automatically from X-ray images. In this system, CNN is used for deep feature extraction and LSTM is used for detection using the extracted feature. A collection of 4575 X-ray images, including 1525 images of COVID-19, were used as a dataset in this system. The experimental results show that our proposed system achieved an accuracy of 99.4%, AUC of 99.9%, specificity of 99.2%, sensitivity of 99.3%, and F1-score of 98.9%. The system achieved desired results on the currently available dataset, which can be further improved when more COVID-19 images become available. The proposed system can help doctors to diagnose and treat COVID-19 patients easily.

Survey: smartphone-based assessment of cardiovascular diseases using ECG and PPG analysis

A number of resources, every year, being spent to tackle early detection of cardiac abnormalities which is one of the leading causes of deaths all over the Globe. The challenges for healthcare systems includes early detection, portability and mobility of patients. This paper presents a categorical review of smartphone-based systems that can detect cardiac abnormalities by the analysis of Electrocardiogram (ECG) and Photoplethysmography (PPG) and the limitation and challenges of these system. The ECG based systems can monitor, record and forward signals for analysis and an alarm can be triggered in case of abnormality, however the limitation of smart phone’s processing capabilities, lack of storage and speed of network are major challenges. The systems based on PPG signals are non-invasive and provides mobility and portability. This study aims to critically review the existing systems, their limitation, challenges and possible improvements to serve as a reference for researchers and developers.

Vision-Based Mouth Motion Analysis in Epilepsy: A 3D Perspective

Epilepsy monitoring involves the study of videos to assess clinical signs (semiology) to assist with the diagnosis of seizures. Recent advances in the application of vision-based approaches to epilepsy analysis have demonstrated significant potential to automate this assessment. Nevertheless, current proposed computer vision based techniques are unable to accurately quantify specific facial modifications, e.g. mouth motions, which are examined by neurologists to distinguish between seizure types. 2D approaches that analyse facial landmarks have been proposed to quantify mouth motions, however, they are unable to fully represent motions in the mouth and cheeks (ictal pouting) due to a lack of landmarks in the the cheek regions. Additionally, 2D region-based techniques based on the detection of the mouth have limitations when dealing with large pose variations, and thus make a fair comparison between samples difficult due to the variety of poses present. 3D approaches, on the other hand, retain rich information about the shape and appearance of faces, simplifying alignment for comparison between sequences. In this paper, we propose a novel network method based on a 3D reconstruction of the face and deep learning to detect and quantify mouth semiology in our video dataset of 20 seizures, recorded from patients with mesial temporal and extra-temporal lobe epilepsy. The proposed network is capable of distinguishing between seizures of both types of epilepsy. An average classification accuracy of 89% demonstrates the benefits of computer vision and deep learning for clinical applications of non-contact systems to identify semiology commonly encountered in a natural clinical setting.

Facial Weakness Analysis and Quantification of Static Images

Facial weakness is a symptom commonly associated to lack of facial muscle control due to neurological injury. Several diseases are associated with facial weakness such as stroke and Bell's palsy. The use of digital imaging through mobile phones, tablets, personal computers and other devices could provide timely opportunity for detection, which if accurate enough can improve treatment by enabling faster patient triage and recovery progress monitoring. Most of the existing facial weakness detection approaches from static images are based on facial landmarks from which geometric features can be calculated. Landmark-based methods, however, can suffer from inaccuracies in face landmarks localization. In this study, We also experimentally evaluate the performance of several feature extraction methods for measuring facial weakness, including the landmark-based features, as well as intensity-based features on a neurologist-certified dataset that comprises 186 images of normal, 125 images of left facial weakness, and 126 images of right facial weakness. We demonstrate that, for the application of facial weakness detection from single (static) images, approaches that incorporate the Histogram of Oriented Gradients (HoG) features tend to be more accurate.

Facial expression recognition for monitoring neurological disorders based on convolutional neural network

Facial expressions are a significant part of non-verbal communication. Recognizing facial expressions of people with neurological disorders is essential because these people may have lost a significant amount of their verbal communication ability. Such an assessment requires time consuming examination involving medical personnel, which can be quite challenging and expensive. Automated facial expression recognition systems that are low-cost and noninvasive can help experts detect neurological disorders. In this study, an automated facial expression recognition system is developed using a novel deep learning approach. The architecture consists of four-stage networks. The first, second and third networks segment the facial components which are essential for facial expression recognition. Owing to the three networks, an iconize facial image is obtained. The fourth network classifies facial expressions using raw facial images and iconize facial images. This four-stage method combines holistic facial information with local part-based features to achieve more robust facial expression recognition. Preliminary experimental results achieved 94.44% accuracy for facial expression recognition on RaFD database. The proposed system produced 5% improvement than the facial expression recognition system by using raw images. This study presents a quantitative, objective and non-invasive facial expression recognition system to help in the monitoring and diagnosis of neurological disorders influencing facial expressions.

Geometric morphometrics for the study of facial expressions in non-human animals, using the domestic cat as an exemplar

Facial expression is a common channel for the communication of emotion. However, in the case of non-human animals, the analytical methods used to quantify facial expressions can be subjective, relying heavily on extrapolation from human-based systems. Here, we demonstrate how geometric morphometrics can be applied in order to overcome these problems. We used this approach to identify and quantify changes in facial shape associated with pain in a non-human animal species. Our method accommodates individual variability, species-specific facial anatomy, and postural effects. Facial images were captured at four different time points during ovariohysterectomy of domestic short haired cats (n = 29), with time points corresponding to varying intensities of pain. Images were annotated using landmarks specifically chosen for their relationship with underlying musculature, and relevance to cat-specific facial action units. Landmark data were subjected to normalisation before Principal Components (PCs) were extracted to identify key sources of facial shape variation, relative to pain intensity. A significant relationship between PC scores and a well-validated composite measure of post-operative pain in cats (UNESP-Botucatu MCPS tool) was evident, demonstrating good convergent validity between our geometric face model, and other metrics of pain detection. This study lays the foundation for the automatic, objective detection of emotional expressions in a range of non-human animal species.