Analysis of Facial Information for Healthcare Applications: A Survey on Computer Vision-Based Approaches

Emotion recognition in doctor-patient interactions from real-world clinical video database: Initial development of artificial empathy

BACKGROUND AND OBJECTIVE

The promising use of artificial intelligence (AI) to emulate human empathy may help a physician engage with a more empathic doctor-patient relationship. This study demonstrates the application of artificial empathy based on facial emotion recognition to evaluate doctor-patient relationships in clinical practice.

METHODS

A prospective study used recorded video data of doctor-patient clinical encounters in dermatology outpatient clinics, Taipei Municipal Wanfang Hospital, and Taipei Medical University Hospital collected from March to December 2019. Two cameras recorded the facial expressions of four doctors and 348 adult patients during regular clinical practice. Facial emotion recognition was used to analyze the basic emotions of doctors and patients with a temporal resolution of 1 second. In addition, a physician-patient satisfaction questionnaire was administered after each clinical session, and two standard patients gave impartial feedback to avoid bias.

RESULTS

Data from 326 clinical session videos showed that (1) Doctors expressed more emotions than patients (t [326] > = 2.998, p < = 0.003), including anger, happiness, disgust, and sadness; the only emotion that patients showed more than doctors was surprise (t [326] = -4.428, p < .001) (p < .001). (2) Patients felt happier during the latter half of the session (t [326] = -2.860, p = .005), indicating a good doctor-patient relationship.

CONCLUSIONS

Artificial empathy can offer objective observations on how doctors' and patients' emotions change. With the ability to detect emotions in 3/4 view and profile images, artificial empathy could be an accessible evaluation tool to study doctor-patient relationships in practical clinical settings.

Importance of telemedicine in mild cognitive impairment and Alzheimer disease patients population during admission to emergency departments with COVID-19

In March 2020, the World Health Organization declared a global pandemic due to the new coronavirus SARS-CoV-2, and several governments have planned a national quarantine to control the spread of the virus. Acute psychological effects during hospitalization in frail elderly individuals with special needs, such as patients with dementia, have been little studied. The greatest distress manifested by these kinds of patients was isolation from their families during hospitalization. Thus, structured video call interventions were carried out to family caregivers of patients diagnosed with dementia during their hospitalization in the COVID-19 ward. The purpose of this quasi-experimental study was to assess changes in cognitive and behavioral symptoms in both patients and caregivers. All study participants underwent psychological assessments. Specifically, the psychological well-being states of patients and their caregivers were measured at admission (T0) and discharge (T1) using psychometric tests and clinical scales. Each participant received an electronic device to access video calls in addition meetings were scheduled with the psychologist and medical team to keep caregivers updated on the health status of their relatives. A psychological support and cognitive rehabilitation service was also provided. Significant differences were found in all clinical variables of the caregiver group. Results showed a significant relationship in the quality of life score between the patient and caregiver groups. The results of this study has highlighted the importance of maintaining significantly effective relationships during the hospitalization period of patients admitted to COVID wards.

The current state of artificial intelligence-augmented digitized neurocognitive screening test

The cognitive screening test is a brief cognitive examination that could be easily performed in a clinical setting. However, one of the main drawbacks of this test was that only a paper-based version was available, which restricts the test to be manually administered and graded by medical personnel at the health centers. The main solution to these problems was to develop a potential remote assessment for screening individuals with cognitive impairment. Currently, multiple studies have been adopting artificial intelligence (AI) technology into these tests, evolving the conventional paper-based neurocognitive test into a digitized AI-assisted neurocognitive test. These studies provided credible evidence of the potential of AI-augmented cognitive screening tests to be better and provided the framework for future studies to further improve the implementation of AI technology in the cognitive screening test. The objective of this review article is to discuss different types of AI used in digitized cognitive screening tests and their advantages and disadvantages.

Fast 3D Face Reconstruction from a Single Image Using Different Deep Learning Approaches for Facial Palsy Patients

The 3D reconstruction of an accurate face model is essential for delivering reliable feedback for clinical decision support. Medical imaging and specific depth sensors are accurate but not suitable for an easy-to-use and portable tool. The recent development of deep learning (DL) models opens new challenges for 3D shape reconstruction from a single image. However, the 3D face shape reconstruction of facial palsy patients is still a challenge, and this has not been investigated. The contribution of the present study is to apply these state-of-the-art methods to reconstruct the 3D face shape models of facial palsy patients in natural and mimic postures from one single image. Three different methods (3D Basel Morphable model and two 3D Deep Pre-trained models) were applied to the dataset of two healthy subjects and two facial palsy patients. The reconstructed outcomes were compared to the 3D shapes reconstructed using Kinect-driven and MRI-based information. As a result, the best mean error of the reconstructed face according to the Kinect-driven reconstructed shape is 1.5±1.1 mm. The best error range is 1.9±1.4 mm when compared to the MRI-based shapes. Before using the procedure to reconstruct the 3D faces of patients with facial palsy or other facial disorders, several ideas for increasing the accuracy of the reconstruction can be discussed based on the results. This present study opens new avenues for the fast reconstruction of the 3D face shapes of facial palsy patients from a single image. As perspectives, the best DL method will be implemented into our computer-aided decision support system for facial disorders.

Classification of facial paralysis based on machine learning techniques

Facial paralysis (FP) is an inability to move facial muscles voluntarily, affecting daily activities. There is a need for quantitative assessment and severity level classification of FP to evaluate the condition. None of the available tools are widely accepted. A comprehensive FP evaluation system has been developed by the authors. The system extracts real-time facial animation units (FAUs) using the Kinect V2 sensor and includes both FP assessment and classification. This paper describes the development and testing of the FP classification phase. A dataset of 375 records from 13 unilateral FP patients and 1650 records from 50 control subjects was compiled. Artificial Intelligence and Machine Learning methods are used to classify seven FP categories: the normal case and three severity levels: mild, moderate, and severe for the left and right sides. For better prediction results (Accuracy = 96.8%, Sensitivity = 88.9% and Specificity = 99%), an ensemble learning classifier was developed rather than one weak classifier. The ensemble approach based on SVMs was proposed for the high-dimensional data to gather the advantages of stacking and bagging. To address the problem of an imbalanced dataset, a hybrid strategy combining three separate techniques was used. Model robustness and stability was evaluated using fivefold cross-validation. The results showed that the classifier is robust, stable and performs well for different train and test samples. The study demonstrates that FAUs acquired by the Kinect sensor can be used in classifying FP. The developed FP assessment and classification system provides a detailed quantitative report and has significant advantages over existing grading scales.

Using Facial Landmark Detection on Thermal Images as a Novel Prognostic Tool for Emergency Departments

Introduction

Emergency departments (ED) at hospitals sometimes experience unexpected deterioration in patients that were assessed to be in a stable condition upon arrival. Odense University Hospital (OUH) has conducted a retrospective study to investigate the possibilities of prognostic tools that can detect these unexpected deterioration cases at an earlier stage. The study suggests that the temperature difference (gradient) between the core and the peripheral body parts can be used to detect these cases. The temperature between the patient's inner canthus (core temperature) and the tip of the nose (peripheral temperature) can be measured with a thermal camera. Based on the temperature measurement from a thermal image, a gradient value can be calculated, which can be used as an early indicator of potential deterioration.

Problem

The lack of a tool to automatically calculate the gradient has prevented the ED at OUH in conducting a comprehensive prospective study on early indicators of patients at risk of deterioration. The current manual way of doing facial landmark detection on thermal images is too time consuming and not feasible as part of the daily workflow at the ED, where nurses have to triage patients within a few minutes.

Objective

The objective of this study was to automate the process of calculating the gradient by developing a handheld prognostic tool that can be used by nurses for automatically performing facial landmark detection on thermal images of patients as they arrive at the ED.

Methods

A systematic literature review has been conducted to investigate previous studies that have been done for applying computer vision methods on thermal images. Several meetings, interviews and field studies have been conducted with the ED at OUH in order to understand their workflow, formulate and prioritize requirements and co-design the prognostic tool.

Results

The study resulted in a novel Android app that can capture a thermal image of a patient's face with a thermal camera attached to a smartphone. Within a few seconds, the app then automatically calculates the gradient to be used in the triage process. The developed tool is the first of its kind using facial landmark detection on thermal images for calculating a gradient that can serve as a novel prognostic indicator for ED patients.

Stable Gaze Tracking with Filtering Based on Internet of Things

Gaze tracking is basic research in the era of the Internet of Things. This study attempts to improve the performance of gaze tracking in an active infrared source gaze-tracking system. Owing to unavoidable noise interference, the estimated points of regard (PORs) tend to fluctuate within a certain range. To reduce the fluctuation range and obtain more stable results, we introduced a Kalman filter (KF) to filter the gaze parameters. Considering that the effect of filtering is relevant to the motion state of the gaze, we design the measurement noise that varies with the speed of the gaze. In addition, we used a correlation filter-based tracking method to quickly locate the pupil, instead of the detection method. Experiments indicated that the variance of the estimation error decreased by 73.83%, the size of the extracted pupil image decreased by 93.75%, and the extraction speed increased by 1.84 times. We also comprehensively discussed the advantages and disadvantages of the proposed method, which provides a reference for related research. It must be pointed out that the proposed algorithm can also be adopted in any eye camera-based gaze tracker.

Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review

Background

Schizophrenia is a severe psychiatric disorder that causes significant social and functional impairment. Currently, the diagnosis of schizophrenia is based on information gleaned from the patient’s self-report, what the clinician observes directly, and what the clinician gathers from collateral informants, but these elements are prone to subjectivity. Utilizing computer vision to measure facial expressions is a promising approach to adding more objectivity in the evaluation and diagnosis of schizophrenia.

Method

We conducted a systematic review using PubMed and Google Scholar. Relevant publications published before (including) December 2021 were identified and evaluated for inclusion. The objective was to conduct a systematic review of computer vision for facial behavior analysis in schizophrenia studies, the clinical findings, and the corresponding data processing and machine learning methods.

Results

Seventeen studies published between 2007 to 2021 were included, with an increasing trend in the number of publications over time. Only 14 articles used interviews to collect data, of which different combinations of passive to evoked, unstructured to structured interviews were used. Various types of hardware were adopted and different types of visual data were collected. Commercial, open-access, and in-house developed models were used to recognize facial behaviors, where frame-level and subject-level features were extracted. Statistical tests and evaluation metrics varied across studies. The number of subjects ranged from 2-120, with an average of 38. Overall, facial behaviors appear to have a role in estimating diagnosis of schizophrenia and psychotic symptoms. When studies were evaluated with a quality assessment checklist, most had a low reporting quality.

Conclusion

Despite the rapid development of computer vision techniques, there are relatively few studies that have applied this technology to schizophrenia research. There was considerable variation in the clinical paradigm and analytic techniques used. Further research is needed to identify and develop standardized practices, which will help to promote further advances in the field.

Towards Facial Gesture Recognition in Photographs of Patients with Facial Palsy

Humans express their emotions verbally and through actions, and hence emotions play a fundamental role in facial expressions and body gestures. Facial expression recognition is a popular topic in security, healthcare, entertainment, advertisement, education, and robotics. Detecting facial expressions via gesture recognition is a complex and challenging problem, especially in persons who suffer face impairments, such as patients with facial paralysis. Facial palsy or paralysis refers to the incapacity to move the facial muscles on one or both sides of the face. This work proposes a methodology based on neural networks and handcrafted features to recognize six gestures in patients with facial palsy. The proposed facial palsy gesture recognition system is designed and evaluated on a publicly available database with good results as a first attempt to perform this task in the medical field. We conclude that, to recognize facial gestures in patients with facial paralysis, the severity of the damage has to be considered because paralyzed organs exhibit different behavior than do healthy ones, and any recognition system must be capable of discerning these behaviors.

How Can Research on Artificial Empathy Be Enhanced by Applying Deepfakes?

We propose the idea of using an open data set of doctor-patient interactions to develop artificial empathy based on facial emotion recognition. Facial emotion recognition allows a doctor to analyze patients' emotions, so that they can reach out to their patients through empathic care. However, face recognition data sets are often difficult to acquire; many researchers struggle with small samples of face recognition data sets. Further, sharing medical images or videos has not been possible, as this approach may violate patient privacy. The use of deepfake technology is a promising approach to deidentifying video recordings of patients’ clinical encounters. Such technology can revolutionize the implementation of facial emotion recognition by replacing a patient's face in an image or video with an unrecognizable face—one with a facial expression that is similar to that of the original. This technology will further enhance the potential use of artificial empathy in helping doctors provide empathic care to achieve good doctor-patient therapeutic relationships, and this may result in better patient satisfaction and adherence to treatment.

Emotional Expression in Children With ASD: A Pre-Study on a Two-Group Pre-Post-Test Design Comparing Robot-Based and Computer-Based Training

Several studies have found a delay in the development of facial emotion recognition and expression in children with an autism spectrum condition (ASC). Several interventions have been designed to help children to fill this gap. Most of them adopt technological devices (i.e., robots, computers, and avatars) as social mediators and reported evidence of improvement. Few interventions have aimed at promoting emotion recognition and expression abilities and, among these, most have focused on emotion recognition. Moreover, a crucial point is the generalization of the ability acquired during treatment to naturalistic interactions. This study aimed to evaluate the effectiveness of two technological-based interventions focused on the expression of basic emotions comparing a robot-based type of training with a "hybrid" computer-based one. Furthermore, we explored the engagement of the hybrid technological device introduced in the study as an intermediate step to facilitate the generalization of the acquired competencies in naturalistic settings. A two-group pre-post-test design was applied to a sample of 12 children (M = 9.33; ds = 2.19) with autism. The children were included in one of the two groups: group 1 received a robot-based type of training (n = 6); and group 2 received a computer-based type of training (n = 6). Pre- and post-intervention evaluations (i.e., time) of facial expression and production of four basic emotions (happiness, sadness, fear, and anger) were performed. Non-parametric ANOVAs found significant time effects between pre- and post-interventions on the ability to recognize sadness [t ₍₁₎ = 7.35, p = 0.006; pre: M (ds) = 4.58 (0.51); post: M (ds) = 5], and to express happiness [t ₍₁₎ = 5.72, p = 0.016; pre: M (ds) = 3.25 (1.81); post: M (ds) = 4.25 (1.76)], and sadness [t ₍₁₎ = 10.89, p < 0; pre: M (ds) = 1.5 (1.32); post: M (ds) = 3.42 (1.78)]. The group^*time interactions were significant for fear [t ₍₁₎ = 1.019, p = 0.03] and anger expression [t ₍₁₎ = 1.039, p = 0.03]. However, Mann-Whitney comparisons did not show significant differences between robot-based and computer-based training. Finally, no difference was found in the levels of engagement comparing the two groups in terms of the number of voice prompts given during interventions. Albeit the results are preliminary and should be interpreted with caution, this study suggests that two types of technology-based training, one mediated via a humanoid robot and the other via a pre-settled video of a peer, perform similarly in promoting facial recognition and expression of basic emotions in children with an ASC. The findings represent the first step to generalize the abilities acquired in a laboratory-trained situation to naturalistic interactions.

SVM classification of facial functions based on facial landmarks and animation Units

Quantitative assessment and classification of facial paralysis (FP) are essential for treatment selection and progress evaluation of the condition. As part of a comprehensive framework towards this goal, this study aims to classify five normal facial functions: smiling, eye closure, raising the eyebrows, blowing cheeks, and whistling as well as the rest state. 3D facial landmarks and facial animation units (FAUs) were obtained using the Kinect V2, a fast and cost-effective depth camera. These were used to compute the features used in a Support Vector Machine (SVM) classifier. A dataset of 1650 records from 50 normal subjects was compiled for this study. The performances of different SVM kernel models were tested with different feature groups. The best performance (Accuracy = 96.7%, Sensitivity = 90.2%, and Specificity = 98%) was found when using the RBF kernel model applied on just nine differences in FAUs. This research will be developed and extended to include FP classification.

Design Guidelines of a Computer-Based Intervention for Computer Vision Syndrome: Focus Group Study and Real-World Deployment

Background

Prolonged time of computer use increases the prevalence of ocular problems, including eye strain, tired eyes, irritation, redness, blurred vision, and double vision, which are collectively referred to as computer vision syndrome (CVS). Approximately 70% of computer users have vision-related problems. For these reasons, properly designed interventions for users with CVS are required. To design an effective screen intervention for preventing or improving CVS, we must understand the effective interfaces of computer-based interventions.

Objective

In this study, we aimed to explore the interface elements of computer-based interventions for CVS to set design guidelines based on the pros and cons of each interface element.

Methods

We conducted an iterative user study to achieve our research objective. First, we conducted a workshop to evaluate the overall interface elements that were included in previous systems for CVS (n=7). Through the workshop, participants evaluated existing interface elements. Based on the evaluation results, we eliminated the elements that negatively affect intervention outcomes. Second, we designed our prototype system LiquidEye that includes multiple interface options (n=11). Interface options included interface elements that were positively evaluated in the workshop study. Lastly, we deployed LiquidEye in the real world to see how the included elements affected the intervention outcomes. Participants used LiquidEye for 14 days, and during this period, we collected participants’ daily logs (n=680). Additionally, we conducted prestudy and poststudy surveys, and poststudy interviews to explore how each interface element affects participation in the system.

Results

User data logs collected from the 14 days of deployment were analyzed with multiple regression analysis to explore the interface elements affecting user participation in the intervention (LiquidEye). Statistically significant elements were the instruction page of the eye resting strategy (P=.01), goal setting of the resting period (P=.009), compliment feedback after completing resting (P<.001), a mid-size popup window (P=.02), and CVS symptom-like effects (P=.004).

Conclusions

Based on the study results, we suggested design implications to consider when designing computer-based interventions for CVS. The sophisticated design of the customization interface can make it possible for users to use the system more interactively, which can result in higher engagement in managing eye conditions. There are important technical challenges that still need to be addressed, but given the fact that this study was able to clarify the various factors related to computer-based interventions, the findings are expected to contribute greatly to the research of various computer-based intervention designs in the future.

Automated Computer Vision Assessment of Hypomimia in Parkinson Disease: Proof-of-Principle Pilot Study

BACKGROUND

Facial expressions require the complex coordination of 43 different facial muscles. Parkinson disease (PD) affects facial musculature leading to "hypomimia" or "masked facies."

OBJECTIVE

We aimed to determine whether modern computer vision techniques can be applied to detect masked facies and quantify drug states in PD.

METHODS

We trained a convolutional neural network on images extracted from videos of 107 self-identified people with PD, along with 1595 videos of controls, in order to detect PD hypomimia cues. This trained model was applied to clinical interviews of 35 PD patients in their on and off drug motor states, and seven journalist interviews of the actor Alan Alda obtained before and after he was diagnosed with PD.

RESULTS

The algorithm achieved a test set area under the receiver operating characteristic curve of 0.71 on 54 subjects to detect PD hypomimia, compared to a value of 0.75 for trained neurologists using the United Parkinson Disease Rating Scale-III Facial Expression score. Additionally, the model accuracy to classify the on and off drug states in the clinical samples was 63% (22/35), in contrast to an accuracy of 46% (16/35) when using clinical rater scores. Finally, each of Alan Alda's seven interviews were successfully classified as occurring before (versus after) his diagnosis, with 100% accuracy (7/7).

CONCLUSIONS

This proof-of-principle pilot study demonstrated that computer vision holds promise as a valuable tool for PD hypomimia and for monitoring a patient's motor state in an objective and noninvasive way, particularly given the increasing importance of telemedicine.

Coronary Artery Disease Detection by Machine Learning with Coronary Bifurcation Features

Background: Early accurate detection of coronary artery disease (CAD) is one of the most important medical research areas. Researchers are motivated to utilize machine learning techniques for quick and accurate detection of CAD. Methods: To obtain the high quality of features used for machine learning, we here extracted the coronary bifurcation features from the coronary computed tomography angiography (CCTA) images by using the morphometric method. The machine learning classifier algorithms, such as logistic regression (LR), decision tree (DT), linear discriminant analysis (LDA), k-nearest neighbors (k-NN), artificial neural network (ANN), and support vector machine (SVM) were applied for estimating the performance by using the measured features. Results: The results showed that in comparison with other machine learning methods, the polynomial-SVM with the use of the grid search optimization method had the best performance for the detection of CAD and had yielded the classification accuracy of 100.00%. Among six examined coronary bifurcation features, the exponent of vessel diameter (n) and the area expansion ratio (AER) were two key features in the detection of CAD. Conclusions: This study could aid the clinicians to detect CAD accurately, which may probably provide an alternative method for the non-invasive diagnosis in clinical.

A Survey of Computer Vision Methods for 2D Object Detection from Unmanned Aerial Vehicles

The spread of Unmanned Aerial Vehicles (UAVs) in the last decade revolutionized many applications fields. Most investigated research topics focus on increasing autonomy during operational campaigns, environmental monitoring, surveillance, maps, and labeling. To achieve such complex goals, a high-level module is exploited to build semantic knowledge leveraging the outputs of the low-level module that takes data acquired from multiple sensors and extracts information concerning what is sensed. All in all, the detection of the objects is undoubtedly the most important low-level task, and the most employed sensors to accomplish it are by far RGB cameras due to costs, dimensions, and the wide literature on RGB-based object detection. This survey presents recent advancements in 2D object detection for the case of UAVs, focusing on the differences, strategies, and trade-offs between the generic problem of object detection, and the adaptation of such solutions for operations of the UAV. Moreover, a new taxonomy that considers different heights intervals and driven by the methodological approaches introduced by the works in the state of the art instead of hardware, physical and/or technological constraints is proposed.

When I Look into Your Eyes: A Survey on Computer Vision Contributions for Human Gaze Estimation and Tracking

The automatic detection of eye positions, their temporal consistency, and their mapping into a line of sight in the real world (to find where a person is looking at) is reported in the scientific literature as gaze tracking. This has become a very hot topic in the field of computer vision during the last decades, with a surprising and continuously growing number of application fields. A very long journey has been made from the first pioneering works, and this continuous search for more accurate solutions process has been further boosted in the last decade when deep neural networks have revolutionized the whole machine learning area, and gaze tracking as well. In this arena, it is being increasingly useful to find guidance through survey/review articles collecting most relevant works and putting clear pros and cons of existing techniques, also by introducing a precise taxonomy. This kind of manuscripts allows researchers and technicians to choose the better way to move towards their application or scientific goals. In the literature, there exist holistic and specifically technological survey documents (even if not updated), but, unfortunately, there is not an overview discussing how the great advancements in computer vision have impacted gaze tracking. Thus, this work represents an attempt to fill this gap, also introducing a wider point of view that brings to a new taxonomy (extending the consolidated ones) by considering gaze tracking as a more exhaustive task that aims at estimating gaze target from different perspectives: from the eye of the beholder (first-person view), from an external camera framing the beholder's, from a third-person view looking at the scene where the beholder is placed in, and from an external view independent from the beholder.

Machine Learning Techniques with ECG and EEG Data: An Exploratory Study

Electrocardiography (ECG) and electroencephalography (EEG) are powerful tools in medicine for the analysis of various diseases. The emergence of affordable ECG and EEG sensors and ubiquitous mobile devices provides an opportunity to make such analysis accessible to everyone. In this paper, we propose the implementation of a neural network-based method for the automatic identification of the relationship between the previously known conditions of older adults and the different features calculated from the various signals. The data were collected using a smartphone and low-cost ECG and EEG sensors during the performance of the timed-up and go test. Different patterns related to the features extracted, such as heart rate, heart rate variability, average QRS amplitude, average R-R interval, and average R-S interval from ECG data, and the frequency and variability from the EEG data were identified. A combination of these parameters allowed us to identify the presence of certain diseases accurately. The analysis revealed that the different institutions and ages were mainly identified. Still, the various diseases and groups of diseases were difficult to recognize, because the frequency of the different diseases was rare in the considered population. Therefore, the test should be performed with more people to achieve better results.