Head pose estimation in computer vision: a survey

Algorithmic and sensor-based research on Chinese children's and adolescents' screen use behavior and light environment

Background

Myopia poses a global health concern and is influenced by both genetic and environmental factors. The incidence of myopia tends to increase during infectious outbreaks, such as the COVID-19 pandemic. This study examined the screen-time behaviors among Chinese children and adolescents and investigated the efficacy of artificial intelligence (AI)-based alerts in modifying screen-time practices.

Methods

A cross-sectional analysis was performed using data from 6,716 children and adolescents with AI-enhanced tablets that monitored and recorded their behavior and environmental light during screen time.

Results

The median daily screen time of all participants was 58.82 min. Among all age groups, elementary-school students had the longest median daily screen time, which was 87.25 min and exceeded 4 h per week. Children younger than 2 years engaged with tablets for a median of 41.84 min per day. Learning accounted for 54.88% of participants' screen time, and 51.03% (3,390/6,643) of the participants used tablets for 1 h at an average distance <50 cm. The distance and posture alarms were triggered 807,355 and 509,199 times, respectively. In the study, 70.65% of the participants used the tablet under an illuminance of <300 lux during the day and 61.11% under an illuminance of <100 lux at night. The ambient light of 85.19% of the participants exceeded 4,000 K color temperature during night. Most incorrect viewing habits (65.49% in viewing distance; 86.48% in viewing posture) were rectified swiftly following AI notifications (all p < 0.05).

Conclusion

Young children are increasingly using digital screens, with school-age children and adolescents showing longer screen time than preschoolers. The study highlighted inadequate lighting conditions during screen use. AI alerts proved effective in prompting users to correct their screen-related behavior promptly.

Invoking and identifying task-oriented interlocutor confusion in human-robot interaction

Successful conversational interaction with a social robot requires not only an assessment of a user's contribution to an interaction, but also awareness of their emotional and attitudinal states as the interaction unfolds. To this end, our research aims to systematically trigger, but then interpret human behaviors to track different states of potential user confusion in interaction so that systems can be primed to adjust their policies in light of users entering confusion states. In this paper, we present a detailed human-robot interaction study to prompt, investigate, and eventually detect confusion states in users. The study itself employs a Wizard-of-Oz (WoZ) style design with a Pepper robot to prompt confusion states for task-oriented dialogues in a well-defined manner. The data collected from 81 participants includes audio and visual data, from both the robot's perspective and the environment, as well as participant survey data. From these data, we evaluated the correlations of induced confusion conditions with multimodal data, including eye gaze estimation, head pose estimation, facial emotion detection, silence duration time, and user speech analysis-including emotion and pitch analysis. Analysis shows significant differences of participants' behaviors in states of confusion based on these signals, as well as a strong correlation between confusion conditions and participants own self-reported confusion scores. The paper establishes strong correlations between confusion levels and these observable features, and lays the ground or a more complete social and affect oriented strategy for task-oriented human-robot interaction. The contributions of this paper include the methodology applied, dataset, and our systematic analysis.

Attention-Guided Huber Loss for Head Pose Estimation Based on Improved Capsule Network

Head pose estimation is an important technology for analyzing human behavior and has been widely researched and applied in areas such as human-computer interaction and fatigue detection. However, traditional head pose estimation networks suffer from the problem of easily losing spatial structure information, particularly in complex scenarios where occlusions and multiple object detections are common, resulting in low accuracy. To address the above issues, we propose a head pose estimation model based on the residual network and capsule network. Firstly, a deep residual network is used to extract features from three stages, capturing spatial structure information at different levels, and a global attention block is employed to enhance the spatial weight of feature extraction. To effectively avoid the loss of spatial structure information, the features are encoded and transmitted to the output using an improved capsule network, which is enhanced in its generalization ability through self-attention routing mechanisms. To enhance the robustness of the model, we optimize Huber loss, which is first used in head pose estimation. Finally, experiments are conducted on three popular public datasets, 300W-LP, AFLW2000, and BIWI. The results demonstrate that the proposed method achieves state-of-the-art results, particularly in scenarios with occlusions.

Analyzing students' attention by gaze tracking and object detection in classroom teaching

Purpose

Attention is one of the most important factors to affect the academic performance of students. Effectively analyzing students' attention in class can promote teachers' precise teaching and students' personalized learning. To intelligently analyze the students' attention in classroom from the first-person perspective, this paper proposes a fusion model based on gaze tracking and object detection. In particular, the proposed attention analysis model does not depend on any smart equipment.

Design/methodology/approach

Given a first-person view video of students' learning, the authors first estimate the gazing point by using the deep space–time neural network. Second, single shot multi-box detector and fast segmentation convolutional neural network are comparatively adopted to accurately detect the objects in the video. Third, they predict the gazing objects by combining the results of gazing point estimation and object detection. Finally, the personalized attention of students is analyzed based on the predicted gazing objects and the measurable eye movement criteria.

Findings

A large number of experiments are carried out on a public database and a new dataset that is built in a real classroom. The experimental results show that the proposed model not only can accurately track the students' gazing trajectory and effectively analyze the fluctuation of attention of the individual student and all students but also provide a valuable reference to evaluate the process of learning of students.

Originality/value

The contributions of this paper can be summarized as follows. The analysis of students' attention plays an important role in improving teaching quality and student achievement. However, there is little research on how to automatically and intelligently analyze students' attention. To alleviate this problem, this paper focuses on analyzing students' attention by gaze tracking and object detection in classroom teaching, which is significant for practical application in the field of education. The authors proposed an effectively intelligent fusion model based on the deep neural network, which mainly includes the gazing point module and the object detection module, to analyze students' attention in classroom teaching instead of relying on any smart wearable device. They introduce the attention mechanism into the gazing point module to improve the performance of gazing point detection and perform some comparison experiments on the public dataset to prove that the gazing point module can achieve better performance. They associate the eye movement criteria with visual gaze to get quantifiable objective data for students' attention analysis, which can provide a valuable basis to evaluate the learning process of students, provide useful learning information of students for both parents and teachers and support the development of individualized teaching. They built a new database that contains the first-person view videos of 11 subjects in a real classroom and employ it to evaluate the effectiveness and feasibility of the proposed model.

A Novel Zernike Moment-Based Real-Time Head Pose and Gaze Estimation Framework for Accuracy-Sensitive Applications

A real-time head pose and gaze estimation (HPGE) algorithm has excellent potential for technological advancements either in human-machine or human-robot interactions. For example, in high-accuracy advent applications such as Driver's Assistance System (DAS), HPGE plays a crucial role in omitting accidents and road hazards. In this paper, the authors propose a new hybrid framework for improved estimation by combining both the appearance and geometric-based conventional methods to extract local and global features. Therefore, the Zernike moments algorithm has been prominent in extracting rotation, scale, and illumination invariant features. Later, conventional discriminant algorithms were used to classify the head poses and gaze direction. Furthermore, the experiments were performed on standard datasets and real-time images to analyze the accuracy of the proposed algorithm. As a result, the proposed framework has immediately estimated the range of direction changes under different illumination conditions. We obtained an accuracy of ~85%; the average response time was 21.52 and 7.483 ms for estimating head poses and gaze, respectively, independent of illumination, background, and occlusion. The proposed method is promising for future developments of a robust system that is invariant even to blurring conditions and thus reaching much more significant performance enhancement.

Evaluation of Various State of the Art Head Pose Estimation Algorithms for Clinical Scenarios

Head pose assessment can reveal important clinical information on human motor control. Quantitative assessment have the potential to objectively evaluate head pose and movements' specifics, in order to monitor the progression of a disease or the effectiveness of a treatment. Optoelectronic camera-based motion-capture systems, recognized as a gold standard in clinical biomechanics, have been proposed for head pose estimation. However, these systems require markers to be positioned on the person's face which is impractical for everyday clinical practice. Furthermore, the limited access to this type of equipment and the emerging trend to assess mobility in natural environments support the development of algorithms capable of estimating head orientation using off-the-shelf sensors, such as RGB cameras. Although artificial vision is a popular field of research, limited validation of human pose estimation based on image recognition suitable for clinical applications has been performed. This paper first provides a brief review of available head pose estimation algorithms in the literature. Current state-of-the-art head pose algorithms designed to capture the facial geometry from videos, OpenFace 2.0, MediaPipe and 3DDFA_V2, are then further evaluated and compared. Accuracy is assessed by comparing both approaches to a baseline, measured with an optoelectronic camera-based motion-capture system. Results reveal a mean error lower or equal to 5.6∘ for 3DDFA_V2 depending on the plane of movement, while the mean error reaches 14.1∘ and 11.0∘ for OpenFace 2.0 and MediaPipe, respectively. This demonstrates the superiority of the 3DDFA_V2 algorithm in estimating head pose, in different directions of motion, and suggests that this algorithm can be used in clinical scenarios.

“No Chit Chat!” A Warning From a Physical Versus Virtual Robot Invigilator: Which Matters Most?

Past work has not considered social robots as proctors or monitors to prevent cheating or maintain discipline in the context of exam invigilation with adults. Further, we do not see an investigation into the role of invigilation for the robot presented in two different embodiments (physical vs. virtual). We demonstrate a system that enables a robot (physical and virtual) to act as an invigilator and deploy an exam setup with two participants completing a programming task. We conducted two studies (an online video-based survey and an in-person evaluation) to understand participants’ perceptions of the invigilator robot presented in two different embodiments. Additionally, we investigated whether participants showed cheating behaviours in one condition more than the other. The findings showed that participants’ ratings did not differ significantly. Further, participants were more talkative in the virtual robot condition compared to the physical robot condition. These findings are promising and call for further research into the invigilation role of social robots in more subtle and complex exam-like settings.

An Improved Tiered Head Pose Estimation Network with Self-Adjust Loss Function

As an important task in computer vision, head pose estimation has been widely applied in both academia and industry. However, there remains two challenges in the field of head pose estimation: (1) even given the same task (e.g., tiredness detection), the existing algorithms usually consider the estimation of the three angles (i.e., roll, yaw, and pitch) as separate facets, which disregard their interplay as well as differences and thus share the same parameters for all layers; and (2) the discontinuity in angle estimation definitely reduces the accuracy. To solve these two problems, a THESL-Net (tiered head pose estimation with self-adjust loss network) model is proposed in this study. Specifically, first, an idea of stepped estimation using distinct network layers is proposed, gaining a greater freedom during angle estimation. Furthermore, the reasons for the discontinuity in angle estimation are revealed, including not only labeling the dataset with quaternions or Euler angles, but also the loss function that simply adds the classification and regression losses. Subsequently, a self-adjustment constraint on the loss function is applied, making the angle estimation more consistent. Finally, to examine the influence of different angle ranges on the proposed model, experiments are conducted on three popular public benchmark datasets, BIWI, AFLW2000, and UPNA, demonstrating that the proposed model outperforms the state-of-the-art approaches.

Labeled-Robust Regression: Simultaneous Data Recovery and Classification

Rank minimization is widely used to extract low-dimensional subspaces. As a convex relaxation of the rank minimization, the problem of nuclear norm minimization has been attracting widespread attention. However, the standard nuclear norm minimization usually results in overcompression of data in all subspaces and eliminates the discrimination information between different categories of data. To overcome these drawbacks, in this article, we introduce the label information into the nuclear norm minimization problem and propose a labeled-robust principal component analysis (L-RPCA) to realize nuclear norm minimization on multisubspace data. Compared with the standard nuclear norm minimization, our method can effectively utilize the discriminant information in multisubspace rank minimization and avoid excessive elimination of local information and multisubspace characteristics of the data. Then, an effective labeled-robust regression (L-RR) method is proposed to simultaneously recover the data and labels of the observed data. Experiments on real datasets show that our proposed methods are superior to other state-of-the-art methods.

Gaze following requires early visual experience

Gaze understanding—a suggested precursor for understanding others’ intentions—requires recovery of gaze direction from the observed person's head and eye position. This challenging computation is naturally acquired at infancy without explicit external guidance, but can it be learned later if vision is extremely poor throughout early childhood? We addressed this question by studying gaze following in Ethiopian patients with early bilateral congenital cataracts diagnosed and treated by us only at late childhood. This sight restoration provided a unique opportunity to directly address basic issues on the roles of “nature” and “nurture” in development, as it caused a selective perturbation to the natural process, eliminating some gaze-direction cues while leaving others still available. Following surgery, the patients’ visual acuity typically improved substantially, allowing discrimination of pupil position in the eye. Yet, the patients failed to show eye gaze-following effects and fixated less than controls on the eyes—two spontaneous behaviors typically seen in controls. Our model for unsupervised learning of gaze direction explains how head-based gaze following can develop under severe image blur, resembling preoperative conditions. It also suggests why, despite acquiring sufficient resolution to extract eye position, automatic eye gaze following is not established after surgery due to lack of detailed early visual experience. We suggest that visual skills acquired in infancy in an unsupervised manner will be difficult or impossible to acquire when internal guidance is no longer available, even when sufficient image resolution for the task is restored. This creates fundamental barriers to spontaneous vision recovery following prolonged deprivation in early age.

Face Image Analysis Using Machine Learning: A Survey on Recent Trends and Applications

Human face image analysis using machine learning is an important element in computer vision. The human face image conveys information such as age, gender, identity, emotion, race, and attractiveness to both human and computer systems. Over the last ten years, face analysis methods using machine learning have received immense attention due to their diverse applications in various tasks. Although several methods have been reported in the last ten years, face image analysis still represents a complicated challenge, particularly for images obtained from ’in the wild’ conditions. This survey paper presents a comprehensive review focusing on methods in both controlled and uncontrolled conditions. Our work illustrates both merits and demerits of each method previously proposed, starting from seminal works on face image analysis and ending with the latest ideas exploiting deep learning frameworks. We show a comparison of the performance of the previous methods on standard datasets and also present some promising future directions on the topic.

Head Pose Estimation Based on Multivariate Label Distribution

Accurate ground-truth pose is essential to the training of most existing head pose estimation methods. However, in many cases, the "ground truth" pose is obtained in rather subjective ways, such as asking the subjects to stare at different markers on the wall. Thus it is better to use soft labels rather than explicit hard labels to indicate the pose of a face image. This paper proposes to associate a multivariate label distribution (MLD) to each image. An MLD covers a neighborhood around the original pose. Labeling the images with MLD can not only alleviate the problem of inaccurate pose labels, but also boost the training examples associated to each pose without actually increasing the total amount of training examples. Four algorithms are proposed to learn from MLD. Furthermore, an extension of MLD with the hierarchical structure is proposed to deal with fine-grained head pose estimation, which is named hierarchical multivariate label distribution (HMLD). Experimental results show that the MLD-based methods perform significantly better than the compared state-of-the-art head pose estimation algorithms. Moreover, the MLD-based methods appear much more robust against the label noise in the training set than the compared baseline methods.

Real-Time Gender Recognition for Juvenile and Adult Faces

Facial gender recognition is a crucial research topic due to its comprehensive use cases, including a demographic gender survey, visitor profile identification, targeted advertisement, access control, security, and surveillance from CCTV. For these real-time applications, the face of a person can be oriented to any angle from the camera axis, and the person can be of any age group, including juveniles. A child’s face consists of immature craniofacial feature points in texture and edge compared to an adult face, making it very hard to recognize gender using the child’s face. Real-word faces captured in an unconstrained environment make the gender prediction system more complex to identify correctly due to orientation. These factors reduce the accuracy of the existing state-of-the-art models developed so far for real-time facial gender prediction. This paper presents the novelty of facial gender recognition for juveniles, adults, and unconstrained-oriented faces. The progressive calibration network (PCN) detects rotation-invariant faces in the proposed model. Then, a Gabor filter is applied to extract unique edge and texture features from the detected face. The Gabor filter is invariant to illumination and produces texture and edge features with redundant feature coefficients in large dimensions. Gabor has drawbacks such as redundancy and a large dimension resolved by the proposed meanDWT feature optimization method, which optimizes the system’s accuracy, the size of the model, and computational timing. The proposed feature engineering model is classified with different classifiers such as Naïve Bayes, Logistic Regression, SVM with linear, and RBF kernel. Its results are compared with the state-of-the-art techniques; detailed experimental analysis is presented and concluded to support the argument. We also present a review of approaches based on conventional and deep learning methods with their pros and cons for facial gender recognition on different datasets available for facial gender recognition.

Detecting Groups and Estimating F-Formations for Social Human–Robot Interactions

The ability of a robot to detect and join groups of people is of increasing importance in social contexts, and for the collaboration between teams of humans and robots. In this paper, we propose a framework, autonomous group interactions for robots (AGIR), that endows a robot with the ability to detect such groups while following the principles of F-formations. Using on-board sensors, this method accounts for a wide spectrum of different robot systems, ranging from autonomous service robots to telepresence robots. The presented framework detects individuals, estimates their position and orientation, detects groups, determines their F-formations, and is able to suggest a position for the robot to enter the social group. For evaluation, two simulation scenes were developed based on the standard real-world datasets. The 1st scene is built with 20 virtual agents (VAs) interacting in 7 different groups of varying sizes and 3 different formations. The 2nd scene is built with 36 VAs, positioned in 13 different groups of varying sizes and 6 different formations. A model of a Pepper robot is used in both simulated scenes in randomly generated different positions. The ability for the robot to estimate orientation, detect groups, and estimate F-formations at various locations is used to determine the validation of the approaches. The obtained results show a high accuracy within each of the simulated scenarios and demonstrates that the framework is able to work from an egocentric view with a robot in real time.

A Study on the Teaching Design of a Hybrid Civics Course Based on the Improved Attention Mechanism

As an important vehicle for moral education, the moral indicators of civics and political science textbooks are naturally some of the most important criteria for revising textbooks. However, the textbook text dataset has too much textual information, ambiguous features, unbalanced sample distributions, etc. To address these problems, this paper combines a novel data enhancement method to obtain classification results based on word vectors. Additionally, for the problem of unbalanced sample sizes, this paper proposes a network model based on the attention mechanism, which combines the ideas of SMOTE and EDA, and uses a self-built stop word list and synonym word forest to conduct synonym queries, achieve a few categories of oversampling, and randomly disrupt the sentence order and intra-sentence word order to build a balanced dataset. The experimental results also show that the data augmentation method used in this paper’s model can effectively improve the performance of the model, resulting in a higher boost in the F1-measure of the model. The model incorporating the attention mechanism has better model generalization compared to the one without the attention mechanism, as well as a significant advantage compared to the reference model in other settings. The experimental results show that, compared with the original text classifier, the scheme of this paper effectively improves the evaluation effect and the reliability design for teaching a civics course.

Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction

This paper presents an implementation of RoSA, a Robot System Assistant, for safe and intuitive human-machine interaction. The interaction modalities were chosen and previously reviewed using a Wizard of Oz study emphasizing a strong propensity for speech and pointing gestures. Based on these findings, we design and implement a new multi-modal system for contactless human-machine interaction based on speech, facial, and gesture recognition. We evaluate our proposed system in an extensive study with multiple subjects to examine the user experience and interaction efficiency. It reports that our method achieves similar usability scores compared to the entirely human remote-controlled robot interaction in our Wizard of Oz study. Furthermore, our framework’s implementation is based on the Robot Operating System (ROS), allowing modularity and extendability for our multi-device and multi-user method.

Estimation of in-scanner head pose changes during structural MRI using a convolutional neural network trained on eye tracker video

INTRODUCTION

In-scanner head motion is a common cause of reduced image quality in neuroimaging, and causes systematic brain-wide changes in cortical thickness and volumetric estimates derived from structural MRI scans. There are few widely available methods for measuring head motion during structural MRI. Here, we train a deep learning predictive model to estimate changes in head pose using video obtained from an in-scanner eye tracker during an EPI-BOLD acquisition with participants undertaking deliberate in-scanner head movements. The predictive model was used to estimate head pose changes during structural MRI scans, and correlated with cortical thickness and subcortical volume estimates.

METHODS

21 healthy controls (age 32 ± 13 years, 11 female) were studied. Participants carried out a series of stereotyped prompted in-scanner head motions during acquisition of an EPI-BOLD sequence with simultaneous recording of eye tracker video. Motion-affected and motion-free whole brain T1-weighted MRI were also obtained. Image coregistration was used to estimate changes in head pose over the duration of the EPI-BOLD scan, and used to train a predictive model to estimate head pose changes from the video data. Model performance was quantified by assessing the coefficient of determination (R²). We evaluated the utility of our technique by assessing the relationship between video-based head pose changes during structural MRI and (i) vertex-wise cortical thickness and (ii) subcortical volume estimates.

RESULTS

Video-based head pose estimates were significantly correlated with ground truth head pose changes estimated from EPI-BOLD imaging in a hold-out dataset. We observed a general brain-wide overall reduction in cortical thickness with increased head motion, with some isolated regions showing increased cortical thickness estimates with increased motion. Subcortical volumes were generally reduced in motion affected scans.

CONCLUSIONS

We trained a predictive model to estimate changes in head pose during structural MRI scans using in-scanner eye tracker video. The method is independent of individual image acquisition parameters and does not require markers to be to be fixed to the patient, suggesting it may be well suited to clinical imaging and research environments. Head pose changes estimated using our approach can be used as covariates for morphometric image analyses to improve the neurobiological validity of structural imaging studies of brain development and disease.

Landmark-free, parametric hypothesis tests regarding two-dimensional contour shapes using coherent point drift registration and statistical parametric mapping

This paper proposes a computational framework for automated, landmark-free hypothesis testing of 2D contour shapes (i.e., shape outlines), and implements one realization of that framework. The proposed framework consists of point set registration, point correspondence determination, and parametric full-shape hypothesis testing. The results are calculated quickly (<2 s), yield morphologically rich detail in an easy-to-understand visualization, and are complimented by parametrically (or nonparametrically) calculated probability values. These probability values represent the likelihood that, in the absence of a true shape effect, smooth, random Gaussian shape changes would yield an effect as large as the observed one. This proposed framework nevertheless possesses a number of limitations, including sensitivity to algorithm parameters. As a number of algorithms and algorithm parameters could be substituted at each stage in the proposed data processing chain, sensitivity analysis would be necessary for robust statistical conclusions. In this paper, the proposed technique is applied to nine public datasets using a two-sample design, and an ANCOVA design is then applied to a synthetic dataset to demonstrate how the proposed method generalizes to the family of classical hypothesis tests. Extension to the analysis of 3D shapes is discussed.

Head Pose Estimation through Keypoints Matching between Reconstructed 3D Face Model and 2D Image

Mainstream methods treat head pose estimation as a supervised classification/regression problem, whose performance heavily depends on the accuracy of ground-truth labels of training data. However, it is rather difficult to obtain accurate head pose labels in practice, due to the lack of effective equipment and reasonable approaches for head pose labeling. In this paper, we propose a method which does not need to be trained with head pose labels, but matches the keypoints between a reconstructed 3D face model and the 2D input image, for head pose estimation. The proposed head pose estimation method consists of two components: the 3D face reconstruction and the 3D-2D matching keypoints. At the 3D face reconstruction phase, a personalized 3D face model is reconstructed from the input head image using convolutional neural networks, which are jointly optimized by an asymmetric Euclidean loss and a keypoint loss. At the 3D-2D keypoints matching phase, an iterative optimization algorithm is proposed to match the keypoints between the reconstructed 3D face model and the 2D input image efficiently under the constraint of perspective transformation. The proposed method is extensively evaluated on five widely used head pose estimation datasets, including Pointing'04, BIWI, AFLW2000, Multi-PIE, and Pandora. The experimental results demonstrate that the proposed method achieves excellent cross-dataset performance and surpasses most of the existing state-of-the-art approaches, with average MAEs of 4.78∘ on Pointing'04, 6.83∘ on BIWI, 7.05∘ on AFLW2000, 5.47∘ on Multi-PIE, and 5.06∘ on Pandora, although the model of the proposed method is not trained on any of these five datasets.

FASHE: A FrActal Based Strategy for Head Pose Estimation

Head pose estimation (HPE) represents a topic central to many relevant research fields and characterized by a wide application range. In particular, HPE performed using a singular RGB frame is particular suitable to be applied at best-frame-selection problems. This explains a growing interest witnessed by a large number of contributions, most of which exploit deep learning architectures and require extensive training sessions to achieve accuracy and robustness in estimating head rotations on three axes. However, methods alternative to machine learning approaches could be capable of similar if not better performance. To this regard, we present FASHE, an approach based on partitioned iterated function systems (PIFS) to represent auto-similarities within face image through a contractive affine function transforming the domain blocks extracted only once by a single frontal reference image, in a good approximation of the range blocks which the target image has been partitioned into. Pose estimation is achieved by finding the closest match between fractal code of target image and a reference array by means of Hamming distance. The results of experiments conducted exceed the state of the art on both Biwi and Ponting'04 datasets as well as approaching those of the best performing methods on the challenging AFLW2000 database. In addition, the applications to GOTCHA Video Dataset demonstrate that FASHE successfully operates in-the-wild.

Computer Vision-Based Assessment of Motor Functioning in Schizophrenia: Use of Smartphones for Remote Measurement of Schizophrenia Symptomatology

Introduction

Motor abnormalities have been shown to be a distinct component of schizophrenia symptomatology. However, objective and scalable methods for assessment of motor functioning in schizophrenia are lacking. Advancements in machine learning-based digital tools have allowed for automated and remote "digital phenotyping" of disease symptomatology. Here, we assess the performance of a computer vision-based assessment of motor functioning as a characteristic of schizophrenia using video data collected remotely through smartphones.

Methods

Eighteen patients with schizophrenia and 9 healthy controls were asked to remotely participate in smartphone-based assessments daily for 14 days. Video recorded from the smartphone front-facing camera during these assessments was used to quantify the Euclidean distance of head movement between frames through a pretrained computer vision model. The ability of head movement measurements to distinguish between patients and healthy controls as well as their relationship to schizophrenia symptom severity as measured through traditional clinical scores was assessed.

Results

The rate of head movement in participants with schizophrenia (1.48 mm/frame) and those without differed significantly (2.50 mm/frame; p = 0.01), and a logistic regression demonstrated that head movement was a significant predictor of schizophrenia diagnosis (p = 0.02). Linear regression between head movement and clinical scores of schizophrenia showed that head movement has a negative relationship with schizophrenia symptom severity (p = 0.04), primarily with negative symptoms of schizophrenia.

Conclusions

Remote, smartphone-based assessments were able to capture meaningful visual behavior for computer vision-based objective measurement of head movement. The measurements of head movement acquired were able to accurately classify schizophrenia diagnosis and quantify symptom severity in patients with schizophrenia.

Driver Distraction Detection Method Based on Continuous Head Pose Estimation

In view of the fact that the detection of driver’s distraction is a burning issue, this study chooses the driver’s head pose as the evaluation parameter for driving distraction and proposes a driver distraction method based on the head pose. The effects of single regression and classification combined with regression are compared in terms of accuracy, and four kinds of classical networks are improved and trained using 300W-LP and AFLW datasets. The HPE_Resnet50 with the best accuracy is selected as the head pose estimator and applied to the ten-category distracted driving dataset SF3D to obtain 20,000 sets of head pose data. The differences between classes are discussed qualitatively and quantitatively. The analysis of variance shows that there is a statistically significant difference in head posture between safe driving and all kinds of distracted driving at 95% and 90% confidence levels, and the postures of all kinds of driving movements are distributed in a specific Euler angle range, which provides a characteristic basis for the design of subsequent recognition methods. In addition, according to the continuity of human movement, this paper also selects 90 drivers’ videos to analyze the difference in head pose between safe driving and distracted driving frame by frame. By calculating the spatial distance and sample statistics, the results provide the reference point, spatial range, and threshold of safe driving under this driving condition. Experimental results show that the average error of HPE_Resnet50 in AFLW2000 is 6.17° and that there is an average difference of 12.4° to 54.9° in the Euler angle between safe driving and nine kinds of distracted driving on SF3D.

Utilizing minicomputer technology for low-cost photorefraction: a feasibility study

Eccentric photorefraction is an objective technique to determine the refractive errors of the eye. To address the rise in prevalence of visual impairment, especially in rural areas, a minicomputer-based low-cost infrared photorefractor was developed using off-the-shelf hardware components. Clinical validation revealed that the developed infrared photorefractor demonstrated a linear working range between +4.0 D and -6.0 D at 50 cm. Further, measurement of astigmatism from human eye showed absolute error for cylinder of 0.3 D and high correlation for axis assessment. To conclude, feasibility was shown for a low-cost, portable and low-power driven stand-alone device to objectively determine refractive errors, showing potential for screening applications. The developed photorefractor creates a new avenue for telemedicine for ophthalmic measurements.

Customized Access Technology for Children using Head Movement Recognition

Children with cerebral palsy and complex communication needs face limitations in their access technology (AT) usage. Speech recognition software and conventional ATs (e.g., mechanical switches) can be insufficient for those with speech impairment and limited control of voluntary motion. Automatic recognition of head movements represents a promising pathway. Previous studies have shown the robustness of head pose estimation algorithms on adult participants, but further research is needed to use these methods with children. An algorithm for head movement recognition was implemented and evaluated on videos recorded in a naturalistic environment when children were playing a videogame. A face-tracking algorithm was used to detect the main facial landmarks. Head poses were then estimated using the Pose from Orthography and Scaling with Iterations (POSIT) algorithm and three head movements were classified through Hidden Markov Models (HMMs). Preliminary classification results obtained from the analysis of videos of five typically developing children showed an accuracy of up to 95.6% in predicting head movements.

FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network

Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem.

When I Look into Your Eyes: A Survey on Computer Vision Contributions for Human Gaze Estimation and Tracking

The automatic detection of eye positions, their temporal consistency, and their mapping into a line of sight in the real world (to find where a person is looking at) is reported in the scientific literature as gaze tracking. This has become a very hot topic in the field of computer vision during the last decades, with a surprising and continuously growing number of application fields. A very long journey has been made from the first pioneering works, and this continuous search for more accurate solutions process has been further boosted in the last decade when deep neural networks have revolutionized the whole machine learning area, and gaze tracking as well. In this arena, it is being increasingly useful to find guidance through survey/review articles collecting most relevant works and putting clear pros and cons of existing techniques, also by introducing a precise taxonomy. This kind of manuscripts allows researchers and technicians to choose the better way to move towards their application or scientific goals. In the literature, there exist holistic and specifically technological survey documents (even if not updated), but, unfortunately, there is not an overview discussing how the great advancements in computer vision have impacted gaze tracking. Thus, this work represents an attempt to fill this gap, also introducing a wider point of view that brings to a new taxonomy (extending the consolidated ones) by considering gaze tracking as a more exhaustive task that aims at estimating gaze target from different perspectives: from the eye of the beholder (first-person view), from an external camera framing the beholder's, from a third-person view looking at the scene where the beholder is placed in, and from an external view independent from the beholder.

Static, Dynamic, and Cognitive Fit of Exosystems for the Human Operator

OBJECTIVE

To define static, dynamic, and cognitive fit and their interactions as they pertain to exosystems and to document open research needs in using these fit characteristics to inform exosystem design.

BACKGROUND

Initial exosystem sizing and fit evaluations are currently based on scalar anthropometric dimensions and subjective assessments. As fit depends on ongoing interactions related to task setting and user, attempts to tailor equipment have limitations when optimizing for this limited fit definition.

METHOD

A targeted literature review was conducted to inform a conceptual framework defining three characteristics of exosystem fit: static, dynamic, and cognitive. Details are provided on the importance of differentiating fit characteristics for developing exosystems.

RESULTS

Static fit considers alignment between human and equipment and requires understanding anthropometric characteristics of target users and geometric equipment features. Dynamic fit assesses how the human and equipment move and interact with each other, with a focus on the relative alignment between the two systems. Cognitive fit considers the stages of human-information processing, including somatosensation, executive function, and motor selection. Human cognitive capabilities should remain available to process task- and stimulus-related information in the presence of an exosystem. Dynamic and cognitive fit are operationalized in a task-specific manner, while static fit can be considered for predefined postures.

CONCLUSION

A deeper understanding of how an exosystem fits an individual is needed to ensure good human-system performance. Development of methods for evaluating different fit characteristics is necessary.

APPLICATION

Methods are presented to inform exosystem evaluation across physical and cognitive characteristics.

Towards Wide Range Tracking of Head Scanning Movement in Driving

Gaining environmental awareness through lateral head scanning (yaw rotations) is important for driving safety, especially when approaching intersections. Therefore, head scanning movements could be an important behavioral metric for driving safety research and driving risk mitigation systems. Tracking head scanning movements with a single in-car camera is preferred hardware-wise, but it is very challenging to track the head over almost a 180° range. In this paper we investigate two state-of-the-art methods, a multi-loss deep residual learning method with 50 layers (multi-loss ResNet-50) and an ORB feature-based simultaneous localization and mapping method (ORB-SLAM). While deep learning methods have been extensively studied for head pose detection, this is the first study in which SLAM has been employed to innovatively track head scanning over a very wide range. Our laboratory experimental results showed that ORB-SLAM was more accurate than multi-loss ResNet-50, which often failed when many facial features were not in the view. On the contrary, ORB-SLAM was able to continue tracking as it doesn't rely on particular facial features. Testing with real driving videos demonstrated the feasibility of using ORB-SLAM for tracking large lateral head scans in naturalistic video data.

Face-from-Depth for Head Pose Estimation on Depth Images

Depth cameras allow to set up reliable solutions for people monitoring and behavior understanding, especially when unstable or poor illumination conditions make unusable common RGB sensors. Therefore, we propose a complete framework for the estimation of the head and shoulder pose based on depth images only. A head detection and localization module is also included, in order to develop a complete end-to-end system. The core element of the framework is a Convolutional Neural Network, called POSEidon ⁺, that receives as input three types of images and provides the 3D angles of the pose as output. Moreover, a Face-from-Depth component based on a Deterministic Conditional GAN model is able to hallucinate a face from the corresponding depth image. We empirically demonstrate that this positively impacts the system performances. We test the proposed framework on two public datasets, namely Biwi Kinect Head Pose and ICT-3DHP, and on Pandora, a new challenging dataset mainly inspired by the automotive setup. Experimental results show that our method overcomes several recent state-of-art works based on both intensity and depth input data, running in real-time at more than 30 frames per second.

A Multi-Task Framework for Facial Attributes Classification through End-to-End Face Parsing and Deep Convolutional Neural Networks

Human face image analysis is an active research area within computer vision. In this paper we propose a framework for face image analysis, addressing three challenging problems of race, age, and gender recognition through face parsing. We manually labeled face images for training an end-to-end face parsing model through Deep Convolutional Neural Networks. The deep learning-based segmentation model parses a face image into seven dense classes. We use the probabilistic classification method and created probability maps for each face class. The probability maps are used as feature descriptors. We trained another Convolutional Neural Network model by extracting features from probability maps of the corresponding class for each demographic task (race, age, and gender). We perform extensive experiments on state-of-the-art datasets and obtained much better results as compared to previous results.

Use of information modelling techniques to understand research trends in eye gaze estimation methods: An automated review

Eye gaze tracking has been used to study the influence of visual stimuli on consumer behavior and attentional processes. Eye gaze tracking techniques have made substantial contributions in advertisement design, human computer interaction, virtual reality and disease diagnosis. Eye gaze estimation is considered critical for prediction of human attention, and hence indispensable for better understanding human activities. In this paper, Latent Semantic Analysis is used to develop an information model for identifying emerging research trends within eye gaze estimation techniques. An exhaustive collection of 423 titles and abstracts of research papers published during 2005-2018 were used. Five major research areas and ten research trends were classified based upon this study.

Low cost gaze estimation: knowledge-based solutions

Eye tracking technology in low resolution scenarios is not a completely solved issue to date. The possibility of using eye tracking in a mobile gadget is a challenging objective that would permit to spread this technology to non-explored fields. In this paper, a knowledge based approach is presented to solve gaze estimation in low resolution settings. The understanding of the high resolution paradigm permits to propose alternative models to solve gaze estimation. In this manner, three models are presented: a geometrical model, an interpolation model and a compound model, as solutions for gaze estimation for remote low resolution systems. Since this work considers head position essential to improve gaze accuracy, a method for head pose estimation is also proposed. The methods are validated in an optimal framework, I2Head database, which combines head and gaze data. The experimental validation of the models demonstrates their sensitivity to image processing inaccuracies, critical in the case of the geometrical model. Static and extreme movement scenarios are analyzed showing the higher robustness of compound and geometrical models in the presence of user's displacement. Accuracy values of about 3° have been obtained, increasing to values close to 5° in extreme displacement settings, results fully comparable with the state-of-the-art.

3D Approaches and Challenges in Facial Expression Recognition Algorithms—A Literature Review

In recent years, facial expression analysis and recognition (FER) have emerged as an active research topic with applications in several different areas, including the human-computer interaction domain. Solutions based on 2D models are not entirely satisfactory for real-world applications, as they present some problems of pose variations and illumination related to the nature of the data. Thanks to technological development, 3D facial data, both still images and video sequences, have become increasingly used to improve the accuracy of FER systems. Despite the advance in 3D algorithms, these solutions still have some drawbacks that make pure three-dimensional techniques convenient only for a set of specific applications; a viable solution to overcome such limitations is adopting a multimodal 2D+3D analysis. In this paper, we analyze the limits and strengths of traditional and deep-learning FER techniques, intending to provide the research community an overview of the results obtained looking to the next future. Furthermore, we describe in detail the most used databases to address the problem of facial expressions and emotions, highlighting the results obtained by the various authors. The different techniques used are compared, and some conclusions are drawn concerning the best recognition rates achieved.

A Unified Framework for Head Pose, Age and Gender Classification through End-to-End Face Segmentation

Accurate face segmentation strongly benefits the human face image analysis problem. In this paper we propose a unified framework for face image analysis through end-to-end semantic face segmentation. The proposed framework contains a set of stack components for face understanding, which includes head pose estimation, age classification, and gender recognition. A manually labeled face data-set is used for training the Conditional Random Fields (CRFs) based segmentation model. A multi-class face segmentation framework developed through CRFs segments a facial image into six parts. The probabilistic classification strategy is used, and probability maps are generated for each class. The probability maps are used as features descriptors and a Random Decision Forest (RDF) classifier is modeled for each task (head pose, age, and gender). We assess the performance of the proposed framework on several data-sets and report better results as compared to the previously reported results.