1
|
Paparelli A, Sokhn N, Stacchi L, Coutrot A, Richoz AR, Caldara R. Idiosyncratic fixation patterns generalize across dynamic and static facial expression recognition. Sci Rep 2024; 14:16193. [PMID: 39003314 PMCID: PMC11246522 DOI: 10.1038/s41598-024-66619-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2024] [Accepted: 07/02/2024] [Indexed: 07/15/2024] Open
Abstract
Facial expression recognition (FER) is crucial for understanding the emotional state of others during human social interactions. It has been assumed that humans share universal visual sampling strategies to achieve this task. However, recent studies in face identification have revealed striking idiosyncratic fixation patterns, questioning the universality of face processing. More importantly, very little is known about whether such idiosyncrasies extend to the biological relevant recognition of static and dynamic facial expressions of emotion (FEEs). To clarify this issue, we tracked observers' eye movements categorizing static and ecologically valid dynamic faces displaying the six basic FEEs, all normalized for time presentation (1 s), contrast and global luminance across exposure time. We then used robust data-driven analyses combining statistical fixation maps with hidden Markov Models to explore eye-movements across FEEs and stimulus modalities. Our data revealed three spatially and temporally distinct equally occurring face scanning strategies during FER. Crucially, such visual sampling strategies were mostly comparably effective in FER and highly consistent across FEEs and modalities. Our findings show that spatiotemporal idiosyncratic gaze strategies also occur for the biologically relevant recognition of FEEs, further questioning the universality of FER and, more generally, face processing.
Collapse
Affiliation(s)
- Anita Paparelli
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Faucigny 2, 1700, Fribourg, Switzerland
| | - Nayla Sokhn
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Faucigny 2, 1700, Fribourg, Switzerland
| | - Lisa Stacchi
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Faucigny 2, 1700, Fribourg, Switzerland
| | - Antoine Coutrot
- Laboratoire d'Informatique en Image Et Systèmes d'information, French Centre National de La Recherche Scientifique, University of Lyon, Lyon, France
| | - Anne-Raphaëlle Richoz
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Faucigny 2, 1700, Fribourg, Switzerland
| | - Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Faucigny 2, 1700, Fribourg, Switzerland.
| |
Collapse
|
2
|
Hsiao JHW. Understanding Human Cognition Through Computational Modeling. Top Cogn Sci 2024; 16:349-376. [PMID: 38781432 DOI: 10.1111/tops.12737] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 05/07/2024] [Accepted: 05/08/2024] [Indexed: 05/25/2024]
Abstract
One important goal of cognitive science is to understand the mind in terms of its representational and computational capacities, where computational modeling plays an essential role in providing theoretical explanations and predictions of human behavior and mental phenomena. In my research, I have been using computational modeling, together with behavioral experiments and cognitive neuroscience methods, to investigate the information processing mechanisms underlying learning and visual cognition in terms of perceptual representation and attention strategy. In perceptual representation, I have used neural network models to understand how the split architecture in the human visual system influences visual cognition, and to examine perceptual representation development as the results of expertise. In attention strategy, I have developed the Eye Movement analysis with Hidden Markov Models method for quantifying eye movement pattern and consistency using both spatial and temporal information, which has led to novel findings across disciplines not discoverable using traditional methods. By integrating it with deep neural networks (DNN), I have developed DNN+HMM to account for eye movement strategy learning in human visual cognition. The understanding of the human mind through computational modeling also facilitates research on artificial intelligence's (AI) comparability with human cognition, which can in turn help explainable AI systems infer humans' belief on AI's operations and provide human-centered explanations to enhance human-AI interaction and mutual understanding. Together, these demonstrate the essential role of computational modeling methods in providing theoretical accounts of the human mind as well as its interaction with its environment and AI systems.
Collapse
|
3
|
Qi R, Zheng Y, Yang Y, Cao CC, Hsiao JH. Explanation strategies in humans versus current explainable artificial intelligence: Insights from image classification. Br J Psychol 2024. [PMID: 38858823 DOI: 10.1111/bjop.12714] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2023] [Accepted: 05/22/2024] [Indexed: 06/12/2024]
Abstract
Explainable AI (XAI) methods provide explanations of AI models, but our understanding of how they compare with human explanations remains limited. Here, we examined human participants' attention strategies when classifying images and when explaining how they classified the images through eye-tracking and compared their attention strategies with saliency-based explanations from current XAI methods. We found that humans adopted more explorative attention strategies for the explanation task than the classification task itself. Two representative explanation strategies were identified through clustering: One involved focused visual scanning on foreground objects with more conceptual explanations, which contained more specific information for inferring class labels, whereas the other involved explorative scanning with more visual explanations, which were rated higher in effectiveness for early category learning. Interestingly, XAI saliency map explanations had the highest similarity to the explorative attention strategy in humans, and explanations highlighting discriminative features from invoking observable causality through perturbation had higher similarity to human strategies than those highlighting internal features associated with higher class score. Thus, humans use both visual and conceptual information during explanation, which serve different purposes, and XAI methods that highlight features informing observable causality match better with human explanations, potentially more accessible to users.
Collapse
Affiliation(s)
- Ruoxi Qi
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China
| | - Yueyuan Zheng
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China
- Huawei Research Hong Kong, Hong Kong SAR, China
| | - Yi Yang
- Huawei Research Hong Kong, Hong Kong SAR, China
| | - Caleb Chen Cao
- Huawei Research Hong Kong, Hong Kong SAR, China
- Big Data Institute, Hong Kong University of Science and Technology, Hong Kong SAR, China
| | - Janet H Hsiao
- Division of Social Science, Hong Kong University of Science and Technology, Hong Kong SAR, China
| |
Collapse
|
4
|
Mohamed Selim A, Barz M, Bhatti OS, Alam HMT, Sonntag D. A review of machine learning in scanpath analysis for passive gaze-based interaction. Front Artif Intell 2024; 7:1391745. [PMID: 38903158 PMCID: PMC11188426 DOI: 10.3389/frai.2024.1391745] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 05/15/2024] [Indexed: 06/22/2024] Open
Abstract
The scanpath is an important concept in eye tracking. It refers to a person's eye movements over a period of time, commonly represented as a series of alternating fixations and saccades. Machine learning has been increasingly used for the automatic interpretation of scanpaths over the past few years, particularly in research on passive gaze-based interaction, i.e., interfaces that implicitly observe and interpret human eye movements, with the goal of improving the interaction. This literature review investigates research on machine learning applications in scanpath analysis for passive gaze-based interaction between 2012 and 2022, starting from 2,425 publications and focussing on 77 publications. We provide insights on research domains and common learning tasks in passive gaze-based interaction and present common machine learning practices from data collection and preparation to model selection and evaluation. We discuss commonly followed practices and identify gaps and challenges, especially concerning emerging machine learning topics, to guide future research in the field.
Collapse
Affiliation(s)
- Abdulrahman Mohamed Selim
- German Research Center for Artificial Intelligence (DFKI), Interactive Machine Learning Department, Saarbrücken, Germany
| | - Michael Barz
- German Research Center for Artificial Intelligence (DFKI), Interactive Machine Learning Department, Saarbrücken, Germany
- Applied Artificial Intelligence, University of Oldenburg, Oldenburg, Germany
| | - Omair Shahzad Bhatti
- German Research Center for Artificial Intelligence (DFKI), Interactive Machine Learning Department, Saarbrücken, Germany
| | - Hasan Md Tusfiqur Alam
- German Research Center for Artificial Intelligence (DFKI), Interactive Machine Learning Department, Saarbrücken, Germany
| | - Daniel Sonntag
- German Research Center for Artificial Intelligence (DFKI), Interactive Machine Learning Department, Saarbrücken, Germany
- Applied Artificial Intelligence, University of Oldenburg, Oldenburg, Germany
| |
Collapse
|
5
|
Butcher N, Bennetts RJ, Sexton L, Barbanta A, Lander K. Eye movement differences when recognising and learning moving and static faces. Q J Exp Psychol (Hove) 2024:17470218241252145. [PMID: 38644390 DOI: 10.1177/17470218241252145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/23/2024]
Abstract
Seeing a face in motion can help subsequent face recognition. Several explanations have been proposed for this "motion advantage," but other factors that might play a role have received less attention. For example, facial movement might enhance recognition by attracting attention to the internal facial features, thereby facilitating identification. However, there is no direct evidence that motion increases attention to regions of the face that facilitate identification (i.e., internal features) compared with static faces. We tested this hypothesis by recording participants' eye movements while they completed the famous face recognition (Experiment 1, N = 32), and face-learning (Experiment 2, N = 60, Experiment 3, N = 68) tasks, with presentation style manipulated (moving or static). Across all three experiments, a motion advantage was found, and participants directed a higher proportion of fixations to the internal features (i.e., eyes, nose, and mouth) of moving faces versus static. Conversely, the proportion of fixations to the internal non-feature area (i.e., cheeks, forehead, chin) and external area (Experiment 3) was significantly reduced for moving compared with static faces (all ps < .05). Results suggest that during both familiar and unfamiliar face recognition, facial motion is associated with increased attention to internal facial features, but only during familiar face recognition is the magnitude of the motion advantage significantly related functionally to the proportion of fixations directed to the internal features.
Collapse
Affiliation(s)
- Natalie Butcher
- Department of Psychology, Teesside University, Middlesbrough, UK
| | | | - Laura Sexton
- Department of Psychology, Teesside University, Middlesbrough, UK
- School of Psychology, Faculty of Health Sciences and Wellbeing, University of Sunderland, Sunderland, UK
| | | | - Karen Lander
- Division of Psychology, Communication and Human Neuroscience, University of Manchester, Manchester, UK
| |
Collapse
|
6
|
Xu K. Insights into the relationship between eye movements and personality traits in restricted visual fields. Sci Rep 2024; 14:10261. [PMID: 38704441 PMCID: PMC11069522 DOI: 10.1038/s41598-024-60992-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2024] [Accepted: 04/30/2024] [Indexed: 05/06/2024] Open
Abstract
Previous studies have suggested behavioral patterns, such as visual attention and eye movements, relate to individual personality traits. However, these studies mainly focused on free visual tasks, and the impact of visual field restriction remains inadequately understood. The primary objective of this study is to elucidate the patterns of conscious eye movements induced by visual field restriction and to examine how these patterns relate to individual personality traits. Building on previous research, we aim to gain new insights through two behavioral experiments, unraveling the intricate relationship between visual behaviors and individual personality traits. As a result, both Experiment 1 and Experiment 2 revealed differences in eye movements during free observation and visual field restriction. Particularly, simulation results based on the analyzed data showed clear distinctions in eye movements between free observation and visual field restriction conditions. This suggests that eye movements during free observation involve a mixture of conscious and unconscious eye movements. Furthermore, we observed significant correlations between conscious eye movements and personality traits, with more pronounced effects in the visual field restriction condition used in Experiment 2 compared to Experiment 1. These analytical findings provide a novel perspective on human cognitive processes through visual perception.
Collapse
Affiliation(s)
- Kuangzhe Xu
- Institute for Promotion of Higher Education, Hirosaki University, Aomori, 036-8560, Japan.
| |
Collapse
|
7
|
Viktorsson C, Valtakari NV, Falck-Ytter T, Hooge ITC, Rudling M, Hessels RS. Stable eye versus mouth preference in a live speech-processing task. Sci Rep 2023; 13:12878. [PMID: 37553414 PMCID: PMC10409748 DOI: 10.1038/s41598-023-40017-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Accepted: 08/03/2023] [Indexed: 08/10/2023] Open
Abstract
Looking at the mouth region is thought to be a useful strategy for speech-perception tasks. The tendency to look at the eyes versus the mouth of another person during speech processing has thus far mainly been studied using screen-based paradigms. In this study, we estimated the eye-mouth-index (EMI) of 38 adult participants in a live setting. Participants were seated across the table from an experimenter, who read sentences out loud for the participant to remember in both a familiar (English) and unfamiliar (Finnish) language. No statistically significant difference in the EMI between the familiar and the unfamiliar languages was observed. Total relative looking time at the mouth also did not predict the number of correctly identified sentences. Instead, we found that the EMI was higher during an instruction phase than during the speech-processing task. Moreover, we observed high intra-individual correlations in the EMI across the languages and different phases of the experiment. We conclude that there are stable individual differences in looking at the eyes versus the mouth of another person. Furthermore, this behavior appears to be flexible and dependent on the requirements of the situation (speech processing or not).
Collapse
Affiliation(s)
- Charlotte Viktorsson
- Development and Neurodiversity Lab, Department of Psychology, Uppsala University, Uppsala, Sweden.
| | - Niilo V Valtakari
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | - Terje Falck-Ytter
- Development and Neurodiversity Lab, Department of Psychology, Uppsala University, Uppsala, Sweden
- Center of Neurodevelopmental Disorders (KIND), Division of Neuropsychiatry, Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden
| | - Ignace T C Hooge
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | - Maja Rudling
- Development and Neurodiversity Lab, Department of Psychology, Uppsala University, Uppsala, Sweden
| | - Roy S Hessels
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
8
|
Frank K, Ehrl D, Bernardini F, Walbrun A, Moellhoff N, Alfertshofer M, Davidovic K, Mardini S, Gotkin RH, Cotofana S. How We Look At Mature Faces-An Eye-Tracking Investigation Into the Perception of Age. Aesthet Surg J 2023; 43:115-122. [PMID: 36099471 DOI: 10.1093/asj/sjac251] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 09/06/2022] [Accepted: 09/06/2022] [Indexed: 02/07/2023] Open
Abstract
BACKGROUND It is still unclear which facial region contributes most to the perception of an aged face when evaluated by eye-tracking analyses. OBJECTIVES The authors sought to apply eye-tracking technology to identify whether mature faces require longer fixation durations than young faces and which facial region contributes most to the perception of a mature face. METHODS Eye-tracking analyses were conducted in 74 volunteers (37 males, 37 females; 43 ≤ 40 years, 31 > 40 years) evaluating their gaze pattern and the fixation durations for the entire face and 9 facial subregions. Frontal facial images of 16 younger (<40 years) and older (>40 years) gender-matched individuals were presented in a standardized setting. RESULTS Independent of age or gender of the observer, a younger stimulus image was viewed shorter than an older stimulus image with 0.82 (0.63) seconds vs 1.06 (0.73) seconds with P < 0.001. There was no statistically significant difference in their duration of a stable eye fixation when observers inspected a male vs a female stimulus image [0.94 (0.70) seconds vs 0.94 (0.68) seconds; P = 0.657] independent of the observer's age or gender. The facial image that captured the most attention of the observer (rank 9) was the perioral region with 1.61 (0.73) seconds for younger observers and 1.57 (0.73) seconds for older observers. CONCLUSIONS It was revealed that the perioral region attracts the most attention of observers and contributes most to an aged facial appearance. Practitioners should be mindful of the importance of the perioral region when designing an aesthetic treatment plan.
Collapse
Affiliation(s)
- Konstantin Frank
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, Germany
| | - Denis Ehrl
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, Germany
| | | | - Alina Walbrun
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, Germany
| | - Nicholas Moellhoff
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, Germany
| | - Michael Alfertshofer
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, Germany
| | - Kristina Davidovic
- Department of Radiology and Medical School, University of Belgrade, Belgrade, Serbia
| | - Samir Mardini
- Department of Plastic and Reconstructive Surgery, Mayo Clinic, Rochester, MN, USA
| | | | - Sebastian Cotofana
- Department of Clinical Anatomy, Mayo Clinic College of Medicine and Science, Rochester, MN, USA
| |
Collapse
|
9
|
Wong HK, Keeble DRT, Stephen ID. Do they 'look' different(ly)? Dynamic face recognition in Malaysians: Chinese, Malays and Indians compared. Br J Psychol 2023; 114 Suppl 1:134-149. [PMID: 36647242 DOI: 10.1111/bjop.12629] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2021] [Revised: 12/26/2022] [Accepted: 12/28/2022] [Indexed: 01/18/2023]
Abstract
Previous cross-cultural eye-tracking studies examining face recognition discovered differences in the eye movement strategies that observers employ when perceiving faces. However, it is unclear (1) the degree to which this effect is fundamentally related to culture and (2) to what extent facial physiognomy can account for the differences in looking strategies when scanning own- and other-race faces. In the current study, Malay, Chinese and Indian young adults who live in the same multiracial country performed a modified yes/no recognition task. Participants' recognition accuracy and eye movements were recorded while viewing muted face videos of own- and other-race individuals. Behavioural results revealed a clear own-race advantage in recognition memory, and eye-tracking results showed that the three ethnic race groups adopted dissimilar fixation patterns when perceiving faces. Chinese participants preferentially attended more to the eyes than Indian participants did, while Indian participants made more and longer fixations on the nose than Malay participants did. In addition, we detected statistically significant, though subtle, differences in fixation patterns between the faces of the three races. These findings suggest that the racial differences in face-scanning patterns may be attributed both to culture and to variations in facial physiognomy between races.
Collapse
Affiliation(s)
- Hoo Keat Wong
- School of Psychology, University of Nottingham Malaysia, Semenyih, Malaysia
| | - David R T Keeble
- School of Psychology, University of Nottingham Malaysia, Semenyih, Malaysia
| | - Ian D Stephen
- School of Social Sciences, Nottingham Trent University, Nottingham, UK
| |
Collapse
|
10
|
Stacchi L, Caldara R. Stimulus size modulates idiosyncratic neural face identity discrimination. J Vis 2022; 22:9. [PMID: 36580295 PMCID: PMC9804033 DOI: 10.1167/jov.22.13.9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/30/2022] Open
Abstract
Humans show individual differences in neural facial identity discrimination (FID) responses across viewing positions. Critically, these variations have been shown to be reliable over time and to directly relate to observers' idiosyncratic preferences in facial information sampling. This functional signature in facial identity processing might relate to observer-specific diagnostic information processing. Although these individual differences are a valuable source of information for interpreting data, they can also be difficult to isolate when it is not possible to test many conditions. To address this potential issue, we explored whether reducing stimulus size would help decrease these interindividual variations in neural FID. We manipulated the size of face stimuli (covering 3°, 5°, 6.7°, 8.5°, and 12° of visual angle), as well as the fixation location (left eye, right eye, below the nasion, nose, and mouth) while recording electrophysiological responses. Same identity faces were presented with a base frequency of 6 Hz. Different identity faces were periodically inserted within this sequence to trigger an objective index of neural FID. Our data show robust and consistent individual differences in neural face identity discrimination across viewing positions for all face sizes. Nevertheless, FID was optimal for a larger number of observers when faces subtended 6.7° of visual angle and fixation was below the nasion. This condition is the most suited to reduce natural interindividual variations in neural FID patterns, defining an important benchmark to measure neural FID when it is not possible to assess and control for observers' idiosyncrasies.
Collapse
Affiliation(s)
- Lisa Stacchi
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland,
| | - Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland,
| |
Collapse
|
11
|
Hsiao JH, An J, Hui VKS, Zheng Y, Chan AB. Understanding the role of eye movement consistency in face recognition and autism through integrating deep neural networks and hidden Markov models. NPJ SCIENCE OF LEARNING 2022; 7:28. [PMID: 36284113 PMCID: PMC9596700 DOI: 10.1038/s41539-022-00139-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Accepted: 08/31/2022] [Indexed: 06/16/2023]
Abstract
Greater eyes-focused eye movement pattern during face recognition is associated with better performance in adults but not in children. We test the hypothesis that higher eye movement consistency across trials, instead of a greater eyes-focused pattern, predicts better performance in children since it reflects capacity in developing visual routines. We first simulated visual routine development through combining deep neural network and hidden Markov model that jointly learn perceptual representations and eye movement strategies for face recognition. The model accounted for the advantage of eyes-focused pattern in adults, and predicted that in children (partially trained models) consistency but not pattern of eye movements predicted recognition performance. This result was then verified with data from typically developing children. In addition, lower eye movement consistency in children was associated with autism diagnosis, particularly autistic traits in social skills. Thus, children's face recognition involves visual routine development through social exposure, indexed by eye movement consistency.
Collapse
Affiliation(s)
- Janet H Hsiao
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China.
- The State Key Laboratory of Brain and Cognitive Sciences, University of Hong Kong, Hong Kong SAR, China.
- The Institute of Data Science, University of Hong Kong, Hong Kong SAR, China.
| | - Jeehye An
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China
| | | | - Yueyuan Zheng
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China
| | - Antoni B Chan
- Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China
| |
Collapse
|
12
|
Linka M, Broda MD, Alsheimer T, de Haas B, Ramon M. Characteristic fixation biases in Super-Recognizers. J Vis 2022; 22:17. [PMID: 35900724 PMCID: PMC9344214 DOI: 10.1167/jov.22.8.17] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022] Open
Abstract
Neurotypical observers show large and reliable individual differences in gaze behavior along several semantic object dimensions. Individual gaze behavior toward faces has been linked to face identity processing, including that of neurotypical observers. Here, we investigated potential gaze biases in Super-Recognizers (SRs), individuals with exceptional face identity processing skills. Ten SRs, identified with a novel conservative diagnostic framework, and 43 controls freely viewed 700 complex scenes depicting more than 5000 objects. First, we tested whether SRs and controls differ in fixation biases along four semantic dimensions: faces, text, objects being touched, and bodies. Second, we tested potential group differences in fixation biases toward eyes and mouths. Finally, we tested whether SRs fixate closer to the theoretical optimal fixation point for face identification. SRs showed a stronger gaze bias toward faces and away from text and touched objects, starting from the first fixation onward. Further, SRs spent a significantly smaller proportion of first fixations and dwell time toward faces on mouths but did not differ in dwell time or first fixations devoted to eyes. Face fixation of SRs also fell significantly closer to the theoretical optimal fixation point for identification, just below the eyes. Our findings suggest that reliable superiority for face identity processing is accompanied by early fixation biases toward faces and preferred saccadic landing positions close to the theoretical optimum for face identification. We discuss future directions to investigate the functional basis of individual fixation behavior and face identity processing ability.
Collapse
Affiliation(s)
- Marcel Linka
- Experimental Psychology, Justus Liebig University Giessen, Giessen, Germany.,
| | | | - Tamara Alsheimer
- Experimental Psychology, Justus Liebig University Giessen, Giessen, Germany.,Applied Face Cognition Lab, University of Lausanne, Institute of Psychology, Lausanne, Switzerland.,
| | - Benjamin de Haas
- Experimental Psychology, Justus Liebig University Giessen, Giessen, Germany.,
| | - Meike Ramon
- Applied Face Cognition Lab, University of Lausanne, Institute of Psychology, Lausanne, Switzerland.,
| |
Collapse
|
13
|
Gupta N, White H, Trott S, Spiegel JH. Observer Gaze Patterns of Patient Photographs Before and After Facial Feminization. Aesthet Surg J 2022; 42:725-732. [PMID: 34994379 DOI: 10.1093/asj/sjab434] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Human interaction begins with visual evaluation of others, and this often centers on the face. Objective measurement of this evaluation gives clues to social perception. OBJECTIVES The objective of this study was to use eye-tracking technology to evaluate if there are scanpath differences when observers view faces of cisgender men, cisgender women, and transgender women before and after facial feminization surgery (FFS) including when assigning tasks assessing femininity, attractiveness, and likability. METHODS Undergraduate psychology students were prospectively recruited as observers at a single institution. Their eye movements were recorded by eye-tracking technology when they were presented with frontal photographs of prototypical male, prototypical female, and pre- and post-FFS faces in a random order and then with prompting to assess femininity, attractiveness, and likability. RESULTS Twenty-seven observers performed the tasks. Participants focused their attention more on the central triangle of post-FFS and prototypical female images and on the forehead of pre-FFS and prototypical male images. Higher femininity ratings were associated with longer proportional fixations to the central triangle and lower proportional fixations to the forehead. CONCLUSIONS This preliminary study implies the scanpath for viewing a post-FFS face is closer to that for viewing a prototypical female than a prototypical male based on differences viewing the forehead and brow vs the central triangle.
Collapse
Affiliation(s)
- Nikita Gupta
- Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, University of Kentucky , Lexington, KY , USA
| | - Hannah White
- Department of Psychological Sciences, University of Missouri-St. Louis , St. Louis, MO , USA
| | - Skylar Trott
- Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, University of Kentucky , Lexington, KY , USA
| | - Jeffrey H Spiegel
- Division of Facial Plastic and Reconstructive Surgery, Department of Otolaryngology-Head and Neck Surgery, Boston University School of Medicine , Boston, MA , USA
| |
Collapse
|
14
|
Liao W, Li STK, Hsiao JHW. Music reading experience modulates eye movement pattern in English reading but not in Chinese reading. Sci Rep 2022; 12:9144. [PMID: 35650229 PMCID: PMC9397380 DOI: 10.1038/s41598-022-12978-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2022] [Accepted: 05/19/2022] [Indexed: 11/30/2022] Open
Abstract
Here we tested the hypothesis that in Chinese-English bilinguals,
music reading experience may modulate eye movement planning in reading English but
not Chinese sentences due to the similarity in perceptual demands on processing
sequential symbol strings separated by spaces between music notation and English
sentence reading. Chinese–English bilingual musicians and non-musicians read legal,
semantically incorrect, and syntactically (and semantically) incorrect sentences in
both English and Chinese. In English reading, musicians showed more dispersed eye
movement patterns in reading syntactically incorrect sentences than legal sentences,
whereas non-musicians did not. This effect was not observed in Chinese reading.
Musicians also had shorter saccade lengths when viewing syntactically incorrect than
correct musical notations and sentences in an unfamiliar alphabetic language
(Tibetan), whereas non-musicians did not. Thus, musicians’ eye movement planning was
disturbed by syntactic violations in both music and English reading but not in
Chinese reading, and this effect was generalized to an unfamiliar alphabetic
language. These results suggested that music reading experience may modulate
perceptual processes in reading differentially in bilinguals’ two languages,
depending on their processing similarities.
Collapse
Affiliation(s)
- Weiyan Liao
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China
| | - Sara Tze Kwan Li
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China.,Department of Social Sciences, School of Arts and Social Sciences, Hong Kong Metropolitan University, Hong Kong SAR, China
| | - Janet Hui-Wen Hsiao
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China. .,The State Key Laboratory of Brain and Cognitive Sciences, University of Hong Kong, Hong Kong SAR, China. .,The Institute of Data Science, University of Hong Kong, Hong Kong SAR, China.
| |
Collapse
|
15
|
Duran N, Atkinson AP. Foveal processing of emotion-informative facial features. PLoS One 2021; 16:e0260814. [PMID: 34855898 PMCID: PMC8638924 DOI: 10.1371/journal.pone.0260814] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2021] [Accepted: 11/17/2021] [Indexed: 11/18/2022] Open
Abstract
Certain facial features provide useful information for recognition of facial expressions. In two experiments, we investigated whether foveating informative features of briefly presented expressions improves recognition accuracy and whether these features are targeted reflexively when not foveated. Angry, fearful, surprised, and sad or disgusted expressions were presented briefly at locations which would ensure foveation of specific features. Foveating the mouth of fearful, surprised and disgusted expressions improved emotion recognition compared to foveating an eye or cheek or the central brow. Foveating the brow led to equivocal results in anger recognition across the two experiments, which might be due to the different combination of emotions used. There was no consistent evidence suggesting that reflexive first saccades targeted emotion-relevant features; instead, they targeted the closest feature to initial fixation. In a third experiment, angry, fearful, surprised and disgusted expressions were presented for 5 seconds. Duration of task-related fixations in the eyes, brow, nose and mouth regions was modulated by the presented expression. Moreover, longer fixation at the mouth positively correlated with anger and disgust accuracy both when these expressions were freely viewed (Experiment 2b) and when briefly presented at the mouth (Experiment 2a). Finally, an overall preference to fixate the mouth across all expressions correlated positively with anger and disgust accuracy. These findings suggest that foveal processing of informative features is functional/contributory to emotion recognition, but they are not automatically sought out when not foveated, and that facial emotion recognition performance is related to idiosyncratic gaze behaviour.
Collapse
Affiliation(s)
- Nazire Duran
- Department of Psychology, Durham University, Durham, United Kingdom
| | - Anthony P. Atkinson
- Department of Psychology, Durham University, Durham, United Kingdom
- * E-mail:
| |
Collapse
|
16
|
Holleman GA, Hooge ITC, Huijding J, Deković M, Kemner C, Hessels RS. Gaze and speech behavior in parent–child interactions: The role of conflict and cooperation. CURRENT PSYCHOLOGY 2021. [DOI: 10.1007/s12144-021-02532-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
AbstractA primary mode of human social behavior is face-to-face interaction. In this study, we investigated the characteristics of gaze and its relation to speech behavior during video-mediated face-to-face interactions between parents and their preadolescent children. 81 parent–child dyads engaged in conversations about cooperative and conflictive family topics. We used a dual-eye tracking setup that is capable of concurrently recording eye movements, frontal video, and audio from two conversational partners. Our results show that children spoke more in the cooperation-scenario whereas parents spoke more in the conflict-scenario. Parents gazed slightly more at the eyes of their children in the conflict-scenario compared to the cooperation-scenario. Both parents and children looked more at the other's mouth region while listening compared to while speaking. Results are discussed in terms of the role that parents and children take during cooperative and conflictive interactions and how gaze behavior may support and coordinate such interactions.
Collapse
|
17
|
Han NX, Chakravarthula PN, Eckstein MP. Peripheral facial features guiding eye movements and reducing fixational variability. J Vis 2021; 21:7. [PMID: 34347018 PMCID: PMC8340657 DOI: 10.1167/jov.21.8.7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Face processing is a fast and efficient process due to its evolutionary and social importance. A majority of people direct their first eye movement to a featureless point just below the eyes that maximizes accuracy in recognizing a person's identity and gender. Yet, the exact properties or features of the face that guide the first eye movements and reduce fixational variability are unknown. Here, we manipulated the presence of the facial features and the spatial configuration of features to investigate their effect on the location and variability of first and second fixations to peripherally presented faces. Our results showed that observers can utilize the face outline, individual facial features, and feature spatial configuration to guide the first eye movements to their preferred point of fixation. The eyes have a preferential role in guiding the first eye movements and reducing fixation variability. Eliminating the eyes or altering their position had the greatest influence on the location and variability of fixations and resulted in the largest detriment to face identification performance. The other internal features (nose and mouth) also contribute to reducing fixation variability. A subsequent experiment measuring detection of single features showed that the eyes have the highest detectability (relative to other features) in the visual periphery providing a strong sensory signal to guide the oculomotor system. Together, the results suggest a flexible multiple-cue approach that might be a robust solution to cope with how the varying eccentricities in the real world influence the ability to resolve individual feature properties and the preferential role of the eyes.
Collapse
Affiliation(s)
- Nicole X Han
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.,
| | - Puneeth N Chakravarthula
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.,
| | - Miguel P Eckstein
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.,
| |
Collapse
|
18
|
Prunty JE, Keemink JR, Kelly DJ. Infants scan static and dynamic facial expressions differently. INFANCY 2021; 26:831-856. [PMID: 34288344 DOI: 10.1111/infa.12426] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2020] [Revised: 07/02/2021] [Accepted: 07/08/2021] [Indexed: 11/30/2022]
Abstract
Despite being inherently dynamic phenomena, much of our understanding of how infants attend and scan facial expressions is based on static face stimuli. Here we investigate how six-, nine-, and twelve-month infants allocate their visual attention toward dynamic-interactive videos of the six basic emotional expressions, and compare their responses with static images of the same stimuli. We find infants show clear differences in how they attend and scan dynamic and static expressions, looking longer toward the dynamic-face and lower-face regions. Infants across all age groups show differential interest in expressions, and show precise scanning of regions "diagnostic" for emotion recognition. These data also indicate that infants' attention toward dynamic expressions develops over the first year of life, including relative increases in interest and scanning precision toward some negative facial expressions (e.g., anger, fear, and disgust).
Collapse
Affiliation(s)
| | | | - David J Kelly
- School of Psychology, University of Kent, Canterbury, UK
| |
Collapse
|
19
|
An algorithmic approach to determine expertise development using object-related gaze pattern sequences. Behav Res Methods 2021; 54:493-507. [PMID: 34258709 PMCID: PMC8863757 DOI: 10.3758/s13428-021-01652-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/14/2021] [Indexed: 12/20/2022]
Abstract
Eye tracking (ET) technology is increasingly utilized to quantify visual behavior in the study of the development of domain-specific expertise. However, the identification and measurement of distinct gaze patterns using traditional ET metrics has been challenging, and the insights gained shown to be inconclusive about the nature of expert gaze behavior. In this article, we introduce an algorithmic approach for the extraction of object-related gaze sequences and determine task-related expertise by investigating the development of gaze sequence patterns during a multi-trial study of a simplified airplane assembly task. We demonstrate the algorithm in a study where novice (n = 28) and expert (n = 2) eye movements were recorded in successive trials (n = 8), allowing us to verify whether similar patterns develop with increasing expertise. In the proposed approach, AOI sequences were transformed to string representation and processed using the k-mer method, a well-known method from the field of computational biology. Our results for expertise development suggest that basic tendencies are visible in traditional ET metrics, such as the fixation duration, but are much more evident for k-mers of k > 2. With increased on-task experience, the appearance of expert k-mer patterns in novice gaze sequences was shown to increase significantly (p < 0.001). The results illustrate that the multi-trial k-mer approach is suitable for revealing specific cognitive processes and can quantify learning progress using gaze patterns that include both spatial and temporal information, which could provide a valuable tool for novice training and expert assessment.
Collapse
|
20
|
Abstract
The eye movement analysis with hidden Markov models (EMHMM) method provides quantitative measures of individual differences in eye-movement pattern. However, it is limited to tasks where stimuli have the same feature layout (e.g., faces). Here we proposed to combine EMHMM with the data mining technique co-clustering to discover participant groups with consistent eye-movement patterns across stimuli for tasks involving stimuli with different feature layouts. Through applying this method to eye movements in scene perception, we discovered explorative (switching between the foreground and background information or different regions of interest) and focused (mainly looking at the foreground with less switching) eye-movement patterns among Asian participants. Higher similarity to the explorative pattern predicted better foreground object recognition performance, whereas higher similarity to the focused pattern was associated with better feature integration in the flanker task. These results have important implications for using eye tracking as a window into individual differences in cognitive abilities and styles. Thus, EMHMM with co-clustering provides quantitative assessments on eye-movement patterns across stimuli and tasks. It can be applied to many other real-life visual tasks, making a significant impact on the use of eye tracking to study cognitive behavior across disciplines.
Collapse
|
21
|
Alexithymia explains atypical spatiotemporal dynamics of eye gaze in autism. Cognition 2021; 212:104710. [PMID: 33862441 DOI: 10.1016/j.cognition.2021.104710] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2020] [Revised: 03/24/2021] [Accepted: 03/26/2021] [Indexed: 11/23/2022]
Abstract
Recognition of emotional facial expressions is considered to be atypical in autism. This difficulty is thought to be due to the way that facial expressions are visually explored. Evidence for atypical visual exploration of emotional faces in autism is, however, equivocal. We propose that, where observed, atypical visual exploration of emotional facial expressions is due to alexithymia, a distinct but frequently co-occurring condition. In this eye-tracking study we tested the alexithymia hypothesis using a number of recent methodological advances to study eye gaze during several emotion processing tasks (emotion recognition, intensity judgements, free gaze), in 25 adults with, and 45 without, autism. A multilevel polynomial modelling strategy was used to describe the spatiotemporal dynamics of eye gaze to emotional facial expressions. Converging evidence from traditional and novel analysis methods revealed that atypical gaze to the eyes is best predicted by alexithymia in both autistic and non-autistic individuals. Information theoretic analyses also revealed differential effects of task on gaze patterns as a function of alexithymia, but not autism. These findings highlight factors underlying atypical emotion processing in autistic individuals, with wide-ranging implications for emotion research.
Collapse
|
22
|
Hsiao JH, An J, Zheng Y, Chan AB. Do portrait artists have enhanced face processing abilities? Evidence from hidden Markov modeling of eye movements. Cognition 2021; 211:104616. [PMID: 33592393 DOI: 10.1016/j.cognition.2021.104616] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Revised: 01/24/2021] [Accepted: 01/27/2021] [Indexed: 11/16/2022]
Abstract
Recent research has suggested the importance of part-based information in face recognition in addition to global, whole-face information. Nevertheless, face drawing experience was reported to enhance selective attention to the eyes but did not improve face recognition performance, leading to speculations about limited plasticity in adult face recognition. Here we examined the mechanism underlying the limited advantage of face drawing experience in face recognition through the Eye Movement analysis with Hidden Markov Models (EMHMM) approach. We found that portrait artists showed more eyes-focused eye movement patterns and outperformed novices in face matching, and participants' drawing rating was correlated with both eye movement pattern and performance. In contrast, portrait artists did not outperform novices and did not differ from novices in eye movement pattern in either the face recognition or part-whole tasks, although the eyes-focused pattern was associated with better recognition performance and longer response times in the whole condition relative to the part condition. Interestingly, in contrast to the face recognition and part-whole tasks, participants' performance in face matching was predicted by their drawing rating but not eye movement pattern. These results suggested that artists' advantage in face processing is specific to tasks similar to their drawing experience such as face matching, and may be related to their better ability in extracting identity-invariant information between two faces rather than more eyes-focused eye movement patterns.
Collapse
Affiliation(s)
- Janet H Hsiao
- Department of Psychology, University of Hong Kong, Hong Kong Special Administrative Region; The State Key Laboratory of Brain and Cognitive Sciences, University of Hong Kong, Hong Kong Special Administrative Region.
| | - Jeehye An
- Department of Psychology, University of Hong Kong, Hong Kong Special Administrative Region
| | - Yueyuan Zheng
- Department of Psychology, University of Hong Kong, Hong Kong Special Administrative Region
| | - Antoni B Chan
- Department of Computer Science, City University of Hong Kong, Hong Kong Special Administrative Region
| |
Collapse
|
23
|
Abstract
Gaze-where one looks, how long, and when-plays an essential part in human social behavior. While many aspects of social gaze have been reviewed, there is no comprehensive review or theoretical framework that describes how gaze to faces supports face-to-face interaction. In this review, I address the following questions: (1) When does gaze need to be allocated to a particular region of a face in order to provide the relevant information for successful interaction; (2) How do humans look at other people, and faces in particular, regardless of whether gaze needs to be directed at a particular region to acquire the relevant visual information; (3) How does gaze support the regulation of interaction? The work reviewed spans psychophysical research, observational research, and eye-tracking research in both lab-based and interactive contexts. Based on the literature overview, I sketch a framework for future research based on dynamic systems theory. The framework holds that gaze should be investigated in relation to sub-states of the interaction, encompassing sub-states of the interactors, the content of the interaction as well as the interactive context. The relevant sub-states for understanding gaze in interaction vary over different timescales from microgenesis to ontogenesis and phylogenesis. The framework has important implications for vision science, psychopathology, developmental science, and social robotics.
Collapse
Affiliation(s)
- Roy S Hessels
- Experimental Psychology, Helmholtz Institute, Utrecht University, Heidelberglaan 1, 3584CS, Utrecht, The Netherlands.
- Developmental Psychology, Heidelberglaan 1, 3584CS, Utrecht, The Netherlands.
| |
Collapse
|
24
|
Hessels RS, Benjamins JS, van Doorn AJ, Koenderink JJ, Holleman GA, Hooge ITC. Looking behavior and potential human interactions during locomotion. J Vis 2020; 20:5. [PMID: 33007079 PMCID: PMC7545070 DOI: 10.1167/jov.20.10.5] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
As humans move through parts of their environment, they meet others that may or may not try to interact with them. Where do people look when they meet others? We had participants wearing an eye tracker walk through a university building. On the way, they encountered nine “walkers.” Walkers were instructed to e.g. ignore the participant, greet him or her, or attempt to hand out a flyer. The participant's gaze was mostly directed to the currently relevant body parts of the walker. Thus, the participants gaze depended on the walker's action. Individual differences in participant's looking behavior were consistent across walkers. Participants who did not respond to the walker seemed to look less at that walker, although this difference was not statistically significant. We suggest that models of gaze allocation should take social motivation into account.
Collapse
Affiliation(s)
- Roy S Hessels
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands.,
| | - Jeroen S Benjamins
- Experimental Psychology, Helmholtz Institute, and Social, Health and Organizational Psychology, Utrecht University, Utrecht, the Netherlands.,
| | - Andrea J van Doorn
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands.,
| | - Jan J Koenderink
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands.,
| | - Gijs A Holleman
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands.,
| | - Ignace T C Hooge
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands.,
| |
Collapse
|
25
|
Papinutto M, Lao J, Lalanne D, Caldara R. Watchers do not follow the eye movements of Walkers. Vision Res 2020; 176:130-140. [PMID: 32882595 DOI: 10.1016/j.visres.2020.08.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2020] [Revised: 08/03/2020] [Accepted: 08/05/2020] [Indexed: 11/27/2022]
Abstract
Eye movements are a functional signature of how the visual system effectively decodes and adapts to the environment. However, scientific knowledge in eye movements mostly arises from studies conducted in laboratories, with well-controlled stimuli presented in constrained unnatural settings. Only a few studies have attempted to directly compare and assess whether eye movement data acquired in the real world generalize with those in laboratory settings, with same visual inputs. However, none of these studies controlled for both the auditory signals typical of real-world settings and the top-down task effects across conditions, leaving this question unresolved. To minimize this inherent gap across conditions, we compared the eye movements recorded from observers during ecological spatial navigation in the wild (the Walkers) with those recorded in laboratory (the Watchers) on the same visual and auditory inputs, with both groups performing the very same active cognitive task. We derived robust data-driven statistical saliency and motion maps. The Walkers and Watchers differed in terms of eye movement characteristics: fixation number and duration, saccade amplitude. The Watchers relied significantly more on saliency and motion than the Walkers. Interestingly, both groups exhibited similar fixation patterns towards social agents and objects. Altogether, our data show that eye movements patterns obtained in laboratory do not fully generalize to real world, even when task and auditory information is controlled. These observations invite to caution when generalizing the eye movements obtained in laboratory with those of ecological spatial navigation.
Collapse
Affiliation(s)
- M Papinutto
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Switzerland; Human-IST Institute, Department of Informatics, University of Fribourg, Switzerland.
| | - J Lao
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Switzerland
| | - D Lalanne
- Human-IST Institute, Department of Informatics, University of Fribourg, Switzerland
| | - R Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Switzerland
| |
Collapse
|
26
|
David EJ, Lebranchu P, Perreira Da Silva M, Le Callet P. Predicting artificial visual field losses: A gaze-based inference study. J Vis 2020; 19:22. [PMID: 31868896 DOI: 10.1167/19.14.22] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Visual field defects are a world-wide concern, and the proportion of the population experiencing vision loss is ever increasing. Macular degeneration and glaucoma are among the four leading causes of permanent vision loss. Identifying and characterizing visual field losses from gaze alone could prove crucial in the future for screening tests, rehabilitation therapies, and monitoring. In this experiment, 54 participants took part in a free-viewing task of visual scenes while experiencing artificial scotomas (central and peripheral) of varying radii in a gaze-contingent paradigm. We studied the importance of a set of gaze features as predictors to best differentiate between artificial scotoma conditions. Linear mixed models were utilized to measure differences between scotoma conditions. Correlation and factorial analyses revealed redundancies in our data. Finally, hidden Markov models and recurrent neural networks were implemented as classifiers in order to measure the predictive usefulness of gaze features. The results show separate saccade direction biases depending on scotoma type. We demonstrate that the saccade relative angle, amplitude, and peak velocity of saccades are the best features on the basis of which to distinguish between artificial scotomas in a free-viewing task. Finally, we discuss the usefulness of our protocol and analyses as a gaze-feature identifier tool that discriminates between artificial scotomas of different types and sizes.
Collapse
Affiliation(s)
| | - Pierre Lebranchu
- University of Nantes and Nantes University Hospital, Nantes, France
| | | | | |
Collapse
|
27
|
Wegner-Clemens K, Rennig J, Beauchamp MS. A relationship between Autism-Spectrum Quotient and face viewing behavior in 98 participants. PLoS One 2020; 15:e0230866. [PMID: 32352984 PMCID: PMC7192493 DOI: 10.1371/journal.pone.0230866] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2019] [Accepted: 03/10/2020] [Indexed: 01/18/2023] Open
Abstract
Faces are one of the most important stimuli that we encounter, but humans vary dramatically in their behavior when viewing a face: some individuals preferentially fixate the eyes, others fixate the mouth, and still others show an intermediate pattern. The determinants of these large individual differences are unknown. However, individuals with Autism Spectrum Disorder (ASD) spend less time fixating the eyes of a viewed face than controls, suggesting the hypothesis that autistic traits in healthy adults might explain individual differences in face viewing behavior. Autistic traits were measured in 98 healthy adults recruited from an academic setting using the Autism-Spectrum Quotient, a validated 50-statement questionnaire. Fixations were measured using a video-based eye tracker while participants viewed two different types of audiovisual movies: short videos of talker speaking single syllables and longer videos of talkers speaking sentences in a social context. For both types of movies, there was a positive correlation between Autism-Spectrum Quotient score and percent of time fixating the lower half of the face that explained from 4% to 10% of the variance in individual face viewing behavior. This effect suggests that in healthy adults, autistic traits are one of many factors that contribute to individual differences in face viewing behavior.
Collapse
Affiliation(s)
- Kira Wegner-Clemens
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, Texas, United States of America
| | - Johannes Rennig
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, Texas, United States of America
| | - Michael S. Beauchamp
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, Texas, United States of America
| |
Collapse
|
28
|
Rennig J, Wegner-Clemens K, Beauchamp MS. Face viewing behavior predicts multisensory gain during speech perception. Psychon Bull Rev 2020; 27:70-77. [PMID: 31845209 PMCID: PMC7004844 DOI: 10.3758/s13423-019-01665-y] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Visual information from the face of an interlocutor complements auditory information from their voice, enhancing intelligibility. However, there are large individual differences in the ability to comprehend noisy audiovisual speech. Another axis of individual variability is the extent to which humans fixate the mouth or the eyes of a viewed face. We speculated that across a lifetime of face viewing, individuals who prefer to fixate the mouth of a viewed face might accumulate stronger associations between visual and auditory speech, resulting in improved comprehension of noisy audiovisual speech. To test this idea, we assessed interindividual variability in two tasks. Participants (n = 102) varied greatly in their ability to understand noisy audiovisual sentences (accuracy from 2-58%) and in the time they spent fixating the mouth of a talker enunciating clear audiovisual syllables (3-98% of total time). These two variables were positively correlated: a 10% increase in time spent fixating the mouth equated to a 5.6% increase in multisensory gain. This finding demonstrates an unexpected link, mediated by histories of visual exposure, between two fundamental human abilities: processing faces and understanding speech.
Collapse
Affiliation(s)
- Johannes Rennig
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, 1 Baylor Plaza Suite S104, Houston, TX, 77030, USA
| | - Kira Wegner-Clemens
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, 1 Baylor Plaza Suite S104, Houston, TX, 77030, USA
| | - Michael S Beauchamp
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, 1 Baylor Plaza Suite S104, Houston, TX, 77030, USA.
| |
Collapse
|
29
|
Zhang J, Chan AB, Lau EYY, Hsiao JH. Individuals with insomnia misrecognize angry faces as fearful faces while missing the eyes: an eye-tracking study. Sleep 2019; 42:5185639. [PMID: 30452735 DOI: 10.1093/sleep/zsy220] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2018] [Indexed: 11/12/2022] Open
Abstract
Individuals with insomnia have been found to have disturbed perception of facial expressions. Through eye movement examinations, here we test the hypothesis that this effect is due to impaired visual attention functions for retrieving diagnostic features in facial expression judgments. Twenty-three individuals with insomnia symptoms and 23 controls without insomnia completed a task to categorize happy, sad, fearful, and angry facial expressions. The participants with insomnia were less accurate in recognizing angry faces and misidentified them as fearful faces more often than the controls. A hidden Markov modeling approach for eye movement data analysis revealed that when viewing facial expressions, more individuals with insomnia adopted a nose-mouth eye movement pattern focusing on the vertical face midline while more controls adopted an eyes-mouth pattern preferentially attending to lateral features, particularly the two eyes. As previous studies found that the primary diagnostic feature for recognizing angry faces is the eyes while the diagnostic features for other facial expressions involve the mouth region, missing the eye region may contribute to specific difficulties in recognizing angry facial expressions, consistent with our behavioral finding in participants with insomnia symptoms. Taken together, the findings suggest that impaired information selection through visual attention control may be related to the compromised emotion perception in individuals with insomnia.
Collapse
Affiliation(s)
- Jinxiao Zhang
- Department of Psychology, The University of Hong Kong, Pok Fu Lam, Hong Kong
| | - Antoni B Chan
- Department of Computer Science, City University of Hong Kong, Kowloon Tong, Hong Kong
| | - Esther Yuet Ying Lau
- Department of Psychology, Education University of Hong Kong, Tai Po, Hong Kong.,Centre for Psychosocial Health, Education University of Hong Kong, Tai Po, Hong Kong
| | - Janet H Hsiao
- Department of Psychology, The University of Hong Kong, Pok Fu Lam, Hong Kong
| |
Collapse
|
30
|
Abstract
Here we propose the eye movement analysis with switching hidden Markov model (EMSHMM) approach to analyzing eye movement data in cognitive tasks involving cognitive state changes. We used a switching hidden Markov model (SHMM) to capture a participant's cognitive state transitions during the task, with eye movement patterns during each cognitive state being summarized using a regular HMM. We applied EMSHMM to a face preference decision-making task with two pre-assumed cognitive states-exploration and preference-biased periods-and we discovered two common eye movement patterns through clustering the cognitive state transitions. One pattern showed both a later transition from the exploration to the preference-biased cognitive state and a stronger tendency to look at the preferred stimulus at the end, and was associated with higher decision inference accuracy at the end; the other pattern entered the preference-biased cognitive state earlier, leading to earlier above-chance inference accuracy in a trial but lower inference accuracy at the end. This finding was not revealed by any other method. As compared with our previous HMM method, which assumes no cognitive state change (i.e., EMHMM), EMSHMM captured eye movement behavior in the task better, resulting in higher decision inference accuracy. Thus, EMSHMM reveals and provides quantitative measures of individual differences in cognitive behavior/style, making a significant impact on the use of eyetracking to study cognitive behavior across disciplines.
Collapse
|
31
|
Wegner-Clemens K, Rennig J, Magnotti JF, Beauchamp MS. Using principal component analysis to characterize eye movement fixation patterns during face viewing. J Vis 2019; 19:2. [PMID: 31689715 PMCID: PMC6833982 DOI: 10.1167/19.13.2] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2019] [Accepted: 08/23/2019] [Indexed: 01/22/2023] Open
Abstract
Human faces contain dozens of visual features, but viewers preferentially fixate just two of them: the eyes and the mouth. Face-viewing behavior is usually studied by manually drawing regions of interest (ROIs) on the eyes, mouth, and other facial features. ROI analyses are problematic as they require arbitrary experimenter decisions about the location and number of ROIs, and they discard data because all fixations within each ROI are treated identically and fixations outside of any ROI are ignored. We introduce a data-driven method that uses principal component analysis (PCA) to characterize human face-viewing behavior. All fixations are entered into a PCA, and the resulting eigenimages provide a quantitative measure of variability in face-viewing behavior. In fixation data from 41 participants viewing four face exemplars under three stimulus and task conditions, the first principal component (PC1) separated the eye and mouth regions of the face. PC1 scores varied widely across participants, revealing large individual differences in preference for eye or mouth fixation, and PC1 scores varied by condition, revealing the importance of behavioral task in determining fixation location. Linear mixed effects modeling of the PC1 scores demonstrated that task condition accounted for 41% of the variance, individual differences accounted for 28% of the variance, and stimulus exemplar for less than 1% of the variance. Fixation eigenimages provide a useful tool for investigating the relative importance of the different factors that drive human face-viewing behavior.
Collapse
Affiliation(s)
- Kira Wegner-Clemens
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, TX
| | - Johannes Rennig
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, TX
| | - John F Magnotti
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, TX
| | - Michael S Beauchamp
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, TX
| |
Collapse
|
32
|
Kampermann L, Wilming N, Alink A, Büchel C, Onat S. Fixation-pattern similarity analysis reveals adaptive changes in face-viewing strategies following aversive learning. eLife 2019; 8:e44111. [PMID: 31635690 PMCID: PMC6805121 DOI: 10.7554/elife.44111] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2018] [Accepted: 09/17/2019] [Indexed: 11/28/2022] Open
Abstract
Animals can effortlessly adapt their behavior by generalizing from past aversive experiences, allowing to avoid harm in novel situations. We studied how visual information was sampled by eye-movements during this process called fear generalization, using faces organized along a circular two-dimensional perceptual continuum. During learning, one face was conditioned to predict a harmful event, whereas the most dissimilar face stayed neutral. This introduced an adversity gradient along one specific dimension, while the other, unspecific dimension was defined solely by perceptual similarity. Aversive learning changed scanning patterns selectively along the adversity-related dimension, but not the orthogonal dimension. This effect was mainly located within the eye region of faces. Our results provide evidence for adaptive changes in viewing strategies of faces following aversive learning. This is compatible with the view that these changes serve to sample information in a way that allows discriminating between safe and adverse for a better threat prediction.
Collapse
Affiliation(s)
- Lea Kampermann
- Department of Systems NeuroscienceUniversity Medical Center Hamburg-EppendorfHamburgGermany
| | - Niklas Wilming
- Department of Neurophysiology and PathophysiologyUniversity Medical Center Hamburg-EppendorfHamburgGermany
| | - Arjen Alink
- Department of Systems NeuroscienceUniversity Medical Center Hamburg-EppendorfHamburgGermany
| | - Christian Büchel
- Department of Systems NeuroscienceUniversity Medical Center Hamburg-EppendorfHamburgGermany
| | - Selim Onat
- Department of Systems NeuroscienceUniversity Medical Center Hamburg-EppendorfHamburgGermany
| |
Collapse
|
33
|
Stacchi L, Liu-Shuang J, Ramon M, Caldara R. Reliability of individual differences in neural face identity discrimination. Neuroimage 2019; 189:468-475. [DOI: 10.1016/j.neuroimage.2019.01.023] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2018] [Revised: 12/19/2018] [Accepted: 01/09/2019] [Indexed: 11/27/2022] Open
|
34
|
Perception and Deception: Human Beauty and the Brain. Behav Sci (Basel) 2019; 9:bs9040034. [PMID: 30934856 PMCID: PMC6523404 DOI: 10.3390/bs9040034] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2019] [Revised: 03/19/2019] [Accepted: 03/25/2019] [Indexed: 01/31/2023] Open
Abstract
Human physical characteristics and their perception by the brain are under pressure by natural selection to optimize reproductive success. Men and women have different strategies to appear attractive and have different interests in identifying beauty in people. Nevertheless, men and women from all cultures agree on who is and who is not attractive, and throughout the world attractive people show greater acquisition of resources and greater reproductive success than others. The brain employs at least three modules, composed of interconnected brain regions, to judge facial attractiveness: one for identification, one for interpretation and one for valuing. Key elements that go into the judgment are age and health, as well as symmetry, averageness, face and body proportions, facial color and texture. These elements are all Costly Signals of reproductive fitness because they are difficult to fake. However, people deceive others using tricks such as coloring hair, cosmetics and clothing styles, while at the same time they also focus on detecting fakes. People may also deceive themselves, especially about their own attractiveness, and use self-signally actions to demonstrate to themselves their own true value. The neuroscience of beauty is best understood by considering the evolutionary pressures to maximize reproductive fitness.
Collapse
|
35
|
Neural Representations of Faces Are Tuned to Eye Movements. J Neurosci 2019; 39:4113-4123. [PMID: 30867260 DOI: 10.1523/jneurosci.2968-18.2019] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2018] [Revised: 02/07/2019] [Accepted: 03/05/2019] [Indexed: 01/23/2023] Open
Abstract
Eye movements provide a functional signature of how human vision is achieved. Many recent studies have consistently reported robust idiosyncratic visual sampling strategies during face recognition. Whether these interindividual differences are mirrored by idiosyncratic neural responses remains unknown. To this aim, we first tracked eye movements of male and female observers during face recognition. Additionally, for every observer we obtained an objective index of neural face discrimination through EEG that was recorded while they fixated different facial information. We found that foveation of facial features fixated longer during face recognition elicited stronger neural face discrimination responses across all observers. This relationship occurred independently of interindividual differences in preferential facial information sampling (e.g., eye vs mouth lookers), and started as early as the first fixation. Our data show that eye movements play a functional role during face processing by providing the neural system with the information that is diagnostic to a specific observer. The effective processing of identity involves idiosyncratic, rather than universal face representations.SIGNIFICANCE STATEMENT When engaging in face recognition, observers deploy idiosyncratic fixation patterns to sample facial information. Whether these individual differences concur with idiosyncratic face-sensitive neural responses remains unclear. To address this issue, we recorded observers' fixation patterns, as well as their neural face discrimination responses elicited during fixation of 10 different locations on the face, corresponding to different types of facial information. Our data reveal a clear interplay between individuals' face-sensitive neural responses and their idiosyncratic eye-movement patterns during identity processing, which emerges as early as the first fixation. Collectively, our findings favor the existence of idiosyncratic, rather than universal face representations.
Collapse
|
36
|
Hessels RS, Holleman GA, Kingstone A, Hooge IT, Kemner C. Gaze allocation in face-to-face communication is affected primarily by task structure and social context, not stimulus-driven factors. Cognition 2019; 184:28-43. [DOI: 10.1016/j.cognition.2018.12.005] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2018] [Revised: 11/30/2018] [Accepted: 12/06/2018] [Indexed: 12/28/2022]
|
37
|
Abstract
Humans use saccades to inspect objects of interest with the foveola, the small region of the retina with highest acuity. This process of visual exploration is normally studied over large scenes. However, in everyday tasks, the stimulus within the foveola is complex, and the need for visual exploration may extend to this smaller scale. We have previously shown that fixational eye movements, in particular microsaccades, play an important role in fine spatial vision. Here, we investigate whether task-driven visual exploration occurs during the fixation pauses in between large saccades. Observers judged the expression of faces covering approximately 1°, as if viewed from a distance of many meters. We use a custom system for accurately localizing the line of sight and continually track gaze position at high resolution. Our findings reveal that active spatial exploration, a process driven by the goals of the task, takes place at the foveal scale. The scanning strategies used at this scale resemble those used when examining larger scenes, with idiosyncrasies maintained across spatial scales. These findings suggest that the visual system possesses not only a coarser priority map of the extrafoveal space to guide saccades, but also a finer-grained priority map that is used to guide microsaccades once the region of interest is foveated.
Collapse
|
38
|
Crosby F, Hermens F. Does it look safe? An eye tracking study into the visual aspects of fear of crime. Q J Exp Psychol (Hove) 2019; 72:599-615. [DOI: 10.1177/1747021818769203] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Studies of fear of crime often focus on demographic and social factors, but these can be difficult to change. Studies of visual aspects have suggested that features reflecting incivilities, such as litter, graffiti, and vandalism increase fear of crime, but methods often rely on participants actively mentioning such aspects, and more subtle, less conscious aspects may be overlooked. To address these concerns, this study examined people’s eye movements while they judged scenes for safety. In total, 40 current and former university students were asked to rate images of day-time and night-time scenes of Lincoln, UK (where they studied) and Egham, UK (unfamiliar location) for safety, maintenance, and familiarity while their eye movements were recorded. Another 25 observers not from Lincoln or Egham rated the same images in an Internet survey. Ratings showed a strong association between safety and maintenance and lower safety ratings for night-time scenes for both groups, in agreement with earlier findings. Eye movements of the Lincoln participants showed increased dwell times on buildings, houses, and vehicles during safety judgements and increased dwell times on streets, pavements, and markers of incivilities for maintenance. Results confirm that maintenance plays an important role in perceptions of safety, but eye movements suggest that observers also look for indicators of current or recent presence of people.
Collapse
Affiliation(s)
- Freya Crosby
- School of Psychology, University of Lincoln, Lincoln, UK
| | - Frouke Hermens
- School of Psychology, University of Lincoln, Lincoln, UK
| |
Collapse
|
39
|
Arizpe JM, Noles DL, Tsao JW, Chan AWY. Eye Movement Dynamics Differ between Encoding and Recognition of Faces. Vision (Basel) 2019; 3:vision3010009. [PMID: 31735810 PMCID: PMC6802769 DOI: 10.3390/vision3010009] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2018] [Revised: 11/15/2018] [Accepted: 12/26/2018] [Indexed: 11/16/2022] Open
Abstract
Facial recognition is widely thought to involve a holistic perceptual process, and optimal recognition performance can be rapidly achieved within two fixations. However, is facial identity encoding likewise holistic and rapid, and how do gaze dynamics during encoding relate to recognition? While having eye movements tracked, participants completed an encoding ("study") phase and subsequent recognition ("test") phase, each divided into blocks of one- or five-second stimulus presentation time conditions to distinguish the influences of experimental phase (encoding/recognition) and stimulus presentation time (short/long). Within the first two fixations, several differences between encoding and recognition were evident in the temporal and spatial dynamics of the eye-movements. Most importantly, in behavior, the long study phase presentation time alone caused improved recognition performance (i.e., longer time at recognition did not improve performance), revealing that encoding is not as rapid as recognition, since longer sequences of eye-movements are functionally required to achieve optimal encoding than to achieve optimal recognition. Together, these results are inconsistent with a scan path replay hypothesis. Rather, feature information seems to have been gradually integrated over many fixations during encoding, enabling recognition that could subsequently occur rapidly and holistically within a small number of fixations.
Collapse
Affiliation(s)
- Joseph M. Arizpe
- Department of Neurology, University of Tennessee Health Science Center, Memphis, TN 38163, USA
- Children’s Foundation Research Institute, Le Bonheur Children’s Hospital, Memphis, TN 38103, USA
- Science Applications International Corporation (SAIC), Fort Sam Houston, TX 78234, USA
- Correspondence:
| | - Danielle L. Noles
- Department of Neurology, University of Tennessee Health Science Center, Memphis, TN 38163, USA
- Children’s Foundation Research Institute, Le Bonheur Children’s Hospital, Memphis, TN 38103, USA
- School of Medicine, University of Tennessee Health Science Center, Memphis, TN 38163, USA
| | - Jack W. Tsao
- Department of Neurology, University of Tennessee Health Science Center, Memphis, TN 38163, USA
- Children’s Foundation Research Institute, Le Bonheur Children’s Hospital, Memphis, TN 38103, USA
- Department of Anatomy & Neurobiology, University of Tennessee Health Science Center, Memphis, TN 38163, USA
- Memphis Veterans Affairs Medical Center, Memphis, TN 38104, USA
| | - Annie W.-Y. Chan
- Department of Neurology, University of Tennessee Health Science Center, Memphis, TN 38163, USA
- Children’s Foundation Research Institute, Le Bonheur Children’s Hospital, Memphis, TN 38103, USA
- Department of Radiology, University of Tennessee Health Science Center, Memphis, TN 38163, USA
- Department of Life Sciences, Centre for Cognitive Neuroscience, Division of Psychology, Brunel University London, London, UB8 3PH, UK
| |
Collapse
|
40
|
Thompson SJ, Foulsham T, Leekam SR, Jones CR. Attention to the face is characterised by a difficult to inhibit first fixation to the eyes. Acta Psychol (Amst) 2019; 193:229-238. [PMID: 30690268 DOI: 10.1016/j.actpsy.2019.01.006] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2018] [Revised: 12/21/2018] [Accepted: 01/07/2019] [Indexed: 10/27/2022] Open
Abstract
The eyes are preferentially attended over other facial features and recent evidence suggests this bias is difficult to suppress. To further examine the automatic and volitional nature of this bias for eye information, we used a novel prompting face recognition paradigm in 41 adults and measured the location of their first fixations, overall dwell time and behavioural responses. First, patterns of eye gaze were measured during a free-viewing forced choice face recognition paradigm. Second, the task was repeated but with prompts to look to either the eyes or the mouth. Participants showed significantly more first fixations to the eyes than mouth, both when prompted to look at the eyes and when prompted to look at the mouth. The pattern of looking to the eyes when prompted was indistinguishable from the unprompted condition in which participants were free to look where they chose. Notably, the dwell time data demonstrated that the eye bias did not persist over the entire presentation period. Our results suggest a difficult-to-inhibit bias to initially orient to the eyes, which is superseded by volitional, top-down control of eye gaze. Further, the amount of looking to the eyes is at a maximum level spontaneously and cannot be enhanced by explicit instructions.
Collapse
|
41
|
Lüthold P, Lao J, He L, Zhou X, Caldara R. Waldo reveals cultural differences in return fixations. VISUAL COGNITION 2019. [DOI: 10.1080/13506285.2018.1561567] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
Affiliation(s)
- Patrick Lüthold
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Junpeng Lao
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Lingnan He
- School of Communication and Design, Sun Yat-Sen University, Guangzhou, People’s Republic of China
| | - Xinyue Zhou
- School of Management, Zhejiang University, Zhejiang, People’s Republic of China
| | - Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg, Fribourg, Switzerland
| |
Collapse
|
42
|
Eye-movement patterns in face recognition are associated with cognitive decline in older adults. Psychon Bull Rev 2019; 25:2200-2207. [PMID: 29313315 DOI: 10.3758/s13423-017-1419-0] [Citation(s) in RCA: 30] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
The Hidden Markov Modeling approach for eye-movement data analysis is able to quantitatively assess differences and similarities among individual patterns. Here we applied this approach to examine the relationships between eye-movement patterns in face recognition and age-related cognitive decline. We found that significantly more older than young adults adopted "holistic" patterns, in which most eye fixations landed around the face center, as opposed to "analytic" patterns, in which eye movements switched among the two eyes and the face center. Participants showing analytic patterns had better performance than those with holistic patterns regardless of age. Interestingly, older adults with lower cognitive status (as assessed by the Montreal Cognitive Assessment), particularly in executive and visual attention functioning (as assessed by Tower of London and Trail Making Tests) were associated with a higher likelihood of holistic patterns. This result suggests the possibility of using eye movements as an easily deployable screening assessment for cognitive decline in older adults.
Collapse
|
43
|
Calvo MG, Fernández-Martín A, Gutiérrez-García A, Lundqvist D. Selective eye fixations on diagnostic face regions of dynamic emotional expressions: KDEF-dyn database. Sci Rep 2018; 8:17039. [PMID: 30451919 PMCID: PMC6242984 DOI: 10.1038/s41598-018-35259-w] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2018] [Accepted: 10/28/2018] [Indexed: 12/20/2022] Open
Abstract
Prior research using static facial stimuli (photographs) has identified diagnostic face regions (i.e., functional for recognition) of emotional expressions. In the current study, we aimed to determine attentional orienting, engagement, and time course of fixation on diagnostic regions. To this end, we assessed the eye movements of observers inspecting dynamic expressions that changed from a neutral to an emotional face. A new stimulus set (KDEF-dyn) was developed, which comprises 240 video-clips of 40 human models portraying six basic emotions (happy, sad, angry, fearful, disgusted, and surprised). For validation purposes, 72 observers categorized the expressions while gaze behavior was measured (probability of first fixation, entry time, gaze duration, and number of fixations). Specific visual scanpath profiles characterized each emotional expression: The eye region was looked at earlier and longer for angry and sad faces; the mouth region, for happy faces; and the nose/cheek region, for disgusted faces; the eye and the mouth regions attracted attention in a more balanced manner for surprise and fear. These profiles reflected enhanced selective attention to expression-specific diagnostic face regions. The KDEF-dyn stimuli and the validation data will be available to the scientific community as a useful tool for research on emotional facial expression processing.
Collapse
Affiliation(s)
- Manuel G Calvo
- Department of Cognitive Psychology, Universidad de La Laguna, Tenerife, Spain.
- Instituto Universitario de Neurociencia (IUNE), Universidad de La Laguna, Tenerife, Spain.
| | | | | | | |
Collapse
|
44
|
Hermens F, Golubickis M, Macrae CN. Eye movements while judging faces for trustworthiness and dominance. PeerJ 2018; 6:e5702. [PMID: 30324015 PMCID: PMC6186410 DOI: 10.7717/peerj.5702] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2018] [Accepted: 09/06/2018] [Indexed: 11/20/2022] Open
Abstract
Past studies examining how people judge faces for trustworthiness and dominance have suggested that they use particular facial features (e.g. mouth features for trustworthiness, eyebrow and cheek features for dominance ratings) to complete the task. Here, we examine whether eye movements during the task reflect the importance of these features. We here compared eye movements for trustworthiness and dominance ratings of face images under three stimulus configurations: Small images (mimicking large viewing distances), large images (mimicking face to face viewing), and a moving window condition (removing extrafoveal information). Whereas first area fixated, dwell times, and number of fixations depended on the size of the stimuli and the availability of extrafoveal vision, and varied substantially across participants, no clear task differences were found. These results indicate that gaze patterns for face stimuli are highly individual, do not vary between trustworthiness and dominance ratings, but are influenced by the size of the stimuli and the availability of extrafoveal vision.
Collapse
Affiliation(s)
- Frouke Hermens
- School of Psychology, University of Lincoln, Lincoln, Lincolnshire, UK
| | | | - C. Neil Macrae
- School of Psychology, University of Aberdeen, Aberdeen, UK
| |
Collapse
|
45
|
One versus two eyes makes a difference! Early face perception is modulated by featural fixation and feature context. Cortex 2018; 109:35-49. [PMID: 30286305 DOI: 10.1016/j.cortex.2018.08.025] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2017] [Revised: 07/11/2018] [Accepted: 08/30/2018] [Indexed: 11/23/2022]
Abstract
The N170 event-related potential component is an early marker of face perception that is particularly sensitive to isolated eye regions and to eye fixations within a face. Here, this eye sensitivity was tested further by measuring the N170 to isolated facial features and to the same features fixated within a face, using a gaze-contingent procedure. The neural response to single isolated eyes and eye regions (two eyes) was also compared. Pixel intensity and contrast were controlled at the global (image) and local (featural) levels. Consistent with previous findings, larger N170 amplitudes were elicited when the left or right eye was fixated within a face, compared to the mouth or nose, demonstrating that the N170 eye sensitivity reflects higher-order perceptual processes and not merely low-level perceptual effects. The N170 was also largest and most delayed for isolated features, compared to equivalent fixations within a face. Specifically, mouth fixation yielded the largest amplitude difference, and nose fixation yielded the largest latency difference between these two contexts, suggesting the N170 may reflect a complex interplay between holistic and featural processes. Critically, eye regions elicited consistently larger and shorter N170 responses compared to single eyes, with enhanced responses for contralateral eye content, irrespective of eye or nasion fixation. These results confirm the importance of the eyes in early face perception, and provide novel evidence of an increased sensitivity to the presence of two symmetric eyes compared to only one eye, consistent with a neural eye region detector rather than an eye detector per se.
Collapse
|
46
|
Calvo MG, Krumhuber EG, Fernández-Martín A. Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions. Q J Exp Psychol (Hove) 2018; 72:729-741. [PMID: 29471708 DOI: 10.1177/1747021818763747] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
Abstract
A happy facial expression makes a person look (more) trustworthy. Do perceptions of happiness and trustworthiness rely on the same face regions and visual attention processes? In an eye-tracking study, eye movements and fixations were recorded while participants judged the un/happiness or the un/trustworthiness of dynamic facial expressions in which the eyes and/or the mouth unfolded from neutral to happy or vice versa. A smiling mouth and happy eyes enhanced perceived happiness and trustworthiness similarly, with a greater contribution of the smile relative to the eyes. This comparable judgement output for happiness and trustworthiness was reached through shared as well as distinct attentional mechanisms: (a) entry times and (b) initial fixation thresholds for each face region were equivalent for both judgements, thereby revealing the same attentional orienting in happiness and trustworthiness processing. However, (c) greater and (d) longer fixation density for the mouth region in the happiness task, and for the eye region in the trustworthiness task, demonstrated different selective attentional engagement. Relatedly, (e) mean fixation duration across face regions was longer in the trustworthiness task, thus showing increased attentional intensity or processing effort.
Collapse
Affiliation(s)
- Manuel G Calvo
- 1 Department of Cognitive Psychology, Universidad de La Laguna, Tenerife, Spain
| | | | - Andrés Fernández-Martín
- 3 Universidad Internacional de La Rioja, Logroño, Spain and Instituto Universitario de Neurociencia (IUNE), Universidad de La Laguna, Tenerife, Spain
| |
Collapse
|
47
|
Abstract
We report the personal eye gaze patterns of people engaged in face-to-face getting acquainted conversation. Considerable differences between individuals are underscored by a stability of eye gaze patterns within individuals. Results suggest the existence of an eye-mouth gaze continuum. This continuum includes some people showing a strong preference for eye gaze, some with a strong preference for mouth gaze, and others distributing their gaze between the eyes and mouth to varying extents. Additionally, we found evidence of within-participant consistency not just for location preference but also for the duration of fixations upon the eye and mouth regions. We also estimate that during a 4-minute getting acquainted conversation mutual face gaze constitutes about 60% of conversation that occurs via typically brief instances of 2.2 seconds. Mutual eye contact ranged from 0-45% of conversation, via very brief instances. This was despite participants subjectively perceiving eye contact occurring for about 70% of conversation. We argue that the subjective perception of eye contact is a product of mutual face gaze instead of actual mutual eye contact. We also outline the fast activity of gaze movements upon various locations both on and off face during a typical face-to-face conversation.
Collapse
|
48
|
Abstract
How people look at visual information reveals fundamental information about them; their interests and their states of mind. Previous studies showed that scanpath, i.e., the sequence of eye movements made by an observer exploring a visual stimulus, can be used to infer observer-related (e.g., task at hand) and stimuli-related (e.g., image semantic category) information. However, eye movements are complex signals and many of these studies rely on limited gaze descriptors and bespoke datasets. Here, we provide a turnkey method for scanpath modeling and classification. This method relies on variational hidden Markov models (HMMs) and discriminant analysis (DA). HMMs encapsulate the dynamic and individualistic dimensions of gaze behavior, allowing DA to capture systematic patterns diagnostic of a given class of observers and/or stimuli. We test our approach on two very different datasets. Firstly, we use fixations recorded while viewing 800 static natural scene images, and infer an observer-related characteristic: the task at hand. We achieve an average of 55.9% correct classification rate (chance = 33%). We show that correct classification rates positively correlate with the number of salient regions present in the stimuli. Secondly, we use eye positions recorded while viewing 15 conversational videos, and infer a stimulus-related characteristic: the presence or absence of original soundtrack. We achieve an average 81.2% correct classification rate (chance = 50%). HMMs allow to integrate bottom-up, top-down, and oculomotor influences into a single model of gaze behavior. This synergistic approach between behavior and machine learning will open new avenues for simple quantification of gazing behavior. We release SMAC with HMM, a Matlab toolbox freely available to the community under an open-source license agreement.
Collapse
Affiliation(s)
| | - Janet H Hsiao
- Department of Psychology, The University of Hong Kong, Pok Fu Lam, Hong Kong
| | - Antoni B Chan
- Department of Computer Science, City University of Hong Kong, Kowloon Tong, Hong Kong
| |
Collapse
|
49
|
Hidden Markov model analysis reveals the advantage of analytic eye movement patterns in face recognition across cultures. Cognition 2017; 169:102-117. [PMID: 28869811 DOI: 10.1016/j.cognition.2017.08.003] [Citation(s) in RCA: 31] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2016] [Revised: 08/08/2017] [Accepted: 08/08/2017] [Indexed: 11/21/2022]
Abstract
It remains controversial whether culture modulates eye movement behavior in face recognition. Inconsistent results have been reported regarding whether cultural differences in eye movement patterns exist, whether these differences affect recognition performance, and whether participants use similar eye movement patterns when viewing faces from different ethnicities. These inconsistencies may be due to substantial individual differences in eye movement patterns within a cultural group. Here we addressed this issue by conducting individual-level eye movement data analysis using hidden Markov models (HMMs). Each individual's eye movements were modeled with an HMM. We clustered the individual HMMs according to their similarities and discovered three common patterns in both Asian and Caucasian participants: holistic (looking mostly at the face center), left-eye-biased analytic (looking mostly at the two individual eyes in addition to the face center with a slight bias to the left eye), and right-eye-based analytic (looking mostly at the right eye in addition to the face center). The frequency of participants adopting the three patterns did not differ significantly between Asians and Caucasians, suggesting little modulation from culture. Significantly more participants (75%) showed similar eye movement patterns when viewing own- and other-race faces than different patterns. Most importantly, participants with left-eye-biased analytic patterns performed significantly better than those using either holistic or right-eye-biased analytic patterns. These results suggest that active retrieval of facial feature information through an analytic eye movement pattern may be optimal for face recognition regardless of culture.
Collapse
|
50
|
Caldara R. Culture Reveals a Flexible System for Face Processing. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE 2017. [DOI: 10.1177/0963721417710036] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
The human face transmits a wealth of signals that readily provide crucial information for social interactions, such as facial identity and emotional expression. Nonetheless, a fundamental question remains debated: Is face processing governed by universal perceptual processes? It has long been presumed that this is the case. However, over the past decade, our work at the Eye and Brain Mapping Laboratory has called into question this widely held assumption. We have investigated the eye movements of Western and Eastern observers across various face-processing tasks to determine the effect of culture on perceptual processing. Commonalities aside, we found that Westerners distribute local fixations across the eye and mouth regions, whereas Easterners preferentially deploy central, global fixations during face recognition. Moreover, during the recognition of facial expressions of emotion, Westerners fixate the mouth relatively more to discriminate across expressions, whereas Easterners favor the eye region. Both observations demonstrate that the face system relies on different strategies to perform a range of socially relevant face-processing tasks with comparable levels of efficiency. Overall, these cultural perceptual biases challenge the view that the processes dedicated to face processing are universal, favoring instead the existence of distinct, flexible strategies. The way humans perceive the world and process faces is determined by experience and environmental factors.
Collapse
Affiliation(s)
- Roberto Caldara
- Eye and Brain Mapping Laboratory (iBMLab), Department of Psychology, University of Fribourg
| |
Collapse
|