Effects of Caricaturing in Shape or Color on Familiarity Decisions for Familiar and Unfamiliar Faces

Finding neural correlates of depersonalisation/derealisation disorder via explainable CNN-based analysis guided by clinical assessment scores

Mental health disorders are typically diagnosed based on subjective reports (e.g., through questionnaires) followed by clinical interviews to evaluate the self-reported symptoms. Therefore, considering the interconnected nature of psychiatric disorders, their accurate diagnosis is a real challenge without indicators of underlying physiological dysfunction. Depersonalisation/derealisation disorder (DPD) is an example of dissociative disorder affecting 1-2 % of the population. DPD is characterised mainly by persistent disembodiment, detachment from surroundings, and feelings of emotional numbness, which can significantly impact patients' quality of life. The underlying neural correlates of DPD have been investigated for years to understand and help with a more accurate and in-time diagnosis of the disorder. However, in terms of EEG studies, which hold great importance due to their convenient and inexpensive nature, the literature has often been based on hypotheses proposed by experts in the field, which require prior knowledge of the disorder. In addition, participants' labelling in research experiments is often derived from the outcome of the Cambridge Depersonalisation Scale (CDS), a subjective assessment to quantify the level of depersonalisation/derealisation, the threshold and reliability of which might be challenged. As a result, we aimed to propose a novel end-to-end EEG processing pipeline based on deep neural networks for DPD biomarker discovery, which requires no prior handcrafted labelled data. Alternatively, it can assimilate knowledge from clinical outcomes like CDS as well as data-driven patterns that differentiate individual brain responses. In addition, the structure of the proposed model targets the uncertainty in CDS scores by using them as prior information only to guide the unsupervised learning task in a multi-task learning scenario. A comprehensive evaluation has been done to confirm the significance of the proposed deep structure, including new ways of network visualisation to investigate spectral, spatial, and temporal information derived in the learning process. We argued that the proposed EEG analytics could also be applied to investigate other psychological and mental disorders currently indicated on the basis of clinical assessment scores. The code to reproduce the results presented in this paper is openly accessible at https://github.com/AbbasSalami/DPD_Analysis.

A narrow band of image dimensions is critical for face recognition

A key challenge in human and computer face recognition is to differentiate information that is diagnostic for identity from other sources of image variation. Here, we used a combined computational and behavioural approach to reveal critical image dimensions for face recognition. Behavioural data were collected using a sorting and matching task with unfamiliar faces and a recognition task with familiar faces. Principal components analysis was used to reveal the dimensions across which the shape and texture of faces in these tasks varied. We then asked which image dimensions were able to predict behavioural performance across these tasks. We found that the ability to predict behavioural responses in the unfamiliar face tasks increased when the early PCA dimensions (i.e. those accounting for most variance) of shape and texture were removed from the analysis. Image similarity also predicted the output of a computer model of face recognition, but again only when the early image dimensions were removed from the analysis. Finally, we found that recognition of familiar faces increased when the early image dimensions were removed, decreased when intermediate dimensions were removed, but then returned to baseline recognition when only later dimensions were removed. Together, these findings suggest that early image dimensions reflect ambient changes, such as changes in viewpoint or lighting, that do not contribute to face recognition. However, there is a narrow band of image dimensions for shape and texture that are critical for the recognition of identity in humans and computer models of face recognition.

Neural Correlates of Voice Learning with Distinctive and Non-Distinctive Faces

Recognizing people from their voices may be facilitated by a voice's distinctiveness, in a manner similar to that which has been reported for faces. However, little is known about the neural time-course of voice learning and the role of facial information in voice learning. Based on evidence for audiovisual integration in the recognition of familiar people, we studied the behavioral and electrophysiological correlates of voice learning associated with distinctive or non-distinctive faces. We repeated twelve unfamiliar voices uttering short sentences, together with either distinctive or non-distinctive faces (depicted before and during voice presentation) in six learning-test cycles. During learning, distinctive faces increased early visually-evoked (N170, P200, N250) potentials relative to non-distinctive faces, and face distinctiveness modulated voice-elicited slow EEG activity at the occipito-temporal and fronto-central electrodes. At the test, unimodally-presented voices previously learned with distinctive faces were classified more quickly than were voices learned with non-distinctive faces, and also more quickly than novel voices. Moreover, voices previously learned with faces elicited an N250-like component that was similar in topography to that typically observed for facial stimuli. The preliminary source localization of this voice-induced N250 was compatible with a source in the fusiform gyrus. Taken together, our findings provide support for a theory of early interaction between voice and face processing areas during both learning and voice recognition.

Stylized faces enhance ERP features used for the detection of emotional responses

For their ease of accessibility and low cost, current Brain-Computer Interfaces (BCI) used to detect subjective emotional and affective states rely largely on electroencephalographic (EEG) signals. Public datasets are available for researchers to design models for affect detection from EEG. However, few designs focus on optimally exploiting the nature of the stimulus elicitation to improve accuracy. The RSVP protocol is used in this experiment to present human faces of emotion to 28 participants while EEG was measured. We found that artificially enhanced human faces with exaggerated, cartoonish visual features significantly improve some commonly used neural correlates of emotion as measured by event-related potentials (ERPs). These images elicit an enhanced N170 component, well known to relate to the facial visual encoding process. Our findings suggest that the study of emotion elicitation could exploit consistent, high detail, AI generated stimuli transformations to study the characteristics of electrical brain activity related to visual affective stimuli. Furthermore, this specific result might be useful in the context of affective BCI design, where a higher accuracy in affect decoding from EEG can improve the experience of a user.

Incomplete faces are completed using a more average face

Facial first impressions are known to influence how we behave towards others. As a result of the COVID-19 pandemic, we often view incomplete faces due to the commonplace wearing of face masks. Previous research has shown that perceptions of attractiveness are often increased due to these coverings, with initial evidence suggesting that this may be caused by viewers using a mental representation of the average face to complete any missing information. Here, we directly address this hypothesis by presenting participants with incomplete faces (either the lower or upper half removed) and asking them to decide how they thought the actual, full face looked. Participants were able to manipulate the missing half of the face onscreen by increasing or decreasing the averageness of its shape. Our results demonstrated that participants did not select the original versions of the faces but instead chose more average versions when manipulating both the lower and upper face. Further, the typicality of the original image influenced responses, with less typical faces (in comparison with more typical ones) being completed using an even more average version of the missing half of the faces. Taken together, these findings provide the first direct evidence that people utilise an average/typical internal representation when inferring information about incomplete faces. This result has theoretical importance in terms of visual perception, as well as real-world relevance in a time where face masks are commonplace due to the COVID-19 pandemic.

Similar use of shape and texture cues for own- and other-race faces during face learning and recognition

Although the other-race effect (ORE; superior recognition of own- relative to other-race faces) is well established, the mechanisms underlying it are not well understood. We examined whether the ORE is attributable to differential use of shape and texture cues for own- vs. other-race faces. Shape cues are particularly important for detecting that an own-race face is unfamiliar, whereas texture cues are more important for recognizing familiar and newly learned own-race faces. We compared the influence of shape and texture cues on Caucasian participants' recognition of Caucasian and East Asian faces using two complementary approaches. In Experiment 1, participants studied veridical, shape-caricatured, or texture-caricatured faces and then were asked to recognize them in an old/new recognition task. In Experiment 2, all study faces were veridical and we independently removed the diagnosticity of shape (or texture) cues in the test phase by replacing original shape (or texture) with average shape (or texture). Despite an overall own-race advantage, participants' use of shape and texture cues was comparable for own- and other-race faces. These results suggest that the other-race effect is not attributable to qualitative differences in the use of shape and texture cues.

Contrast Negation Impairs Sorting Unfamiliar Faces by Identity: A Comparison With Original (Contrast-Positive) and Stretched Images

Recognition of unfamiliar faces is difficult in part due to variations in expressions, angles, and image quality. Studies suggest shape and surface properties play varied roles in face learning, and identification of unfamiliar faces uses diagnostic pigmentation/surface reflectance relative to shape information. Here, participants sorted photo-cards of unfamiliar faces by identity, which were shown in their original, stretched, and contrast-negated forms, to examine the utility of diagnostic shape and surface properties in sorting unfamiliar faces by identity. In four experiments, we varied the presentation order of conditions (contrast-negated first or original first with stretched second across experiments) and whether the same or different photo-cards were seen across conditions. Stretching the images did not impair performance in any measures relative to other conditions. Contrast negation generally exacerbated poor sorting by identity compared with the other conditions. However, seeing the contrast-negated photo-cards last mitigated some of the effects of contrast negation. Together, results suggest an important role for surface properties such as pigmentation and reflectance for sorting by identity and add to literatures on informational content and appearance variability in discrimination of facial identity.

Different and common brain signals of altered neurocognitive mechanisms for unfamiliar face processing in acquired and developmental prosopagnosia

Neuropsychological studies have shown that prosopagnosic individuals perceive face structure in an atypical way. This might preclude the formation of appropriate face representations and, consequently, hamper effective recognition. The present ERP study, in combination with Bayesian source reconstruction, investigates how information related to both external (E) and internal (I) features was processed by E.C. and I.P., suffering from acquired and developmental prosopagnosia, respectively. They carried out a face-feature matching task with new faces. E.C. showed poor performance and remarkable lack of early face-sensitive P1, N170 and P2 responses on right (damaged) posterior cortex. Although she presented the expected mismatch effect to target faces in the E-I sequence, it was of shorter duration than in Controls, and involved left parietal, right frontocentral and dorsofrontal regions, suggestive of reduced neural circuitry to process face configurations. In turn, I.P. performed efficiently but with a remarkable bias to give "match" responses. His face-sensitive potentials P1-N170 were comparable to those from Controls, however, he showed no subsequent P2 response and a mismatch effect only in the I-E sequence, reflecting activation confined to those regions that sustain typically the initial stages of face processing. Relevantly, neither of the prosopagnosics exhibited conspicuous P3 responses to features acting as primes, indicating that diagnostic information for constructing face representations could not be sufficiently attended nor deeply encoded. Our findings suggest a different locus for altered neurocognitive mechanisms in the face network in participants with different types of prosopagnosia, but common indicators of a deficient allocation of attentional resources for further recognition.

Hemodynamic response to familiar faces in children with ADHD

Background

School-age children with attention deficit hyperactivity disorder (ADHD) have difficulties in interpersonal relationships, in addition to impaired facial expression perception and recognition. For successful social interactions, the ability to discriminate between familiar and unfamiliar faces is critical. However, there are no published reports on the recognition of familiar and unfamiliar faces by children with ADHD.

Methods

We evaluated the neural correlates of familiar and unfamiliar facial recognition in children with ADHD compared to typically developing (TD) children. We used functional near-infrared spectroscopy (fNIRS) to measure hemodynamic responses on the bilateral temporal regions while participants looked at photographs of familiar and unfamiliar faces. Nine boys with ADHD and 14 age-matched TD boys participated in the study. fNIRS data were Z-scored prior to analysis.

Results

During familiar face processing, TD children only showed significant activity in the late phase, while ADHD children showed significant activity in both the early and late phases. Additionally, the boys with ADHD did not show right hemispheric lateralization to familiar faces.

Conclusions

This study is the first to assess brain activity during familiar face processing in boys with ADHD using fNIRS. These findings of atypical patterns of brain activity in boys with ADHD may be related to social cognitive impairments from ADHD.

Improving face identity perception in age-related macular degeneration via caricaturing

Patients with age-related macular degeneration (AMD) have difficulty recognising people's faces. We tested whether this could be improved using caricaturing: an image enhancement procedure derived from cortical coding in a perceptual 'face-space'. Caricaturing exaggerates the distinctive ways in which an individual's face shape differs from the average. We tested 19 AMD-affected eyes (from 12 patients; ages 66-93 years) monocularly, selected to cover the full range of vision loss. Patients rated how different in identity people's faces appeared when compared in pairs (e.g., two young men, both Caucasian), at four caricature strengths (0, 20, 40, 60% exaggeration). This task gives data reliable enough to analyse statistically at the individual-eye level. All 9 eyes with mild vision loss (acuity ≥ 6/18) showed significant improvement in identity discrimination (higher dissimilarity ratings) with caricaturing. The size of improvement matched that in normal-vision young adults. The caricature benefit became less stable as visual acuity further decreased, but caricaturing was still effective in half the eyes with moderate and severe vision loss (significant improvement in 5 of 10 eyes; at acuities from 6/24 to poorer than <6/360). We conclude caricaturing has the potential to help many AMD patients recognise faces.

Familiar Face Priming: The Role of Second-Order Configuration and Individual Face Recognition Abilities

The role of second-order configuration-that is, metric distances between individual features-for familiar face recognition has been the subject of debate. Recent reports suggest that better face recognition abilities coincide with a weaker reliance on shape information for face recognition. We examined contributions of second-order configuration to familiar face repetition priming by manipulating metric distances between facial features. S1 comprised familiar face primes as either: unaltered, with increased or decreased interocular distance, with increased or decreased distance between nose and mouth; or a different familiar face (unprimed). Participants performed a familiarity decision task on familiar and unfamiliar S2 targets, and completed a test battery consisting of three face identity processing tests. Accuracies, reaction times, and inverse efficiency scores were assessed for the priming experiment, and potential priming costs in inverse efficiency scores were correlated with test battery scores. Overall, priming was found, and priming effects were reduced only by primes with interocular distance distortions. Correlational data showed that better face recognition skills coincided with a weaker reliance on second-order configurations. Our findings (a) suggest an importance of interocular, but not mouth-to-nose, distances for familiar face recognition and (b) show that good face recognizers are less sensitive to second-order configuration.

Face identity recognition in simulated prosthetic vision is poorer than previously reported and can be improved by caricaturing

The visual prosthesis (or "bionic eye") has become a reality but provides a low resolution view of the world. Simulating prosthetic vision in normal-vision observers, previous studies report good face recognition ability using tasks that allow recognition to be achieved on the basis of information that survives low resolution well, including basic category (sex, age) and extra-face information (hairstyle, glasses). Here, we test within-category individuation for face-only information (e.g., distinguishing between multiple Caucasian young men with hair covered). Under these conditions, recognition was poor (although above chance) even for a simulated 40×40 array with all phosphene elements assumed functional, a resolution above the upper end of current-generation prosthetic implants. This indicates that a significant challenge is to develop methods to improve face identity recognition. Inspired by "bionic ear" improvements achieved by altering signal input to match high-level perceptual (speech) requirements, we test a high-level perceptual enhancement of face images, namely face caricaturing (exaggerating identity information away from an average face). Results show caricaturing improved identity recognition in memory and/or perception (degree by which two faces look dissimilar) down to a resolution of 32×32 with 30% phosphene dropout. Findings imply caricaturing may offer benefits for patients at resolutions realistic for some current-generation or in-development implants.