Recognition by association: Within- and cross-modality associative priming with faces and voices

Bridging the gap between EEG and DCNNs reveals a fatigue mechanism of facial repetition suppression

Facial repetition suppression, a well-studied phenomenon characterized by decreased neural responses to repeated faces in visual cortices, remains a subject of ongoing debate regarding its underlying neural mechanisms. Our research harnesses advanced multivariate analysis techniques and the prowess of deep convolutional neural networks (DCNNs) in face recognition to bridge the gap between human electroencephalogram (EEG) data and DCNNs, especially in the context of facial repetition suppression. Our innovative reverse engineering approach, manipulating the neuronal activity in DCNNs and conducted representational comparisons between brain activations derived from human EEG and manipulated DCNN activations, provided insights into the underlying facial repetition suppression. Significantly, our findings advocate the fatigue mechanism as the dominant force behind the facial repetition suppression effect. Broadly, this integrative framework, bridging the human brain and DCNNs, offers a promising tool for simulating brain activity and making inferences regarding the neural mechanisms underpinning complex human behaviors.

Unimodal and cross-modal identity judgements using an audio-visual sorting task: Evidence for independent processing of faces and voices

Unimodal and cross-modal information provided by faces and voices contribute to identity percepts. To examine how these sources of information interact, we devised a novel audio-visual sorting task in which participants were required to group video-only and audio-only clips into two identities. In a series of three experiments, we show that unimodal face and voice sorting were more accurate than cross-modal sorting: While face sorting was consistently most accurate followed by voice sorting, cross-modal sorting was at chancel level or below. In Experiment 1, we compared performance in our novel audio-visual sorting task to a traditional identity matching task, showing that unimodal and cross-modal identity perception were overall moderately more accurate than the traditional identity matching task. In Experiment 2, separating unimodal from cross-modal sorting led to small improvements in accuracy for unimodal sorting, but no change in cross-modal sorting performance. In Experiment 3, we explored the effect of minimal audio-visual training: Participants were shown a clip of the two identities in conversation prior to completing the sorting task. This led to small, nonsignificant improvements in accuracy for unimodal and cross-modal sorting. Our results indicate that unfamiliar face and voice perception operate relatively independently with no evidence of mutual benefit, suggesting that extracting reliable cross-modal identity information is challenging.

Validity of an eyetracking method for capturing auditory-visual cross-format semantic priming

INTRODUCTION

Semantic priming paradigms are important for understanding lexical-semantic processing and the nature of linguistic deficits accompanying language performance in neurologically impaired individuals such as people with aphasia. Reaction-time-(RT)-based traditional semantic priming tasks entail potential confounds, especially problematic when applied to people with aphasia, who may have concomitant neurocognitive challenges that limit task performance. Some of these confounds include requirements of following complex instructions, making metalinguistic judgments, and using speech or limb-based motor actions to indicate overt responses. Eyetracking methods have great potential for avoiding some of these confounds. We tested the validity of an eyetracking method in capturing semantic priming in an auditory-visual cross-format priming paradigm (auditory word prime-visual image target).

METHOD

A total of 72 neurologically unimpaired adults participated in two phases: a stimulus development phase using traditional priming (n = 32) and an experimental eyetracking phase (n = 40). Each phase included two conditions, representing distinct levels of prime-target semantic relatedness: unrelated and related. Mean RT data from the traditional priming (stimuli development) phase guided image selection for the eyetracking experiment. Eyetracking indices of fixation duration and latency of fixation were recorded to capture semantic priming in the eyetracking experiment.

RESULTS

Eye fixation data indicated that images related to auditory primes were attended to earlier and attracted significantly greater visual attention than unrelated images. These results mirrored RT data from the traditional priming method, which showed faster RT latencies and more accurate naming performance for related images than for unrelated images.

CONCLUSIONS

Results support the validity of eyetracking indices of semantic priming and offer a robust testing protocol for future studies in this line of research. Current clinical relevance for people with aphasia is highlighted. Further empirical testing of the psychometric properties of the eyetracking measures in various semantic priming contexts is recommended.