Using image reconstruction to investigate face perception in amnesia

Perception and memory-based representations of facial emotions: Associations with personality functioning, affective states and recognition abilities

Personality traits and affective states are associated with biases in facial emotion perception. However, the precise personality impairments and affective states that underlie these biases remain largely unknown. To investigate how relevant factors influence facial emotion perception and recollection, Experiment 1 employed an image reconstruction approach in which community-dwelling adults (N = 89) rated the similarity of pairs of facial expressions, including those recalled from memory. Subsequently, perception- and memory-based expression representations derived from such ratings were assessed across participants and related to measures of personality impairment, state affect, and visual recognition abilities. Impairment in self-direction and level of positive affect accounted for the largest components of individual variability in perception and memory representations, respectively. Additionally, individual differences in these representations were impacted by face recognition ability. In Experiment 2, adult participants (N = 81) rated facial image reconstructions derived in Experiment 1, revealing that individual variability was associated with specific visual face properties, such as expressiveness, representation accuracy, and positivity/negativity. These findings highlight and clarify the influence of personality, affective state, and recognition abilities on individual differences in the perception and recollection of facial expressions.

Scene-selectivity in CA1/subicular complex: Multivoxel pattern analysis at 7T

Prior univariate functional magnetic resonance imaging (fMRI) studies in humans suggest that the anteromedial subicular complex of the hippocampus is a hub for scene-based cognition. However, it is possible that univariate approaches were not sufficiently sensitive to detect scene-related activity in other subfields that have been implicated in spatial processing (e.g., CA1). Further, as connectivity-based functional gradients in the hippocampus do not respect classical subfield boundary definitions, category selectivity may be distributed across anatomical subfields. Region-of-interest approaches, therefore, may limit our ability to observe category selectivity across discrete subfield boundaries. To address these issues, we applied searchlight multivariate pattern analysis to 7T fMRI data of healthy adults who undertook a simultaneous visual odd-one-out discrimination task for scene and non-scene (including face) visual stimuli, hypothesising that scene classification would be possible in multiple hippocampal regions within, but not constrained to, anteromedial subicular complex and CA1. Indeed, we found that the scene-selective searchlight map overlapped not only with anteromedial subicular complex (distal subiculum, pre/para subiculum), but also inferior CA1, alongside posteromedial (including retrosplenial) and parahippocampal cortices. Probabilistic overlap maps revealed gradients of scene category selectivity, with the strongest overlap located in the medial hippocampus, converging with searchlight findings. This was contrasted with gradients of face category selectivity, which had stronger overlap in more lateral hippocampus, supporting ideas of parallel processing streams for these two categories. Our work helps to map the scene, in contrast to, face processing networks within, and connected to, the human hippocampus.

Mental image reconstruction from human brain activity: Neural decoding of mental imagery via deep neural network-based Bayesian estimation

Visual images observed by humans can be reconstructed from their brain activity. However, the visualization (externalization) of mental imagery is challenging. Only a few studies have reported successful visualization of mental imagery, and their visualizable images have been limited to specific domains such as human faces or alphabetical letters. Therefore, visualizing mental imagery for arbitrary natural images stands as a significant milestone. In this study, we achieved this by enhancing a previous method. Specifically, we demonstrated that the visual image reconstruction method proposed in the seminal study by Shen et al. (2019) heavily relied on low-level visual information decoded from the brain and could not efficiently utilize the semantic information that would be recruited during mental imagery. To address this limitation, we extended the previous method to a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen images (i.e., those observed by the human eye) and imagined images from brain activity. Quantitative evaluation showed that our framework could identify seen and imagined images highly accurately compared to the chance accuracy (seen: 90.7%, imagery: 75.6%, chance accuracy: 50.0%). In contrast, the previous method could only identify seen images (seen: 64.3%, imagery: 50.4%). These results suggest that our framework would provide a unique tool for directly investigating the subjective contents of the brain such as illusions, hallucinations, and dreams.