Neural dynamics of visual and semantic object processing

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Cortical time-course of evidence accumulation during semantic processing

Our understanding of the surrounding world and communication with other people are tied to mental representations of concepts. In order for the brain to recognize an object, it must determine which concept to access based on information available from sensory inputs. In this study, we combine magnetoencephalography and machine learning to investigate how concepts are represented and accessed in the brain over time. Using brain responses from a silent picture naming task, we track the dynamics of visual and semantic information processing, and show that the brain gradually accumulates information on different levels before eventually reaching a plateau. The timing of this plateau point varies across individuals and feature models, indicating notable temporal variation in visual object recognition and semantic processing.

Recurrent connectivity supports higher-level visual and semantic object representations in the brain

Visual object recognition has been traditionally conceptualised as a predominantly feedforward process through the ventral visual pathway. While feedforward artificial neural networks (ANNs) can achieve human-level classification on some image-labelling tasks, it's unclear whether computational models of vision alone can accurately capture the evolving spatiotemporal neural dynamics. Here, we probe these dynamics using a combination of representational similarity and connectivity analyses of fMRI and MEG data recorded during the recognition of familiar, unambiguous objects. Modelling the visual and semantic properties of our stimuli using an artificial neural network as well as a semantic feature model, we find that unique aspects of the neural architecture and connectivity dynamics relate to visual and semantic object properties. Critically, we show that recurrent processing between the anterior and posterior ventral temporal cortex relates to higher-level visual properties prior to semantic object properties, in addition to semantic-related feedback from the frontal lobe to the ventral temporal lobe between 250 and 500 ms after stimulus onset. These results demonstrate the distinct contributions made by semantic object properties in explaining neural activity and connectivity, highlighting it as a core part of object recognition not fully accounted for by current biologically inspired neural networks.

Semantic cognition in healthy ageing: Neural signatures of representation and control mechanisms in naming typical and atypical objects

Effective use of conceptual knowledge engages semantic representation and control processes to access information in a goal-driven manner. Neuropsychological findings of patients presenting either degraded knowledge (e.g., semantic dementia) or disrupted control (e.g., semantic aphasia) converge with neuroimaging evidence from young adults, and delineate the neural segregation of representation and control mechanisms. However, there is still scarce research on the neurofunctional underpinnings of such mechanisms in healthy ageing. To address this, we conducted an fMRI study, wherein young and older adults performed a covert naming task of typical and atypical objects. Three main age-related differences were found. As shown by age group and typicality interactions, older adults exhibited overactivation during naming of atypical (e.g., avocado) relative to typical concepts in brain regions associated to semantic representation, including anterior and medial portions of left temporal lobe (respectively, ATL and MTG). This provides evidence for the reorganization of neural activity in these brain regions contingent to the enrichment of semantic repositories in older ages. The medial orbitofrontal gyrus was also overactivated, indicating that the processing of atypical concepts (relative to typical items) taxes additional control resources in the elderly. Increased activation in the inferior frontal gyrus (IFG) was observed in naming typical items (relative to atypical ones), but only for young adults. This suggests that naming typical items (e.g., strawberry) taxes more on control processes in younger ages, presumably due to the semantic competition set by other items that share multiple features with the target (e.g., raspberry, blackberry, cherry). Together, these results reveal the dynamic nature of semantic control interplaying with conceptual representations as people grow older, by indicating that distinct neural bases uphold semantic performance from young to older ages. These findings may be explained by neural compensation mechanisms coming into play to support neurocognitive changes in healthy ageing.

Semantic Expectation Effects on Object Detection: Using Figure Assignment to Elucidate Mechanisms

Recent evidence suggesting that object detection is improved following valid rather than invalid labels implies that semantics influence object detection. It is not clear, however, whether the results index object detection or feature detection. Further, because control conditions were absent and labels and objects were repeated multiple times, the mechanisms are unknown. We assessed object detection via figure assignment, whereby objects are segmented from backgrounds. Masked bipartite displays depicting a portion of a mono-oriented object (a familiar configuration) on one side of a central border were shown once only for 90 or 100 ms. Familiar configuration is a figural prior. Accurate detection was indexed by reports of an object on the familiar configuration side of the border. Compared to control experiments without labels, valid labels improved accuracy and reduced response times (RTs) more for upright than inverted objects (Studies 1 and 2). Invalid labels denoting different superordinate-level objects (DSC; Study 1) or same superordinate-level objects (SSC; Study 2) reduced accuracy for upright displays only. Orientation dependency indicates that effects are mediated by activated object representations rather than features which are invariant over orientation. Following invalid SSC labels (Study 2), accurate detection RTs were longer than control for both orientations, implicating conflict between semantic representations that had to be resolved before object detection. These results demonstrate that object detection is not just affected by semantics, it entails semantics.