What is changing when: Decoding visual information in movies from human intracranial recordings

Proximity to boundaries reveals spatial context representation in human hippocampal CA1

Recollection of real-world events is often accompanied by a sense of being in the place where the event transpired. Convergent evidence suggests the hippocampus plays a key role in supporting episodic memory by associating information with the time and place it was originally encountered. This representation is reinstated during memory retrieval. However, little is known about the roles of different subfields of the human hippocampus in this process. Research in humans and non-human animal models has suggested that spatial environmental boundaries have a powerful influence on spatial and episodic memory, as well as hippocampal representations of contexts and events. Here, we used high-resolution fMRI to investigate how boundaries influence hippocampal activity patterns during the recollection of objects encountered in different spatial contexts. During the encoding phase, participants viewed objects once in a naturalistic virtual reality task in which they passively explored two rooms in one of two houses. Following the encoding phase, participants were scanned while they recollected items in the absence of any spatial contextual information. Our behavioral results demonstrated that spatial context memory was enhanced for objects encountered near a boundary. Activity patterns in CA1 carried information about the spatial context associated with each of these boundary items. Exploratory analyses revealed that recollection performance was correlated with the fidelity of retrieved spatial context representations in anterior parahippocampal cortex and subiculum. Our results highlight the privileged role of boundaries in CA1 and suggest more generally a close relationship between memory for spatial contexts and representations in the hippocampus and parahippocampal region.

Semantic novelty modulates neural responses to visual change across the human brain

Our continuous visual experience in daily life is dominated by change. Previous research has focused on visual change due to stimulus motion, eye movements or unfolding events, but not their combined impact across the brain, or their interactions with semantic novelty. We investigate the neural responses to these sources of novelty during film viewing. We analyzed intracranial recordings in humans across 6328 electrodes from 23 individuals. Responses associated with saccades and film cuts were dominant across the entire brain. Film cuts at semantic event boundaries were particularly effective in the temporal and medial temporal lobe. Saccades to visual targets with high visual novelty were also associated with strong neural responses. Specific locations in higher-order association areas showed selectivity to either high or low-novelty saccades. We conclude that neural activity associated with film cuts and eye movements is widespread across the brain and is modulated by semantic novelty.

Modulation of Spectral Representation and Connectivity Patterns in Response to Visual Narrative in the Human Brain

The regional brain networks and the underlying neurophysiological mechanisms subserving the cognition of visual narrative in humans have largely been studied with non-invasive brain recording. In this study, we specifically investigated how regional and cross-regional cortical activities support visual narrative interpretation using intracranial stereotactic electroencephalograms recordings from thirteen human subjects (6 females, and 7 males). Widely distributed recording sites across the brain were sampled while subjects were explicitly instructed to observe images from fables presented in “sequential” order, and a set of images drawn from multiple fables presented in “scrambled” order. Broadband activity mainly within the frontal and temporal lobes were found to encode if a presented image is part of a visual narrative (sequential) or random image set (scrambled). Moreover, the temporal lobe exhibits strong activation in response to visual narratives while the frontal lobe is more engaged when contextually novel stimuli are presented. We also investigated the dynamics of interregional interactions between visual narratives and contextually novel series of images. Interestingly, the interregional connectivity is also altered between sequential and scrambled sequences. Together, these results suggest that both changes in regional neuronal activity and cross-regional interactions subserve visual narrative and contextual novelty processing.

Neurons detect cognitive boundaries to structure episodic memories in humans

While experience is continuous, memories are organized as discrete events. Cognitive boundaries are thought to segment experience and structure memory, but how this process is implemented remains unclear. We recorded the activity of single neurons in the human medial temporal lobe during the formation and retrieval of memories with complex narratives. Here we show that neurons responded to abstract cognitive boundaries between different episodes. Boundary-induced neural state changes during encoding predicted subsequent recognition accuracy but impaired event order memory, mirroring a fundamental behavioral tradeoff between content and time memory. Furthermore, the neural state following boundaries was reinstated during both successful retrieval and false memories. These findings reveal a neuronal substrate for detecting cognitive boundaries that transform experience into mnemonic episodes and structure mental time travel during retrieval.

Growing evidence for separate neural mechanisms for attention and consciousness

Our conscious experience of the world seems to go in lockstep with our attentional focus: We tend to see, hear, taste, and feel what we attend to, and vice versa. This tight coupling between attention and consciousness has given rise to the idea that these two phenomena are indivisible. In the late 1950s, the honoree of this special issue, Charles Eriksen, was among a small group of early pioneers that sought to investigate whether a transient increase in overall level of attention (alertness) in response to a noxious stimulus can be decoupled from conscious perception using experimental techniques. Recent years saw a similar debate regarding whether attention and consciousness are two dissociable processes. Initial evidence that attention and consciousness are two separate processes primarily rested on behavioral data. However, the past couple of years witnessed an explosion of studies aimed at testing this conjecture using neuroscientific techniques. Here we provide an overview of these and related empirical studies on the distinction between the neuronal correlates of attention and consciousness, and detail how advancements in theory and technology can bring about a more detailed understanding of the two. We argue that the most promising approach will combine ever-evolving neurophysiological and interventionist tools with quantitative, empirically testable theories of consciousness that are grounded in a mathematically formalized understanding of phenomenology.

Neurological Soft Signs Are Associated With Altered Cerebellar-Cerebral Functional Connectivity in Schizophrenia

Cerebellar dysfunction is associated with neurological soft signs (NSS), which is a promising endophenotype for schizophrenia spectrum disorders. However, the relationship between cerebellar-cerebral resting-state functional connectivity (rsFC) and NSS is largely unexplored. Moreover, both NSS and cerebellar-cerebral rsFC have been found to be correlated with negative symptoms of schizophrenia. Here, we investigated the correlations between NSS and cerebellar-cerebral rsFC, explored their relationship with negative symptoms in a main dataset, and validated the significant findings in a replication dataset. Both datasets comprised schizophrenia patients and healthy controls. In schizophrenia patients, we found positive correlations between NSS and rsFC of the cerebellum with the inferior frontal gyrus and the precuneus, and negative correlations between NSS and rsFC of the cerebellum with the inferior temporal gyrus. In healthy controls, NSS scores were positively correlated with rsFC of the cerebellum with the superior frontal gyrus and negatively correlated with rsFC between the cerebellum and the middle occipital gyrus. Cerebellar-prefrontal rsFC was also positively correlated with negative symptoms in schizophrenia patients. These findings were validated in the replication dataset. Our results suggest that the uncoupling of rsFC between the cerebellum and the cerebral cortex may underlie the expression of NSS in schizophrenia. NSS-related cerebellar-prefrontal rsFC may be a potential neural pathway for possible neural modulation to alleviate negative symptoms.

Encoding schemes in somatosensation: From micro- to meta-topography

Encoding schemes are systematic large-scale arrangements that convert incoming sensory information into a format required for further information processing. The increased spatial resolution of brain images obtained with ultra-high field magnetic resonance imaging at 7 T (7T-MRI) and above increases the granularity and precision of processing units that mediate the link between neuronal encoding and functional readouts. Here, these new developments are reviewed with a focus on human tactile encoding schemes derived from small-scale processing units (in the order of 0.5-5 mm) that are relevant for theoretical and practical concepts of somatosensory encoding and cortical plasticity. Precisely, we review recent approaches to characterize meso-scale maps, layer units, and cortical fields in the sensorimotor cortex of the living human brain and discuss their impact on theories of perception, motor control, topographic encoding, and cortical plasticity. Finally, we discuss concepts on the integration of small-scale processing units into functional networks that span multiple topographic maps and multiple cortical areas. Novel research areas are highlighted that may help to bridge the gap between cortical microstructure and meta-topographic models on brain anatomy and function.

Human stereoEEG recordings reveal network dynamics of decision-making in a rule-switching task

The processing steps that lead up to a decision, i.e., the transformation of sensory evidence into motor output, are not fully understood. Here, we combine stereoEEG recordings from the human cortex, with single-lead and time-resolved decoding, using a wide range of temporal frequencies, to characterize decision processing during a rule-switching task. Our data reveal the contribution of rostral inferior parietal lobule (IPL) regions, in particular PFt, and the parietal opercular regions in decision processing and demonstrate that the network representing the decision is common to both task rules. We reconstruct the sequence in which regions engage in decision processing on single trials, thereby providing a detailed picture of the network dynamics involved in decision-making. The reconstructed timeline suggests that the supramarginal gyrus in IPL links decision regions in prefrontal cortex with premotor regions, where the motor plan for the response is elaborated.

Using Coherence-based spectro-spatial filters for stimulus features prediction from electro-corticographic recordings

The traditional approach in neuroscience relies on encoding models where brain responses are related to different stimuli in order to establish dependencies. In decoding tasks, on the contrary, brain responses are used to predict the stimuli, and traditionally, the signals are assumed stationary within trials, which is rarely the case for natural stimuli. We hypothesize that a decoding model assuming each experimental trial as a realization of a random process more likely reflects the statistical properties of the undergoing process compared to the assumption of stationarity. Here, we propose a Coherence-based spectro-spatial filter that allows for reconstructing stimulus features from brain signal's features. The proposed method extracts common patterns between features of the brain signals and the stimuli that produced them. These patterns, originating from different recording electrodes are combined, forming a spatial filter that produces a unified prediction of the presented stimulus. This approach takes into account frequency, phase, and spatial distribution of brain features, hence avoiding the need to predefine specific frequency bands of interest or phase relationships between stimulus and brain responses manually. Furthermore, the model does not require the tuning of hyper-parameters, reducing significantly the computational load attached to it. Using three different cognitive tasks (motor movements, speech perception, and speech production), we show that the proposed method consistently improves stimulus feature predictions in terms of correlation (group averages of 0.74 for motor movements, 0.84 for speech perception, and 0.74 for speech production) in comparison with other methods based on regularized multivariate regression, probabilistic graphical models and artificial neural networks. Furthermore, the model parameters revealed those anatomical regions and spectral components that were discriminant in the different cognitive tasks. This novel method does not only provide a useful tool to address fundamental neuroscience questions, but could also be applied to neuroprosthetics.

Individualized psychiatric imaging based on inter-subject neural synchronization in movie watching

The individual heterogeneity is a challenge to the prosperous promises of cutting-edge neuroimaging techniques for better diagnosis and early detection of psychiatric disorders. Individuals with similar clinical manifestations may result from very different pathophysiology. Conventional approaches based on comparing group-averages provide insufficient information to support the individualized diagnosis. Here we present an individualized imaging methodology that combines naturalistic imaging and the normative model. This paradigm adopts video clips with rich cognitive, social, and emotional contents to evoke synchronized brain dynamics of healthy participants and builds a spatiotemporal response norm. By comparing individual brain responses with the response norm, we could recognize patients using machine learning techniques. We applied this methodology to recognize first-episode drug-naïve schizophrenia patients in a dataset containing 72 patients and 54 healthy controls. Some segments of the video evoked more synchronized brain activity in the healthy controls than in the schizophrenia patients. We built a spatiotemporal response norm by averaging the brain responses of the healthy controls in a training set, and trained a classifier to recognize patients based on the differences between individual brain responses and the norm. The performance of the classifier was then evaluated using an independent test set. The mean accuracies from a 5-fold cross-validation were 0.71-0.78 depending on the parameters such as the number of features and the width of the sliding windows. These findings reflected the potential of this methodology towards a clinical tool for individualized diagnosis.