Cortical response to naturalistic stimuli is largely predictable with deep neural networks

Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision

Objective. Voxel-wise visual encoding models based on convolutional neural networks (CNNs) have emerged as one of the prominent predictive tools of human brain activity via functional magnetic resonance imaging signals. While CNN-based models imitate the hierarchical structure of the human visual cortex to generate explainable features in response to natural visual stimuli, there is still a need for a brain-inspired model to predict brain responses accurately based on biomedical data.Approach. To bridge this gap, we propose a response prediction module called the Structurally Constrained Multi-Output (SCMO) module to include homologous correlations that arise between a group of voxels in a cortical region and predict more accurate responses.Main results. This module employs all the responses across a visual area to predict individual voxel-wise BOLD responses and therefore accounts for the population activity and collective behavior of voxels. Such a module can determine the relationships within each visual region by creating a structure matrix that represents the underlying voxel-to-voxel interactions. Moreover, since each response module in visual encoding tasks relies on the image features, we conducted experiments using two different feature extraction modules to assess the predictive performance of our proposed module. Specifically, we employed a recurrent CNN that integrates both feedforward and recurrent interactions, as well as the popular AlexNet model that utilizes feedforward connections.Significance.We demonstrate that the proposed framework provides a reliable predictive ability to generate brain responses across multiple areas, outperforming benchmark models in terms of stability and coherency of features.

Dissonant music engages early visual processing

The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character's intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Cognitive neuroscience aims to develop computational models that can accurately predict and explain neural responses to sensory inputs in the cortex. Recent studies attempt to leverage the representation power of deep neural networks (DNNs) to predict the brain response and suggest a correspondence between artificial and biological neural networks in their feature representations. However, typical voxel-wise encoding models tend to rely on specific networks designed for computer vision tasks, leading to suboptimal brain-wide correspondence during cognitive tasks. To address this challenge, this work proposes a novel approach that upgrades voxel-wise encoding models through multi-level integration of features from DNNs and information from brain networks. Our approach combines DNN feature-level ensemble learning and brain atlas-level model integration, resulting in significant improvements in predicting whole-brain neural activity during naturalistic video perception. Furthermore, this multi-level integration framework enables a deeper understanding of the brain's neural representation mechanism, accurately predicting the neural response to complex visual concepts. We demonstrate that neural encoding models can be optimized by leveraging a framework that integrates both data-driven approaches and theoretical insights into the functional structure of the cortical networks.

A hierarchy of processing complexity and timescales for natural sounds in human auditory cortex

Efficient behavior is supported by humans' ability to rapidly recognize acoustically distinct sounds as members of a common category. Within auditory cortex, there are critical unanswered questions regarding the organization and dynamics of sound categorization. Here, we performed intracerebral recordings in the context of epilepsy surgery as 20 patient-participants listened to natural sounds. We built encoding models to predict neural responses using features of these sounds extracted from different layers within a sound-categorization deep neural network (DNN). This approach yielded highly accurate models of neural responses throughout auditory cortex. The complexity of a cortical site's representation (measured by the depth of the DNN layer that produced the best model) was closely related to its anatomical location, with shallow, middle, and deep layers of the DNN associated with core (primary auditory cortex), lateral belt, and parabelt regions, respectively. Smoothly varying gradients of representational complexity also existed within these regions, with complexity increasing along a posteromedial-to-anterolateral direction in core and lateral belt, and along posterior-to-anterior and dorsal-to-ventral dimensions in parabelt. When we estimated the time window over which each recording site integrates information, we found shorter integration windows in core relative to lateral belt and parabelt. Lastly, we found a relationship between the length of the integration window and the complexity of information processing within core (but not lateral belt or parabelt). These findings suggest hierarchies of timescales and processing complexity, and their interrelationship, represent a functional organizational principle of the auditory stream that underlies our perception of complex, abstract auditory information.

Exploring Types of Photonic Neural Networks for Imaging and Computing-A Review

Photonic neural networks (PNNs), utilizing light-based technologies, show immense potential in artificial intelligence (AI) and computing. Compared to traditional electronic neural networks, they offer faster processing speeds, lower energy usage, and improved parallelism. Leveraging light's properties for information processing could revolutionize diverse applications, including complex calculations and advanced machine learning (ML). Furthermore, these networks could address scalability and efficiency challenges in large-scale AI systems, potentially reshaping the future of computing and AI research. In this comprehensive review, we provide current, cutting-edge insights into diverse types of PNNs crafted for both imaging and computing purposes. Additionally, we delve into the intricate challenges they encounter during implementation, while also illuminating the promising perspectives they introduce to the field.

Divergences in color perception between deep neural networks and humans

Deep neural networks (DNNs) are increasingly proposed as models of human vision, bolstered by their impressive performance on image classification and object recognition tasks. Yet, the extent to which DNNs capture fundamental aspects of human vision such as color perception remains unclear. Here, we develop novel experiments for evaluating the perceptual coherence of color embeddings in DNNs, and we assess how well these algorithms predict human color similarity judgments collected via an online survey. We find that state-of-the-art DNN architectures - including convolutional neural networks and vision transformers - provide color similarity judgments that strikingly diverge from human color judgments of (i) images with controlled color properties, (ii) images generated from online searches, and (iii) real-world images from the canonical CIFAR-10 dataset. We compare DNN performance against an interpretable and cognitively plausible model of color perception based on wavelet decomposition, inspired by foundational theories in computational neuroscience. While one deep learning model - a convolutional DNN trained on a style transfer task - captures some aspects of human color perception, our wavelet algorithm provides more coherent color embeddings that better predict human color judgments compared to all DNNs we examine. These results hold when altering the high-level visual task used to train similar DNN architectures (e.g., image classification versus image segmentation), as well as when examining the color embeddings of different layers in a given DNN architecture. These findings break new ground in the effort to analyze the perceptual representations of machine learning algorithms and to improve their ability to serve as cognitively plausible models of human vision. Implications for machine learning, human perception, and embodied cognition are discussed.

Neural encoding with unsupervised spiking convolutional neural network

Accurately predicting the brain responses to various stimuli poses a significant challenge in neuroscience. Despite recent breakthroughs in neural encoding using convolutional neural networks (CNNs) in fMRI studies, there remain critical gaps between the computational rules of traditional artificial neurons and real biological neurons. To address this issue, a spiking CNN (SCNN)-based framework is presented in this study to achieve neural encoding in a more biologically plausible manner. The framework utilizes unsupervised SCNN to extract visual features of image stimuli and employs a receptive field-based regression algorithm to predict fMRI responses from the SCNN features. Experimental results on handwritten characters, handwritten digits and natural images demonstrate that the proposed approach can achieve remarkably good encoding performance and can be utilized for "brain reading" tasks such as image reconstruction and identification. This work suggests that SNN can serve as a promising tool for neural encoding.

A large-scale fMRI dataset for the visual processing of naturalistic scenes

One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.

The default network dominates neural responses to evolving movie stories

Neuroscientific studies exploring real-world dynamic perception often overlook the influence of continuous changes in narrative content. In our research, we utilize machine learning tools for natural language processing to examine the relationship between movie narratives and neural responses. By analyzing over 50,000 brain images of participants watching Forrest Gump from the studyforrest dataset, we find distinct brain states that capture unique semantic aspects of the unfolding story. The default network, associated with semantic information integration, is the most engaged during movie watching. Furthermore, we identify two mechanisms that underlie how the default network liaises with the amygdala and hippocampus. Our findings demonstrate effective approaches to understanding neural processes in everyday situations and their relation to conscious awareness.

Effects of remifentanil on brain responses to noxious stimuli during deep propofol sedation

BACKGROUND

The safety of anaesthesia has improved as a result of better control of anaesthetic depth. However, conventional monitoring does not inform on the nature of nociceptive processes during unconsciousness. A means of inferring the quality of potentially painful experiences could derive from analysis of brain activity using neuroimaging. We have evaluated the dose effects of remifentanil on brain response to noxious stimuli during deep sedation and spontaneous breathing.

METHODS

Optimal data were obtained in 26 healthy subjects. Pressure stimulation that proved to be moderately painful before the experiment was applied to the thumbnail. Functional MRI was acquired in 4-min periods at low (0.5 ng ml^-1), medium (1 ng ml^-1), and high (1.5 ng ml^-1) target plasma concentrations of remifentanil at a stable background infusion of propofol adjusted to induce a state of light unconsciousness.

RESULTS

At low remifentanil doses, we observed partial activation in brain areas processing sensory-discriminative and emotional-affective aspects of pain. At medium doses, relevant changes were identified in structures highly sensitive to general brain arousal, including the brainstem, cerebellum, thalamus, auditory and visual cortices, and the frontal lobe. At high doses, no significant activation was observed.

CONCLUSIONS

The response to moderately intense focal pressure in pain-related brain networks is effectively eliminated with safe remifentanil doses. However, the safety margin in deep sedation-analgesia would be narrowed in minimising not only nociceptive responses, but also arousal-related biological stress.

A rubric for human-like agents and NeuroAI

Researchers across cognitive, neuro- and computer sciences increasingly reference 'human-like' artificial intelligence and 'neuroAI'. However, the scope and use of the terms are often inconsistent. Contributed research ranges widely from mimicking behaviour, to testing machine learning methods as neurally plausible hypotheses at the cellular or functional levels, or solving engineering problems. However, it cannot be assumed nor expected that progress on one of these three goals will automatically translate to progress in others. Here, a simple rubric is proposed to clarify the scope of individual contributions, grounded in their commitments to human-like behaviour, neural plausibility or benchmark/engineering/computer science goals. This is clarified using examples of weak and strong neuroAI and human-like agents, and discussing the generative, corroborate and corrective ways in which the three dimensions interact with one another. The author maintains that future progress in artificial intelligence will need strong interactions across the disciplines, with iterative feedback loops and meticulous validity tests-leading to both known and yet-unknown advances that may span decades to come. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Personalized visual encoding model construction with small data

Quantifying population heterogeneity in brain stimuli-response mapping may allow insight into variability in bottom-up neural systems that can in turn be related to individual's behavior or pathological state. Encoding models that predict brain responses to stimuli are one way to capture this relationship. However, they generally need a large amount of fMRI data to achieve optimal accuracy. Here, we propose an ensemble approach to create encoding models for novel individuals with relatively little data by modeling each subject's predicted response vector as a linear combination of the other subjects' predicted response vectors. We show that these ensemble encoding models trained with hundreds of image-response pairs, achieve accuracy not different from models trained on 20,000 image-response pairs. Importantly, the ensemble encoding models preserve patterns of inter-individual differences in the image-response relationship. We also show the proposed approach is robust against domain shift by validating on data with a different scanner and experimental setup. Additionally, we show that the ensemble encoding models are able to discover the inter-individual differences in various face areas' responses to images of animal vs human faces using a recently developed NeuroGen framework. Our approach shows the potential to use existing densely-sampled data, i.e. large amounts of data collected from a single individual, to efficiently create accurate, personalized encoding models and, subsequently, personalized optimal synthetic images for new individuals scanned under different experimental conditions.

A large and rich EEG dataset for modeling human visual object recognition

The human brain achieves visual object recognition through multiple stages of linear and nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models' prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.

Behavioral Studies Using Large-Scale Brain Networks – Methods and Validations

Mapping human behaviors to brain activity has become a key focus in modern cognitive neuroscience. As methods such as functional MRI (fMRI) advance cognitive scientists show an increasing interest in investigating neural activity in terms of functional connectivity and brain networks, rather than activation in a single brain region. Due to the noisy nature of neural activity, determining how behaviors are associated with specific neural signals is not well-established. Previous research has suggested graph theory techniques as a solution. Graph theory provides an opportunity to interpret human behaviors in terms of the topological organization of brain network architecture. Graph theory-based approaches, however, only scratch the surface of what neural connections relate to human behavior. Recently, the development of data-driven methods, e.g., machine learning and deep learning approaches, provide a new perspective to study the relationship between brain networks and human behaviors across the whole brain, expanding upon past literatures. In this review, we sought to revisit these data-driven approaches to facilitate our understanding of neural mechanisms and build models of human behaviors. We start with the popular graph theory approach and then discuss other data-driven approaches such as connectome-based predictive modeling, multivariate pattern analysis, network dynamic modeling, and deep learning techniques that quantify meaningful networks and connectivity related to cognition and behaviors. Importantly, for each topic, we discuss the pros and cons of the methods in addition to providing examples using our own data for each technique to describe how these methods can be applied to real-world neuroimaging data.

Naturalistic stimuli in touch research

Neural mechanisms of touch are typically studied in laboratory settings using robotic or other types of well-controlled devices. Such stimuli are very different from highly complex naturalistic human-to-human touch interactions. The lack of scientifically useful naturalistic stimuli hampers progress, particularly in social touch research. Vision science, on the other hand, has benefitted from inventions such as virtual reality systems that have provided researchers with precision control of naturalistic stimuli. In the field of touch research, producing and manipulating stimuli is particularly challenging due to the complexity of skin mechanics. Here, we review the history of touch neuroscience focusing on the contrast between strictly controlled and naturalistic stimuli, and compare the field to vision science. We discuss new methods that may overcome obstacles with precision-controlled tactile stimuli, and recent successes in naturalistic texture production. In social touch research, precise tracking and measurement of naturalistic human-to-human touch interactions offer exciting new possibilities.

Movie Events Detecting Reveals Inter-Subject Synchrony Difference of Functional Brain Activity in Autism Spectrum Disorder

Recently, movie-watching fMRI has been recognized as a novel method to explore brain working patterns. Previous researchers correlated natural stimuli with brain responses to explore brain functional specialization by “reverse correlation” methods, which were based on within-group analysis. However, what external stimuli drove significantly different brain responses in two groups of different subjects were still unknown. To address this, sliding time windows technique combined with inter-Subject functional correlation (ISFC) was proposed to detect movie events with significant group differences between autism spectrum disorder (ASD) and typical development (TD) subjects. Then, using inter-Subject correlation (ISC) and ISFC analysis, we found that in three movie events involving character emotions, the ASD group showed significantly lower ISC in the middle temporal gyrus, temporal pole, cerebellum, caudate, precuneus, and showed decreased functional connectivity between large scale networks than that in TD. Under the movie event focusing on objects and scenes shot, the dorsal and ventral attentional networks of ASD had a strong synchronous response. Meanwhile, ASD also displayed increased functional connectivity between the frontoparietal network (FPN) and dorsal attention network (DAN), FPN, and sensorimotor network (SMN) than TD. ASD has its own unique synchronous response rather than being “unresponsive” in natural movie-watching. Our findings provide a new method and valuable insight for exploring the inconsistency of the brain “tick collectively” to same natural stimuli. This analytic approach has the potential to explore pathological mechanisms and promote training methods of ASD.

Mapping the neural systems driving breathing at the transition to unconsciousness

After falling asleep, the brain needs to detach from waking activity and reorganize into a functionally distinct state. A functional MRI (fMRI) study has recently revealed that the transition to unconsciousness induced by propofol involves a global decline of brain activity followed by a transient reduction in cortico-subcortical coupling. We have analyzed the relationships between transitional brain activity and breathing changes as one example of a vital function that needs the brain to readapt. Thirty healthy participants were originally examined. The analysis involved the correlation between breathing and fMRI signal upon loss of consciousness. We proposed that a decrease in ventilation would be coupled to the initial decline in fMRI signal in brain areas relevant for modulating breathing in the awake state, and that the subsequent recovery would be coupled to fMRI signal in structures relevant for controlling breathing during the unconscious state. Results showed that a slight reduction in breathing from wakefulness to unconsciousness was distinctively associated with decreased activity in brain systems underlying different aspects of consciousness including the prefrontal cortex, the default mode network and somatosensory areas. Breathing recovery was distinctively coupled to activity in deep brain structures controlling basic behaviors such as the hypothalamus and amygdala. Activity in the brainstem, cerebellum and hippocampus was associated with breathing variations in both states. Therefore, our brain maps illustrate potential drives to breathe, unique to wakefulness, in the form of brain systems underlying cognitive awareness, self-awareness and sensory awareness, and to unconsciousness involving structures controlling instinctive and homeostatic behaviors.