GAUDIE: Development, validation, and exploration of a naturalistic German AUDItory Emotional database

Since thoroughly validated naturalistic affective German speech stimulus databases are rare, we present here a novel validated database of speech sequences assembled with the purpose of emotion induction. The database comprises 37 audio speech sequences with a total duration of 92 minutes for the induction of positive, neutral, and negative emotion: comedian shows intending to elicit humorous and amusing feelings, weather forecasts, and arguments between couples and relatives from movies or television series. Multiple continuous and discrete ratings are used to validate the database to capture the time course and variabilities of valence and arousal. We analyse and quantify how well the audio sequences fulfil quality criteria of differentiation, salience/strength, and generalizability across participants. Hence, we provide a validated speech database of naturalistic scenarios suitable to investigate emotion processing and its time course with German-speaking participants. Information on using the stimulus database for research purposes can be found at the OSF project repository GAUDIE: https://osf.io/xyr6j/ .

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Cognitive neuroscience aims to develop computational models that can accurately predict and explain neural responses to sensory inputs in the cortex. Recent studies attempt to leverage the representation power of deep neural networks (DNNs) to predict the brain response and suggest a correspondence between artificial and biological neural networks in their feature representations. However, typical voxel-wise encoding models tend to rely on specific networks designed for computer vision tasks, leading to suboptimal brain-wide correspondence during cognitive tasks. To address this challenge, this work proposes a novel approach that upgrades voxel-wise encoding models through multi-level integration of features from DNNs and information from brain networks. Our approach combines DNN feature-level ensemble learning and brain atlas-level model integration, resulting in significant improvements in predicting whole-brain neural activity during naturalistic video perception. Furthermore, this multi-level integration framework enables a deeper understanding of the brain's neural representation mechanism, accurately predicting the neural response to complex visual concepts. We demonstrate that neural encoding models can be optimized by leveraging a framework that integrates both data-driven approaches and theoretical insights into the functional structure of the cortical networks.

The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognition

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

Machine-learning as a validated tool to characterize individual differences in free recall of naturalistic events

The use of naturalistic stimuli, such as narrative movies, is gaining popularity in many fields, characterizing memory, affect, and decision-making. Narrative recall paradigms are often used to capture the complexity and richness of memory for naturalistic events. However, scoring narrative recalls is time-consuming and prone to human biases. Here, we show the validity and reliability of using a natural language processing tool, the Universal Sentence Encoder (USE), to automatically score narrative recalls. We compared the reliability in scoring made between two independent raters (i.e., hand scored) and between our automated algorithm and individual raters (i.e., automated) on trial-unique video clips of magic tricks. Study 1 showed that our automated segmentation approaches yielded high reliability and reflected measures yielded by hand scoring. Study 1 further showed that the results using USE outperformed another popular natural language processing tool, GloVe. In Study 2, we tested whether our automated approach remained valid when testing individuals varying on clinically relevant dimensions that influence episodic memory, age, and anxiety. We found that our automated approach was equally reliable across both age groups and anxiety groups, which shows the efficacy of our approach to assess narrative recall in large-scale individual difference analysis. In sum, these findings suggested that machine learning approach implementing USE is a promising tool for scoring large-scale narrative recalls and perform individual difference analysis for research using naturalistic stimuli.

A functional neuroimaging dataset acquired during naturalistic movie watching and narrated recall of a series of short cinematic films

Whole-brain functional magnetic resonance imaging (fMRI) data from twenty healthy human participants were collected during naturalistic movie watching and free spoken recall tasks. Participants watched ten short (approximately 2 - 8 min) audiovisual movies and then verbally described what they remembered about the movies in their own words. Participants' verbal responses were audio recorded using an MR-compatible microphone. The audio recordings were transcribed and timestamped by independent coders. The neural and behavioral data were organized in the Brain Imaging Data Structure (BIDS) format and made publicly available via OpenNeuro.org. The dataset can be used to explore the neural bases of naturalistic memory and other cognitive functions including but not limited to visual/auditory perception, language comprehension, and speech generation.

Dynamic and stationary brain connectivity during movie watching as revealed by functional MRI

Spatially remote brain regions show synchronized activity as typically revealed by correlated functional MRI (fMRI) signals. An emerging line of research has focused on the temporal fluctuations of connectivity; however, its relationships with stationary connectivity have not been clearly illustrated. We examined dynamic and stationary connectivity when the participants watched four different movie clips. We calculated point-by-point multiplication between two regional time series to estimate the time-resolved dynamic connectivity, and estimated the inter-individual consistency of the dynamic connectivity time series. Widespread consistent dynamic connectivity was observed for each movie clip, which also showed differences between the clips. For example, a cartoon movie clip, Wall-E, showed more consistent of dynamic connectivity with the posterior cingulate cortex and supramarginal gyrus, while a court drama clip, A Few Good Men, showed more consistent of dynamic connectivity with the auditory cortex and temporoparietal junction, which might suggest the involvement of specific brain processing for different movie contents. In contrast, the stationary connectivity as measured by the correlations between regional time series was highly similar among the movie clips, and showed fewer statistically significant differences. The patterns of consistent dynamic connectivity could be used to classify different movie clips with higher accuracy than the stationary connectivity and regional activity. These results support the functional significance of dynamic connectivity in reflecting functional brain changes, which could provide more functionally relevant information than stationary connectivity.

Inferring Brain State Dynamics Underlying Naturalistic Stimuli Evoked Emotion Changes With dHA-HMM

The brain functional mechanisms underlying emotional changes have been primarily studied based on the traditional task design with discrete and simple stimuli. However, the brain state transitions when exposed to continuous and naturalistic stimuli with rich affection variations remain poorly understood. This study proposes a dynamic hyperalignment algorithm (dHA) to functionally align the inter-subject neural activity. The hidden Markov model (HMM) was used to study how the brain dynamics responds to emotion during long-time movie-viewing activity. The results showed that dHA significantly improved inter-subject consistency and allowed more consistent temporal HMM states across participants. Afterward, grouping the emotions in a clustering dendrogram revealed a hierarchical grouping of the HMM states. Further emotional sensitivity and specificity analyses of ordered states revealed the most significant differences in happiness and sadness. We then compared the activation map in HMM states during happiness and sadness and found significant differences in the whole brain, but strong activation was observed during both in the superior temporal gyrus, which is related to the early process of emotional prosody processing. A comparison of the inter-network functional connections indicates unique functional connections of the memory retrieval and cognitive network with the cerebellum network during happiness. Moreover, the persistent bilateral connections among salience, cognitive, and sensorimotor networks during sadness may reflect the interaction between high-level cognitive networks and low-level sensory networks. The main results were verified by the second session of the dataset. All these findings enrich our understanding of the brain states related to emotional variation during naturalistic stimuli.

Act natural: Functional connectivity from naturalistic stimuli fMRI outperforms resting-state in predicting brain activity

The search for an 'ideal' approach to investigate the functional connections in the human brain is an ongoing challenge for the neuroscience community. While resting-state functional magnetic resonance imaging (fMRI) has been widely used to study individual functional connectivity patterns, recent work has highlighted the benefits of collecting functional connectivity data while participants are exposed to naturalistic stimuli, such as watching a movie or listening to a story. For example, functional connectivity data collected during movie-watching were shown to predict cognitive and emotional scores more accurately than resting-state-derived functional connectivity. We have previously reported a tight link between resting-state functional connectivity and task-derived neural activity, such that the former successfully predicts the latter. In the current work we use data from the Human Connectome Project to demonstrate that naturalistic-stimulus-derived functional connectivity predicts task-induced brain activation maps more accurately than resting-state-derived functional connectivity. We then show that activation maps predicted using naturalistic stimuli are better predictors of individual intelligence scores than activation maps predicted using resting-state. We additionally examine the influence of naturalistic-stimulus type on prediction accuracy. Our findings emphasize the potential of naturalistic stimuli as a promising alternative to resting-state fMRI for connectome-based predictive modelling of individual brain activity and cognitive traits.

The role of low-frequency oscillations in three-dimensional perception with depth cues in virtual reality

Currently, vision-related neuroscience studies are undergoing a trend from simplified image stimuli toward more naturalistic stimuli. Virtual reality (VR), as an emerging technology for visual immersion, provides more depth cues for three-dimensional (3D) presentation than two-dimensional (2D) image. It is still unclear whether the depth cues used to create 3D visual perception modulate specific cortical activation. Here, we constructed two visual stimuli presented by stereoscopic vision in VR and graphical projection with 2D image, respectively, and used electroencephalography to examine neural oscillations and their functional connectivity during 3D perception. We find that neural oscillations are specific to delta and theta bands in stereoscopic vision and the functional connectivity in the both bands increase in cortical areas related to visual pathways. These findings indicate that low-frequency oscillations play an important role in 3D perception with depth cues.

Discovering hidden brain network responses to naturalistic stimuli via tensor component analysis of multi-subject fMRI data

The study of brain network interactions during naturalistic stimuli facilitates a deeper understanding of human brain function. To estimate large-scale brain networks evoked with naturalistic stimuli, a tensor component analysis (TCA) based framework was used to characterize shared spatio-temporal patterns across subjects in a purely data-driven manner. In this framework, a third-order tensor is constructed from the timeseries extracted from all brain regions from a given parcellation, for all participants, with modes of the tensor corresponding to spatial distribution, time series and participants. TCA then reveals spatially and temporally shared components, i.e., evoked networks with the naturalistic stimuli, their time courses of activity and subject loadings of each component. To enhance the reproducibility of the estimation with the adaptive TCA algorithm, a novel spectral clustering method, tensor spectral clustering, was proposed and applied to evaluate the stability of the TCA algorithm. We demonstrated the effectiveness of the proposed framework via simulations and real fMRI data collected during a motor task with a traditional fMRI study design. We also applied the proposed framework to fMRI data collected during passive movie watching to illustrate how reproducible brain networks are evoked by naturalistic movie viewing.

EmoCodes: a Standardized Coding System for Socio-emotional Content in Complex Video Stimuli

Social information processing is vital for inferring emotional states in others, yet affective neuroscience has only begun to scratch the surface of how we represent emotional information in the brain. Most previous affective neuroscience work has used isolated stimuli such as static images of affective faces or scenes to probe affective processing. While this work has provided rich insight to the initial stages of emotion processing (encoding cues), activation to isolated stimuli provides limited insight into later phases of emotion processing such as interpretation of cues or interactions between cues and established cognitive schemas. Recent work has highlighted the potential value of using complex video stimuli to probe socio-emotional processing, highlighting the need to develop standardized video coding schemas as this exciting field expands. Toward that end, we present a standardized and open-source coding system for complex videos, two fully coded videos, and a video and code processing Python library. The EmoCodes manual coding system provides an externally validated and replicable system for coding complex cartoon stimuli, with future plans to validate the system for other video types. The emocodes Python library provides automated tools for extracting low-level features from video files as well as tools for summarizing and analyzing the manual codes for suitability of use in neuroimaging analysis. Materials can be freely accessed at https://emocodes.org/. These tools represent an important step toward replicable and standardized study of socio-emotional processing using complex video stimuli.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42761-021-00100-7.

Hybrid hyperalignment: A single high-dimensional model of shared information embedded in cortical patterns of response and functional connectivity

Shared information content is represented across brains in idiosyncratic functional topographies. Hyperalignment addresses these idiosyncrasies by using neural responses to project individuals’ brain data into a common model space while maintaining the geometric relationships between distinct patterns of activity or connectivity. The dimensions of this common model capture functional profiles that are shared across individuals such as cortical response profiles collected during a common time-locked stimulus presentation (e.g. movie viewing) or functional connectivity profiles. Hyperalignment can use either response-based or connectivity-based input data to derive transformations that project individuals’ neural data from anatomical space into the common model space. Previously, only response or connectivity profiles were used in the derivation of these transformations. In this study, we developed a new hyperalignment algorithm, hybrid hyperalignment, that derives transformations based on both response-based and connectivity-based information. We used three different movie-viewing fMRI datasets to test the performance of our new algorithm. Hybrid hyperalignment derives a single common model space that aligns response-based information as well as or better than response hyperalignment while simultaneously aligning connectivity-based information better than connectivity hyperalignment. These results suggest that a single common information space can encode both shared cortical response and functional connectivity profiles across individuals.

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

Individualized assessment of residual cognition in patients with disorders of consciousness

•

Single-trial electrical recordings index higher-order cognitive processing of movie stimuli.

•

Common patterns of neural activity associated with the brain’s executive network.

•

The time course of common neural activity correlates with ratings of suspense.

•

38% of non-responsive patients correlate with controls during movie-watching tasks.

•

Novel bedside assessment of complex cognition in behaviourally non-responsive patients.

Patients diagnosed with disorders of consciousness show minimal or inconsistent behavioural evidence of conscious awareness. However, using functional neuroimaging, recent research in clinical neuroscience has identified a subpopulation of these patients who reliably produce neural markers indicative of awareness. In this study, we recorded electroencephalograms during a response-free movie task to assess narrative processing in patients with disorders of consciousness. Thirteen patients diagnosed with a disorder of consciousness and 28 healthy controls participated in this study. We designed a movie-watching/listening paradigm involving two suspenseful movie clips, one audiovisual and one audio-only, and used electroencephalography to extract patterns of brain activity that were maximally correlated between subjects. These activity patterns served as electrophysiological indices of narrative processing, which were compared to the neural responses of patients during the same movies. Our analysis revealed two patterns of neural activity, one for each movie condition, that were significantly and reliably correlated between healthy participants. Of the twelve patients who watched the audiovisual movie, 25% produced a pattern of activity that was significantly correlated with the healthy group, while of the ten who listened to the audio narrative, 30% produced electrophysiological patterns similar to controls (one patient responded appropriately to both). The method presented here allows for rapid bedside assessment of higher-order cognitive processing in patients with disorders of consciousness. By leveraging the common neural response to movie stimuli, we were able to identify comparable patterns of brain activity in individual, behaviourally non-responsive patients, reflecting a capacity for narrative processing.

Movies and narratives as naturalistic stimuli in neuroimaging

Using movies and narratives as naturalistic stimuli in human neuroimaging studies has yielded significant advances in understanding of cognitive and emotional functions. The relevant literature was reviewed, with emphasis on how the use of naturalistic stimuli has helped advance scientific understanding of human memory, attention, language, emotions, and social cognition in ways that would have been difficult otherwise. These advances include discovering a cortical hierarchy of temporal receptive windows, which supports processing of dynamic information that accumulates over several time scales, such as immediate reactions vs. slowly emerging patterns in social interactions. Naturalistic stimuli have also helped elucidate how the hippocampus supports segmentation and memorization of events in day-to-day life and have afforded insights into attentional brain mechanisms underlying our ability to adopt specific perspectives during natural viewing. Further, neuroimaging studies with naturalistic stimuli have revealed the role of the default-mode network in narrative-processing and in social cognition. Finally, by robustly eliciting genuine emotions, these stimuli have helped elucidate the brain basis of both basic and social emotions apparently manifested as highly overlapping yet distinguishable patterns of brain activity.

Modeling an auditory stimulated brain under altered states of consciousness using the generalized Ising model

Propofol is a short-acting medication that results in decreased levels of consciousness and is used for general anesthesia. Although it is the most commonly used anesthetic in the world, much remains unknown about the mechanisms by which it induces a loss of consciousness. Characterizing anesthesia-induced alterations to brain network activity might provide a powerful framework for understanding the neural mechanisms of unconsciousness. The aim of this work was to model brain activity in healthy brains during various stages of consciousness, as induced by propofol, in the auditory paradigm. We used the generalized Ising model (GIM) to fit the empirical fMRI data of healthy subjects while they listened to an audio clip from a movie. The external stimulus (audio clip) is believed to be at least partially driving a synchronization process of the brain activity and provides a similar conscious experience in different subjects. In order to observe the common synchronization among the subjects, a novel technique called the inter subject correlation (ISC) was implemented. We showed that the GIM-modified to incorporate the naturalistic external field-was able to fit the empirical task fMRI data in the awake state, in mild sedation, in deep sedation, and in recovery, at a temperature T* which is well above the critical temperature. To our knowledge this is the first study that captures human brain activity in response to real-life external stimuli at different levels of conscious awareness using mathematical modeling. This study might be helpful in the future to assess the level of consciousness of patients with disorders of consciousness and help in regaining their consciousness.

Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space

Connectivity hyperalignment can be used to estimate a single shared response space across disjoint datasets. We develop a connectivity-based shared response model that factorizes aggregated fMRI datasets into a single reduced-dimension shared connectivity space and subject-specific topographic transformations. These transformations resolve idiosyncratic functional topographies and can be used to project response time series into shared space. We evaluate this algorithm on a large collection of heterogeneous, naturalistic fMRI datasets acquired while subjects listened to spoken stories. Projecting subject data into shared space dramatically improves between-subject story time-segment classification and increases the dimensionality of shared information across subjects. This improvement generalizes to subjects and stories excluded when estimating the shared space. We demonstrate that estimating a simple semantic encoding model in shared space improves between-subject forward encoding and inverted encoding model performance. The shared space estimated across all datasets is distinct from the shared space derived from any particular constituent dataset; the algorithm leverages shared connectivity to yield a consensus shared space conjoining diverse story stimuli.

Nature abhors a paywall: How open science can realize the potential of naturalistic stimuli

Naturalistic stimuli show significant potential to inform behavioral, cognitive, and clinical neuroscience. To date, this impact is still limited by the relative inaccessibility of both generated neuroimaging data as well as the supporting naturalistic stimuli. In this perspective, we highlight currently available naturalistic datasets and technical solutions such as DataLad that continue to advance our ability to share this data. We also review scientific and sociological challenges in selecting naturalistic stimuli for reproducible research. Overall, we encourage researchers to share their naturalistic datasets to the full extent possible under local copyright law.

Functional connectivity of major depression disorder using ongoing EEG during music perception

OBJECTIVE

The functional connectivity (FC) of major depression disorder (MDD) has not been well studied under naturalistic and continuous stimuli conditions. In this study, we investigated the frequency-specific FC of MDD patients exposed to conditions of music perception using ongoing electroencephalogram (EEG).

METHODS

First, we applied the phase lag index (PLI) method to calculate the connectivity matrices and graph theory-based methods to measure the topology of brain networks across different frequency bands. Then, classification methods were adopted to identify the most discriminate frequency band for the diagnosis of MDD.

RESULTS

During music perception, MDD patients exhibited a decreased connectivity pattern in the delta band but an increased connectivity pattern in the beta band. Healthy people showed a left hemisphere-dominant phenomenon, but MDD patients did not show such a lateralized effect. Support vector machine (SVM) achieved the best classification performance in the beta frequency band with an accuracy of 89.7%, sensitivity of 89.4% and specificity of 89.9%.

CONCLUSIONS

MDD patients exhibited an altered FC in delta and beta bands, and the beta band showed a superiority in the diagnosis of MDD.

SIGNIFICANCE

Our study provided a promising reference for the diagnosis of MDD, and revealed a new perspective for understanding the topology of MDD brain networks during music perception.

Intracranial-EEG evidence for medial temporal pole driving amygdala activity induced by multi-modal emotional stimuli

The temporal pole (TP) is an associative cortical region required for complex cognitive functions such as social and emotional cognition. However, mapping the TP with functional magnetic resonance imaging is technically challenging and thus understanding its interaction with other key emotional circuitry, such as the amygdala, remains elusive. We exploited the unique advantages of stereo-electroencephalography (sEEG) to assess the responses of the TP and the amygdala during the perception of emotionally salient stimuli of pictures, music and movies. These stimuli consistently elicited high gamma responses (70-140 Hz) in both the TP and the amygdala, accompanied by functional connectivity in the low frequency range (2-12 Hz). Computational analyses suggested that the TP drove this effect in the theta frequency range, modulated by the emotional valence of the stimuli. Notably, cross-frequency analysis indicated the phase of theta oscillations in the TP modulated the amplitude of high gamma activity in the amygdala. These results were reproducible across three types of sensory inputs including naturalistic stimuli. Our results suggest that multimodal emotional stimuli induce a hierarchical influence of the TP over the amygdala.

Dynamic intersubject neural synchronization reflects affective responses to sad music

Psychological theories of emotion often highlight the dynamic quality of the affective experience, yet neuroimaging studies of affect have traditionally relied on static stimuli that lack ecological validity. Consequently, the brain regions that represent emotions and feelings as they unfold remain unclear. Recently, dynamic, model-free analytical techniques have been employed with naturalistic stimuli to better capture time-varying patterns of activity in the brain; yet, few studies have focused on relating these patterns to changes in subjective feelings. Here, we address this gap, using intersubject correlation and phase synchronization to assess how stimulus-driven changes in brain activity and connectivity are related to two aspects of emotional experience: emotional intensity and enjoyment. During fMRI scanning, healthy volunteers listened to a full-length piece of music selected to induce sadness. After scanning, participants listened to the piece twice while simultaneously rating the intensity of felt sadness or felt enjoyment. Activity in the auditory cortex, insula, and inferior frontal gyrus was significantly synchronized across participants. Synchronization in auditory, visual, and prefrontal regions was significantly greater in participants with higher measures of a subscale of trait empathy related to feeling emotions in response to music. When assessed dynamically, continuous enjoyment ratings positively predicted a moment-to-moment measure of intersubject synchronization in auditory, default mode, and striatal networks, as well as the orbitofrontal cortex, whereas sadness predicted intersubject synchronization in limbic and striatal networks. The results suggest that stimulus-driven patterns of neural communication in emotional processing and high-level cortical regions carry meaningful information with regards to our feeling in response to a naturalistic stimulus.

Social network proximity predicts similar trajectories of psychological states: Evidence from multi-voxel spatiotemporal dynamics

Homophily is a prevalent characteristic of human social networks: individuals tend to associate and bond with others who are similar to themselves with respect to physical traits and demographic attributes, such as age, gender, and ethnicity. Recent research using functional magnetic resonance imaging has demonstrated a positive relationship between individuals' real-world social network proximity (i.e., whether they are friends, friends-of-friends, or farther removed in social ties) and inter-subject correlation (ISC) in their time series of neural responses when viewing audiovisual movies. However, conventional ISC methods only capture information about similarity in the temporal evolution of region-averaged neural responses, and ignore information carried in fine-grained, spatially distributed response topographies. Here, we demonstrate that temporal trajectories of multi-voxel response patterns to naturalistic stimuli are exceptionally similar among friends and predictive of social network proximity, over and above the effects of response magnitude fluctuations. Furthermore, inter-subject similarity in the temporal trajectory of multi-voxel response patterns across distant points in time was particularly positively associated with individuals' proximity in their real-world social network. The fact that exceptional similarities among friends were most pronounced in long-range temporal fluctuations of response patterns located in multimodal cortical regions (e.g., regions of posterior parietal cortex) suggests that aspects of high-level processing during naturalistic stimulation may be particularly similar among friends. Given the localization of results, we speculate that socially close individuals may be particularly similar in endogenously driven shifts in how they distribute their attention (e.g., across the environment, within internal representations) over time. These results suggest that friends may experience exceptionally similar trajectories of psychological states when exposed to a common stimulus, and, more generally, that there are meaningful individual differences in the temporal evolution of multi-voxel response patterns during naturalistic stimulation.

Predicting individual face-selective topography using naturalistic stimuli

Subject-specific, functionally defined areas are conventionally estimated with functional localizers and a simple contrast analysis between responses to different stimulus categories. Compared with functional localizers, naturalistic stimuli provide several advantages such as stronger and widespread brain activation, greater engagement, and increased subject compliance. In this study we demonstrate that a subject's idiosyncratic functional topography can be estimated with high fidelity from that subject's fMRI data obtained while watching a naturalistic movie using hyperalignment to project other subjects' localizer data into that subject's idiosyncratic cortical anatomy. These findings lay the foundation for developing an efficient tool for mapping functional topographies for a wide range of perceptual and cognitive functions in new subjects based only on fMRI data collected while watching an engaging, naturalistic stimulus and other subjects' localizer data from a normative sample.

Event Representations and Predictive Processing: The Role of the Midline Default Network Core

The human brain is tightly coupled to the world through its sensory‐motor systems—but it also spends a lot of its metabolism talking to itself. One important function of this intrinsic activity is the establishment and updating of event models—representations of the current situation that can predictively guide perception, learning, and action control. Here, we propose that event models largely depend on the default network (DN) midline core that includes the posterior cingulate and anterior medial prefrontal cortex. An increasing body of data indeed suggests that this subnetwork can facilitate stimuli processing during both naturalistic event comprehension and cognitive tasks in which mental representations of prior situations, trials, and task rules can predictively guide attention and performance. This midline core involvement in supporting predictions through event models can make sense of an otherwise complex and conflicting pattern of results regarding the possible cognitive functions subserved by the DN.

Stawarczyk, Bezdek, and Zacks offer neuroscience evidence for a midline default network core, which appears to coordinate internal, top‐down mentation with externally‐triggered, bottom‐up attention in a push‐pull relationship. The network may enable the flexible pursuance of thoughts tuned into or detached from the current environment.

Effects of spatial fMRI resolution on the classification of naturalistic movies

Studies involving multivariate pattern analysis (MVPA) of BOLD fMRI data generally attribute the success of the information-theoretic approach to BOLD signal contrast on the fine spatial scale of millimeters facilitating the classification or decoding of perceptual stimuli. However, to date MVPA studies that have actually explored fMRI resolutions at less than 2 mm voxel size are rare and limited to small sets of unnatural stimuli (like visual gratings) as well as specific sub-regions of the brain, notably the primary somatosensory cortices. To investigate what spatial scale best supports high information extraction under more general conditions this study combined naturalistic movie stimuli with high-resolution fMRI at 7 T and linear discriminant analysis (LDA) of global and local BOLD signal patterns. Contrary to predictions, LDA and similar classifiers reached a maximum in classification accuracy (CA) at a smoothed resolution close to 3 mm, well above the 1.2 mm voxel size of the fMRI acquisition. Maximal CAs around 90% were contingent upon global fMRI signal patterns comprising 4 k-16 k of the most reactive voxels distributed sparsely throughout the occipital and ventro-temporal cortices. A Searchlight analysis of local fMRI patterns largely confirmed the global results, but also revealed a small subset of brain regions in early visual cortex showing limited increases in CA with higher resolution. Principal component analysis of the global and local fMRI signal patterns suggested that reproducible neuronal contributions were spatially auto-correlated and smooth, while other components of higher spatial frequency were likely related to physiological noise and responsible for the reduced CA at higher resolution. Systematic differences between experiments and subjects suggested that higher CA was significantly correlated with more consistent behavior revealed by eye tracking. Thus, the optimal resolution of fMRI data for MVPA was mainly limited by physiological noise of high spatial frequency as well as behavioral (in-)consistency.

Assessing the effects of cocaine dependence and pathological gambling using group-wise sparse representation of natural stimulus FMRI data

Assessing functional brain activation patterns in neuropsychiatric disorders such as cocaine dependence (CD) or pathological gambling (PG) under naturalistic stimuli has received rising interest in recent years. In this paper, we propose and apply a novel group-wise sparse representation framework to assess differences in neural responses to naturalistic stimuli across multiple groups of participants (healthy control, cocaine dependence, pathological gambling). Specifically, natural stimulus fMRI (N-fMRI) signals from all three groups of subjects are aggregated into a big data matrix, which is then decomposed into a common signal basis dictionary and associated weight coefficient matrices via an effective online dictionary learning and sparse coding method. The coefficient matrices associated with each common dictionary atom are statistically assessed for each group separately. With the inter-group comparisons based on the group-wise correspondence established by the common dictionary, our experimental results demonstrated that the group-wise sparse coding and representation strategy can effectively and specifically detect brain networks/regions affected by different pathological conditions of the brain under naturalistic stimuli.

The integration of the internal and external milieu in the insula during dynamic emotional experiences

Whilst external events trigger emotional responses, interoception (the perception of internal physiological states) is fundamental to core emotional experience. By combining high resolution functional neuroimaging with concurrent physiological recordings, we investigated the neural mechanisms of interoceptive integration during free listening to an emotionally salient audio film. We found that cardiac activity, a key interoceptive signal, was robustly synchronised across participants and centrally represented in the posterior insula. Effective connectivity analysis revealed that the anterior insula, specifically tuned to the emotionally salient moments of the audio stream, serves as an integration hub of interoceptive processing: interoceptive states represented in the posterior insula are integrated with exteroceptive representations by the anterior insula to highlight these emotionally salient moments. Our study for the first time demonstrates the insular hierarchy for interoceptive processing during natural emotional experience. These findings provide an ecologically-valid framework for elucidating the neural underpinnings of emotional deficits in neuropsychiatric disorders.

Optimizing methods for linking cinematic features to fMRI data

One of the challenges of naturalistic neurosciences using movie-viewing experiments is how to interpret observed brain activations in relation to the multiplicity of time-locked stimulus features. As previous studies have shown less inter-subject synchronization across viewers of random video footage than story-driven films, new methods need to be developed for analysis of less story-driven contents. To optimize the linkage between our fMRI data collected during viewing of a deliberately non-narrative silent film 'At Land' by Maya Deren (1944) and its annotated content, we combined the method of elastic-net regularization with the model-driven linear regression and the well-established data-driven independent component analysis (ICA) and inter-subject correlation (ISC) methods. In the linear regression analysis, both IC and region-of-interest (ROI) time-series were fitted with time-series of a total of 36 binary-valued and one real-valued tactile annotation of film features. The elastic-net regularization and cross-validation were applied in the ordinary least-squares linear regression in order to avoid over-fitting due to the multicollinearity of regressors, the results were compared against both the partial least-squares (PLS) regression and the un-regularized full-model regression. Non-parametric permutation testing scheme was applied to evaluate the statistical significance of regression. We found statistically significant correlation between the annotation model and 9 ICs out of 40 ICs. Regression analysis was also repeated for a large set of cubic ROIs covering the grey matter. Both IC- and ROI-based regression analyses revealed activations in parietal and occipital regions, with additional smaller clusters in the frontal lobe. Furthermore, we found elastic-net based regression more sensitive than PLS and un-regularized regression since it detected a larger number of significant ICs and ROIs. Along with the ISC ranking methods, our regression analysis proved a feasible method for ordering the ICs based on their functional relevance to the annotated cinematic features. The novelty of our method is - in comparison to the hypothesis-driven manual pre-selection and observation of some individual regressors biased by choice - in applying data-driven approach to all content features simultaneously. We found especially the combination of regularized regression and ICA useful when analyzing fMRI data obtained using non-narrative movie stimulus with a large set of complex and correlated features.