Decoding human brain activity during real-world experiences

Inter- and intra-subject similarity in network functional connectivity across a full narrative movie

Naturalistic paradigms, such as watching movies during functional magnetic resonance imaging (fMRI), are thought to prompt the emotional and cognitive processes typically elicited in real life situations. Therefore, naturalistic viewing (NV) holds great potential for studying individual differences. However, in how far NV elicits similarity within and between subjects on a network level, particularly depending on emotions portrayed in movies, is currently unknown. We used the studyforrest dataset to investigate the inter- and intra-subject similarity in network functional connectivity (NFC) of 14 meta-analytically defined networks across a full narrative, audio-visual movie split into 8 consecutive movie segments. We characterized the movie segments by valence and arousal portrayed within the sequences, before utilizing a linear mixed model to analyze which factors explain inter- and intra-subject similarity. Our results showed that the model best explaining inter-subject similarity comprised network, movie segment, valence and a movie segment by valence interaction. Intra-subject similarity was influenced significantly by the same factors and an additional three-way interaction between movie segment, valence and arousal. Overall, inter- and intra-subject similarity in NFC were sensitive to the ongoing narrative and emotions in the movie. Lowest similarity both within and between subjects was seen in the emotional regulation network and networks associated with long-term memory processing, which might be explained by specific features and content of the movie. We conclude that detailed characterization of movie features is crucial for NV research.

Functional organization of the caudal part of the human superior parietal lobule

Like in macaque, the caudal portion of the human superior parietal lobule (SPL) plays a key role in a series of perceptive, visuomotor and somatosensory processes. Here, we review the functional properties of three separate portions of the caudal SPL, i.e., the posterior parieto-occipital sulcus (POs), the anterior POs, and the anterior part of the caudal SPL. We propose that the posterior POs is mainly dedicated to the analysis of visual motion cues useful for object motion detection during self-motion and for spatial navigation, while the more anterior parts are implicated in visuomotor control of limb actions. The anterior POs is mainly involved in using the spotlight of attention to guide reach-to-grasp hand movements, especially in dynamic environments. The anterior part of the caudal SPL plays a central role in visually guided locomotion, being implicated in controlling leg-related movements as well as the four limbs interaction with the environment, and in encoding egomotion-compatible optic flow. Together, these functions reveal how the caudal SPL is strongly implicated in skilled visually-guided behaviors.

Release of cognitive and multimodal MRI data including real-world tasks and hippocampal subfield segmentations

We share data from N = 217 healthy adults (mean age 29 years, range 20-41; 109 females, 108 males) who underwent extensive cognitive assessment and neuroimaging to examine the neural basis of individual differences, with a particular focus on a brain structure called the hippocampus. Cognitive data were collected using a wide array of questionnaires, naturalistic tests that examined imagination, autobiographical memory recall and spatial navigation, traditional laboratory-based tests such as recalling word pairs, and comprehensive characterisation of the strategies used to perform the cognitive tests. 3 Tesla MRI data were also acquired and include multi-parameter mapping to examine tissue microstructure, diffusion-weighted MRI, T2-weighted high-resolution partial volume structural MRI scans (with the masks of hippocampal subfields manually segmented from these scans), whole brain resting state functional MRI scans and partial volume high resolution resting state functional MRI scans. This rich dataset will be of value to cognitive and clinical neuroscientists researching individual differences, real-world cognition, brain-behaviour associations, hippocampal subfields and more. All data are freely available on Dryad.

Posterior hippocampal CA2/3 volume is associated with autobiographical memory recall ability in lower performing individuals

People vary substantially in their capacity to recall past experiences, known as autobiographical memories. Here we investigated whether the volumes of specific hippocampal subfields were associated with autobiographical memory retrieval ability. We manually segmented the full length of the two hippocampi in 201 healthy young adults into DG/CA4, CA2/3, CA1, subiculum, pre/parasubiculum and uncus, in the largest such manually segmented subfield sample yet reported. Across the group we found no evidence for an association between any subfield volume and autobiographical memory recall ability. However, when participants were assigned to lower and higher performing groups based on their memory recall scores, we found that bilateral CA2/3 volume was significantly and positively associated with autobiographical memory recall performance specifically in the lower performing group. We further observed that this effect was attributable to posterior CA2/3. By contrast, semantic details from autobiographical memories, and performance on a range of laboratory-based memory tests, did not correlate with CA2/3 volume. Overall, our findings highlight that posterior CA2/3 may be particularly pertinent for autobiographical memory recall. They also reveal that there may not be direct one-to-one mapping of posterior CA2/3 volume with autobiographical memory ability, with size mattering perhaps only in those with poorer memory recall.

Integrating media content analysis, reception analysis, and media effects studies

Every day, the world of media is at our fingertips, whether it is watching movies, listening to the radio, or browsing online media. On average, people spend over 8 h per day consuming messages from the mass media, amounting to a total lifetime dose of more than 20 years in which conceptual content stimulates our brains. Effects from this flood of information range from short-term attention bursts (e.g., by breaking news features or viral 'memes') to life-long memories (e.g., of one's favorite childhood movie), and from micro-level impacts on an individual's memory, attitudes, and behaviors to macro-level effects on nations or generations. The modern study of media's influence on society dates back to the 1940s. This body of mass communication scholarship has largely asked, "what is media's effect on the individual?" Around the time of the cognitive revolution, media psychologists began to ask, "what cognitive processes are involved in media processing?" More recently, neuroimaging researchers started using real-life media as stimuli to examine perception and cognition under more natural conditions. Such research asks: "what can media tell us about brain function?" With some exceptions, these bodies of scholarship often talk past each other. An integration offers new insights into the neurocognitive mechanisms through which media affect single individuals and entire audiences. However, this endeavor faces the same challenges as all interdisciplinary approaches: Researchers with different backgrounds have different levels of expertise, goals, and foci. For instance, neuroimaging researchers label media stimuli as "naturalistic" although they are in many ways rather artificial. Similarly, media experts are typically unfamiliar with the brain. Neither media creators nor neuroscientifically oriented researchers approach media effects from a social scientific perspective, which is the domain of yet another species. In this article, we provide an overview of approaches and traditions to studying media, and we review the emerging literature that aims to connect these streams. We introduce an organizing scheme that connects the causal paths from media content → brain responses → media effects and discuss network control theory as a promising framework to integrate media content, reception, and effects analyses.

Does memory research have a realistic future?

How do we remember our past experiences? This question remains stubbornly resistant to resolution. The next 25 years may see significant traction on this and other outstanding issues if memory researchers capitalise on exciting technological developments that allow embodied cognition to be studied in ways that closely approximate real life.

Musical Emotions Recognition Using Entropy Features and Channel Optimization Based on EEG

The dynamic of music is an important factor to arouse emotional experience, but current research mainly uses short-term artificial stimulus materials, which cannot effectively awaken complex emotions and reflect their dynamic brain response. In this paper, we used three long-term stimulus materials with many dynamic emotions inside: the "Waltz No. 2" containing pleasure and excitement, the "No. 14 Couplets" containing excitement, briskness, and nervousness, and the first movement of "Symphony No. 5 in C minor" containing passion, relaxation, cheerfulness, and nervousness. Approximate entropy (ApEn) and sample entropy (SampEn) were applied to extract the non-linear features of electroencephalogram (EEG) signals under long-term dynamic stimulation, and the K-Nearest Neighbor (KNN) method was used to recognize emotions. Further, a supervised feature vector dimensionality reduction method was proposed. Firstly, the optimal channel set for each subject was obtained by using a particle swarm optimization (PSO) algorithm, and then the number of times to select each channel in the optimal channel set of all subjects was counted. If the number was greater than or equal to the threshold, it was a common channel suitable for all subjects. The recognition results based on the optimal channel set demonstrated that each accuracy of two categories of emotions based on "Waltz No. 2" and three categories of emotions based on "No. 14 Couplets" was generally above 80%, respectively, and the recognition accuracy of four categories based on the first movement of "Symphony No. 5 in C minor" was about 70%. The recognition accuracy based on the common channel set was about 10% lower than that based on the optimal channel set, but not much different from that based on the whole channel set. This result suggested that the common channel could basically reflect the universal features of the whole subjects while realizing feature dimension reduction. The common channels were mainly distributed in the frontal lobe, central region, parietal lobe, occipital lobe, and temporal lobe. The channel number distributed in the frontal lobe was greater than the ones in other regions, indicating that the frontal lobe was the main emotional response region. Brain region topographic map based on the common channel set showed that there were differences in entropy intensity between different brain regions of the same emotion and the same brain region of different emotions. The number of times to select each channel in the optimal channel set of all 30 subjects showed that the principal component channels representing five brain regions were Fp1/F3 in the frontal lobe, CP5 in the central region, Pz in the parietal lobe, O2 in the occipital lobe, and T8 in the temporal lobe, respectively.

Retrospective behavioral sampling (RBS): A method to effectively track the cognitive fluctuations driven by naturalistic stimulation

Everyday experiences are dynamic, driving fluctuations across simultaneous cognitive processes. A key challenge in the study of naturalistic cognition is to disentangle the complexity of these dynamic processes, without altering the natural experience itself. Retrospective behavioral sampling (RBS) is a novel approach to model the cognitive fluctuations corresponding to the time-course of naturalistic stimulation, across a variety of cognitive dimensions. We tested the effectiveness and reliability of RBS in a web-based experiment, in which 53 participants viewed short movies and listened to a story, followed by retrospective reporting. Participants recalled their experience of 55 discrete events from the stimuli, rating their quality of memory, magnitude of surprise, intensity of negative and positive emotions, perceived importance, reflectivity state, and mental time travel. In addition, a subset of the original cohort re-rated their memory of events in a follow-up questionnaire. Results show highly replicable fluctuation patterns across distinct cognitive dimensions, thereby revealing a stimulus-driven experience that is substantially shared among individuals. Remarkably, memory ratings more than a week after stimulation resulted in an almost identical time-course of memorability as measured immediately following stimulation. In addition, idiosyncratic response patterns were preserved across different stimuli, indicating that RBS characterizes individual differences that are stimulus invariant. The current findings highlight the potential of RBS as a powerful tool for measuring dynamic processes of naturalistic cognition. We discuss the promising approach of matching RBS fluctuations with dynamic processes measured via other testing modalities, such as neuroimaging, to study the neural manifestations of naturalistic cognitive processing.

Neural substrates of spatial processing and navigation in blindness: An activation likelihood estimation meta-analysis

Even though vision is considered the best suited sensory modality to acquire spatial information, blind individuals can form spatial representations to navigate and orient themselves efficiently in space. Consequently, many studies support the amodality hypothesis of spatial representations since sensory modalities other than vision contribute to the formation of spatial representations, independently of visual experience and imagery. However, given the high variability in abilities and deficits observed in blind populations, a clear consensus about the neural representations of space has yet to be established. To this end, we performed a meta-analysis of the literature on the neural correlates of spatial processing and navigation via sensory modalities other than vision, like touch and audition, in individuals with early and late onset blindness. An activation likelihood estimation (ALE) analysis of the neuroimaging literature revealed that early blind individuals and sighted controls activate the same neural networks in the processing of non-visual spatial information and navigation, including the posterior parietal cortex, frontal eye fields, insula, and the hippocampal complex. Furthermore, blind individuals also recruit primary and associative occipital areas involved in visuo-spatial processing via cross-modal plasticity mechanisms. The scarcity of studies involving late blind individuals did not allow us to establish a clear consensus about the neural substrates of spatial representations in this specific population. In conclusion, the results of our analysis on neuroimaging studies involving early blind individuals support the amodality hypothesis of spatial representations.

Conduction velocity along a key white matter tract is associated with autobiographical memory recall ability

Conduction velocity is the speed at which electrical signals travel along axons and is a crucial determinant of neural communication. Inferences about conduction velocity can now be made in vivo in humans using a measure called the magnetic resonance (MR) g-ratio. This is the ratio of the inner axon diameter relative to that of the axon plus the myelin sheath that encases it. Here, in the first application to cognition, we found that variations in MR g-ratio, and by inference conduction velocity, of the parahippocampal cingulum bundle were associated with autobiographical memory recall ability in 217 healthy adults. This tract connects the hippocampus with a range of other brain areas. We further observed that the association seemed to be with inner axon diameter rather than myelin content. The extent to which neurites were coherently organised within the parahippocampal cingulum bundle was also linked with autobiographical memory recall ability. Moreover, these findings were specific to autobiographical memory recall and were not apparent for laboratory-based memory tests. Our results offer a new perspective on individual differences in autobiographical memory recall ability, highlighting the possible influence of specific white matter microstructure features on conduction velocity when recalling detailed memories of real-life past experiences.

Discovering hidden brain network responses to naturalistic stimuli via tensor component analysis of multi-subject fMRI data

The study of brain network interactions during naturalistic stimuli facilitates a deeper understanding of human brain function. To estimate large-scale brain networks evoked with naturalistic stimuli, a tensor component analysis (TCA) based framework was used to characterize shared spatio-temporal patterns across subjects in a purely data-driven manner. In this framework, a third-order tensor is constructed from the timeseries extracted from all brain regions from a given parcellation, for all participants, with modes of the tensor corresponding to spatial distribution, time series and participants. TCA then reveals spatially and temporally shared components, i.e., evoked networks with the naturalistic stimuli, their time courses of activity and subject loadings of each component. To enhance the reproducibility of the estimation with the adaptive TCA algorithm, a novel spectral clustering method, tensor spectral clustering, was proposed and applied to evaluate the stability of the TCA algorithm. We demonstrated the effectiveness of the proposed framework via simulations and real fMRI data collected during a motor task with a traditional fMRI study design. We also applied the proposed framework to fMRI data collected during passive movie watching to illustrate how reproducible brain networks are evoked by naturalistic movie viewing.

Decision thresholding on fMRI activation maps using the Hilbert-Huang transform

OBJECTIVE

Functional magnetic resonance imaging (fMRI) requires thresholds by which to identify brain regions with significant activation, particularly for experiments involving real-life paradigms. One conventional non-parametric approach to generating surrogate data involves decomposition of the original fMRI time series using the Fourier transform, after which the phase is randomized without altering the magnitude of individual frequency components. However, it has been reported that spontaneous brain signals could be non-stationary, which, if true, could lead to false-positive results.

APPROACH

This paper introduces a randomization procedure based on the Hilbert-Huang transform by which to account for non-stationarity in fMRI time series derived from two fMRI datasets (stationary or non-stationary). The significance of individual voxels was determined by comparing the distribution of empirical data versus a surrogate distribution.

MAIN RESULTS

In a comparison with conventional phase-randomization and wavelet-based permutation methods, the proposed method proved highly effective in generating activation maps indicating essential brain regions, while filtering out noise in the white matter.

SIGNIFICANCE

This work demonstrated the importance of considering the non-stationary nature of fMRI time series when selecting resampling methods by which to probe brain activity or identify functional networks in real-life fMRI experiments. We propose a statistical testing method to deal with the non-stationarity of continuous brain signals.

Sensorimotor anticipation of others' actions in real-world and video settings: modulation by level of engagement?

Electroencephalography (EEG) studies investigating social cognition have used both video and real-world stimuli, often without a strong reasoning why one or the other was chosen. Video stimuli can be selected for practical reasons, while naturalistic real-world stimuli are ecologically valid. The current study investigated modulatory effects on EEG mu (8 - 13 Hz) suppression, directly prior to the onset - and during the course - of observed actions, related to real-world and video settings. Recordings were made over sensorimotor cortex and stimuli in both settings consisted of identical (un)predictable object-related grasping and placing actions. In both settings a very similar mu suppression was found during unfolding of the action, irrespective of predictability. However, mu suppression related to the anticipation of upcoming predictable actions was found exclusively in the real-world setting. Thus, even though the presentation setting does not seem to modulate mu suppression during action observation, it does affect the anticipation-related mu suppression. We discuss the possibility that this may be due to increased social engagement in real-world settings, which in particular affects anticipation. The findings emphasise the importance of using real-world stimuli to bring out the subtle, anticipatory, aspects related to action observation.

Clinical applications of magnetic resonance imaging based functional and structural connectivity

Advances in computational neuroimaging techniques have expanded the armamentarium of imaging tools available for clinical applications in clinical neuroscience. Non-invasive, in vivo brain MRI structural and functional network mapping has been used to identify therapeutic targets, define eloquent brain regions to preserve, and gain insight into pathological processes and treatments as well as prognostic biomarkers. These tools have the real potential to inform patient-specific treatment strategies. Nevertheless, a realistic appraisal of clinical utility is needed that balances the growing excitement and interest in the field with important limitations associated with these techniques. Quality of the raw data, minutiae of the processing methodology, and the statistical models applied can all impact on the results and their interpretation. A lack of standardization in data acquisition and processing has also resulted in issues with reproducibility. This limitation has had a direct impact on the reliability of these tools and ultimately, confidence in their clinical use. Advances in MRI technology and computational power as well as automation and standardization of processing methods, including machine learning approaches, may help address some of these issues and make these tools more reliable in clinical use. In this review, we will highlight the current clinical uses of MRI connectomics in the diagnosis and treatment of neurological disorders; balancing emerging applications and technologies with limitations of connectivity analytic approaches to present an encompassing and appropriate perspective.

The fronto-parietal network is not a flexible hub during naturalistic cognition

The fronto‐parietal network (FPN) is crucial for cognitively demanding tasks as it selectively represents task‐relevant information and controls other brain regions. To implement these functions, it has been argued that it is a flexible hub that reconfigures its functional connectivity with other networks. This was supported by a study in which a set of demanding tasks were presented, that varied in their sensory features, comparison rules, and response mappings, and the FPN showed greater reconfiguration of functional connectivity between tasks than any other network. However, this task set was designed to engage the FPN, and therefore it remains an open question whether the FPN is in a flexible hub in general or only for such task sets. Using two freely available datasets (Experiment 1, N = 15, Experiment 2, N = 644), we examined dynamic functional connectivity during naturalistic cognition, while participants watched a movie. Many differences in the flexibility were found across networks but the FPN was not the most flexible hub in the brain, during either movie for any of two measures, using a regression model or a correlation model and across five timescales. We, therefore, conclude that the FPN does not have the trait of being a flexible hub, although it may adopt this state for particular task sets.

Sleep Leads to Brain-Wide Neural Changes Independent of Allocentric and Egocentric Spatial Training in Humans and Rats

Sleep is important for memory consolidation and systems consolidation in particular, which is thought to occur during sleep. While there has been a significant amount of research regarding the effect of sleep on behavior and certain mechanisms during sleep, evidence that sleep leads to consolidation across the system has been lacking until now. We investigated the role of sleep in the consolidation of spatial memory in both rats and humans using a watermaze task involving allocentric- and egocentric-based training. Analysis of immediate early gene expression in rodents, combined with functional magnetic resonance imaging in humans, elucidated similar behavioral and neural effects in both species. Sleep had a beneficial effect on behavior in rats and a marginally significant effect in humans. Interestingly, sleep led to changes across multiple brain regions at the time of retrieval in both species and in both training conditions. In rats, sleep led to increased gene expression in the hippocampus, striatum, and prefrontal cortex. In the humans, sleep led to an activity increase in brain regions belonging to the executive control network and a decrease in activity in regions belonging to the default mode network. Thus, we provide cross-species evidence for system-level memory consolidation occurring during sleep.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Promises and challenges of human computational ethology

The movements an organism makes provide insights into its internal states and motives. This principle is the foundation of the new field of computational ethology, which links rich automatic measurements of natural behaviors to motivational states and neural activity. Computational ethology has proven transformative for animal behavioral neuroscience. This success raises the question of whether rich automatic measurements of behavior can similarly drive progress in human neuroscience and psychology. New technologies for capturing and analyzing complex behaviors in real and virtual environments enable us to probe the human brain during naturalistic dynamic interactions with the environment that so far were beyond experimental investigation. Inspired by nonhuman computational ethology, we explore how these new tools can be used to test important questions in human neuroscience. We argue that application of this methodology will help human neuroscience and psychology extend limited behavioral measurements such as reaction time and accuracy, permit novel insights into how the human brain produces behavior, and ultimately reduce the growing measurement gap between human and animal neuroscience.

The Treachery of Images: How Realism Influences Brain and Behavior

Although the cognitive sciences aim to ultimately understand behavior and brain function in the real world, for historical and practical reasons, the field has relied heavily on artificial stimuli, typically pictures. We review a growing body of evidence that both behavior and brain function differ between image proxies and real, tangible objects. We also propose a new framework for immersive neuroscience to combine two approaches: (i) the traditional build-up approach of gradually combining simplified stimuli, tasks, and processes; and (ii) a newer tear-down approach that begins with reality and compelling simulations such as virtual reality to determine which elements critically affect behavior and brain processing.

An Analysis of the External Validity of EEG Spectral Power in an Uncontrolled Outdoor Environment during Default and Complex Neurocognitive States

Traditionally, quantitative electroencephalography (QEEG) studies collect data within controlled laboratory environments that limit the external validity of scientific conclusions. To probe these validity limits, we used a mobile EEG system to record electrophysiological signals from human participants while they were located within a controlled laboratory environment and an uncontrolled outdoor environment exhibiting several moderate background influences. Participants performed two tasks during these recordings, one engaging brain activity related to several complex cognitive functions (number sense, attention, memory, executive function) and the other engaging two default brain states. We computed EEG spectral power over three frequency bands (theta: 4-7 Hz, alpha: 8-13 Hz, low beta: 14-20 Hz) where EEG oscillatory activity is known to correlate with the neurocognitive states engaged by these tasks. Null hypothesis significance testing yielded significant EEG power effects typical of the neurocognitive states engaged by each task, but only a beta-band power difference between the two background recording environments during the default brain state. Bayesian analysis showed that the remaining environment null effects were unlikely to reflect measurement insensitivities. This overall pattern of results supports the external validity of laboratory EEG power findings for complex and default neurocognitive states engaged within moderately uncontrolled environments.

The generalizability crisis

Most theories and hypotheses in psychology are verbal in nature, yet their evaluation overwhelmingly relies on inferential statistical procedures. The validity of the move from qualitative to quantitative analysis depends on the verbal and statistical expressions of a hypothesis being closely aligned - that is, that the two must refer to roughly the same set of hypothetical observations. Here, I argue that many applications of statistical inference in psychology fail to meet this basic condition. Focusing on the most widely used class of model in psychology - the linear mixed model - I explore the consequences of failing to statistically operationalize verbal hypotheses in a way that respects researchers' actual generalization intentions. I demonstrate that although the "random effect" formalism is used pervasively in psychology to model intersubject variability, few researchers accord the same treatment to other variables they clearly intend to generalize over (e.g., stimuli, tasks, or research sites). The under-specification of random effects imposes far stronger constraints on the generalizability of results than most researchers appreciate. Ignoring these constraints can dramatically inflate false-positive rates, and often leads researchers to draw sweeping verbal generalizations that lack a meaningful connection to the statistical quantities they are putatively based on. I argue that failure to take the alignment between verbal and statistical expressions seriously lies at the heart of many of psychology's ongoing problems (e.g., the replication crisis), and conclude with a discussion of several potential avenues for improvement.

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

Characterizing Inscapes and resting-state in MEG: Effects in typical and atypical development

Examining the brain at rest is a powerful approach used to understand the intrinsic properties of typical and disordered human brain function, yet task-free paradigms are associated with greater head motion, particularly in young and/or clinical populations such as autism spectrum disorder (ASD) and attention-deficit/hyperactivity disorder (ADHD). Inscapes, a non-social and non-verbal movie paradigm, has been introduced to increase attention, thus mitigating head motion, while reducing the task-induced activations found during typical movie watching. Inscapes has not yet been validated for use in magnetoencephalography (MEG), and it has yet to be shown whether its effects are stable in clinical populations. Across typically developing (N = 32) children and adolescents and those with ASD (N = 46) and ADHD (N = 42), we demonstrate that head motion is reduced during Inscapes. Due to the task state evoked by movie paradigms, we also expectedly observed concomitant modulations in local neural activity (oscillatory power) and functional connectivity (phase and envelope coupling) in intrinsic resting-state networks and across the frequency spectra compared to a fixation cross resting-state. Increases in local activity were accompanied by decreases in low-frequency connectivity within and between resting-state networks, primarily the visual network, suggesting that task-state evoked by Inscapes moderates ongoing and spontaneous cortical inhibition that forms the idling intrinsic networks found during a fixation cross resting-state. Importantly, these effects were similar in ASD and ADHD, making Inscapes a well-suited advancement for investigations of resting brain function in young and clinical populations.

A naturalistic neuroimaging database for understanding the brain using ecological stimuli

Neuroimaging has advanced our understanding of human psychology using reductionist stimuli that often do not resemble information the brain naturally encounters. It has improved our understanding of the network organization of the brain mostly through analyses of 'resting-state' data for which the functions of networks cannot be verifiably labelled. We make a 'Naturalistic Neuroimaging Database' (NNDb v1.0) publically available to allow for a more complete understanding of the brain under more ecological conditions during which networks can be labelled. Eighty-six participants underwent behavioural testing and watched one of 10 full-length movies while functional magnetic resonance imaging was acquired. Resulting timeseries data are shown to be of high quality, with good signal-to-noise ratio, few outliers and low movement. Data-driven functional analyses provide further evidence of data quality. They also demonstrate accurate timeseries/movie alignment and how movie annotations might be used to label networks. The NNDb can be used to answer questions previously unaddressed with standard neuroimaging approaches, progressing our knowledge of how the brain works in the real world.

Movies and narratives as naturalistic stimuli in neuroimaging

Using movies and narratives as naturalistic stimuli in human neuroimaging studies has yielded significant advances in understanding of cognitive and emotional functions. The relevant literature was reviewed, with emphasis on how the use of naturalistic stimuli has helped advance scientific understanding of human memory, attention, language, emotions, and social cognition in ways that would have been difficult otherwise. These advances include discovering a cortical hierarchy of temporal receptive windows, which supports processing of dynamic information that accumulates over several time scales, such as immediate reactions vs. slowly emerging patterns in social interactions. Naturalistic stimuli have also helped elucidate how the hippocampus supports segmentation and memorization of events in day-to-day life and have afforded insights into attentional brain mechanisms underlying our ability to adopt specific perspectives during natural viewing. Further, neuroimaging studies with naturalistic stimuli have revealed the role of the default-mode network in narrative-processing and in social cognition. Finally, by robustly eliciting genuine emotions, these stimuli have helped elucidate the brain basis of both basic and social emotions apparently manifested as highly overlapping yet distinguishable patterns of brain activity.

Resting-State Activity in High-Order Visual Areas as a Window into Natural Human Brain Activations

A major limitation of conventional human brain research has been its basis in highly artificial laboratory experiments. Due to technical constraints, little is known about the nature of cortical activations during ecological real life. We have previously proposed the "spontaneous trait reactivation (STR)" hypothesis arguing that resting-state patterns, which emerge spontaneously in the absence of external stimulus, reflect the statistics of habitual cortical activations during real life. Therefore, these patterns can serve as a window into daily life cortical activity. A straightforward prediction of this hypothesis is that spontaneous patterns should preferentially correlate to patterns generated by naturalistic stimuli compared with artificial ones. Here we targeted high-level category-selective visual areas and tested this prediction by comparing BOLD functional connectivity patterns formed during rest to patterns formed in response to naturalistic stimuli, as well as to more artificial category-selective, dynamic stimuli. Our results revealed a significant correlation between the resting-state patterns and functional connectivity patterns generated by naturalistic stimuli. Furthermore, the correlations to naturalistic stimuli were significantly higher than those found between resting-state patterns and those generated by artificial control stimuli. These findings provide evidence of a stringent link between spontaneous patterns and the activation patterns during natural vision.

Uncertainty promotes information-seeking actions, but what information?

Navigating an unfamiliar city almost certainly brings out uncertainty about getting from place to place. This uncertainty, in turn, triggers information gathering. While navigational uncertainty is common, little is known about what type of information people seek when they are uncertain. The primary choices for information types with environments include landmarks (distal or local), landmark configurations (relation between two or more landmarks), and a distinct geometry, at least for some environments. Uncertainty could lead individuals to more likely seek one of these information types. Extant research informs both predictions about and empirical work exploring this question. This review covers relevant cognitive literature and then suggests empirical approaches to better understand information-seeking actions triggered by uncertainty. Notably, we propose that examining continuous navigation data can provide important insights into information seeking. Benefits of continuous data will be elaborated through one paradigm, spatial reorientation, which intentionally induces uncertainty through disorientation and cue conflict. While this and other methods have been used previously, data have primarily reflected only the final choice. Continuous behavior during a task can better reveal the cognition-action loop contributing to spatial learning and decision making.

Brain annotation toolbox: exploring the functional and genetic associations of neuroimaging results

MOTIVATION

Advances in neuroimaging and sequencing techniques provide an unprecedented opportunity to map the function of brain regions and identify the roots of psychiatric diseases. However, the results from most neuroimaging studies, i.e. activated clusters/regions or functional connectivities between brain regions, frequently cannot be conveniently and systematically interpreted, rendering the biological meaning unclear.

RESULTS

We describe a brain annotation toolbox that generates functional and genetic annotations for neuroimaging results. The voxel-level functional description from the Neurosynth database and gene expression profile from the Allen Human Brain Atlas are used to generate functional/genetic information for region-level neuroimaging results. The validity of the approach is demonstrated by showing that the functional and genetic annotations for specific brain regions are consistent with each other; and further the region by region functional similarity network and genetic similarity network are highly correlated for major brain atlases. One application of brain annotation toolbox is to help provide functional/genetic annotations for newly discovered regions with unknown functions, e.g. the 97 new regions identified in the Human Connectome Project. Importantly, this toolbox can help understand differences between psychiatric patients and controls, and this is demonstrated using schizophrenia and autism data, for which the functional and genetic annotations for the neuroimaging changes in patients are consistent with each other and help interpret the results.

AVAILABILITY AND IMPLEMENTATION

BAT is implemented as a free and open-source MATLAB toolbox and is publicly available at http://123.56.224.61:1313/post/bat.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Towards clinical applications of movie fMRI

As evidenced by the present special issue, movie fMRI is emerging as a powerful tool for exploring brain function and characterizing its variation across individuals. Here, we provide a brief perspective on the potential of movie fMRI for advancing the discovery of brain imaging-based markers of psychiatric illness. We discuss relevant gaps and opportunities in movie fMRI, and propose community-level models that might accelerate the pace of discovery of fMRI-based biomarkers in psychiatry.

Exploring Frequency-Dependent Brain Networks from Ongoing EEG Using Spatial ICA During Music Listening

Recently, exploring brain activity based on functional networks during naturalistic stimuli especially music and video represents an attractive challenge because of the low signal-to-noise ratio in collected brain data. Although most efforts focusing on exploring the listening brain have been made through functional magnetic resonance imaging (fMRI), sensor-level electro- or magnetoencephalography (EEG/MEG) technique, little is known about how neural rhythms are involved in the brain network activity under naturalistic stimuli. This study exploited cortical oscillations through analysis of ongoing EEG and musical feature during freely listening to music. We used a data-driven method that combined music information retrieval with spatial Fourier Independent Components Analysis (spatial Fourier-ICA) to probe the interplay between the spatial profiles and the spectral patterns of the brain network emerging from music listening. Correlation analysis was performed between time courses of brain networks extracted from EEG data and musical feature time series extracted from music stimuli to derive the musical feature related oscillatory patterns in the listening brain. We found brain networks of musical feature processing were frequency-dependent. Musical feature time series, especially fluctuation centroid and key feature, were associated with an increased beta activation in the bilateral superior temporal gyrus. An increased alpha oscillation in the bilateral occipital cortex emerged during music listening, which was consistent with alpha functional suppression hypothesis in task-irrelevant regions. We also observed an increased delta-beta oscillatory activity in the prefrontal cortex associated with musical feature processing. In addition to these findings, the proposed method seems valuable for characterizing the large-scale frequency-dependent brain activity engaged in musical feature processing.

Decoding dynamic affective responses to naturalistic videos with shared neural patterns

This study explored the feasibility of using shared neural patterns from brief affective episodes (viewing affective pictures) to decode extended, dynamic affective sequences in a naturalistic experience (watching movie-trailers). Twenty-eight participants viewed pictures from the International Affective Picture System (IAPS) and, in a separate session, watched various movie-trailers. We first located voxels at bilateral occipital cortex (LOC) responsive to affective picture categories by GLM analysis, then performed between-subject hyperalignment on the LOC voxels based on their responses during movie-trailer watching. After hyperalignment, we trained between-subject machine learning classifiers on the affective pictures, and used the classifiers to decode affective states of an out-of-sample participant both during picture viewing and during movie-trailer watching. Within participants, neural classifiers identified valence and arousal categories of pictures, and tracked self-reported valence and arousal during video watching. In aggregate, neural classifiers produced valence and arousal time series that tracked the dynamic ratings of the movie-trailers obtained from a separate sample. Our findings provide further support for the possibility of using pre-trained neural representations to decode dynamic affective responses during a naturalistic experience.

Information Content of Prefrontal Cortex Activity Quantifies the Difficulty of Narrated Stories

The ability to realize the individuals' impressions during the verbal communication allows social robots to significantly facilitate their social interactions in such areas as child education and elderly care. However, such impressions are highly subjective and internalized and therefore cannot be easily comprehended through behavioural observations. Although brain-machine interface suggests the utility of the brain information in human-robot interaction, previous studies did not consider its potential for estimating the internal impressions during verbal communication. In this article, we introduce a novel approach to estimation of the individuals' perceived difficulty of stories using the quantified information content of their prefrontal cortex activity. We demonstrate the robustness of our approach by showing its comparable performance in face-to-face, humanoid, speaker, and video-chat settings. Our results contribute to the field of socially assistive robotics by taking a step toward enabling robots determine their human companions' perceived difficulty of conversations, thereby enabling these media to sustain their communication with humans by adapting to individuals' pace and interest in response to conversational nuances and complexity.

A manually denoised audio-visual movie watching fMRI dataset for the studyforrest project

The data presented here are related to the studyforrest project that uses the movie ‘Forrest Gump’ to map brain functions in a real-life context using functional magnetic resonance imaging (fMRI). However, neural-related fMRI signals are often small and confounded by various noise sources (i.e., artifacts) that makes searching for the signals induced by specific cognitive processes significantly challenging. To make neural-related signals stand out from the noise, the audio-visual movie watching fMRI dataset from the project was denoised by a combination of spatial independent component analysis and manual identification of signals or noise. Here, both the denoised data and the labeled decomposed components are shared to facilitate further study. Compared with the original data, the denoised data showed a substantial improvement in the temporal signal-to-noise ratio and provided a higher sensitivity in subsequent analyses such as in an inter-subject correlation analysis.

Measurement(s)	Blood Oxygen Level-Dependent Functional MRI
Technology Type(s)	data transformation
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.10266554

Learning Desire Is Predicted by Similar Neural Processing of Naturalistic Educational Materials

Naturalistic stimuli can elicit highly similar brain activity across viewers. How do naturalistic educational materials engage human brains and evoke learning desire? Here, we presented 15 audiovisual course clips (each lasting ∼120 s) to university students and recorded their neural activity through electroencephalography. Upon finishing all the video viewings, subjects ranked 15 courses in order of learning desire and reported the reasons for high learning desire (i.e., “value” and “interest”). The brain activity during the video viewing was measured as the neural similarity via intersubject correlation (ISC), that is, correlation between each subject’s neural responses and those of others. Based on averaged learning desire rankings across subjects, course clips were classified with high versus medium versus low motivational effectiveness. We found that the ISC of high effective course clips was larger than that of low effective ones. The ISC difference (high vs low) was positively associated with subjects’ learning desire difference (high vs low). Such an association occurred when viewing time accumulated to ∼80 s. Moreover, ISC was correlated with “interest-based” rather than “value-based” learning desire. These findings advance our understanding of learning motivation via the neural similarity in the context of on-line education and provide potential neurophysiological suggestions for pedagogical practices.

Neural Evidence of Superior Memory: How to Capture Brain Activities of Encoding Processes Underlying Superior Memory

Relatively little attention has been paid to the neural basis of superior memory despite its potential in providing important insight into efforts to improve memory in the general population or to offset age-related cognitive decline. The current study reports a rare opportunity to reproduce and isolate specific neural activities directly associated with exceptional memory. To capture the brain processes responsible for superior memory, we returned to a laboratory task and analytic approach used to explore the nature of exceptional memory, namely, digit-span task combined with verbal protocol analysis. One participant with average memory received approximately 50 h of digit-span training and the participant's digit-span increased from normative (8 digits) to exceptional (30 digits). Event-related potentials were recorded while the participant's digit span increased from 19 to 30 digits. Protocol analysis allowed us to identify direct behavioral indices of idiosyncratic encoding processes underlying the superior memory performance. EEG indices directly corresponding to the behavioral indices of encoding processes were identified. The results suggest that the early attention-related encoding processes were reflected in theta and delta whereas the later attention-independent encoding processes were reflected in time-domain slow-wave. This fine-grained approach offers new insights into studying neural mechanism mediating superior memory and the cognitive effort necessary to develop it.

Real-Life Neuroscience: An Ecological Approach to Brain and Behavior Research

Owing to advances in neuroimaging technology, the past couple of decades have witnessed a surge of research on brain mechanisms that underlie human cognition. Despite the immense development in cognitive neuroscience, the vast majority of neuroimaging experiments examine isolated agents carrying out artificial tasks in sensory and socially deprived environments. Thus, the understanding of the mechanisms of various domains in cognitive neuroscience, including social cognition and episodic memory, is sorely lacking. Here we focus on social and memory research as representatives of cognitive functions and propose that mainstream, lab-based experimental designs in these fields suffer from two fundamental limitations, pertaining to person-dependent and situation-dependent factors. The person-dependent factor addresses the issue of limiting the active role of the participants in lab-based paradigms that may interfere with their sense of agency and embodiment. The situation-dependent factor addresses the issue of the artificial decontextualized environment in most available paradigms. Building on recent findings showing that real-life as opposed to controlled experimental paradigms involve different mechanisms, we argue that adopting a real-life approach may radically change our understanding of brain and behavior. Therefore, we advocate in favor of a paradigm shift toward a nonreductionist approach, exploiting portable technology in semicontrolled environments, to explore behavior in real life.

Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging

The use of movie-watching as an acquisition state for functional connectivity (FC) MRI has recently enabled multiple groups to obtain rich data sets in younger children with both substantial sample sizes and scan durations. Using naturalistic paradigms such as movies has also provided analytic flexibility for these developmental studies that extends beyond conventional resting state approaches. This review highlights the advantages and challenges of using movies for developmental neuroimaging and explores some of the methodological issues involved in designing pediatric studies with movies. Emerging themes from movie-watching studies are discussed, including an emphasis on intersubject correlations, developmental changes in network interactions under complex naturalistic conditions, and dynamic age-related changes in both sensory and higher-order network FC even in narrow age ranges. Converging evidence suggests an enhanced ability to identify brain-behavior correlations in children when using movie-watching data relative to both resting state and conventional tasks. Future directions and cautionary notes highlight the potential and the limitations of using movies to study FC in pediatric populations.

Role of the Default Mode Network in Cognitive Transitions

A frequently repeated finding is that the default mode network (DMN) shows activation decreases during externally focused tasks. This finding has led to an emphasis in DMN research on internally focused self-relevant thought processes. A recent study, in contrast, implicates the DMN in substantial externally focused task switches. Using functional magnetic resonance imaging, we scanned 24 participants performing a task switch experiment. Whilst replicating previous DMN task switch effects, we also found large DMN increases for brief rests as well as task restarts after rest. Our findings are difficult to explain using theories strictly linked to internal or self-directed cognition. In line with principal results from the literature, we suggest that the DMN encodes scene, episode or context, by integrating spatial, self-referential, and temporal information. Context representations are strong at rest, but rereference to context also occurs at major cognitive transitions.

Interpretation of Social Interactions: Functional Imaging of Cognitive-Semiotic Categories During Naturalistic Viewing

Social interactions arise from patterns of communicative signs, whose perception and interpretation require a multitude of cognitive functions. The semiotic framework of Peirce's Universal Categories (UCs) laid ground for a novel cognitive-semiotic typology of social interactions. During functional magnetic resonance imaging (fMRI), 16 volunteers watched a movie narrative encompassing verbal and non-verbal social interactions. Three types of non-verbal interactions were coded ("unresolved," "non-habitual," and "habitual") based on a typology reflecting Peirce's UCs. As expected, the auditory cortex responded to verbal interactions, but non-verbal interactions modulated temporal areas as well. Conceivably, when speech was lacking, ambiguous visual information (unresolved interactions) primed auditory processing in contrast to learned behavioral patterns (habitual interactions). The latter recruited a parahippocampal-occipital network supporting conceptual processing and associative memory retrieval. Requesting semiotic contextualization, non-habitual interactions activated visuo-spatial and contextual rule-learning areas such as the temporo-parietal junction and right lateral prefrontal cortex. In summary, the cognitive-semiotic typology reflected distinct sensory and association networks underlying the interpretation of observed non-verbal social interactions.

Broadband time-resolved multi-channel functional near-infrared spectroscopy system to monitor in vivo physiological changes of human brain activity

We have developed a broadband time-resolved multi-channel near-infrared spectroscopy system that can monitor the physiological responses of the adult human brain. This system is composed of a supercontinuum laser for the source part and of an intensified charge-coupled device camera coupled with an imaging spectrometer for the detection part. It allows the detection of the spectral, from 600 to 900 nm, and spatial dimensions as well as the arrival time of photon information simultaneously. We describe the setup and its characterization in terms of temporal instrument response function, wavelength sensitivity, and stability. The ability of the system to detect the hemodynamic response is then demonstrated. First, an in vivo experiment on an adult volunteer was performed to monitor the response in the arm during a cuff occlusion. Second, the response in the brain during a cognitive task was monitored on a group of five healthy volunteers. Moreover, looking at the response at different time windows, we could monitor the hemodynamic response in depth, enhancing the detection of the cortical activation. Those first results demonstrate the ability of our system to discriminate between the responses of superficial and deep tissues, addressing an important issue in functional near-infrared spectroscopy.

Neural correlates of the natural observation of an emotionally loaded video

Studies based on a paradigm of free or natural viewing have revealed characteristics that allow us to know how the brain processes stimuli within a natural environment. This method has been little used to study brain function. With a connectivity approach, we examine the processing of emotions using an exploratory method to analyze functional magnetic resonance imaging (fMRI) data. This research describes our approach to modeling stress paradigms suitable for neuroimaging environments. We showed a short film (4.54 minutes) with high negative emotional valence and high arousal content to 24 healthy male subjects (36.42 years old; SD = 12.14) during fMRI. Independent component analysis (ICA) was used to identify networks based on spatial statistical independence. Through this analysis we identified the sensorimotor system and its influence on the dorsal attention and default-mode networks, which in turn have reciprocal activity and modulate networks described as emotional.

Differential Entropy Preserves Variational Information of Near-Infrared Spectroscopy Time Series Associated With Working Memory

Neuroscience research shows a growing interest in the application of Near-Infrared Spectroscopy (NIRS) in analysis and decoding of the brain activity of human subjects. Given the correlation that is observed between the Blood Oxygen Dependent Level (BOLD) responses that are exhibited by the time series data of functional Magnetic Resonance Imaging (fMRI) and the hemoglobin oxy/deoxy-genation that is captured by NIRS, linear models play a central role in these applications. This, in turn, results in adaptation of the feature extraction strategies that are well-suited for discretization of data that exhibit a high degree of linearity, namely, slope and the mean as well as their combination, to summarize the informational contents of the NIRS time series. In this article, we demonstrate that these features are inefficient in capturing the variational information of NIRS data, limiting the reliability and the adequacy of the conclusion on their results. Alternatively, we propose the linear estimate of differential entropy of these time series as a natural representation of such information. We provide evidence for our claim through comparative analysis of the application of these features on NIRS data pertinent to several working memory tasks as well as naturalistic conversational stimuli.

Network Dynamics of Attention During a Naturalistic Behavioral Paradigm

This study investigates the dynamics of attention during continuous, naturalistic interactions in a video game. Specifically, the effect of repeated distraction on a continuous primary task is related to a functional model of network connectivity. We introduce the Non-linear Attentional Saturation Hypothesis (NASH), which predicts that effective connectivity within attentional networks increases non-linearly with decreasing distraction over time, and exhibits dampening at critical parameter values. Functional magnetic resonance imaging (fMRI) data collected using a naturalistic behavioral paradigm coupled with an interactive video game is used to test the hypothesis. As predicted, connectivity in pre-defined regions corresponding to attentional networks increases as distraction decreases. Moreover, the functional relationship between connectivity and distraction is convex, that is, network connectivity somewhat increases as distraction decreases during the continuous primary task, however, connectivity increases considerably as distraction falls below critical levels. This result characterizes the non-linear pattern of connectivity within attentional networks, particularly with respect to their dynamics during behavior. These results are also summarized in the form of a network structure analysis, which underscores the role of various nodes in regulating the global network state. In conclusion, we situate the implications of this research in the context of cognitive complexity and an emerging theory of flow during media exposure.

Exploring collective experience in watching dance through intersubject correlation and functional connectivity of fMRI brain activity

How the brain contends with naturalistic viewing conditions when it must cope with concurrent streams of diverse sensory inputs and internally generated thoughts is still largely an open question. In this study, we used fMRI to record brain activity while a group of 18 participants watched an edited dance duet accompanied by a soundtrack. After scanning, participants performed a short behavioral task to identify neural correlates of dance segments that could later be recalled. Intersubject correlation (ISC) analysis was used to identify the brain regions correlated among observers, and the results of this ISC map were used to define a set of regions for subsequent analysis of functional connectivity. The resulting network was found to be composed of eight subnetworks and the significance of these subnetworks is discussed. While most subnetworks could be explained by sensory and motor processes, two subnetworks appeared related more to complex cognition. These results inform our understanding of the neural basis of common experience in watching dance and open new directions for the study of complex cognition.

On application of kernel PCA for generating stimulus features for fMRI during continuous music listening

BACKGROUND

There has been growing interest towards naturalistic neuroimaging experiments, which deepen our understanding of how human brain processes and integrates incoming streams of multifaceted sensory information, as commonly occurs in real world. Music is a good example of such complex continuous phenomenon. In a few recent fMRI studies examining neural correlates of music in continuous listening settings, multiple perceptual attributes of music stimulus were represented by a set of high-level features, produced as the linear combination of the acoustic descriptors computationally extracted from the stimulus audio. NEW METHOD: fMRI data from naturalistic music listening experiment were employed here. Kernel principal component analysis (KPCA) was applied to acoustic descriptors extracted from the stimulus audio to generate a set of nonlinear stimulus features. Subsequently, perceptual and neural correlates of the generated high-level features were examined.

RESULTS

The generated features captured musical percepts that were hidden from the linear PCA features, namely Rhythmic Complexity and Event Synchronicity. Neural correlates of the new features revealed activations associated to processing of complex rhythms, including auditory, motor, and frontal areas.

COMPARISON WITH EXISTING METHOD

Results were compared with the findings in the previously published study, which analyzed the same fMRI data but applied linear PCA for generating stimulus features. To enable comparison of the results, methodology for finding stimulus-driven functional maps was adopted from the previous study.

CONCLUSIONS

Exploiting nonlinear relationships among acoustic descriptors can lead to the novel high-level stimulus features, which can in turn reveal new brain structures involved in music processing.