Future trends in Neuroimaging: Neural processes as expressed within real-life contexts

Measuring shared responses across subjects using intersubject correlation

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual's brain activity to model another's, we can measure shared information across brains-even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis and outline best practices.

Aberrant Cortical Integration in First-Episode Psychosis During Natural Audiovisual Processing

BACKGROUND

Functional magnetic resonance imaging studies of psychotic disorders have reported both hypoactivity and hyperactivity in numerous brain regions. In line with the dysconnection hypothesis, these regions include cortical integrative hub regions. However, most earlier studies focused on a single cognitive function at a time, assessed by delivering artificial stimuli to patients with chronic psychosis. Thus, it remains unresolved whether these findings are present already in early psychosis and whether they translate to real-life-like conditions that require multisensory processing and integration.

METHODS

Scenes from the movie Alice in Wonderland (2010) were shown to 51 patients with first-episode psychosis (16 women) and 32 community-based control subjects (17 women) during 3T functional magnetic resonance imaging. We compared intersubject correlation, a measure of similarity of brain signal time courses in each voxel, between the groups. We also quantified the hubness as the number of connections each region has.

RESULTS

Intersubject correlation was significantly lower in patients with first-episode psychosis than in control subjects in the medial and lateral prefrontal, cingulate, precuneal, and parietotemporal regions, including the default mode network. Regional magnitude of between-group difference in intersubject correlation was associated with the hubness.

CONCLUSIONS

Our findings provide novel evidence for the dysconnection hypothesis by showing that during complex real-life-like stimulation, the most prominent functional alterations in psychotic disorders relate to integrative brain functions. Presence of such abnormalities in first-episode psychosis rules out long-term effects of illness or medication. These methods can be used in further studies to map widespread hub alterations in a single functional magnetic resonance imaging session and link them to potential downstream and upstream pathways.

Mapping Meanings

In a 2016 paper, Huth and colleagues probed, in a general way, how word meanings map onto cortical locations. By comparing the fit between alternative maps, this methodology offered a means to evaluate what sorts of meaning representations the brain handles under ecologically realistic conditions.

Cooperating yet distinct brain networks engaged during naturalistic paradigms: A meta-analysis of functional MRI results

Cognitive processes do not occur by pure insertion and instead depend on the full complement of co-occurring mental processes, including perceptual and motor functions. As such, there is limited ecological validity to human neuroimaging experiments that use highly controlled tasks to isolate mental processes of interest. However, a growing literature shows how dynamic, interactive tasks have allowed researchers to study cognition as it more naturally occurs. Collective analysis across such neuroimaging experiments may answer broader questions regarding how naturalistic cognition is biologically distributed throughout the brain. We applied an unbiased, data-driven, meta-analytic approach that uses k-means clustering to identify core brain networks engaged across the naturalistic functional neuroimaging literature. Functional decoding allowed us to, then, delineate how information is distributed between these networks throughout the execution of dynamical cognition in realistic settings. This analysis revealed six recurrent patterns of brain activation, representing sensory, domain-specific, and attentional neural networks that support the cognitive demands of naturalistic paradigms. Although gaps in the literature remain, these results suggest that naturalistic fMRI paradigms recruit a common set of networks that allow both separate processing of different streams of information and integration of relevant information to enable flexible cognition and complex behavior.

Connectivity of the precuneus-posterior cingulate cortex with the anterior cingulate cortex-medial prefrontal cortex differs consistently between control subjects and first-episode psychosis patients during a movie stimulus

BACKGROUND

Functional connectivity is altered in psychotic disorders. Multiple findings concentrate on the default mode network, anchored on the precuneus-posterior cingulate cortex (PC-PCC). However, the nature of the alterations varies between studies and connectivity alterations have not been studied during an ecologically valid natural stimulus. In the present study, we investigated the functional and structural connectivity of a PC-PCC region, where functioning differentiated first-episode psychosis patients from control subjects during free viewing of a movie in our earlier study.

METHODS

14 first-episode psychosis patients and 12 control subjects were imaged with GE 3T, and 29 patients and 19 control subjects were imaged with a Siemens Skyra 3T scanner while watching scenes from the movie Alice in Wonderland. Group differences in functional connectivity were analysed for both scanners separately and results were compared to identify any overlap. Diffusion tensor measures of 26 patients and 19 control subjects were compared for the related white matter tracts, identified by deterministic tractography.

RESULTS

Functional connectivity was increased in patients across scanners between the midline regions of the PC-PCC and the anterior cingulate cortex-medial prefrontal cortex (ACC-mPFC). We found no group differences in any of the diffusion tensor imaging measures.

CONCLUSIONS

Already in the early stages of psychosis functional connectivity between the midline structures of the PC-PCC and the ACC-mPFC is consistently increased during naturalistic stimulus.

Interpretation of Social Interactions: Functional Imaging of Cognitive-Semiotic Categories During Naturalistic Viewing

Social interactions arise from patterns of communicative signs, whose perception and interpretation require a multitude of cognitive functions. The semiotic framework of Peirce's Universal Categories (UCs) laid ground for a novel cognitive-semiotic typology of social interactions. During functional magnetic resonance imaging (fMRI), 16 volunteers watched a movie narrative encompassing verbal and non-verbal social interactions. Three types of non-verbal interactions were coded ("unresolved," "non-habitual," and "habitual") based on a typology reflecting Peirce's UCs. As expected, the auditory cortex responded to verbal interactions, but non-verbal interactions modulated temporal areas as well. Conceivably, when speech was lacking, ambiguous visual information (unresolved interactions) primed auditory processing in contrast to learned behavioral patterns (habitual interactions). The latter recruited a parahippocampal-occipital network supporting conceptual processing and associative memory retrieval. Requesting semiotic contextualization, non-habitual interactions activated visuo-spatial and contextual rule-learning areas such as the temporo-parietal junction and right lateral prefrontal cortex. In summary, the cognitive-semiotic typology reflected distinct sensory and association networks underlying the interpretation of observed non-verbal social interactions.

Neural correlates of the natural observation of an emotionally loaded video

Studies based on a paradigm of free or natural viewing have revealed characteristics that allow us to know how the brain processes stimuli within a natural environment. This method has been little used to study brain function. With a connectivity approach, we examine the processing of emotions using an exploratory method to analyze functional magnetic resonance imaging (fMRI) data. This research describes our approach to modeling stress paradigms suitable for neuroimaging environments. We showed a short film (4.54 minutes) with high negative emotional valence and high arousal content to 24 healthy male subjects (36.42 years old; SD = 12.14) during fMRI. Independent component analysis (ICA) was used to identify networks based on spatial statistical independence. Through this analysis we identified the sensorimotor system and its influence on the dorsal attention and default-mode networks, which in turn have reciprocal activity and modulate networks described as emotional.

Differential Entropy Preserves Variational Information of Near-Infrared Spectroscopy Time Series Associated With Working Memory

Neuroscience research shows a growing interest in the application of Near-Infrared Spectroscopy (NIRS) in analysis and decoding of the brain activity of human subjects. Given the correlation that is observed between the Blood Oxygen Dependent Level (BOLD) responses that are exhibited by the time series data of functional Magnetic Resonance Imaging (fMRI) and the hemoglobin oxy/deoxy-genation that is captured by NIRS, linear models play a central role in these applications. This, in turn, results in adaptation of the feature extraction strategies that are well-suited for discretization of data that exhibit a high degree of linearity, namely, slope and the mean as well as their combination, to summarize the informational contents of the NIRS time series. In this article, we demonstrate that these features are inefficient in capturing the variational information of NIRS data, limiting the reliability and the adequacy of the conclusion on their results. Alternatively, we propose the linear estimate of differential entropy of these time series as a natural representation of such information. We provide evidence for our claim through comparative analysis of the application of these features on NIRS data pertinent to several working memory tasks as well as naturalistic conversational stimuli.

Latent source mining in FMRI via restricted Boltzmann machine

Blind source separation (BSS) is commonly used in functional magnetic resonance imaging (fMRI) data analysis. Recently, BSS models based on restricted Boltzmann machine (RBM), one of the building blocks of deep learning models, have been shown to improve brain network identification compared to conventional single matrix factorization models such as independent component analysis (ICA). These models, however, trained RBM on fMRI volumes, and are hence challenged by model complexity and limited training set. In this article, we propose to apply RBM to fMRI time courses instead of volumes for BSS. The proposed method not only interprets fMRI time courses explicitly to take advantages of deep learning models in latent feature learning but also substantially reduces model complexity and increases the scale of training set to improve training efficiency. Our experimental results based on Human Connectome Project (HCP) datasets demonstrated the superiority of the proposed method over ICA and the one that applied RBM to fMRI volumes in identifying task-related components, resulted in more accurate and specific representations of task-related activations. Moreover, our method separated out components representing intermixed effects between task events, which could reflect inherent interactions among functionally connected brain regions. Our study demonstrates the value of RBM in mining complex structures embedded in large-scale fMRI data and its potential as a building block for deeper models in fMRI data analysis.

On application of kernel PCA for generating stimulus features for fMRI during continuous music listening

BACKGROUND

There has been growing interest towards naturalistic neuroimaging experiments, which deepen our understanding of how human brain processes and integrates incoming streams of multifaceted sensory information, as commonly occurs in real world. Music is a good example of such complex continuous phenomenon. In a few recent fMRI studies examining neural correlates of music in continuous listening settings, multiple perceptual attributes of music stimulus were represented by a set of high-level features, produced as the linear combination of the acoustic descriptors computationally extracted from the stimulus audio. NEW METHOD: fMRI data from naturalistic music listening experiment were employed here. Kernel principal component analysis (KPCA) was applied to acoustic descriptors extracted from the stimulus audio to generate a set of nonlinear stimulus features. Subsequently, perceptual and neural correlates of the generated high-level features were examined.

RESULTS

The generated features captured musical percepts that were hidden from the linear PCA features, namely Rhythmic Complexity and Event Synchronicity. Neural correlates of the new features revealed activations associated to processing of complex rhythms, including auditory, motor, and frontal areas.

COMPARISON WITH EXISTING METHOD

Results were compared with the findings in the previously published study, which analyzed the same fMRI data but applied linear PCA for generating stimulus features. To enable comparison of the results, methodology for finding stimulus-driven functional maps was adopted from the previous study.

CONCLUSIONS

Exploiting nonlinear relationships among acoustic descriptors can lead to the novel high-level stimulus features, which can in turn reveal new brain structures involved in music processing.

Inter-subject synchrony as an index of functional specialization in early childhood

Early childhood is a time of significant change within multiple cognitive domains including social cognition, memory, executive function, and language; however, the corresponding neural changes remain poorly understood. This is likely due to the difficulty in acquiring artifact-free functional MRI data during complex task-based or unconstrained resting-state experiments in young children. In addition, task-based and resting state experiments may not capture dynamic real-world processing. Here we overcome both of these challenges through use of naturalistic viewing (i.e., passively watching a movie in the scanner) combined with inter-subject neural synchrony to examine functional specialization within 4- and 6-year old children. Using a novel and stringent crossed random effect statistical analysis, we find that children show more variable patterns of activation compared to adults, particularly within regions of the default mode network (DMN). In addition, we found partial evidence that child-to-adult synchrony increased as a function of age within a DMN region: the temporoparietal junction. Our results suggest age-related differences in functional brain organization within a cross-sectional sample during an ecologically valid context and demonstrate that neural synchrony during naturalistic viewing fMRI can be used to examine functional specialization during early childhood - a time when neural and cognitive systems are in flux.

Variation in the speech signal as a window into the cognitive architecture of language production

The pronunciation of words is highly variable. This variation provides crucial information about the cognitive architecture of the language production system. This review summarizes key empirical findings about variation phenomena, integrating corpus, acoustic, articulatory, and chronometric data from phonetic and psycholinguistic studies. It examines how these data constrain our current understanding of word production processes and highlights major challenges and open issues that should be addressed in future research.

Decoding power-spectral profiles from FMRI brain activities during naturalistic auditory experience

Recent studies have demonstrated a close relationship between computational acoustic features and neural brain activities, and have largely advanced our understanding of auditory information processing in the human brain. Along this line, we proposed a multidisciplinary study to examine whether power spectral density (PSD) profiles can be decoded from brain activities during naturalistic auditory experience. The study was performed on a high resolution functional magnetic resonance imaging (fMRI) dataset acquired when participants freely listened to the audio-description of the movie "Forrest Gump". Representative PSD profiles existing in the audio-movie were identified by clustering the audio samples according to their PSD descriptors. Support vector machine (SVM) classifiers were trained to differentiate the representative PSD profiles using corresponding fMRI brain activities. Based on PSD profile decoding, we explored how the neural decodability correlated to power intensity and frequency deviants. Our experimental results demonstrated that PSD profiles can be reliably decoded from brain activities. We also suggested a sigmoidal relationship between the neural decodability and power intensity deviants of PSD profiles. Our study in addition substantiates the feasibility and advantage of naturalistic paradigm for studying neural encoding of complex auditory information.

Is It Possible to Predict the Future in First-Episode Psychosis?

The outcome of first-episode psychosis (FEP) is highly variable, ranging from early sustained recovery to antipsychotic treatment resistance from the onset of illness. For clinicians, a possibility to predict patient outcomes would be highly valuable for the selection of antipsychotic treatment and in tailoring psychosocial treatments and psychoeducation. This selective review summarizes current knowledge of prognostic markers in FEP. We sought potential outcome predictors from clinical and sociodemographic factors, cognition, brain imaging, genetics, and blood-based biomarkers, and we considered different outcomes, like remission, recovery, physical comorbidities, and suicide risk. Based on the review, it is currently possible to predict the future for FEP patients to some extent. Some clinical features-like the longer duration of untreated psychosis (DUP), poor premorbid adjustment, the insidious mode of onset, the greater severity of negative symptoms, comorbid substance use disorders (SUDs), a history of suicide attempts and suicidal ideation and having non-affective psychosis-are associated with a worse outcome. Of the social and demographic factors, male gender, social disadvantage, neighborhood deprivation, dysfunctional family environment, and ethnicity may be relevant. Treatment non-adherence is a substantial risk factor for relapse, but a small minority of patients with acute onset of FEP and early remission may benefit from antipsychotic discontinuation. Cognitive functioning is associated with functional outcomes. Brain imaging currently has limited utility as an outcome predictor, but this may change with methodological advancements. Polygenic risk scores (PRSs) might be useful as one component of a predictive tool, and pharmacogenetic testing is already available and valuable for patients who have problems in treatment response or with side effects. Most blood-based biomarkers need further validation. None of the currently available predictive markers has adequate sensitivity or specificity used alone. However, personalized treatment of FEP will need predictive tools. We discuss some methodologies, such as machine learning (ML), and tools that could lead to the improved prediction and clinical utility of different prognostic markers in FEP. Combination of different markers in ML models with a user friendly interface, or novel findings from e.g., molecular genetics or neuroimaging, may result in computer-assisted clinical applications in the near future.

Neuroimaging of learning and development: improving ecological validity

Modern neuroscience research, including neuroimaging techniques such as functional magnetic resonance imaging (fMRI), has provided valuable insights that advanced our understanding of brain development and learning processes significantly. However, there is a lively discussion about whether and how these insights can be meaningful to the educational practice. One of the main challenges is the low ecological validity of neuroimaging studies, making it hard to translate neuroimaging findings to real-life learning situations. Here, we describe four approaches that increase the ecological validity of neuroimaging experiments: using more naturalistic stimuli and tasks, moving the research to more naturalistic settings by using portable neuroimaging devices, combining tightly controlled lab-based neuroimaging measurements with real-life variables and follow-up field studies, and including stakeholders from the practice at all stages of the research. We illustrate these approaches with examples and explain how these directions of research optimize the benefits of neuroimaging techniques to study learning and development. This paper provides a frontline overview of methodological approaches that can be used for future neuroimaging studies to increase their ecological validity and thereby their relevance and applicability to the learning practice.

Electrophysiology Reveals the Neural Dynamics of Naturalistic Auditory Language Processing: Event-Related Potentials Reflect Continuous Model Updates

The recent trend away from ANOVA-based analyses places experimental investigations into the neurobiology of cognition in more naturalistic and ecologically valid designs within reach. Using mixed-effects models for epoch-based regression, we demonstrate the feasibility of examining event-related potentials (ERPs), and in particular the N400, to study the neural dynamics of human auditory language processing in a naturalistic setting. Despite the large variability between trials during naturalistic stimulation, we replicated previous findings from the literature: the effects of frequency, animacy, and word order and find previously unexplored interaction effects. This suggests a new perspective on ERPs, namely, as a continuous modulation reflecting continuous stimulation instead of a series of discrete and essentially sequential processes locked to discrete events.

Affective creativity meets classic creativity in the scanner

The investigation of neurocognitive processes underlying more real-life creative behavior is among the greatest challenges in creativity research. In this fMRI study, we addressed this issue by investigating functional patterns of brain activity while participants were required to be creative in an affective context. Affective creativity was assessed in terms of individual's inventiveness in generating alternative appraisals for anger-evoking events, which has recently emerged as a new ability concept in cognitive reappraisal research. In addition, a classic divergent thinking task was administered. Both creativity tasks yielded strong activation in left prefrontal regions, indicating their shared cognitive processing demands like the inhibition of prepotent responses, shifting between different perspectives and controlled memory retrieval. Regarding task-specific differences, classic creative ideation activated a characteristic divergent thinking network comprising the left supramarginal, inferior temporal, and inferior frontal gyri. Affective creativity on the other hand specifically recruited the right superior frontal gyrus, presumably involved in the postretrieval monitoring of reappraisal success, and core hubs of the default-mode network, which are also implicated in social cognition. As a whole, by taking creativity research to the realm of emotion, this study advances our understanding of how more real-life creativity is rooted in the brain. Hum Brain Mapp 39:393-406, 2018. © 2017 Wiley Periodicals, Inc.

Probabilistic language models in cognitive neuroscience: Promises and pitfalls

Cognitive neuroscientists of language comprehension study how neural computations relate to cognitive computations during comprehension. On the cognitive part of the equation, it is important that the computations and processing complexity are explicitly defined. Probabilistic language models can be used to give a computationally explicit account of language complexity during comprehension. Whereas such models have so far predominantly been evaluated against behavioral data, only recently have the models been used to explain neurobiological signals. Measures obtained from these models emphasize the probabilistic, information-processing view of language understanding and provide a set of tools that can be used for testing neural hypotheses about language comprehension. Here, we provide a cursory review of the theoretical foundations and example neuroimaging studies employing probabilistic language models. We highlight the advantages and potential pitfalls of this approach and indicate avenues for future research.

Neurodevelopmental maturation as a function of irritable temperament: Insights From a Naturalistic Emotional Video Viewing Paradigm

Few studies have investigated the neural systems involved in decreasing behavioral reactivity to emotional stimuli as children age. It has been suggested that this process may interact with temperament-linked variations in neurodevelopment to better explain individual differences in the maturation of emotion regulation. In this investigation, children ages 4 to 12 (n = 30, mean age = 7.62 years, SD = 1.71 years) and adults (n = 21, mean age = 26.67 years) watched clips from popular children's films containing positive, negative, or neutral emotional content during functional magnetic resonance imaging. Compared to adults, children demonstrated greater activation in subcortical and visual regions (hippocampus, thalamus, visual cortex, fusiform) during negative clips and greater activation of subcortical and prefrontal regions during positive clips (hippocampus, thalamus, caudate, ACC, OFC, superior frontal cortex). In children only, we found an age by temperament interaction in frontal and subcortical regions indicating that activation increased as a function of age in the most irritable children, but decreased as a function of age in the least irritable children. Findings were not present in the temperament domain of fear. Findings replicate and extend the existing irritability literature, indicating that healthy children highest in irritability may develop comparatively greater activation of the lateral prefrontal cortex in order to support adaptive regulation during emotional challenges. These results are discussed within the context of the emerging literature on the utility of complex, multidimensional, and naturalistic stimuli, which present a complementary alternative to understanding ecologically valid and sustained neural responses to emotionally evocative stimuli. Hum Brain Mapp 38:5307-5321, 2017. © 2017 Wiley Periodicals, Inc.

The Dorsal Medial Prefrontal Cortex Responds Preferentially to Social Interactions during Natural Viewing

Humans display a strong tendency to make spontaneous inferences concerning the thoughts and intentions of others. Although this ability relies upon the concerted effort of multiple brain regions, the dorsal medial prefrontal cortex (DMPFC) is most closely associated with the ability to reason about other people's mental states and form impressions of their character. Here, we investigated this region's putative social category preference using fMRI as 34 participants engaged in uninstructed viewing of a complex naturalistic stimulus. Using a data-driven "reverse correlation" approach, we characterize the DMPFC's stimulus response profile from ongoing neural responses to a dynamic movie stimulus. Results of this analysis demonstrate that the DMPFC's response profile is dominated by the presence of scenes involving social interactions between characters. Subsequent content analysis of video clips created from this response profile confirmed this finding. In contrast, regions of the inferotemporal and parietal cortex were selectively tuned to faces and actions, both features that often covary with social interaction but may be difficult to disentangle using standard event-related approaches. Together, these findings suggest that the DMPFC is finely tuned for processing social interaction above other categories and that this preference is maintained during unrestricted viewing of complex natural stimuli such as movies.

SIGNIFICANCE STATEMENT

Recently, studies have brought into question whether the dorsal medial prefrontal cortex (DMPFC), a region long associated with social cognition, is specialized for the processing of social information. We examine the response profile of this region during natural viewing of a reasonably naturalistic stimulus (i.e., a Hollywood movie) using a data-driven reverse correlation technique. Our findings demonstrate that, during natural viewing, the DMPFC is strongly tuned to the social features of the stimulus above other categories. Moreover, this response differs from other areas with previously well characterized response profiles such as the lateral and medial fusiform gyrus. These findings suggest that this region's dominant function in everyday situations is to support reasoning about the thoughts and intentions of conspecifics.

Measuring speaker-listener neural coupling with functional near infrared spectroscopy

The present study investigates brain-to-brain coupling, defined as inter-subject correlations in the hemodynamic response, during natural verbal communication. We used functional near-infrared spectroscopy (fNIRS) to record brain activity of 3 speakers telling stories and 15 listeners comprehending audio recordings of these stories. Listeners’ brain activity was significantly correlated with speakers’ with a delay. This between-brain correlation disappeared when verbal communication failed. We further compared the fNIRS and functional Magnetic Resonance Imaging (fMRI) recordings of listeners comprehending the same story and found a significant relationship between the fNIRS oxygenated-hemoglobin concentration changes and the fMRI BOLD in brain areas associated with speech comprehension. This correlation between fNIRS and fMRI was only present when data from the same story were compared between the two modalities and vanished when data from different stories were compared; this cross-modality consistency further highlights the reliability of the spatiotemporal brain activation pattern as a measure of story comprehension. Our findings suggest that fNIRS can be used for investigating brain-to-brain coupling during verbal communication in natural settings.

Precuneus functioning differentiates first-episode psychosis patients during the fantasy movie Alice in Wonderland

BACKGROUND

While group-level functional alterations have been identified in many brain regions of psychotic patients, multivariate machine-learning methods provide a tool to test whether some of such alterations could be used to differentiate an individual patient. Earlier machine-learning studies have focused on data collected from chronic patients during rest or simple tasks. We set out to unravel brain activation patterns during naturalistic stimulation in first-episode psychosis (FEP).

METHOD

We recorded brain activity from 46 FEP patients and 32 control subjects viewing scenes from the fantasy film Alice in Wonderland. Scenes with varying degrees of fantasy were selected based on the distortion of the 'sense of reality' in psychosis. After cleaning the data with a novel maxCorr method, we used machine learning to classify patients and healthy control subjects on the basis of voxel- and time-point patterns.

RESULTS

Most (136/194) of the voxels that best classified the groups were clustered in a bilateral region of the precuneus. Classification accuracies were up to 79.5% (p = 5.69 × 10-8), and correct classification was more likely the higher the patient's positive-symptom score. Precuneus functioning was related to the fantasy content of the movie, and the relationship was stronger in control subjects than patients.

CONCLUSIONS

These findings are the first to show abnormalities in precuneus functioning during naturalistic information processing in FEP patients. Correlational findings suggest that these alterations are associated with positive psychotic symptoms and processing of fantasy. The results may provide new insights into the neuronal basis of reality distortion in psychosis.

Test-retest reliability of functional connectivity networks during naturalistic fMRI paradigms

Functional connectivity analysis has become a powerful tool for probing the human brain function and its breakdown in neuropsychiatry disorders. So far, most studies adopted resting-state paradigm to examine functional connectivity networks in the brain, thanks to its low demand and high tolerance that are essential for clinical studies. However, the test-retest reliability of resting-state connectivity measures is moderate, potentially due to its low behavioral constraint. On the other hand, naturalistic neuroimaging paradigms, an emerging approach for cognitive neuroscience with high ecological validity, could potentially improve the reliability of functional connectivity measures. To test this hypothesis, we characterized the test-retest reliability of functional connectivity measures during a natural viewing condition, and benchmarked it against resting-state connectivity measures acquired within the same functional magnetic resonance imaging (fMRI) session. We found that the reliability of connectivity and graph theoretical measures of brain networks is significantly improved during natural viewing conditions over resting-state conditions, with an average increase of almost 50% across various connectivity measures. Not only sensory networks for audio-visual processing become more reliable, higher order brain networks, such as default mode and attention networks, but also appear to show higher reliability during natural viewing. Our results support the use of natural viewing paradigms in estimating functional connectivity of brain networks, and have important implications for clinical application of fMRI. Hum Brain Mapp 38:2226-2241, 2017. © 2017 Wiley Periodicals, Inc.

Mirroring and beyond: coupled dynamics as a generalized framework for modelling social interactions

When people observe one another, behavioural alignment can be detected at many levels, from the physical to the mental. Likewise, when people process the same highly complex stimulus sequences, such as films and stories, alignment is detected in the elicited brain activity. In early sensory areas, shared neural patterns are coupled to the low-level properties of the stimulus (shape, motion, volume, etc.), while in high-order brain areas, shared neural patterns are coupled to high-levels aspects of the stimulus, such as meaning. Successful social interactions require such alignments (both behavioural and neural), as communication cannot occur without shared understanding. However, we need to go beyond simple, symmetric (mirror) alignment once we start interacting. Interactions are dynamic processes, which involve continuous mutual adaptation, development of complementary behaviour and division of labour such as leader-follower roles. Here, we argue that interacting individuals are dynamically coupled rather than simply aligned. This broader framework for understanding interactions can encompass both processes by which behaviour and brain activity mirror each other (neural alignment), and situations in which behaviour and brain activity in one participant are coupled (but not mirrored) to the dynamics in the other participant. To apply these more sophisticated accounts of social interactions to the study of the underlying neural processes we need to develop new experimental paradigms and novel methods of data analysis.

Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex

During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation

Here we present an update of the studyforrest (http://studyforrest.org) dataset that complements the previously released functional magnetic resonance imaging (fMRI) data for natural language processing with a new two-hour 3 Tesla fMRI acquisition while 15 of the original participants were shown an audio-visual version of the stimulus motion picture. We demonstrate with two validation analyses that these new data support modeling specific properties of the complex natural stimulus, as well as a substantial within-subject BOLD response congruency in brain areas related to the processing of auditory inputs, speech, and narrative when compared to the existing fMRI data for audio-only stimulation. In addition, we provide participants' eye gaze location as recorded simultaneously with fMRI, and an additional sample of 15 control participants whose eye gaze trajectories for the entire movie were recorded in a lab setting—to enable studies on attentional processes and comparative investigations on the potential impact of the stimulation setting on these processes.

Neural Plasticity following Abacus Training in Humans: A Review and Future Directions

The human brain has an enormous capacity to adapt to a broad variety of environmental demands. Previous studies in the field of abacus training have shown that this training can induce specific changes in the brain. However, the neural mechanism underlying these changes remains elusive. Here, we reviewed the behavioral and imaging findings of comparisons between abacus experts and average control subjects and focused on changes in activation patterns and changes in brain structure. Finally, we noted the limitations and the future directions of this field. We concluded that although current studies have provided us with information about the mechanisms of abacus training, more research on abacus training is needed to understand its neural impact.

The reliability of continuous brain responses during naturalistic listening to music

Low-level (timbral) and high-level (tonal and rhythmical) musical features during continuous listening to music, studied by functional magnetic resonance imaging (fMRI), have been shown to elicit large-scale responses in cognitive, motor, and limbic brain networks. Using a similar methodological approach and a similar group of participants, we aimed to study the replicability of previous findings. Participants' fMRI responses during continuous listening of a tango Nuevo piece were correlated voxelwise against the time series of a set of perceptually validated musical features computationally extracted from the music. The replicability of previous results and the present study was assessed by two approaches: (a) correlating the respective activation maps, and (b) computing the overlap of active voxels between datasets at variable levels of ranked significance. Activity elicited by timbral features was better replicable than activity elicited by tonal and rhythmical ones. These results indicate more reliable processing mechanisms for low-level musical features as compared to more high-level features. The processing of such high-level features is probably more sensitive to the state and traits of the listeners, as well as of their background in music.

'My Virtual Dream': Collective Neurofeedback in an Immersive Art Environment

While human brains are specialized for complex and variable real world tasks, most neuroscience studies reduce environmental complexity, which limits the range of behaviours that can be explored. Motivated to overcome this limitation, we conducted a large-scale experiment with electroencephalography (EEG) based brain-computer interface (BCI) technology as part of an immersive multi-media science-art installation. Data from 523 participants were collected in a single night. The exploratory experiment was designed as a collective computer game where players manipulated mental states of relaxation and concentration with neurofeedback targeting modulation of relative spectral power in alpha and beta frequency ranges. Besides validating robust time-of-night effects, gender differences and distinct spectral power patterns for the two mental states, our results also show differences in neurofeedback learning outcome. The unusually large sample size allowed us to detect unprecedented speed of learning changes in the power spectrum (~ 1 min). Moreover, we found that participants' baseline brain activity predicted subsequent neurofeedback beta training, indicating state-dependent learning. Besides revealing these training effects, which are relevant for BCI applications, our results validate a novel platform engaging art and science and fostering the understanding of brains under natural conditions.

Out-of-sync: disrupted neural activity in emotional circuitry during film viewing in melancholic depression

While a rich body of research in controlled experiments has established changes in the neural circuitry of emotion in major depressive disorders, little is known as to how such alterations might translate into complex, naturalistic settings--namely involving dynamic multimodal stimuli with rich contexts, such as those provided by films. Neuroimaging paradigms employing dynamic natural stimuli alleviate the anxiety often associated with complex tasks and eschew the need for laboratory-style abstractions, hence providing an ecologically valid means of elucidating neural underpinnings of neuropsychiatric disorders. To probe the neurobiological signature of refined depression subtypes, we acquired functional neuroimaging data in patients with the melancholic subtype of major depressive disorder during free viewing of emotionally salient films. We found a marked disengagement of ventromedial prefrontal cortex during natural viewing of a film with negative emotional valence in patients with melancholia. This effect significantly correlated with depression severity. Such changes occurred on the background of diminished consistency of neural activity in visual and auditory sensory networks, as well as higher-order networks involved in emotion and attention, including bilateral intraparietal sulcus and right anterior insula. These findings may reflect a failure to re-allocate resources and diminished reactivity to external emotional stimuli in melancholia.

Portrayed emotions in the movie "Forrest Gump"

Here we present a dataset with a description of portrayed emotions in the movie ”Forrest Gump”. A total of 12 observers independently annotated emotional episodes regarding their temporal location and duration. The nature of an emotion was characterized with basic attributes, such as arousal and valence, as well as explicit emotion category labels. In addition, annotations include a record of the perceptual evidence for the presence of an emotion. Two variants of the movie were annotated separately: 1) an audio-movie version of Forrest Gump that has been used as a stimulus for the acquisition of a large public functional brain imaging dataset, and 2) the original audio-visual movie. We present reliability and consistency estimates that suggest that both stimuli can be used to study visual and auditory emotion cue processing in real-life like situations. Raw annotations from all observers are publicly released in full in order to maximize their utility for a wide range of applications and possible future extensions. In addition, aggregate time series of inter-observer agreement with respect to particular attributes of portrayed emotions are provided to facilitate adoption of these data.

A specific hypoactivation of right temporo-parietal junction/posterior superior temporal sulcus in response to socially awkward situations in autism

People with autism spectrum disorder (ASD) often have difficulty comprehending social situations in the complex, dynamic contexts encountered in the real world. To study the social brain under conditions which approximate naturalistic situations, we measured brain activity with FUNCTIONAL MAGNETIC RESONANCE IMAGING: while participants watched a full-length episode of the sitcom The Office. Having quantified the degree of social awkwardness at each moment of the episode, as judged by an independent sample of controls, we found that both individuals with ASD and control participants showed reliable activation of several brain regions commonly associated with social perception and cognition (e.g. those comprising the 'mentalizing network') during the more awkward moments. However, individuals with ASD showed less activity than controls in a region near right temporo-parietal junction (RTPJ) extending into the posterior end of the right superior temporal sulcus (RSTS). Further analyses suggested that, despite the free-form nature of the experimental design, this group difference was specific to this RTPJ/RSTS area of the mentalizing network; other regions of interest showed similar activity across groups with respect to both location and magnitude. These findings add support to a body of evidence suggesting that RTPJ/RSTS plays a special role in social processes across modalities and may function atypically in individuals with ASD navigating the social world.

The neuroscience of musical improvisation

Researchers have recently begun to examine the neural basis of musical improvisation, one of the most complex forms of creative behavior. The emerging field of improvisation neuroscience has implications not only for the study of artistic expertise, but also for understanding the neural underpinnings of domain-general processes such as motor control and language production. This review synthesizes functional magnetic resonance imagining (fMRI) studies of musical improvisation, including vocal and instrumental improvisation, with samples of jazz pianists, classical musicians, freestyle rap artists, and non-musicians. A network of prefrontal brain regions commonly linked to improvisatory behavior is highlighted, including the pre-supplementary motor area, medial prefrontal cortex, inferior frontal gyrus, dorsolateral prefrontal cortex, and dorsal premotor cortex. Activation of premotor and lateral prefrontal regions suggests that a seemingly unconstrained behavior may actually benefit from motor planning and cognitive control. Yet activation of cortical midline regions points to a role of spontaneous cognition characteristic of the default network. Together, such results may reflect cooperation between large-scale brain networks associated with cognitive control and spontaneous thought. The improvisation literature is integrated with Pressing's theoretical model, and discussed within the broader context of research on the brain basis of creative cognition.

Social in, social out: How the brain responds to social language with more social language

Social connection is a fundamental human need. As such, people's brains are sensitized to social cues, such as those carried by language, and to promoting social communication. The neural mechanisms of certain key building blocks in this process, such as receptivity to and reproduction of social language, however, are not known. We combined quantitative linguistic analysis and neuroimaging to connect neural activity in brain regions used to simulate the mental states of others with exposure to, and re-transmission of, social language. Our results link findings on successful idea transmission from communication science, sociolinguistics and cognitive neuroscience to prospectively predict the degree of social language that participants utilize when re-transmitting ideas as a function of 1) initial language inputs and 2) neural activity during idea exposure.

Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses

Story understanding involves many perceptual and cognitive subprocesses, from perceiving individual words, to parsing sentences, to understanding the relationships among the story characters. We present an integrated computational model of reading that incorporates these and additional subprocesses, simultaneously discovering their fMRI signatures. Our model predicts the fMRI activity associated with reading arbitrary text passages, well enough to distinguish which of two story segments is being read with 74% accuracy. This approach is the first to simultaneously track diverse reading subprocesses during complex story processing and predict the detailed neural representation of diverse story features, ranging from visual word properties to the mention of different story characters and different actions they perform. We construct brain representation maps that replicate many results from a wide range of classical studies that focus each on one aspect of language processing and offer new insights on which type of information is processed by different areas involved in language processing. Additionally, this approach is promising for studying individual differences: it can be used to create single subject maps that may potentially be used to measure reading comprehension and diagnose reading disorders.

Mind the gap: an attempt to bridge computational and neuroscientific approaches to study creativity

Creativity is the hallmark of human cognition and is behind every innovation, scientific discovery, piece of music, artwork, and idea that have shaped our lives, from ancient times till today. Yet scientific understanding of creative processes is quite limited, mostly due to the traditional belief that considers creativity as a mysterious puzzle, a paradox, defying empirical enquiry. Recently, there has been an increasing interest in revealing the neural correlates of human creativity. Though many of these studies, pioneering in nature, help demystification of creativity, but the field is still dominated by popular beliefs in associating creativity with “right brain thinking”, “divergent thinking”, “altered states” and so on (Dietrich and Kanso, 2010). In this article, we discuss a computational framework for creativity based on Baars’ Global Workspace Theory (GWT; Baars, 1988) enhanced with mechanisms based on information theory. Next we propose a neurocognitive architecture of creativity with a strong focus on various facets (i.e., unconscious thought theory, mind wandering, spontaneous brain states) of un/pre-conscious brain responses. Our principal argument is that pre-conscious creativity happens prior to conscious creativity and the proposed computational model may provide a mechanism by which this transition is managed. This integrative approach, albeit unconventional, will hopefully stimulate future neuroscientific studies of the inscrutable phenomenon of creativity.

EEG alpha power and creative ideation

Neuroscientific studies revealed first insights into neural mechanisms underlying creativity, but existing findings are highly variegated and often inconsistent. Despite the disappointing picture on the neuroscience of creativity drawn in recent reviews, there appears to be robust evidence that EEG alpha power is particularly sensitive to various creativity-related demands involved in creative ideation. Alpha power varies as a function of creativity-related task demands and the originality of ideas, is positively related to an individuals' creativity level, and has been observed to increase as a result of creativity interventions. Alpha increases during creative ideation could reflect more internally oriented attention that is characterized by the absence of external bottom-up stimulation and, thus, a form of top-down activity. Moreover, they could indicate the involvement of specific memory processes such as the efficient (re-)combination of unrelated semantic information. We conclude that increased alpha power during creative ideation is among the most consistent findings in neuroscientific research on creativity and discuss possible future directions to better understand the manifold brain mechanisms involved in creativity.