How can EEG/MEG and fMRI/PET data be combined?

Neural shifts in alpha rhythm's dual functioning during empathy maturation

INTRODUCTION

Empathy is a social-cognitive process that operates by relying mainly on the suppression of the cortical alpha rhythm. This phenomenon has been evidenced in dozens of electrophysiological studies targeting adult human subjects. Yet, recent neurodevelopmental studies indicated that at a younger age, empathy involves reversed brain responses (e.g., alpha enhancement patterns). In this multimodal study, we capture neural activity at the alpha range, and hemodynamic response and target subjects at approximately 20 years old as a unique time window in development that allows investigating both low-alpha suppression and high-alpha enhancement. We aim to further investigate the functional role of low-alpha power suppression and high-alpha power enhancement during empathy development.

METHODS

Brain data from 40 healthy individuals were recorded in two consecutive sessions of magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) while subjects perceived vicarious physical pain or no pain.

RESULTS

MEG revealed that the alpha pattern shift during empathy happens in an all-or-none pattern: power enhancement before 18 and suppression after 18 years of age. Additionally, MEG and fMRI highlight a correspondence between high-alpha power increase and blood-oxygen-level-dependent (BOLD) decrease before 18, but low-alpha power decrease and BOLD increase after 18. Importantly, this neurodevelopmental transition was not revealed by four other measures: self-reported (a) ratings of the task stimuli, (b) ratings of naturalistic vignettes of vicarious pain, (c) trait empathy, or neural data from (d) a control neuroimaging task.

DISCUSSION

Findings suggest that at the critical age of around 18, empathy is underpinned by an all-or-none transition from high-alpha power enhancement and functional inhibition to low-alpha power suppression and functional activation in particular brain regions, possibly indicating a marker of maturation in empathic ability. This work advances a recent neurodevelopmental line of studies and provides insight into the functional maturation of empathy at the coming of age.

Visual Selective Attention P300 Source in Frontal-Parietal Lobe: ERP and fMRI Study

Visual selective attention can be achieved into bottom-up and top-down attention. Different selective attention tasks involve different attention control ways. The pop-out task requires more bottom-up attention, whereas the search task involves more top-down attention. P300, which is the positive potential generated by the brain in the latency of 300 ~ 600 ms after stimulus, reflects the processing of attention. There is no consensus on the P300 source. The aim of present study is to study the source of P300 elicited by different visual selective attention. We collected thirteen participants' P300 elicited by pop-out and search tasks with event-related potentials (ERP). We collected twenty-six participants' activation brain regions in pop-out and search tasks with functional magnetic resonance imaging (fMRI). And we analyzed the sources of P300 using the ERP and fMRI integration with high temporal resolution and high spatial resolution. ERP results indicated that the pop-out task induced larger P300 than the search task. P300 induced by the two tasks distributed at frontal and parietal lobes, with P300 induced by the pop-out task mainly at the parietal lobe and that induced by the search task mainly at the frontal lobe. Further ERP and fMRI integration analysis showed that neural difference sources of P300 were the right precentral gyrus, left superior frontal gyrus (medial orbital), left middle temporal gyrus, left rolandic operculum, right postcentral gyrus, and left angular gyrus. Our study suggests that the frontal and parietal lobes contribute to the P300 component of visual selective attention.

Structured sparse multiset canonical correlation analysis of simultaneous fNIRS and EEG provides new insights into the human action-observation network

The action observation network (AON) is a network of brain regions involved in the execution and observation of a given action. The AON has been investigated in humans using mostly electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), but shared neural correlates of action observation and action execution are still unclear due to lack of ecologically valid neuroimaging measures. In this study, we used concurrent EEG and functional Near Infrared Spectroscopy (fNIRS) to examine the AON during a live-action observation and execution paradigm. We developed structured sparse multiset canonical correlation analysis (ssmCCA) to perform EEG-fNIRS data fusion. MCCA is a generalization of CCA to more than two sets of variables and is commonly used in medical multimodal data fusion. However, mCCA suffers from multi-collinearity, high dimensionality, unimodal feature selection, and loss of spatial information in interpreting the results. A limited number of participants (small sample size) is another problem in mCCA, which leads to overfitted models. Here, we adopted graph-guided (structured) fused least absolute shrinkage and selection operator (LASSO) penalty to mCCA to conduct feature selection, incorporating structural information amongst the variables (i.e., brain regions). Benefitting from concurrent recordings of brain hemodynamic and electrophysiological responses, the proposed ssmCCA finds linear transforms of each modality such that the correlation between their projections is maximized. Our analysis of 21 right-handed participants indicated that the left inferior parietal region was active during both action execution and action observation. Our findings provide new insights into the neural correlates of AON which are more fine-tuned than the results from each individual EEG or fNIRS analysis and validate the use of ssmCCA to fuse EEG and fNIRS datasets.

Wearable, Integrated EEG-fNIRS Technologies: A Review

There has been considerable interest in applying electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) simultaneously for multimodal assessment of brain function. EEG–fNIRS can provide a comprehensive picture of brain electrical and hemodynamic function and has been applied across various fields of brain science. The development of wearable, mechanically and electrically integrated EEG–fNIRS technology is a critical next step in the evolution of this field. A suitable system design could significantly increase the data/image quality, the wearability, patient/subject comfort, and capability for long-term monitoring. Here, we present a concise, yet comprehensive, review of the progress that has been made toward achieving a wearable, integrated EEG–fNIRS system. Significant marks of progress include the development of both discrete component-based and microchip-based EEG–fNIRS technologies; modular systems; miniaturized, lightweight form factors; wireless capabilities; and shared analogue-to-digital converter (ADC) architecture between fNIRS and EEG data acquisitions. In describing the attributes, advantages, and disadvantages of current technologies, this review aims to provide a roadmap toward the next generation of wearable, integrated EEG–fNIRS systems.

Spatial attention modulates earliest visual processing: An electrical neuroimaging study

Several studies showed that shifting of visuospatial attention modulates sensory processing at multiple levels of the visual pathways and beyond, including the occipital striate cortices level. However, inconsistent findings have been reported thus leaving these issues still disputed. 21 participants took part to the present study (the EEG signals of 4 of them were discarded due to artifacts). We used ERPs and their neural sources to investigate whether shifting spatial attention in space across the horizontal meridian of the visual field affected striate cortices activation at the earliest latency. Time-series of scalp topographical maps indicated that, unlike ERPs to attentional-neutral central cues, ERPs to attention-directing local cues showed earliest polarity inversions as a function of stimulated field and processing latency range considered, at occipital-parietal electrodes. In between 60-75 ms, attentional shifting cues elicited a positivity for both visual fields, whereas at a later latency (75–90 ms) they elicited a positivity and a negativity for the upper and lower visual hemifields, respectively. Computed neural sources included striate, besides extrastriate, cortices for both visual hemifields and latency ranges. Conjointly, behavioral responses to targets were faster when they were preceded by local than by neutral cues, and when presented in the upper than the lower hemifield. Our findings support the hypothesis that attention shifts may affect early sensory processing in visual cortices.

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

Hand gestures are a form of non-verbal communication used by individuals in conjunction with speech to communicate. Nowadays, with the increasing use of technology, hand-gesture recognition is considered to be an important aspect of Human-Machine Interaction (HMI), allowing the machine to capture and interpret the user's intent and to respond accordingly. The ability to discriminate between human gestures can help in several applications, such as assisted living, healthcare, neuro-rehabilitation, and sports. Recently, multi-sensor data fusion mechanisms have been investigated to improve discrimination accuracy. In this paper, we present a sensor fusion framework that integrates complementary systems: the electromyography (EMG) signal from muscles and visual information. This multi-sensor approach, while improving accuracy and robustness, introduces the disadvantage of high computational cost, which grows exponentially with the number of sensors and the number of measurements. Furthermore, this huge amount of data to process can affect the classification latency which can be crucial in real-case scenarios, such as prosthetic control. Neuromorphic technologies can be deployed to overcome these limitations since they allow real-time processing in parallel at low power consumption. In this paper, we present a fully neuromorphic sensor fusion approach for hand-gesture recognition comprised of an event-based vision sensor and three different neuromorphic processors. In particular, we used the event-based camera, called DVS, and two neuromorphic platforms, Loihi and ODIN + MorphIC. The EMG signals were recorded using traditional electrodes and then converted into spikes to be fed into the chips. We collected a dataset of five gestures from sign language where visual and electromyography signals are synchronized. We compared a fully neuromorphic approach to a baseline implemented using traditional machine learning approaches on a portable GPU system. According to the chip's constraints, we designed specific spiking neural networks (SNNs) for sensor fusion that showed classification accuracy comparable to the software baseline. These neuromorphic alternatives have increased inference time, between 20 and 40%, with respect to the GPU system but have a significantly smaller energy-delay product (EDP) which makes them between 30× and 600× more efficient. The proposed work represents a new benchmark that moves neuromorphic computing toward a real-world scenario.

From Correlational Signs to Markers. Current Trends in Neuroelectric Research on Visual Attentional Processing

Traditionally, electroencephalographic (EEG) and event-related brain potentials (ERPs) research on visual attentional processing attempted to account for mental processes in conceptual terms without reference to the way in which they were physically realized by the anatomical structures and physiological processes of the human brain. The brain science level of analysis, in contrast, attempted to explain the brain as an information processing system and to explain mental events in terms of brain processes. Somehow overcoming the separation between the two abovementioned levels of analysis, the cognitive neuroscience level considered how information was represented and processed in the brain. Neurofunctional processing takes place in a fraction of a second. Hence, the very high time resolution and the reliable sensitivity of EEG and ERPs in detecting fast functional changes in brain activity provided advantages over hemodynamic imaging techniques such as positron emission tomography (PET) or functional magnetic resonance imaging (fMRI), as well as over behavioral measures. However, volume conduction and lack of three-dimensionality limited applications of EEG and ERPs per se more than hemodynamic techniques for revealing locations in which brain processing occurs. These limits could only be overcome by subtraction methods for isolating attentional effects that might endure over time in EEG and may be riding even over several different ERP components, and by intracerebral single and distributed electric source analyses as well as the combining of these signals with high-spatial resolution hemodynamic signals (fMRI), both in healthy individuals and clinical patients. In my view, the articles of the Special Issue concerned with "ERP and EEG Markers of Brain Visual Attentional Processing" of the present journal Brain Sciences provide very good examples of all these levels of analysis.

Integrating Space, Time, and Orientation in Spiking Neural Networks: A Case Study on Multimodal Brain Data Modeling

Recent progress in a noninvasive brain data sampling technology has facilitated simultaneous sampling of multiple modalities of brain data, such as functional magnetic resonance imaging, electroencephalography, diffusion tensor imaging, and so on. In spite of the potential benefits from integrating predictive modeling of multiple modality brain data, this area of research remains mostly unexplored due to a lack of methodological advancements. The difficulty in fusing multiple modalities of brain data within a single model lies in the heterogeneous temporal and spatial characteristics of the data sources. Recent advances in spiking neural network systems, however, provide the flexibility to incorporate multidimensional information within the model. This paper proposes a novel, unsupervised learning algorithm for fusing temporal, spatial, and orientation information in a spiking neural network architecture that could potentially be used to understand and perform predictive modeling using multimodal data. The proposed algorithm is evaluated both qualitatively and quantitatively using synthetically generated data to characterize its behavior and its ability to utilize spatial, temporal, and orientation information within the model. This leads to improved pattern recognition capabilities and performance along with robust interpretability of the brain data. Furthermore, a case study is presented, which aims to build a computational model that discriminates between people with schizophrenia who respond or do not respond to monotherapy with the antipsychotic clozapine.

A Comprehensive Review of Magnetoencephalography (MEG) Studies for Brain Functionality in Healthy Aging and Alzheimer's Disease (AD)

Neural oscillations were established with their association with neurophysiological activities and the altered rhythmic patterns are believed to be linked directly to the progression of cognitive decline. Magnetoencephalography (MEG) is a non-invasive technique to record such neuronal activity due to excellent temporal and fair amount of spatial resolution. Single channel, connectivity as well as brain network analysis using MEG data in resting state and task-based experiments were analyzed from existing literature. Single channel analysis studies reported a less complex, more regular and predictable oscillations in Alzheimer's disease (AD) primarily in the left parietal, temporal and occipital regions. Investigations on both functional connectivity (FC) and effective (EC) connectivity analysis demonstrated a loss of connectivity in AD compared to healthy control (HC) subjects found in higher frequency bands. It has been reported from multiplex network of MEG study in AD in the affected regions of hippocampus, posterior default mode network (DMN) and occipital areas, however, conclusions cannot be drawn due to limited availability of clinical literature. Potential utilization of high spatial resolution in MEG likely to provide information related to in-depth brain functioning and underlying factors responsible for changes in neuronal waves in AD. This review is a comprehensive report to investigate diagnostic biomarkers for AD may be identified by from MEG data. It is also important to note that MEG data can also be utilized for the same pursuit in combination with other imaging modalities.

Investigation of True High Frequency Electrical Substrates of fMRI-Based Resting State Networks Using Parallel Independent Component Analysis of Simultaneous EEG/fMRI Data

Previous work using simultaneously acquired electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data has shown that the slow temporal dynamics of resting state brain networks (RSNs), e.g., default mode network (DMN), visual network (VN), obtained from fMRI are correlated with smoothed and down sampled versions of various EEG features such as microstates and band-limited power envelopes. Therefore, even though the down sampled and smoothed envelope of EEG gamma band power is correlated with fMRI fluctuations in the RSNs, it does not mean that the electrical substrates of the RSNs fluctuate with periods <100 ms. Based on the scale free properties of EEG microstates and their correlation with resting state fMRI fluctuations in the RSNs, researchers have speculated that truly high frequency electrical substrates may exist for the RSNs, which would make resting fluctuations obtained from fMRI more meaningful to typically occurring fast neuronal processes in the sub-100 ms time scale. In this study, we test this critical hypothesis using an integrated framework involving simultaneous EEG/fMRI acquisition, fast fMRI sampling (TR = 200 ms) using multiband EPI (MB EPI), and EEG/fMRI fusion using parallel independent component analysis (pICA) which does not require the down sampling of EEG to fMRI temporal resolution. Our results demonstrate that with faster sampling, high frequency electrical substrates (fluctuating with periods <100 ms time scale) of the RSNs can be observed. This provides a sounder neurophysiological basis for the RSNs.

Overview of MEG

Magnetoencephalography (MEG) is a method to study electrical activity in the human brain by recording the neuromagnetic field outside the head. MEG, like electroencephalography (EEG), provides an excellent, millisecond-scale time resolution, and allows the estimation of the spatial distribution of the underlying activity, in favorable cases with a localization accuracy of a few millimeters. To detect the weak neuromagnetic signals, superconducting sensors, magnetically shielded rooms, and advanced signal processing techniques are used. The analysis and interpretation of MEG data typically involves comparisons between subject groups and experimental conditions using various spatial, temporal, and spectral measures of cortical activity and connectivity. The application of MEG to cognitive neuroscience studies is illustrated with studies of spoken language processing in subjects with normal and impaired reading ability. The mapping of spatiotemporal patterns of activity within networks of cortical areas can provide useful information about the functional architecture of the brain related to sensory and cognitive processing, including language, memory, attention, and perception.

Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness

It is becoming increasingly clear that combining multi-modal brain imaging data is able to provide more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e. capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multi-modal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this paper, we start by introducing the basic reasons why multimodal data fusion is important and what it can do, and importantly how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multi-modal fusion including deep learning and multimodal classification which show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential in order to mitigate against misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.

Understanding the Neuropsychology of Aesthetic Paradox: The Dual Phase Oscillation Hypothesis

Aesthetic delight is a unique and paradoxical psychological experience of simultaneous emotional exaltation and a state of serenity toward a percept when an individual experiences the percept with the approach of an art experiencer or artist. The primary drawback of neuroscientific investigation of art is that it fails to trace the functional coherence of the entire process of generation of aesthetic delight. At least 2 recent seminal works by Vessel et al. (2013) and Cela-Conde et al. (2013) tried to resolve this deficiency using 2 different neuroimaging techniques (fMRI and MEG respectively) and assessed the relevance of the Default Mode Network (DMN) of the brain in the generation of aesthetic pleasure. However, their works are yet to precisely highlight what primarily separates aesthetic experience from similar psychological experiences. This article formulates the dual phase oscillation hypothesis based on the neural correlate of the paradox of aesthetic delight, explaining the precise logic behind linking aesthetic delight and DMN activity. The hypothesis focuses on 2 seemingly paradoxical unique attributes of aesthetic delight: the phenomenon of suspension of disbelief (SOD), whereby the person experiencing art temporarily suspends the belief of surface reality, and the phenomenon of introspective detached contemplation, whereby the same person, while experiencing the same art, reflects on the artistic phenomenon and is simultaneously aware of the surface reality. The hypothesis proposes that aesthetic delight is the dynamic, oscillatory balance between SOD and introspective detached contemplation and is orchestrated by the functional coherence of the DMN. The article integrates the 2 previous works with the fundamentals of this proposal and thus offers a unique neuropsychological solution to the problem of aesthetic paradox.

Single-trial EEG-informed fMRI reveals spatial dependency of BOLD signal on early and late IC-ERP amplitudes during face recognition

Simultaneous EEG-fMRI has opened up new avenues for improving the spatio-temporal resolution of functional brain studies. However, this method usually suffers from poor EEG quality, especially for evoked potentials (ERPs), due to specific artifacts. As such, the use of EEG-informed fMRI analysis in the context of cognitive studies has particularly focused on optimizing narrow ERP time windows of interest, which ignores the rich diverse temporal information of the EEG signal. Here, we propose to use simultaneous EEG-fMRI to investigate the neural cascade occurring during face recognition in 14 healthy volunteers by using the successive ERP peaks recorded during the cognitive part of this process. N170, N400 and P600 peaks, commonly associated with face recognition, were successfully and reproducibly identified for each trial and each subject by using a group independent component analysis (ICA). For the first time we use this group ICA to extract several independent components (IC) corresponding to the sequence of activation and used single-trial peaks as modulation parameters in a general linear model (GLM) of fMRI data. We obtained an occipital-temporal-frontal stream of BOLD signal modulation, in accordance with the three successive IC-ERPs providing an unprecedented spatio-temporal characterization of the whole cognitive process as defined by BOLD signal modulation. By using this approach, the pattern of EEG-informed BOLD modulation provided improved characterization of the network involved than the fMRI-only analysis or the source reconstruction of the three ERPs; the latter techniques showing only two regions in common localized in the occipital lobe.

Present status and future challenges of electroencephalography- and magnetic resonance imaging-based monitoring in preclinical models of focal cerebral ischemia

Animal models are useful tools for better understanding the mechanisms underlying neurological deterioration after an ischemic insult as well as subsequent evolution of changes and recovery of functions. In response to the updated requirements for preclinical investigations of stroke to include relevant functional measurement techniques and biomarker endpoints, we here review the state of knowledge on application of some translational electrophysiological and neuroimaging methods, and in particular, electroencephalography monitoring and magnetic resonance imaging in rodent models of ischemic stroke. This may lead to improvement of diagnostic methods and identification of new therapeutic targets, which would considerably advance the translational value of preclinical stroke research.

High-order interactions observed in multi-task intrinsic networks are dominant indicators of aberrant brain function in schizophrenia

Identifying the complex activity relationships present in rich, modern neuroimaging data sets remains a key challenge for neuroscience. The problem is hard because (a) the underlying spatial and temporal networks may be nonlinear and multivariate and (b) the observed data may be driven by numerous latent factors. Further, modern experiments often produce data sets containing multiple stimulus contexts or tasks processed by the same subjects. Fusing such multi-session data sets may reveal additional structure, but raises further statistical challenges. We present a novel analysis method for extracting complex activity networks from such multifaceted imaging data sets. Compared to previous methods, we choose a new point in the trade-off space, sacrificing detailed generative probability models and explicit latent variable inference in order to achieve robust estimation of multivariate, nonlinear group factors ("network clusters"). We apply our method to identify relationships of task-specific intrinsic networks in schizophrenia patients and control subjects from a large fMRI study. After identifying network-clusters characterized by within- and between-task interactions, we find significant differences between patient and control groups in interaction strength among networks. Our results are consistent with known findings of brain regions exhibiting deviations in schizophrenic patients. However, we also find high-order, nonlinear interactions that discriminate groups but that are not detected by linear, pairwise methods. We additionally identify high-order relationships that provide new insights into schizophrenia but that have not been found by traditional univariate or second-order methods. Overall, our approach can identify key relationships that are missed by existing analysis methods, without losing the ability to find relationships that are known to be important.

Neurophysiological techniques in the study of the excitability, connectivity, and plasticity of the human brain

There is increasing evidence to support the concept that brain plasticity involves distinct functional and structural components, each requiring several cellular mechanisms operating at different time scales, synaptic loci, and developmental phases within an extremely complex framework. However, the precise relationship between functional and structural components of brain plasticity/connectivity phenomena is still unclear and its explanation represents a major challenge within modern neuroscience. The key feature of neurophysiological techniques described in this review paper is their pivotal role in tracking temporal dynamics and inner hierarchies of brain functional and effective connectivities, possibly clarifying some crucial issues underlying brain plasticity. Taken together, the findings presented in this review open an intriguing new field in neuroscience investigation and are important for the adoption of neurophysiological techniques as a tool for basic research and, in future, even for clinical diagnostics purposes.

The virtual brain integrates computational modeling and multimodal neuroimaging

Brain function is thought to emerge from the interactions among neuronal populations. Apart from traditional efforts to reproduce brain dynamics from the micro- to macroscopic scales, complementary approaches develop phenomenological models of lower complexity. Such macroscopic models typically generate only a few selected-ideally functionally relevant-aspects of the brain dynamics. Importantly, they often allow an understanding of the underlying mechanisms beyond computational reproduction. Adding detail to these models will widen their ability to reproduce a broader range of dynamic features of the brain. For instance, such models allow for the exploration of consequences of focal and distributed pathological changes in the system, enabling us to identify and develop approaches to counteract those unfavorable processes. Toward this end, The Virtual Brain (TVB) ( www.thevirtualbrain.org ), a neuroinformatics platform with a brain simulator that incorporates a range of neuronal models and dynamics at its core, has been developed. This integrated framework allows the model-based simulation, analysis, and inference of neurophysiological mechanisms over several brain scales that underlie the generation of macroscopic neuroimaging signals. In this article, we describe how TVB works, and we present the first proof of concept.

Bayesian inference of structural brain networks

Structural brain networks are used to model white-matter connectivity between spatially segregated brain regions. The presence, location and orientation of these white matter tracts can be derived using diffusion-weighted magnetic resonance imaging in combination with probabilistic tractography. Unfortunately, as of yet, none of the existing approaches provide an undisputed way of inferring brain networks from the streamline distributions which tractography produces. State-of-the-art methods rely on an arbitrary threshold or, alternatively, yield weighted results that are difficult to interpret. In this paper, we provide a generative model that explicitly describes how structural brain networks lead to observed streamline distributions. This allows us to draw principled conclusions about brain networks, which we validate using simultaneously acquired resting-state functional MRI data. Inference may be further informed by means of a prior which combines connectivity estimates from multiple subjects. Based on this prior, we obtain networks that significantly improve on the conventional approach.

Effective Connectivity Modeling for fMRI: Six Issues and Possible Solutions Using Linear Dynamic Systems

Analysis of directionally specific or causal interactions between regions in functional magnetic resonance imaging (fMRI) data has proliferated. Here we identify six issues with existing effective connectivity methods that need to be addressed. The issues are discussed within the framework of linear dynamic systems for fMRI (LDSf). The first concerns the use of deterministic models to identify inter-regional effective connectivity. We show that deterministic dynamics are incapable of identifying the trial-to-trial variability typically investigated as the marker of connectivity while stochastic models can capture this variability. The second concerns the simplistic (constant) connectivity modeled by most methods. Connectivity parameters of the LDSf model can vary at the same timescale as the input data. Further, extending LDSf to mixtures of multiple models provides more robust connectivity variation. The third concerns the correct identification of the network itself including the number and anatomical origin of the network nodes. Augmentation of the LDSf state space can identify additional nodes of a network. The fourth concerns the locus of the signal used as a "node" in a network. A novel extension LDSf incorporating sparse canonical correlations can select most relevant voxels from an anatomically defined region based on connectivity. The fifth concerns connection interpretation. Individual parameter differences have received most attention. We present alternative network descriptors of connectivity changes which consider the whole network. The sixth concerns the temporal resolution of fMRI data relative to the timescale of the inter-regional interactions in the brain. LDSf includes an "instantaneous" connection term to capture connectivity occurring at timescales faster than the data resolution. The LDS framework can also be extended to statistically combine fMRI and EEG data. The LDSf framework is a promising foundation for effective connectivity analysis.

Modeling habituation in rat EEG-evoked responses via a neural mass model with feedback

Habituation is a generic property of the neural response to repeated stimuli. Its strength often increases as inter-stimuli relaxation periods decrease. We propose a simple, broadly applicable control structure that enables a neural mass model of the evoked EEG response to exhibit habituated behavior. A key motivation for this investigation is the ongoing effort to develop model-based reconstruction of multi-modal functional neuroimaging data. The control structure proposed here is illustrated and validated in the context of a biophysical neural mass model, developed by Riera et al. (Hum Brain Mapp 27(11):896-914, 2006; 28(4):335-354, 2007), and of simplifications thereof, using data from rat EEG response to medial nerve stimuli presented at frequencies from 1 to 8 Hz. Performance was tested by predictions of both the response to the next stimulus based on the current one, and also of continued stimuli trains over 4-s time intervals based on the first stimulus in the interval, with similar success statistics. These tests demonstrate the ability of simple generative models to capture key features of the evoked response, including habituation.

EEG-fMRI integration: a critical review of biophysical modeling and data analysis approaches

The diverse nature of cerebral activity, as measured using neuroimaging techniques, has been recognised long ago. It seems obvious that using single modality recordings can be limited when it comes to capturing its complex nature. Thus, it has been argued that moving to a multimodal approach will allow neuroscientists to better understand the dynamics and structure of this activity. This means that integrating information from different techniques, such as electroencephalography (EEG) and the blood oxygenated level dependent (BOLD) signal recorded with functional magnetic resonance imaging (fMRI), represents an important methodological challenge. In this work, we review the work that has been done thus far to derive EEG/fMRI integration approaches. This leads us to inspect the conditions under which such an integration approach could work or fail, and to disclose the types of scientific questions one could (and could not) hope to answer with it.

A review of multivariate methods for multimodal fusion of brain imaging data

The development of various neuroimaging techniques is rapidly improving the measurements of brain function/structure. However, despite improvements in individual modalities, it is becoming increasingly clear that the most effective research approaches will utilize multi-modal fusion, which takes advantage of the fact that each modality provides a limited view of the brain. The goal of multi-modal fusion is to capitalize on the strength of each modality in a joint analysis, rather than a separate analysis of each. This is a more complicated endeavor that must be approached more carefully and efficient methods should be developed to draw generalized and valid conclusions from high dimensional data with a limited number of subjects. Numerous research efforts have been reported in the field based on various statistical approaches, e.g. independent component analysis (ICA), canonical correlation analysis (CCA) and partial least squares (PLS). In this review paper, we survey a number of multivariate methods appearing in previous multimodal fusion reports, mostly fMRI with other modality, which were performed with or without prior information. A table for comparing optimization assumptions, purpose of the analysis, the need of priors, dimension reduction strategies and input data types is provided, which may serve as a valuable reference that helps readers understand the trade-offs of the 7 methods comprehensively. Finally, we evaluate 3 representative methods via simulation and give some suggestions on how to select an appropriate method based on a given research.

Towards a model-based integration of co-registered electroencephalography/functional magnetic resonance imaging data with realistic neural population meshes

Brain activity can be measured with several non-invasive neuroimaging modalities, but each modality has inherent limitations with respect to resolution, contrast and interpretability. It is hoped that multimodal integration will address these limitations by using the complementary features of already available data. However, purely statistical integration can prove problematic owing to the disparate signal sources. As an alternative, we propose here an advanced neural population model implemented on an anatomically sound cortical mesh with freely adjustable connectivity, which features proper signal expression through a realistic head model for the electroencephalogram (EEG), as well as a haemodynamic model for functional magnetic resonance imaging based on blood oxygen level dependent contrast (fMRI BOLD). It hence allows simultaneous and realistic predictions of EEG and fMRI BOLD from the same underlying model of neural activity. As proof of principle, we investigate here the influence on simulated brain activity of strengthening visual connectivity. In the future we plan to fit multimodal data with this neural population model. This promises novel, model-based insights into the brain's activity in sleep, rest and task conditions.

A parametric empirical Bayesian framework for fMRI-constrained MEG/EEG source reconstruction

We describe an asymmetric approach to fMRI and MEG/EEG fusion in which fMRI data are treated as empirical priors on electromagnetic sources, such that their influence depends on the MEG/EEG data, by virtue of maximizing the model evidence. This is important if the causes of the MEG/EEG signals differ from those of the fMRI signal. Furthermore, each suprathreshold fMRI cluster is treated as a separate prior, which is important if fMRI data reflect neural activity arising at different times within the EEG/MEG data. We present methodological considerations when mapping from a 3D fMRI Statistical Parametric Map to a 2D cortical surface and thence to the covariance components used within our Parametric Empirical Bayesian framework. Our previous introduction of a canonical (inverse‐normalized) cortical mesh also allows deployment of fMRI priors that live in a template space; for example, from a group analysis of different individuals. We evaluate the ensuing scheme with MEG and EEG data recorded simultaneously from 12 participants, using the same face‐processing paradigm under which independent fMRI data were obtained. Because the fMRI priors become part of the generative model, we use the model evidence to compare (i) multiple versus single, (ii) valid versus invalid, (iii) binary versus continuous, and (iv) variance versus covariance fMRI priors. For these data, multiple, valid, binary, and variance fMRI priors proved best for a standard Minimum Norm inversion. Interestingly, however, inversion using Multiple Sparse Priors benefited little from additional fMRI priors, suggesting that they already provide a sufficiently flexible generative model. Hum Brain Mapp, 2010. © 2010 Wiley‐Liss, Inc.

Challenges and opportunities for drug discovery in psychiatric disorders: the drug hunters' perspective

Innovation is essential for the identification of novel pharmacological therapies to meet the treatment needs of patients with psychiatric disorders. However, over the last 20 yr, in spite of major investments targets falling outside the classical aminergic mechanisms have shown diminished returns. The disappointments are traced to failures in the target identification and target validation effort, as reflected by the poor ability of current bioassays and animal models to predict efficacy and side-effects. Mismatch between disease biology and how psychiatric diseases are categorized has resulted in clinical trials of highly specific agents in heterogeneous patients, leading to variable treatment effects and failed studies. As drug hunters, one sees the opportunity to overhaul the pharmaceutical research and development (R&D) process. Improvements in both preclinical and clinical translational research need to be considered. Linking pharmacodynamic markers with disease biology should provide more predictive and innovative early clinical trials which in turn will increase the success rate of discovering new medicines. However, to exploit these exciting scientific discoveries, pharmaceutical companies need to question the conventional drug research and development model which is silo-driven, non-integrative across the confines of a company, non-disclosing across the pharmaceutical industry, and often independent from academia. This leads to huge redundancy in effort and lack of contextual learning in real time. Nevertheless, there are signs that drug discovery in the 21st century will see more intentional government, academic and industrial collaborations to overcome the above challenges that could eventually link mechanistic disease biology to segments of patients, affording them the benefits of rational and targeted therapy.

Connecting mean field models of neural activity to EEG and fMRI data

Progress in functional neuroimaging of the brain increasingly relies on the integration of data from complementary imaging modalities in order to improve spatiotemporal resolution and interpretability. However, the usefulness of merely statistical combinations is limited, since neural signal sources differ between modalities and are related non-trivially. We demonstrate here that a mean field model of brain activity can simultaneously predict EEG and fMRI BOLD with proper signal generation and expression. Simulations are shown using a realistic head model based on structural MRI, which includes both dense short-range background connectivity and long-range specific connectivity between brain regions. The distribution of modeled neural masses is comparable to the spatial resolution of fMRI BOLD, and the temporal resolution of the modeled dynamics, importantly including activity conduction, matches the fastest known EEG phenomena. The creation of a cortical mean field model with anatomically sound geometry, extensive connectivity, and proper signal expression is an important first step towards the model-based integration of multimodal neuroimages.

Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI

Functional magnetic resonance imaging (fMRI) data and electroencephalography (EEG) data provide complementary spatio-temporal information about brain function. Methods to couple the relative strengths of these modalities usually involve two stages: first forming a feature set from each dataset based on one criterion followed by exploration of connections among the features using a second criterion. We propose a data fusion method for simultaneously acquired fMRI and EEG data that combines these steps using a single criterion for finding the cross-modality associations and performing source separation. Using multi-set canonical correlation analysis (M-CCA), we obtain a decomposition of the two modalities, into spatial maps for fMRI data and a corresponding temporal evolution for EEG data, based on trial-to-trial covariation across the two modalities. Additionally, the analysis is performed on data from a group of subjects in order to make group inferences about the covariation across modalities. Being multivariate, the proposed method facilitates the study of brain connectivity along with localization of brain function. M-CCA can be easily extended to incorporate different data types and additional modalities. We demonstrate the promise of the proposed method in finding covarying trial-to-trial amplitude modulations (AMs) in an auditory task involving implicit pattern learning. The results show approximately linear decreasing trends in AMs for both modalities and the corresponding spatial activations occur mainly in motor, frontal, temporal, inferior parietal, and orbito-frontal areas that are linked both to sensory function as well as learning and expectation--all of which match activations related to the presented paradigm.

Spatiotemporal dipole source localization of face processing ERPs in adolescents: a preliminary study

Background

Despite extensive investigation of the neural systems for face perception and emotion recognition in adults and young children in the past, the precise temporal activation of brain sources specific to the processing of emotional facial expressions in older children and adolescents is not well known. This preliminary study aims to trace the spatiotemporal dynamics of facial emotion processing during adolescence and provide a basis for future developmental studies and comparisons with patient populations that have social-emotional deficits such as autism.

Methods

We presented pictures showing happy, angry, fearful, or neutral facial expressions to healthy adolescents (aged 10–16 years) and recorded 128-channel event-related potentials (ERPs) while they performed an emotion discrimination task. ERP components were analyzed for effects of age and emotion on amplitude and latency. The underlying cortical sources of scalp ERP activity were modeled as multiple equivalent current dipoles using Brain Electrical Source Analysis (BESA).

Results

Initial global/holistic processing of faces (P1) took place in the visual association cortex (lingual gyrus) around 120 ms post-stimulus. Next, structural encoding of facial features (N170) occurred between 160–200 ms in the inferior temporal/fusiform region, and perhaps early emotion processing (Vertex Positive Potential or VPP) in the amygdala and orbitofrontal cortex. Finally, cognitive analysis of facial expressions (P2) in the prefrontal cortex and emotional reactions in somatosensory areas were observed from about 230 ms onwards. The temporal sequence of cortical source activation in response to facial emotion processing was occipital, prefrontal, fusiform, parietal for young adolescents and occipital, limbic, inferior temporal, and prefrontal for older adolescents.

Conclusion

This is a first report of high-density ERP dipole source analysis in healthy adolescents which traces the sequence of neural activity within the first 500 ms of categorizing emotion from faces. Our spatio-temporal brain source models showed the presence of adult-like cortical networks for face processing in adolescents, whose functional specificity to different emotions appear to be not yet fully mature. Age-related differences in brain activation patterns illustrate the continued development and maturation of distinct neural systems for processing facial expressions during adolescence and possible changes in emotion perception, experience, and reaction with age.

Mining EEG-fMRI using independent component analysis

Independent component analysis (ICA) is a multivariate approach that has become increasingly popular for analyzing brain imaging data. In contrast to the widely used general linear model (GLM) that requires the user to parameterize the brain's response to stimuli, ICA allows the researcher to explore the factors that constitute the data and alleviates the need for explicit spatial and temporal priors about the responses. In this paper, we introduce ICA for hemodynamic (fMRI) and electrophysiological (EEG) data processing, and one of the possible extensions to the population level that is available for both data types. We then selectively review some work employing ICA for the decomposition of EEG and fMRI data to facilitate the integration of the two modalities to provide an overview of what is available and for which purposes ICA has been used. An optimized method for symmetric EEG-fMRI decomposition is proposed and the outstanding challenges in multimodal integration are discussed.

Functional imaging in Huntington's disease

Huntington's disease (HD) is a genetic brain disease characterized by loss of capacity in movement control, cognition, and emotional regulation over a period of about 30 years. Since it is well established that clinical impairments and brain atrophy can be detected decades prior to receiving a clinical diagnosis, functional neuroimaging efforts have gained momentum in HD research. In most brain disorders, there is accumulating evidence that the clinical manifestations of disease do not simply depend on the extent of tissue loss, but represent a complex balance among neuronal dysfunction, tissue repair, and circuitry reorganization. Based upon this premise, functional neuroimaging modalities may be more sensitive to the earliest changes in HD than are structural imaging approaches. For this review, PET and fMRI studies conducted in HD samples were summarized. Strengths and limitations of the utilization of functional imaging in HD are discussed and recommendations are offered to facilitate future research endeavors.

A review of group ICA for fMRI data and ICA for joint inference of imaging, genetic, and ERP data

Independent component analysis (ICA) has become an increasingly utilized approach for analyzing brain imaging data. In contrast to the widely used general linear model (GLM) that requires the user to parameterize the data (e.g. the brain's response to stimuli), ICA, by relying upon a general assumption of independence, allows the user to be agnostic regarding the exact form of the response. In addition, ICA is intrinsically a multivariate approach, and hence each component provides a grouping of brain activity into regions that share the same response pattern thus providing a natural measure of functional connectivity. There are a wide variety of ICA approaches that have been proposed, in this paper we focus upon two distinct methods. The first part of this paper reviews the use of ICA for making group inferences from fMRI data. We provide an overview of current approaches for utilizing ICA to make group inferences with a focus upon the group ICA approach implemented in the GIFT software. In the next part of this paper, we provide an overview of the use of ICA to combine or fuse multimodal data. ICA has proven particularly useful for data fusion of multiple tasks or data modalities such as single nucleotide polymorphism (SNP) data or event-related potentials. As demonstrated by a number of examples in this paper, ICA is a powerful and versatile data-driven approach for studying the brain.

Integrated MEG/fMRI model validated using real auditory data

The main objective of this paper is to present methods and results for the estimation of parameters of our proposed integrated magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) model. We use real auditory MEG and fMRI datasets from 7 normal subjects to estimate the parameters of the model. The MEG and fMRI data were acquired at different times, but the stimulus profile was the same for both techniques. We use independent component analysis (ICA) to extract activation-related signal from the MEG data. The stimulus-correlated ICA component is used to estimate MEG parameters of the model. The temporal and spatial information of the fMRI datasets are used to estimate fMRI parameters of the model. The estimated parameters have reasonable means and standard deviations for all subjects. Goodness of fit of the real data to our model shows the possibility of using the proposed model to simulate realistic datasets for evaluation of integrated MEG/fMRI analysis methods.

Feature-based fusion of medical imaging data

The acquisition of multiple brain imaging types for a given study is a very common practice. There have been a number of approaches proposed for combining or fusing multitask or multimodal information. These can be roughly divided into those that attempt to study convergence of multimodal imaging, for example, how function and structure are related in the same region of the brain, and those that attempt to study the complementary nature of modalities, for example, utilizing temporal EEG information and spatial functional magnetic resonance imaging information. Within each of these categories, one can attempt data integration (the use of one imaging modality to improve the results of another) or true data fusion (in which multiple modalities are utilized to inform one another). We review both approaches and present a recent computational approach that first preprocesses the data to compute features of interest. The features are then analyzed in a multivariate manner using independent component analysis. We describe the approach in detail and provide examples of how it has been used for different fusion tasks. We also propose a method for selecting which combination of modalities provides the greatest value in discriminating groups. Finally, we summarize and describe future research topics.

Scanning strategies for simultaneous EEG–fMRI evoked potential studies at 3 T

There are two basic strategies for applying simultaneous EEG-fMRI: either the fMRI data are acquired continuously, or the stimulus is presented during a brief gap in scanning when the EEG data is clear of gradient artefact. The former has the advantage that the protocol for the fMRI data acquisition is not affected by the presence of EEG. This study investigated the effect of these different strategies and the subsequent ballistocardiogram artefact removal methods (Average Artefact Subtraction (AAS) and Optimal Basis Set (OBS)) on EEG data quality recorded in response to a visual stimulus. Continuous scanning generally resulted in VEPs that were no worse, and in some cases were better, than those measured during a gap in scanning. The AAS and OBS methods lead to comparable results at the level of the grand average visual evoked potential (VEP), although when examined at the level of the single trial the OBS method was more effective. The spectral quality of the data was similar across scanning protocols, as demonstrated by the proportion of spectral power in each frequency band, although there was an effect of the artefact removal method on the overall spectral power. Some differences in the VEPs were also noted when a TR of 1.5 s was used relative to a TR of 3 s. The results indicate improved EEG quality when fMRI scanning is continuous and BCG artefacts are removed using the OBS method, confirming that EEG can be added to an fMRI experiment with minimal change to the experimental protocol.

Human brain mapping: hemodynamic response and electrophysiology

In view of the recent advance in functional neuroimaging, the current status of non-invasive techniques applied for human brain mapping was reviewed by integrating two principles: hemodynamic and electrophysiological, from the viewpoint of clinical neurophysiology. The currently available functional neuroimaging techniques based on hemodynamic principles are functional magnetic resonance imaging (fMRI), positron emission tomography (PET) or single-photon emission computed tomography (SPECT), and near-infrared spectroscopy (NIRS). Electrophysiological techniques include electroencephalography (EEG), magnetoencephalography (MEG), and transcranial magnetic stimulation (TMS). As for the coupling between hemodynamic response and neuronal activity (neurovascular coupling), experimental studies suggest that the hemodynamic response is significantly correlated to neuronal activity, especially local field potential (synaptic activity) rather than spiking activity, within a certain range. The hemodynamic response tends to be more widespread in space and lasts longer in time as compared with the neuronal activity. Since each technique has its own characteristic features especially in terms of spatial and temporal resolution, it is important to adopt the most appropriate technique for solving each specific question, and it is useful to combine two techniques either simultaneously or in separate sessions. As for the multi-modal approach, the combined use of EEG and MEG, EEG and PET, or EEG and fMRI is applied for the simultaneous studies, and for the separate use of two different techniques, the information obtained from fMRI is used for estimating the generator source from EEG or MEG data (fMRI-constrained source estimation). Functional connectivity among different brain areas can be studied by using a single technique such as the EEG coherence or the correlation analysis of fMRI or PET data, or by combining the stimulation technique such as TMS with neuroimaging. Further advance of each technology and improvement in the analysis method will promote the understanding of precise functional specialization and inter-areal coupling, and will contribute to the increased efficacy of rapidly developing physiological treatments of neurological and psychiatric disorders.

Dysfunctional and compensatory prefrontal cortical systems, genes and the pathogenesis of schizophrenia

Cognitive deficits are critical determinants of schizophrenia morbidity. In this review, we offer a mechanistic perspective regarding schizophrenia-related changes observed in prefrontal cortical networks engaged in working memory. A body of earlier work converges on aberrations in putative macrocircuit stability and functional efficiency as the underlying pathophysiology of the cognitive deficits in schizophrenia. In parsing the dysfunctional prefrontal cortical dynamics of schizophrenia, recent functional magnetic resonance imaging and electoencephalography works suggest that in the context of reduced capacity for executive aspects of working memory, patients engage a larger network of cortical regions consistent with an interplay between reduced signal-to-noise components and the recruitment of compensatory networks. The genetic programming underlying these systems-level cortical interactions has been examined under the lens of certain schizophrenia susceptibility genes, especially catechol-o-methyltransferase (COMT) and GRM3. Variation in COMT, which presumably impacts on cortical dopamine signaling, translates into variable neural strategies for working memory and altering patterns of intracortical functional correlations. GRM3, which impacts on synaptic glutamate, interacts with COMT and exaggerates the genetic dissection of cortical processing strategies. These findings reveal novel insights into the modulation and parcellation of working memory processing in cortical assemblies and provide a mechanistic link between susceptibility genes and cortical pathophysiology related to schizophrenia.

Combining structural and functional neuroimaging data for studying brain connectivity: a review

Different brain areas are thought to be integrated into large-scale networks to support cognitive function. Recent approaches for investigating structural organization and functional coordination within these networks involve measures of connectivity among brain areas. We review studies combining in vivo structural and functional brain connectivity data, where (a) structural connectivity analysis, mostly based on diffusion tensor imaging is paired with voxel-wise analysis of functional neuroimaging data or (b) the measurement of functional connectivity based on covariance analysis is guided/aided by structural connectivity data. These studies provide insights into the relationships between brain structure and function. Promising trends involve (a) studies where both functional and anatomical connectivity data are collected using high-resolution neuroimaging methods and (b) the development of advanced quantitative models of integration.

Unmixing concurrent EEG-fMRI with parallel independent component analysis

Concurrent event-related EEG-fMRI recordings pick up volume-conducted and hemodynamically convoluted signals from latent neural sources that are spatially and temporally mixed across the brain, i.e. the observed data in both modalities represent multiple, simultaneously active, regionally overlapping neuronal mass responses. This mixing process decreases the sensitivity of voxel-by-voxel prediction of hemodynamic activation by the EEG when multiple sources contribute to either the predictor and/or the response variables. In order to address this problem, we used independent component analysis (ICA) to recover maps from the fMRI and timecourses from the EEG, and matched these components across the modalities by correlating their trial-to-trial modulation. The analysis was implemented as a group-level ICA that extracts a single set of components from the data and directly allows for population inferences about consistently expressed function-relevant spatiotemporal responses. We illustrate the utility of this method by extracting a previously undetected but relevant EEG-fMRI component from a concurrent auditory target detection experiment.

Joint independent component analysis for simultaneous EEG-fMRI: principle and simulation

An optimized scheme for the fusion of electroencephalography and event related potentials with functional magnetic resonance imaging (BOLD-fMRI) data should simultaneously assess all available electrophysiologic and hemodynamic information in a common data space. In doing so, it should be possible to identify features of latent neural sources whose trial-to-trial dynamics are jointly reflected in both modalities. We present a joint independent component analysis (jICA) model for analysis of simultaneous single trial EEG-fMRI measurements from multiple subjects. We outline the general idea underlying the jICA approach and present results from simulated data under realistic noise conditions. Our results indicate that this approach is a feasible and physiologically plausible data-driven way to achieve spatiotemporal mapping of event related responses in the human brain.

Annotation: what electrical brain activity tells us about brain function that other techniques cannot tell us - a child psychiatric perspective

BACKGROUND

Monitoring brain processes in real time requires genuine subsecond resolution to follow the typical timing and frequency of neural events. Non-invasive recordings of electric (EEG/ERP) and magnetic (MEG) fields provide this time resolution. They directly measure neural activations associated with a wide variety of brain states and processes, even during sleep or in infants. Mapping and source estimation can localise these time-varying activation patterns inside the brain.

METHODS

Recent EEG/ERP research on brain functions in the domains of attention and executive functioning, perception, memory, language, emotion and motor processing in ADHD, autism, childhood-onset schizophrenia, Tourette syndrome, specific language disorder and developmental dyslexia, anxiety, obsessive-compulsive disorder, and depression is reviewed.

RESULTS

Over the past two decades, electrophysiology has substantially contributed to the understanding of brain functions during normal development, and psychiatric conditions of children and adolescents. Its time resolution has been important to measure covert processes, and to distinguish cause and effect.

CONCLUSIONS

In the future, EEG/ERP parameters will increasingly characterise the interplay of neural states and information processing. They are particularly promising tools for multilevel investigations of etiological pathways and potential predictors of clinical treatment response.

Single-trial EEG–fMRI reveals the dynamics of cognitive function

Two major non-invasive techniques in cognitive neuroscience, electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), have complementary advantages with regard to their spatial and temporal resolution. Recent hardware and software developments have made it feasible to acquire EEG and fMRI data simultaneously. We emphasize the potential of simultaneous EEG and fMRI recordings to pursue new strategies in cognitive neuroimaging. Specifically, we propose that, by exploiting the combined spatiotemporal resolution of the methods, the integration of EEG and fMRI recordings on a single-trial level enables the rich temporal dynamics of information processing to be characterized within spatially well-defined neural networks.