From coarse to fine? Spatial and temporal dynamics of cortical face processing

Primary vision segregates information along 2 main dimensions: orientation and spatial frequency (SF). An important question is how this primary visual information is integrated to support high-level representations. It is generally assumed that the information carried by different SF is combined following a coarse-to-fine sequence. We directly addressed this assumption by investigating how the network of face-preferring cortical regions processes distinct SF over time. Face stimuli were flashed during 75, 150, or 300 ms and masked. They were filtered to preserve low SF (LSF), middle SF (MSF), or high SF (HSF). Most face-preferring regions robustly responded to coarse LSF, face information in early stages of visual processing (i.e., until 75 ms of exposure duration). LSF processing decayed as a function of exposure duration (mostly until 150 ms). In contrast, the processing of fine HSF, face information became more robust over time in the bilateral fusiform face regions and in the right occipital face area. The present evidence suggests the coarse-to-fine strategy as a plausible modus operandi in high-level visual cortex.

Collicular vision guides nonconscious behavior

Following destruction or deafferentation of primary visual cortex (area V1, striate cortex), clinical blindness ensues, but residual visual functions may, nevertheless, persist without perceptual consciousness (a condition termed blindsight). The study of patients with such lesions thus offers a unique opportunity to investigate what visual capacities are mediated by the extrastriate pathways that bypass V1. Here we provide evidence for a crucial role of the collicular-extrastriate pathway in nonconscious visuomotor integration by showing that, in the absence of V1, the superior colliculus (SC) is essential to translate visual signals that cannot be consciously perceived into motor outputs. We found that a gray stimulus presented in the blind field of a patient with unilateral V1 loss, although not consciously seen, can influence his behavioral and pupillary responses to consciously perceived stimuli in the intact field (implicit bilateral summation). Notably, this effect was accompanied by selective activations in the SC and in occipito-temporal extrastriate areas. However, when instead of gray stimuli we presented purple stimuli, which predominantly draw on S-cones and are thus invisible to the SC, any evidence of implicit visuomotor integration disappeared and activations in the SC dropped significantly. The present findings show that the SC acts as an interface between sensory and motor processing in the human brain, thereby providing a contribution to visually guided behavior that may remain functionally and anatomically segregated from the geniculo-striate pathway and entirely outside conscious visual experience.

Continuous motor sequence learning: cortical efficiency gains accompanied by striatal functional reorganization

The acquisition and generation of action sequences constitute essential elements of purposeful human behavior. However, there is still considerable debate on how experience-driven changes related to skill learning are expressed at the neural systems level. The current functional magnetic resonance imaging (fMRI) study focused on changes in the neural representation of continuous movement sequences as learning evolved. Behavioral and neural manifestations of nonvisual motor practice were studied both within the time frame of a single scanning session, as well as after several days of extended practice. Based on detailed behavioral recordings which enabled the continuous characterization of the ongoing learning process at the single subject level, sequence-specific decreases in activation throughout a learning-related network of cortical areas were identified. Furthermore, the spatial layout of this cortical network remained largely unchanged after extensive practice, although further decreases in activation levels could be observed as learning progressed. In contrast, the posterior part of the left putamen showed increased activation levels when an extensively trained sequence needed to be recalled. Overall, these findings imply that continuous motor sequence learning is mainly associated with more efficient processing in a network of consistently recruited cortical areas, together with co-occurring activation pattern changes at the subcortical level.

Holistic perception of individual faces in the right middle fusiform gyrus as evidenced by the composite face illusion

The perception of a facial feature (e.g., the eyes) is influenced by the position and identity of other features (e.g., the mouth) supporting an integrated, or holistic, representation of individual faces in the human brain. Here we used an event-related adaptation paradigm in functional magnetic resonance imaging (fMRI) to clarify the regions representing faces holistically across the whole brain. In each trial, observers performed the same/different task on top halves (aligned or misaligned) of two faces presented sequentially. For each face pair, the identity of top and bottom parts could be both identical, both different, or different only for the bottom half. The latter manipulation resulted in a composite face illusion, i.e., the erroneous perception of identical top parts as being different, only for aligned faces. Release from adaptation in this condition was found in two sub-areas of the right middle fusiform gyrus responding preferentially to faces, including the "fusiform face area" ("FFA"). There were no significant effects in homologous regions of the left hemisphere or in the inferior occipital cortex. Altogether, these observations indicate that face-sensitive populations of neurons in the right middle fusiform gyrus are optimally tuned to represent individual exemplars of faces holistically.

fMR-adaptation indicates selectivity to audiovisual content congruency in distributed clusters in human superior temporal cortex

BACKGROUND

Efficient multisensory integration is of vital importance for adequate interaction with the environment. In addition to basic binding cues like temporal and spatial coherence, meaningful multisensory information is also bound together by content-based associations. Many functional Magnetic Resonance Imaging (fMRI) studies propose the (posterior) superior temporal cortex (STC) as the key structure for integrating meaningful multisensory information. However, a still unanswered question is how superior temporal cortex encodes content-based associations, especially in light of inconsistent results from studies comparing brain activation to semantically matching (congruent) versus nonmatching (incongruent) multisensory inputs. Here, we used fMR-adaptation (fMR-A) in order to circumvent potential problems with standard fMRI approaches, including spatial averaging and amplitude saturation confounds. We presented repetitions of audiovisual stimuli (letter-speech sound pairs) and manipulated the associative relation between the auditory and visual inputs (congruent/incongruent pairs). We predicted that if multisensory neuronal populations exist in STC and encode audiovisual content relatedness, adaptation should be affected by the manipulated audiovisual relation.

RESULTS

The results revealed an occipital-temporal network that adapted independently of the audiovisual relation. Interestingly, several smaller clusters distributed over superior temporal cortex within that network, adapted stronger to congruent than to incongruent audiovisual repetitions, indicating sensitivity to content congruency.

CONCLUSIONS

These results suggest that the revealed clusters contain multisensory neuronal populations that encode content relatedness by selectively responding to congruent audiovisual inputs, since unisensory neuronal populations are assumed to be insensitive to the audiovisual relation. These findings extend our previously revealed mechanism for the integration of letters and speech sounds and demonstrate that fMR-A is sensitive to multisensory congruency effects that may not be revealed in BOLD amplitude per se.

Multimodal imaging: an evaluation of univariate and multivariate methods for simultaneous EEG/fMRI

The combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) has been proposed as a tool to study brain dynamics with both high temporal and high spatial resolution. Multimodal imaging techniques rely on the assumption of a common neuronal source for the different recorded signals. In order to maximally exploit the combination of these techniques, one needs to understand the coupling (i.e., the relation) between electroencephalographic (EEG) and fMRI blood oxygen level-dependent (BOLD) signals. Recently, simultaneous EEG-fMRI measurements have been used to investigate the relation between the two signals. Previous attempts at the analysis of simultaneous EEG-fMRI data reported significant correlations between regional BOLD activations and modulation of both event-related potential (ERP) and oscillatory EEG power, mostly in the alpha but also in other frequency bands. Beyond the correlation of the two measured brain signals, the relevant issue we address here is the ability of predicting the signal in one modality using information from the other modality. Using multivariate machine learning-based regression, we show how it is possible to predict EEG power oscillations from simultaneously acquired fMRI data during an eyes-open/eyes-closed task using either the original channels or the underlying cortically distributed sources as the relevant EEG signal for the analysis of multimodal data.

Deviant processing of letters and speech sounds as proximate cause of reading failure: a functional magnetic resonance imaging study of dyslexic children

Learning to associate auditory information of speech sounds with visual information of letters is a first and critical step for becoming a skilled reader in alphabetic languages. Nevertheless, it remains largely unknown which brain areas subserve the learning and automation of such associations. Here, we employ functional magnetic resonance imaging to study letter-speech sound integration in children with and without developmental dyslexia. The results demonstrate that dyslexic children show reduced neural integration of letters and speech sounds in the planum temporale/Heschl sulcus and the superior temporal sulcus. While cortical responses to speech sounds in fluent readers were modulated by letter-speech sound congruency with strong suppression effects for incongruent letters, no such modulation was observed in the dyslexic readers. Whole-brain analyses of unisensory visual and auditory group differences additionally revealed reduced unisensory responses to letters in the fusiform gyrus in dyslexic children, as well as reduced activity for processing speech sounds in the anterior superior temporal gyrus, planum temporale/Heschl sulcus and superior temporal sulcus. Importantly, the neural integration of letters and speech sounds in the planum temporale/Heschl sulcus and the neural response to letters in the fusiform gyrus explained almost 40% of the variance in individual reading performance. These findings indicate that an interrelated network of visual, auditory and heteromodal brain areas contributes to the skilled use of letter-speech sound associations necessary for learning to read. By extending similar findings in adults, the data furthermore argue against the notion that reduced neural integration of letters and speech sounds in dyslexia reflect the consequence of a lifetime of reading struggle. Instead, they support the view that letter-speech sound integration is an emergent property of learning to read that develops inadequately in dyslexic readers, presumably as a result of a deviant interactive specialization of neural systems for processing auditory and visual linguistic inputs.

Robustness of optimal design of fMRI experiments with application of a genetic algorithm

In this paper we apply the genetic algorithm developed by Kao et al. (2009) to find designs which are robust against misspecification of the error autocorrelation. Two common optimality criteria, the A-optimality criterion and the D-optimality criterion, based upon a general linear model are employed to obtain locally optimal designs for a given value of the autocorrelation. The maximin criterion is then used to obtain designs which are robust against misspecification of the autocorrelation. Furthermore, robustness depending on the choice of optimality criterion is evaluated. We show analytically and empirically that the A- and D-optimality criterion will result in different optimal designs, e.g. with different stimulus frequencies. Optimal stimulus frequency for the A-optimality criterion has been derived by Liu et al. (2004) whereas we derive here the optimal stimulus frequency for the D-optimality criterion. Conclusions about the robustness of an optimal design against misspecification of model parameters and choice of optimality criterion are drawn based upon our results.

Performance- and stimulus-dependent oscillations in monkey prefrontal cortex during short-term memory

Short-term memory requires the coordination of sub-processes like encoding, retention, retrieval and comparison of stored material to subsequent input. Neuronal oscillations have an inherent time structure, can effectively coordinate synaptic integration of large neuron populations and could therefore organize and integrate distributed sub-processes in time and space. We observed field potential oscillations (14–95 Hz) in ventral prefrontal cortex of monkeys performing a visual memory task. Stimulus-selective and performance-dependent oscillations occurred simultaneously at 65–95 Hz and 14–50 Hz, the latter being phase-locked throughout memory maintenance. We propose that prefrontal oscillatory activity may be instrumental for the dynamical integration of local and global neuronal processes underlying short-term memory.

Multisensory functional magnetic resonance imaging: a future perspective

Advances in functional magnetic resonance imaging (fMRI) technology and analytic tools provide a powerful approach to unravel how the human brain combines the different sensory systems. In this perspective, we outline promising future directions of fMRI to make optimal use of its strengths in multisensory research, and to meet its weaker sides by combining it with other imaging modalities and computational modeling.

Playing charades in the fMRI: are mirror and/or mentalizing areas involved in gestural communication?

Communication is an important aspect of human life, allowing us to powerfully coordinate our behaviour with that of others. Boiled down to its mere essentials, communication entails transferring a mental content from one brain to another. Spoken language obviously plays an important role in communication between human individuals. Manual gestures however often aid the semantic interpretation of the spoken message, and gestures may have played a central role in the earlier evolution of communication. Here we used the social game of charades to investigate the neural basis of gestural communication by having participants produce and interpret meaningful gestures while their brain activity was measured using functional magnetic resonance imaging. While participants decoded observed gestures, the putative mirror neuron system (pMNS: premotor, parietal and posterior mid-temporal cortex), associated with motor simulation, and the temporo-parietal junction (TPJ), associated with mentalizing and agency attribution, were significantly recruited. Of these areas only the pMNS was recruited during the production of gestures. This suggests that gestural communication relies on a combination of simulation and, during decoding, mentalizing/agency attribution brain areas. Comparing the decoding of gestures with a condition in which participants viewed the same gestures with an instruction not to interpret the gestures showed that although parts of the pMNS responded more strongly during active decoding, most of the pMNS and the TPJ did not show such significant task effects. This suggests that the mere observation of gestures recruits most of the system involved in voluntary interpretation.

Type of featural attention differentially modulates hMT+ responses to illusory motion aftereffects

Activity in the human motion complex (hMT(+)/V5) is related to the perception of motion, be it either real surface motion or an illusion of motion such as apparent motion (AM) or motion aftereffect (MAE). It is a long-lasting debate whether illusory motion-related activations in hMT(+) represent the motion itself or attention to it. We have asked whether hMT(+) responses to MAEs are present when shifts in arousal are suppressed and attention is focused on concurrent motion versus nonmotion features. Significant enhancement of hMT(+) activity was observed during MAEs when attention was focused either on concurrent spatial angle or color features. This observation was confirmed by direct comparison of adapting (MAE inducing) versus nonadapting conditions. In contrast, this effect was diminished when subjects had to report on concomitant speed changes of superimposed AM. The same finding was observed for concomitant orthogonal real motion (RM), suggesting that selective attention to concurrent illusory or real motion was interfering with the saliency of MAE signals in hMT(+). We conclude that MAE-related changes in the global activity of hMT(+) are present provided selective attention is not focused on an interfering feature such as concurrent motion. Accordingly, there is a genuine MAE-related motion signal in hMT(+) that is neither explained by shifts in arousal nor by selective attention.