Human Visual Cortex and Deep Convolutional Neural Network Care Deeply about Object Background

Deep convolutional neural networks (DCNNs) are able to partially predict brain activity during object categorization tasks, but factors contributing to this predictive power are not fully understood. Our study aimed to investigate the factors contributing to the predictive power of DCNNs in object categorization tasks. We compared the activity of four DCNN architectures with EEG recordings obtained from 62 human participants during an object categorization task. Previous physiological studies on object categorization have highlighted the importance of figure-ground segregation-the ability to distinguish objects from their backgrounds. Therefore, we investigated whether figure-ground segregation could explain the predictive power of DCNNs. Using a stimulus set consisting of identical target objects embedded in different backgrounds, we examined the influence of object background versus object category within both EEG and DCNN activity. Crucially, the recombination of naturalistic objects and experimentally controlled backgrounds creates a challenging and naturalistic task, while retaining experimental control. Our results showed that early EEG activity (< 100 msec) and early DCNN layers represent object background rather than object category. We also found that the ability of DCNNs to predict EEG activity is primarily influenced by how both systems process object backgrounds, rather than object categories. We demonstrated the role of figure-ground segregation as a potential prerequisite for recognition of object features, by contrasting the activations of trained and untrained (i.e., random weights) DCNNs. These findings suggest that both human visual cortex and DCNNs prioritize the segregation of object backgrounds and target objects to perform object categorization. Altogether, our study provides new insights into the mechanisms underlying object categorization as we demonstrated that both human visual cortex and DCNNs care deeply about object background.

Mechanisms of human dynamic object recognition revealed by sequential deep neural networks

Humans can quickly recognize objects in a dynamically changing world. This ability is showcased by the fact that observers succeed at recognizing objects in rapidly changing image sequences, at up to 13 ms/image. To date, the mechanisms that govern dynamic object recognition remain poorly understood. Here, we developed deep learning models for dynamic recognition and compared different computational mechanisms, contrasting feedforward and recurrent, single-image and sequential processing as well as different forms of adaptation. We found that only models that integrate images sequentially via lateral recurrence mirrored human performance (N = 36) and were predictive of trial-by-trial responses across image durations (13-80 ms/image). Importantly, models with sequential lateral-recurrent integration also captured how human performance changes as a function of image presentation durations, with models processing images for a few time steps capturing human object recognition at shorter presentation durations and models processing images for more time steps capturing human object recognition at longer presentation durations. Furthermore, augmenting such a recurrent model with adaptation markedly improved dynamic recognition performance and accelerated its representational dynamics, thereby predicting human trial-by-trial responses using fewer processing resources. Together, these findings provide new insights into the mechanisms rendering object recognition so fast and effective in a dynamic visual world.

Testing, explaining, and exploring models of facial expressions of emotions

Models are the hallmark of mature scientific inquiry. In psychology, this maturity has been reached in a pervasive question-what models best represent facial expressions of emotion? Several hypotheses propose different combinations of facial movements [action units (AUs)] as best representing the six basic emotions and four conversational signals across cultures. We developed a new framework to formalize such hypotheses as predictive models, compare their ability to predict human emotion categorizations in Western and East Asian cultures, explain the causal role of individual AUs, and explore updated, culture-accented models that improve performance by reducing a prevalent Western bias. Our predictive models also provide a noise ceiling to inform the explanatory power and limitations of different factors (e.g., AUs and individual differences). Thus, our framework provides a new approach to test models of social signals, explain their predictive power, and explore their optimization, with direct implications for theory development.

A Critical Test of Deep Convolutional Neural Networks' Ability to Capture Recurrent Processing in the Brain Using Visual Masking

Recurrent processing is a crucial feature in human visual processing supporting perceptual grouping, figure-ground segmentation, and recognition under challenging conditions. There is a clear need to incorporate recurrent processing in deep convolutional neural networks, but the computations underlying recurrent processing remain unclear. In this article, we tested a form of recurrence in deep residual networks (ResNets) to capture recurrent processing signals in the human brain. Although ResNets are feedforward networks, they approximate an excitatory additive form of recurrence. Essentially, this form of recurrence consists of repeating excitatory activations in response to a static stimulus. Here, we used ResNets of varying depths (reflecting varying levels of recurrent processing) to explain EEG activity within a visual masking paradigm. Sixty-two humans and 50 artificial agents (10 ResNet models of depths -4, 6, 10, 18, and 34) completed an object categorization task. We show that deeper networks explained more variance in brain activity compared with shallower networks. Furthermore, all ResNets captured differences in brain activity between unmasked and masked trials, with differences starting at ∼98 msec (from stimulus onset). These early differences indicated that EEG activity reflected "pure" feedforward signals only briefly (up to ∼98 msec). After ∼98 msec, deeper networks showed a significant increase in explained variance, which peaks at ∼200 msec, but only within unmasked trials, not masked trials. In summary, we provided clear evidence that excitatory additive recurrent processing in ResNets captures some of the recurrent processing in humans.

Arousal state affects perceptual decision-making by modulating hierarchical sensory processing in a large-scale visual system model

Arousal levels strongly affect task performance. Yet, what arousal level is optimal for a task depends on its difficulty. Easy task performance peaks at higher arousal levels, whereas performance on difficult tasks displays an inverted U-shape relationship with arousal, peaking at medium arousal levels, an observation first made by Yerkes and Dodson in 1908. It is commonly proposed that the noradrenergic locus coeruleus system regulates these effects on performance through a widespread release of noradrenaline resulting in changes of cortical gain. This account, however, does not explain why performance decays with high arousal levels only in difficult, but not in simple tasks. Here, we present a mechanistic model that revisits the Yerkes-Dodson effect from a sensory perspective: a deep convolutional neural network augmented with a global gain mechanism reproduced the same interaction between arousal state and task difficulty in its performance. Investigating this model revealed that global gain states differentially modulated sensory information encoding across the processing hierarchy, which explained their differential effects on performance on simple versus difficult tasks. These findings offer a novel hierarchical sensory processing account of how, and why, arousal state affects task performance.

Leveraging Spiking Deep Neural Networks to Understand the Neural Mechanisms Underlying Selective Attention

Spatial attention enhances sensory processing of goal-relevant information and improves perceptual sensitivity. Yet, the specific neural mechanisms underlying the effects of spatial attention on performance are still contested. Here, we examine different attention mechanisms in spiking deep convolutional neural networks. We directly contrast effects of precision (internal noise suppression) and two different gain modulation mechanisms on performance on a visual search task with complex real-world images. Unlike standard artificial neurons, biological neurons have saturating activation functions, permitting implementation of attentional gain as gain on a neuron's input or on its outgoing connection. We show that modulating the connection is most effective in selectively enhancing information processing by redistributing spiking activity and by introducing additional task-relevant information, as shown by representational similarity analyses. Precision only produced minor attentional effects in performance. Our results, which mirror empirical findings, show that it is possible to adjudicate between attention mechanisms using more biologically realistic models and natural stimuli.

Robust BOLD Responses to Faces But Not to Conditioned Threat: Challenging the Amygdala's Reputation in Human Fear and Extinction Learning

Most of our knowledge about human emotional memory comes from animal research. Based on this work, the amygdala is often labeled the brain's "fear center", but it is unclear to what degree neural circuitries underlying fear and extinction learning are conserved across species. Neuroimaging studies in humans yield conflicting findings, with many studies failing to show amygdala activation in response to learned threat. Such null findings are often treated as resulting from MRI-specific problems related to measuring deep brain structures. Here we test this assumption in a mega-analysis of three studies on fear acquisition (n = 98; 68 female) and extinction learning (n = 79; 53 female). The conditioning procedure involved the presentation of two pictures of faces and two pictures of houses: one of each pair was followed by an electric shock [a conditioned stimulus (CS+)], the other one was never followed by a shock (CS-), and participants were instructed to learn these contingencies. Results revealed widespread responses to the CS+ compared with the CS- in the fear network, including anterior insula, midcingulate cortex, thalamus, and bed nucleus of the stria terminalis, but not the amygdala, which actually responded stronger to the CS- Results were independent of spatial smoothing, and of individual differences in trait anxiety and conditioned pupil responses. In contrast, robust amygdala activation distinguished faces from houses, refuting the idea that a poor signal could account for the absence of effects. Moving forward, we suggest that, apart from imaging larger samples at higher resolution, alternative statistical approaches may be used to identify cross-species similarities in fear and extinction learning.SIGNIFICANCE STATEMENT The science of emotional memory provides the foundation of numerous theories on psychopathology, including stress and anxiety disorders. This field relies heavily on animal research, which suggests a central role of the amygdala in fear learning and memory. However, this finding is not strongly corroborated by neuroimaging evidence in humans, and null findings are too easily explained away by methodological limitations inherent to imaging deep brain structures. In a large nonclinical sample, we find widespread BOLD activation in response to learned fear, but not in the amygdala. A poor signal could not account for the absence of effects. While these findings do not disprove the involvement of the amygdala in human fear learning, they challenge its typical portrayals and illustrate the complexities of translational science.

Visual features drive the category-specific impairments on categorization tasks in a patient with object agnosia

Object and scene recognition both require mapping of incoming sensory information to existing conceptual knowledge about the world. A notable finding in brain-damaged patients is that they may show differentially impaired performance for specific categories, such as for "living exemplars". While numerous patients with category-specific impairments have been reported, the explanations for these deficits remain controversial. In the current study, we investigate the ability of a brain injured patient with a well-established category-specific impairment of semantic memory to perform two categorization experiments: 'natural' vs. 'manmade' scenes (experiment 1) and objects (experiment 2). Our findings show that the pattern of categorical impairment does not respect the natural versus manmade distinction. This suggests that the impairments may be better explained by differences in visual features, rather than by category membership. Using Deep Convolutional Neural Networks (DCNNs) as 'artificial animal models' we further explored this idea. Results indicated that DCNNs with 'lesions' in higher order layers showed similar response patterns, with decreased relative performance for manmade scenes (experiment 1) and natural objects (experiment 2), even though they have no semantic category knowledge, apart from a mapping between pictures and labels. Collectively, these results suggest that the direction of category-effects to a large extent depends, at least in MS' case, on the degree of perceptual differentiation called for, and not semantic knowledge.

On the Necessity of Recurrent Processing during Object Recognition: It Depends on the Need for Scene Segmentation

Although feedforward activity may suffice for recognizing objects in isolation, additional visual operations that aid object recognition might be needed for real-world scenes. One such additional operation is figure-ground segmentation, extracting the relevant features and locations of the target object while ignoring irrelevant features. In this study of 60 human participants (female and male), we show objects on backgrounds of increasing complexity to investigate whether recurrent computations are increasingly important for segmenting objects from more complex backgrounds. Three lines of evidence show that recurrent processing is critical for recognition of objects embedded in complex scenes. First, behavioral results indicated a greater reduction in performance after masking objects presented on more complex backgrounds, with the degree of impairment increasing with increasing background complexity. Second, electroencephalography (EEG) measurements showed clear differences in the evoked response potentials between conditions around time points beyond feedforward activity, and exploratory object decoding analyses based on the EEG signal indicated later decoding onsets for objects embedded in more complex backgrounds. Third, deep convolutional neural network performance confirmed this interpretation. Feedforward and less deep networks showed a higher degree of impairment in recognition for objects in complex backgrounds compared with recurrent and deeper networks. Together, these results support the notion that recurrent computations drive figure-ground segmentation of objects in complex scenes.SIGNIFICANCE STATEMENT The incredible speed of object recognition suggests that it relies purely on a fast feedforward buildup of perceptual activity. However, this view is contradicted by studies showing that disruption of recurrent processing leads to decreased object recognition performance. Here, we resolve this issue by showing that how object recognition is resolved and whether recurrent processing is crucial depends on the context in which it is presented. For objects presented in isolation or in simple environments, feedforward activity could be sufficient for successful object recognition. However, when the environment is more complex, additional processing seems necessary to select the elements that belong to the object and by that segregate them from the background.

The Amsterdam Open MRI Collection, a set of multimodal MRI datasets for individual difference analyses

We present the Amsterdam Open MRI Collection (AOMIC): three datasets with multimodal (3 T) MRI data including structural (T1-weighted), diffusion-weighted, and (resting-state and task-based) functional BOLD MRI data, as well as detailed demographics and psychometric variables from a large set of healthy participants (N = 928, N = 226, and N = 216). Notably, task-based fMRI was collected during various robust paradigms (targeting naturalistic vision, emotion perception, working memory, face perception, cognitive conflict and control, and response inhibition) for which extensively annotated event-files are available. For each dataset and data modality, we provide the data in both raw and preprocessed form (both compliant with the Brain Imaging Data Structure), which were subjected to extensive (automated and manual) quality control. All data is publicly available from the OpenNeuro data sharing platform.

Depth in convolutional neural networks solves scene segmentation

Feed-forward deep convolutional neural networks (DCNNs) are, under specific conditions, matching and even surpassing human performance in object recognition in natural scenes. This performance suggests that the analysis of a loose collection of image features could support the recognition of natural object categories, without dedicated systems to solve specific visual subtasks. Research in humans however suggests that while feedforward activity may suffice for sparse scenes with isolated objects, additional visual operations ('routines') that aid the recognition process (e.g. segmentation or grouping) are needed for more complex scenes. Linking human visual processing to performance of DCNNs with increasing depth, we here explored if, how, and when object information is differentiated from the backgrounds they appear on. To this end, we controlled the information in both objects and backgrounds, as well as the relationship between them by adding noise, manipulating background congruence and systematically occluding parts of the image. Results indicate that with an increase in network depth, there is an increase in the distinction between object- and background information. For more shallow networks, results indicated a benefit of training on segmented objects. Overall, these results indicate that, de facto, scene segmentation can be performed by a network of sufficient depth. We conclude that the human brain could perform scene segmentation in the context of object identification without an explicit mechanism, by selecting or "binding" features that belong to the object and ignoring other features, in a manner similar to a very deep convolutional neural network.

Low-level image statistics in natural scenes influence perceptual decision-making

A fundamental component of interacting with our environment is gathering and interpretation of sensory information. When investigating how perceptual information influences decision-making, most researchers have relied on manipulated or unnatural information as perceptual input, resulting in findings that may not generalize to real-world scenes. Unlike simplified, artificial stimuli, real-world scenes contain low-level regularities that are informative about the structural complexity, which the brain could exploit. In this study, participants performed an animal detection task on low, medium or high complexity scenes as determined by two biologically plausible natural scene statistics, contrast energy (CE) or spatial coherence (SC). In experiment 1, stimuli were sampled such that CE and SC both influenced scene complexity. Diffusion modelling showed that the speed of information processing was affected by low-level scene complexity. Experiment 2a/b refined these observations by showing how isolated manipulation of SC resulted in weaker but comparable effects, with an additional change in response boundary, whereas manipulation of only CE had no effect. Overall, performance was best for scenes with intermediate complexity. Our systematic definition quantifies how natural scene complexity interacts with decision-making. We speculate that CE and SC serve as an indication to adjust perceptual decision-making based on the complexity of the input.

Neurocognitive Underpinnings of Aggressive Predation in Economic Contests

Competitions are part and parcel of daily life and require people to invest time and energy to gain advantage over others and to avoid (the risk of) falling behind. Whereas the behavioral mechanisms underlying competition are well documented, its neurocognitive underpinnings remain poorly understood. We addressed this using neuroimaging and computational modeling of individual investment decisions aimed at exploiting one's counterpart ("attack") or at protecting against exploitation by one's counterpart ("defense"). Analyses revealed that during attack relative to defense (i) individuals invest less and are less successful; (ii) computations of expected reward are strategically more sophisticated (reasoning level k = 4 vs. k = 3 during defense); (iii) ventral striatum activity tracks reward prediction errors; (iv) risk prediction errors were not correlated with neural activity in either ROI or whole-brain analyses; and (v) successful exploitation correlated with neural activity in the bilateral ventral striatum, left OFC, left anterior insula, left TPJ, and lateral occipital cortex. We conclude that, in economic contests, coming out ahead (vs. not falling behind) involves sophisticated strategic reasoning that engages both reward and value computation areas and areas associated with theory of mind.

Exome Chip Meta-analysis Fine Maps Causal Variants and Elucidates the Genetic Architecture of Rare Coding Variants in Smoking and Alcohol Use

BACKGROUND

Smoking and alcohol use have been associated with common genetic variants in multiple loci. Rare variants within these loci hold promise in the identification of biological mechanisms in substance use. Exome arrays and genotype imputation can now efficiently genotype rare nonsynonymous and loss of function variants. Such variants are expected to have deleterious functional consequences and to contribute to disease risk.

METHODS

We analyzed ∼250,000 rare variants from 16 independent studies genotyped with exome arrays and augmented this dataset with imputed data from the UK Biobank. Associations were tested for five phenotypes: cigarettes per day, pack-years, smoking initiation, age of smoking initiation, and alcoholic drinks per week. We conducted stratified heritability analyses, single-variant tests, and gene-based burden tests of nonsynonymous/loss-of-function coding variants. We performed a novel fine-mapping analysis to winnow the number of putative causal variants within associated loci.

RESULTS

Meta-analytic sample sizes ranged from 152,348 to 433,216, depending on the phenotype. Rare coding variation explained 1.1% to 2.2% of phenotypic variance, reflecting 11% to 18% of the total single nucleotide polymorphism heritability of these phenotypes. We identified 171 genome-wide associated loci across all phenotypes. Fine mapping identified putative causal variants with double base-pair resolution at 24 of these loci, and between three and 10 variants for 65 loci. Twenty loci contained rare coding variants in the 95% credible intervals.

CONCLUSIONS

Rare coding variation significantly contributes to the heritability of smoking and alcohol use. Fine-mapping genome-wide association study loci identifies specific variants contributing to the biological etiology of substance use behavior.

Sparse Computation in Adaptive Spiking Neural Networks

Artificial Neural Networks (ANNs) are bio-inspired models of neural computation that have proven highly effective. Still, ANNs lack a natural notion of time, and neural units in ANNs exchange analog values in a frame-based manner, a computationally and energetically inefficient form of communication. This contrasts sharply with biological neurons that communicate sparingly and efficiently using isomorphic binary spikes. While Spiking Neural Networks (SNNs) can be constructed by replacing the units of an ANN with spiking neurons (Cao et al., 2015; Diehl et al., 2015) to obtain reasonable performance, these SNNs use Poisson spiking mechanisms with exceedingly high firing rates compared to their biological counterparts. Here we show how spiking neurons that employ a form of neural coding can be used to construct SNNs that match high-performance ANNs and match or exceed state-of-the-art in SNNs on important benchmarks, while requiring firing rates compatible with biological findings. For this, we use spike-based coding based on the firing rate limiting adaptation phenomenon observed in biological spiking neurons. This phenomenon can be captured in fast adapting spiking neuron models, for which we derive the effective transfer function. Neural units in ANNs trained with this transfer function can be substituted directly with adaptive spiking neurons, and the resulting Adaptive SNNs (AdSNNs) can carry out competitive classification in deep neural networks without further modifications. Adaptive spike-based coding additionally allows for the dynamic control of neural coding precision: we show empirically how a simple model of arousal in AdSNNs further halves the average required firing rate and this notion naturally extends to other forms of attention as studied in neuroscience. AdSNNs thus hold promise as a novel and sparsely active model for neural computation that naturally fits to temporally continuous and asynchronous applications.

Scene complexity modulates degree of feedback activity during object detection in natural scenes

Selective brain responses to objects arise within a few hundreds of milliseconds of neural processing, suggesting that visual object recognition is mediated by rapid feed-forward activations. Yet disruption of neural responses in early visual cortex beyond feed-forward processing stages affects object recognition performance. Here, we unite these discrepant findings by reporting that object recognition involves enhanced feedback activity (recurrent processing within early visual cortex) when target objects are embedded in natural scenes that are characterized by high complexity. Human participants performed an animal target detection task on natural scenes with low, medium or high complexity as determined by a computational model of low-level contrast statistics. Three converging lines of evidence indicate that feedback was selectively enhanced for high complexity scenes. First, functional magnetic resonance imaging (fMRI) activity in early visual cortex (V1) was enhanced for target objects in scenes with high, but not low or medium complexity. Second, event-related potentials (ERPs) evoked by target objects were selectively enhanced at feedback stages of visual processing (from ~220 ms onwards) for high complexity scenes only. Third, behavioral performance for high complexity scenes deteriorated when participants were pressed for time and thus less able to incorporate the feedback activity. Modeling of the reaction time distributions using drift diffusion revealed that object information accumulated more slowly for high complexity scenes, with evidence accumulation being coupled to trial-to-trial variation in the EEG feedback response. Together, these results suggest that while feed-forward activity may suffice to recognize isolated objects, the brain employs recurrent processing more adaptively in naturalistic settings, using minimal feedback for simple scenes and increasing feedback for complex scenes.

Shared states: using MVPA to test neural overlap between self-focused emotion imagery and other-focused emotion understanding

The present study tested whether the neural patterns that support imagining ‘performing an action’, ‘feeling a bodily sensation’ or ‘being in a situation’ are directly involved in understanding other people’s actions, bodily sensations and situations. Subjects imagined the content of short sentences describing emotional actions, interoceptive sensations and situations (self-focused task), and processed scenes and focused on how the target person was expressing an emotion, what this person was feeling, and why this person was feeling an emotion (other-focused task). Using a linear support vector machine classifier on brain-wide multi-voxel patterns, we accurately decoded each individual class in the self-focused task. When generalizing the classifier from the self-focused task to the other-focused task, we also accurately decoded whether subjects focused on the emotional actions, interoceptive sensations and situations of others. These results show that the neural patterns that underlie self-imagined experience are involved in understanding the experience of other people. This supports the theoretical assumption that the basic components of emotion experience and understanding share resources in the brain.

Creating Colored Letters: Familial Markers of Grapheme–Color Synesthesia in Parietal Lobe Activation and Structure

Perception is inherently subjective, and individual differences in phenomenology are well illustrated by the phenomenon of synesthesia (highly specific, consistent, and automatic cross-modal experiences, in which the external stimulus corresponding to the additional sensation is absent). It is unknown why some people develop synesthesia and others do not. In the current study, we tested whether neural markers related to having synesthesia in the family were evident in brain function and structure. Relatives of synesthetes (who did not have any type of synesthesia themselves) and matched controls read specially prepared books with colored letters for several weeks and were scanned before and after reading using magnetic resonance imaging. Effects of acquired letter–color associations were evident in brain activation. Training-related activation (while viewing black letters) in the right angular gyrus of the parietal lobe was directly related to the strength of the learned letter–color associations (behavioral Stroop effect). Within this obtained angular gyrus ROI, the familial trait of synesthesia related to brain activation differences while participants viewed both black and colored letters. Finally, we compared brain structure using voxel-based morphometry and diffusion tensor imaging to test for group differences and training effects. One cluster in the left superior parietal lobe had significantly more coherent white matter in the relatives compared with controls. No evidence for experience-dependent plasticity was obtained. For the first time, we present evidence suggesting that the (nonsynesthete) relatives of grapheme–color synesthetes show atypical grapheme processing as well as increased brain connectivity.

The Desire for Amputation or Paralyzation: Evidence for Structural Brain Anomalies in Body Integrity Identity Disorder (BIID)

Background

Body Integrity Identity Disorder (BIID) is a condition in which individuals perceive a mismatch between their internal body scheme and physical body shape, resulting in an absolute desire to be either amputated or paralyzed. The condition is hypothesized to be of congenital nature, but evidence for a neuro-anatomical basis is sparse.

Methods

We collected T1-weighted structural magnetic resonance imaging scans on a 3T scanner in eight individuals with BIID and 24 matched healthy controls, and analyzed the data using voxel-based morphometry.

Results

The results showed reduced grey matter volume in the left dorsal and ventral premotor cortices and larger grey matter volume in the cerebellum (lobule VIIa) in individuals with BIID compared to controls.

Conclusion

The premotor cortex and cerebellum are thought to be crucial for the experience of body-ownership and the integration of multisensory information. Our results suggest that BIID is associated with structural brain anomalies and might result from a dysfunction in the integration of multisensory information, leading to the feeling of disunity between the mental and physical body shape.

First Steps in Using Multi-Voxel Pattern Analysis to Disentangle Neural Processes Underlying Generalization of Spider Fear

A core symptom of anxiety disorders is the tendency to interpret ambiguous information as threatening. Using electroencephalography and blood oxygenation level dependent magnetic resonance imaging (BOLD-MRI), several studies have begun to elucidate brain processes involved in fear-related perceptual biases, but thus far mainly found evidence for general hypervigilance in high fearful individuals. Recently, multi-voxel pattern analysis (MVPA) has become popular for decoding cognitive states from distributed patterns of neural activation. Here, we used this technique to assess whether biased fear generalization, characteristic of clinical fear, is already present during the initial perception and categorization of a stimulus, or emerges during the subsequent interpretation of a stimulus. Individuals with low spider fear (n = 20) and high spider fear (n = 18) underwent functional MRI scanning while viewing series of schematic flowers morphing to spiders. In line with previous studies, individuals with high fear of spiders were behaviorally more likely to classify ambiguous morphs as spiders than individuals with low fear of spiders. Univariate analyses of BOLD-MRI data revealed stronger activation toward spider pictures in high fearful individuals compared to low fearful individuals in numerous areas. Yet, neither average activation, nor support vector machine classification (i.e., a form of MVPA) matched the behavioral results – i.e., a biased response toward ambiguous stimuli – in any of the regions of interest. This may point to limitations of the current design, and to challenges associated with classifying emotional and neutral stimuli in groups that differ in their judgment of emotionality. Improvements for future research are suggested.

Are Autistic Traits in the General Population Related to Global and Regional Brain Differences?

There is accumulating evidence that autistic-related traits in the general population lie on a continuum, with autism spectrum disorders representing the extreme end of this distribution. Here, we tested the hypothesis of a possible relationship between autistic traits and brain morphometry in the general population. Participants completed the short autism-spectrum quotient-questionnaire (AQ); T1-anatomical and DWI-scans were acquired. Associations between autistic traits and gray matter, and white matter microstructural-integrity were performed on the exploration-group (N = 204; 105 males, M-age = 22.85), and validated in the validation-group (N = 304; 155 males, M-age = 22.82). No significant associations were found between AQ-scores and brain morphometry in the exploration-group, or after pooling the data. This questions the assumption that autistic traits and their morphological associations do lie on a continuum in the general population.

Machine learning and dyslexia: Classification of individual structural neuro-imaging scans of students with and without dyslexia

Meta-analytic studies suggest that dyslexia is characterized by subtle and spatially distributed variations in brain anatomy, although many variations failed to be significant after corrections of multiple comparisons. To circumvent issues of significance which are characteristic for conventional analysis techniques, and to provide predictive value, we applied a machine learning technique--support vector machine--to differentiate between subjects with and without dyslexia. In a sample of 22 students with dyslexia (20 women) and 27 students without dyslexia (25 women) (18-21 years), a classification performance of 80% (p < 0.001; d-prime = 1.67) was achieved on the basis of differences in gray matter (sensitivity 82%, specificity 78%). The voxels that were most reliable for classification were found in the left occipital fusiform gyrus (LOFG), in the right occipital fusiform gyrus (ROFG), and in the left inferior parietal lobule (LIPL). Additionally, we found that classification certainty (e.g. the percentage of times a subject was correctly classified) correlated with severity of dyslexia (r = 0.47). Furthermore, various significant correlations were found between the three anatomical regions and behavioural measures of spelling, phonology and whole-word-reading. No correlations were found with behavioural measures of short-term memory and visual/attentional confusion. These data indicate that the LOFG, ROFG and the LIPL are neuro-endophenotype and potentially biomarkers for types of dyslexia related to reading, spelling and phonology. In a second and independent sample of 876 young adults of a general population, the trained classifier of the first sample was tested, resulting in a classification performance of 59% (p = 0.07; d-prime = 0.65). This decline in classification performance resulted from a large percentage of false alarms. This study provided support for the use of machine learning in anatomical brain imaging.

The time course of natural scene perception with reduced attention

Attention is thought to impose an informational bottleneck on vision by selecting particular information from visual scenes for enhanced processing. Behavioral evidence suggests, however, that some scene information is extracted even when attention is directed elsewhere. Here, we investigated the neural correlates of this ability by examining how attention affects electrophysiological markers of scene perception. In two electro-encephalography (EEG) experiments, human subjects categorized real-world scenes as manmade or natural (full attention condition) or performed tasks on unrelated stimuli in the center or periphery of the scenes (reduced attention conditions). Scene processing was examined in two ways: traditional trial averaging was used to assess the presence of a categorical manmade/natural distinction in event-related potentials, whereas single-trial analyses assessed whether EEG activity was modulated by scene statistics that are diagnostic of naturalness of individual scenes. The results indicated that evoked activity up to 250 ms was unaffected by reduced attention, showing intact categorical differences between manmade and natural scenes and strong modulations of single-trial activity by scene statistics in all conditions. Thus initial processing of both categorical and individual scene information remained intact with reduced attention. Importantly, however, attention did have profound effects on later evoked activity; full attention on the scene resulted in prolonged manmade/natural differences, increased neural sensitivity to scene statistics, and enhanced scene memory. These results show that initial processing of real-world scene information is intact with diminished attention but that the depth of processing of this information does depend on attention.

Dyslexia and voxel-based morphometry: correlations between five behavioural measures of dyslexia and gray and white matter volumes

In voxel-based morphometry studies of dyslexia, the relation between causal theories of dyslexia and gray matter (GM) and white matter (WM) volume alterations is still under debate. Some alterations are consistently reported, but others failed to reach significance. We investigated GM alterations in a large sample of Dutch students (37 dyslexics and 57 non-dyslexics) with two analyses: group differences in local GM and total GM and WM volume and correlations between GM and WM volumes and five behavioural measures. We found no significant group differences after corrections for multiple comparisons although total WM volume was lower in the group of dyslexics when age was partialled out. We presented an overview of uncorrected clusters of voxels (p < 0.05, cluster size k > 200) with reduced or increased GM volume. We found four significant correlations between factors of dyslexia representing various behavioural measures and the clusters found in the first analysis. In the whole sample, a factor related to performances in spelling correlated negatively with GM volume in the left posterior cerebellum. Within the group of dyslexics, a factor related to performances in Dutch-English rhyme words correlated positively with GM volume in the left and right caudate nucleus and negatively with increased total WM volume. Most of our findings were in accordance with previous reports. A relatively new finding was the involvement of the caudate nucleus. We confirmed the multiple cognitive nature of dyslexia and suggested that experience greatly influences anatomical alterations depending on various subtypes of dyslexia, especially in a student sample.

Visual Information Shapes the Dynamics of Corticobasal Ganglia Pathways during Response Selection and Inhibition

Action selection often requires the transformation of visual information into motor plans. Preventing premature responses may entail the suppression of visual input and/or of prepared muscle activity. This study examined how the quality of visual information affects frontobasal ganglia (BG) routes associated with response selection and inhibition. Human fMRI data were collected from a stop task with visually degraded or intact face stimuli. During go trials, degraded spatial frequency information reduced the speed of information accumulation and response cautiousness. Effective connectivity analysis of the fMRI data showed action selection to emerge through the classic direct and indirect BG pathways, with inputs deriving form both prefrontal and visual regions. When stimuli were degraded, visual and prefrontal regions processing the stimulus information increased connectivity strengths toward BG, whereas regions evaluating visual scene content or response strategies reduced connectivity toward BG. Response inhibition during stop trials recruited the indirect and hyperdirect BG pathways, with input from visual and prefrontal regions. Importantly, when stimuli were nondegraded and processed fast, the optimal stop model contained additional connections from prefrontal to visual cortex. Individual differences analysis revealed that stronger prefrontal-to-visual connectivity covaried with faster inhibition times. Therefore, prefrontal-to-visual cortex connections appear to suppress the fast flow of visual input for the go task, such that the inhibition process can finish before the selection process. These results indicate response selection and inhibition within the BG to emerge through the interplay of top–down adjustments from prefrontal and bottom–up input from sensory cortex.

A functional MRI marker may predict the outcome of electroconvulsive therapy in severe and treatment-resistant depression

Electroconvulsive therapy (ECT) is effective even in treatment-resistant patients with major depression. Currently, there are no markers available that can assist in identifying those patients most likely to benefit from ECT. In the present study, we investigated whether resting-state network connectivity can predict treatment outcome for individual patients. We included forty-five patients with severe and treatment-resistant unipolar depression and collected functional magnetic resonance imaging scans before the course of ECT. We extracted resting-state networks and used multivariate pattern analysis to discover networks that predicted recovery from depression. Cross-validation revealed two resting-state networks with significant classification accuracy after correction for multiple comparisons. A network centered in the dorsomedial prefrontal cortex (including the dorsolateral prefrontal cortex, orbitofrontal cortex and posterior cingulate cortex) showed a sensitivity of 84% and specificity of 85%. Another network centered in the anterior cingulate cortex (including the dorsolateral prefrontal cortex, sensorimotor cortex, parahippocampal gyrus and midbrain) showed a sensitivity of 80% and a specificity of 75%. These preliminary results demonstrate that resting-state networks may predict treatment outcome for individual patients and suggest that resting-state networks have the potential to serve as prognostic neuroimaging biomarkers to guide personalized treatment decisions.

NMDA Receptor Antagonist Ketamine Distorts Object Recognition by Reducing Feedback to Early Visual Cortex

It is a well-established fact that top-down processes influence neural representations in lower-level visual areas. Electrophysiological recordings in monkeys as well as theoretical models suggest that these top-down processes depend on NMDA receptor functioning. However, this underlying neural mechanism has not been tested in humans. We used fMRI multivoxel pattern analysis to compare the neural representations of ambiguous Mooney images before and after they were recognized with their unambiguous grayscale version. Additionally, we administered ketamine, an NMDA receptor antagonist, to interfere with this process. Our results demonstrate that after recognition, the pattern of brain activation elicited by a Mooney image is more similar to that of its easily recognizable grayscale version than to the pattern evoked by the identical Mooney image before recognition. Moreover, recognition of Mooney images decreased mean response; however, neural representations of separate images became more dissimilar. So from the neural perspective, unrecognizable Mooney images all "look the same", whereas recognized Mooneys look different. We observed these effects in posterior fusiform part of lateral occipital cortex and in early visual cortex. Ketamine distorted these effects of recognition, but in early visual cortex only. This suggests that top-down processes from higher- to lower-level visual areas might operate via an NMDA pathway.

Visual dictionaries as intermediate features in the human brain

The human visual system is assumed to transform low level visual features to object and scene representations via features of intermediate complexity. How the brain computationally represents intermediate features is still unclear. To further elucidate this, we compared the biologically plausible HMAX model and Bag of Words (BoW) model from computer vision. Both these computational models use visual dictionaries, candidate features of intermediate complexity, to represent visual scenes, and the models have been proven effective in automatic object and scene recognition. These models however differ in the computation of visual dictionaries and pooling techniques. We investigated where in the brain and to what extent human fMRI responses to short video can be accounted for by multiple hierarchical levels of the HMAX and BoW models. Brain activity of 20 subjects obtained while viewing a short video clip was analyzed voxel-wise using a distance-based variation partitioning method. Results revealed that both HMAX and BoW explain a significant amount of brain activity in early visual regions V1, V2, and V3. However, BoW exhibits more consistency across subjects in accounting for brain activity compared to HMAX. Furthermore, visual dictionary representations by HMAX and BoW explain significantly some brain activity in higher areas which are believed to process intermediate features. Overall our results indicate that, although both HMAX and BoW account for activity in the human visual system, the BoW seems to more faithfully represent neural responses in low and intermediate level visual areas of the brain.

Oxytocin tempers calculated greed but not impulsive defense in predator-prey contests

Human cooperation and competition is modulated by oxytocin, a hypothalamic neuropeptide that functions as both hormone and neurotransmitter. Oxytocin's functions can be captured in two explanatory yet largely contradictory frameworks: the fear-dampening (FD) hypothesis that oxytocin has anxiolytic effects and reduces fear-motivated action; and the social approach/avoidance (SAA) hypothesis that oxytocin increases cooperative approach and facilitates protection against aversive stimuli and threat. We tested derivations from both frameworks in a novel predator-prey contest game. Healthy males given oxytocin or placebo invested as predator to win their prey's endowment, or as prey to protect their endowment against predation. Neural activity was registered using 3T-MRI. In prey, (fear-motivated) investments were fast and conditioned on the amygdala. Inconsistent with FD, oxytocin did not modulate neural and behavioral responding in prey. In predators, (greed-motivated) investments were slower, and conditioned on the superior frontal gyrus (SFG). Consistent with SAA, oxytocin reduced predator investment, time to decide and activation in SFG. Thus, whereas oxytocin does not incapacitate the impulsive ability to protect and defend oneself, it lowers the greedy and more calculated appetite for coming out ahead.

Latent memory of unattended stimuli reactivated by practice: an FMRI study on the role of consciousness and attention in learning

Although we can only report about what is in the focus of our attention, much more than that is actually processed. And even when attended, stimuli may not always be reportable, for instance when they are masked. A stimulus can thus be unreportable for different reasons: the absence of attention or the absence of a conscious percept. But to what extent does the brain learn from exposure to these unreportable stimuli? In this fMRI experiment subjects were exposed to textured figure-ground stimuli, of which reportability was manipulated either by masking (which only interferes with consciousness) or with an inattention paradigm (which only interferes with attention). One day later learning was assessed neurally and behaviorally. Positive neural learning effects were found for stimuli presented in the inattention paradigm; for attended yet masked stimuli negative adaptation effects were found. Interestingly, these inattentional learning effects only became apparent in a second session after a behavioral detection task had been administered during which performance feedback was provided. This suggests that the memory trace that is formed during inattention is latent until reactivated by behavioral practice. However, no behavioral learning effects were found, therefore we cannot conclude that perceptual learning has taken place for these unattended stimuli.

Opposing Dorsal/Ventral Stream Dynamics during Figure-ground Segregation

The visual system has been commonly subdivided into two segregated visual processing streams: The dorsal pathway processes mainly spatial information, and the ventral pathway specializes in object perception. Recent findings, however, indicate that different forms of interaction (cross-talk) exist between the dorsal and the ventral stream. Here, we used TMS and concurrent EEG recordings to explore these interactions between the dorsal and ventral stream during figure-ground segregation. In two separate experiments, we used repetitive TMS and single-pulse TMS to disrupt processing in the dorsal (V5/HMT+) and the ventral (lateral occipital area) stream during a motion-defined figure discrimination task. We presented stimuli that made it possible to differentiate between relatively low-level (figure boundary detection) from higher-level (surface segregation) processing steps during figure-ground segregation. Results show that disruption of V5/HMT+ impaired performance related to surface segregation; this effect was mainly found when V5/HMT+ was perturbed in an early time window (100 msec) after stimulus presentation. Surprisingly, disruption of the lateral occipital area resulted in increased performance scores and enhanced neural correlates of surface segregation. This facilitatory effect was also mainly found in an early time window (100 msec) after stimulus presentation. These results suggest a “push–pull” interaction in which dorsal and ventral extrastriate areas are being recruited or inhibited depending on stimulus category and task demands.

From image statistics to scene gist: evoked neural activity reveals transition from low-level natural image structure to scene category

The visual system processes natural scenes in a split second. Part of this process is the extraction of "gist," a global first impression. It is unclear, however, how the human visual system computes this information. Here, we show that, when human observers categorize global information in real-world scenes, the brain exhibits strong sensitivity to low-level summary statistics. Subjects rated a specific instance of a global scene property, naturalness, for a large set of natural scenes while EEG was recorded. For each individual scene, we derived two physiologically plausible summary statistics by spatially pooling local contrast filter outputs: contrast energy (CE), indexing contrast strength, and spatial coherence (SC), indexing scene fragmentation. We show that behavioral performance is directly related to these statistics, with naturalness rating being influenced in particular by SC. At the neural level, both statistics parametrically modulated single-trial event-related potential amplitudes during an early, transient window (100-150 ms), but SC continued to influence activity levels later in time (up to 250 ms). In addition, the magnitude of neural activity that discriminated between man-made versus natural ratings of individual trials was related to SC, but not CE. These results suggest that global scene information may be computed by spatial pooling of responses from early visual areas (e.g., LGN or V1). The increased sensitivity over time to SC in particular, which reflects scene fragmentation, suggests that this statistic is actively exploited to estimate scene naturalness.

NMDA receptor antagonist ketamine impairs feature integration in visual perception

Recurrent interactions between neurons in the visual cortex are crucial for the integration of image elements into coherent objects, such as in figure-ground segregation of textured images. Blocking N-methyl-D-aspartate (NMDA) receptors in monkeys can abolish neural signals related to figure-ground segregation and feature integration. However, it is unknown whether this also affects perceptual integration itself. Therefore, we tested whether ketamine, a non-competitive NMDA receptor antagonist, reduces feature integration in humans. We administered a subanesthetic dose of ketamine to healthy subjects who performed a texture discrimination task in a placebo-controlled double blind within-subject design. We found that ketamine significantly impaired performance on the texture discrimination task compared to the placebo condition, while performance on a control fixation task was much less impaired. This effect is not merely due to task difficulty or a difference in sedation levels. We are the first to show a behavioral effect on feature integration by manipulating the NMDA receptor in humans.

Spatial frequency information modulates response inhibition and decision-making processes

We interact with the world through the assessment of available, but sometimes imperfect, sensory information. However, little is known about how variance in the quality of sensory information affects the regulation of controlled actions. In a series of three experiments, comprising a total of seven behavioral studies, we examined how different types of spatial frequency information affect underlying processes of response inhibition and selection. Participants underwent a stop-signal task, a two choice speed/accuracy balance experiment, and a variant of both these tasks where prior information was given about the nature of stimuli. In all experiments, stimuli were either intact, or contained only high-, or low- spatial frequencies. Overall, drift diffusion model analysis showed a decreased rate of information processing when spatial frequencies were removed, whereas the criterion for information accumulation was lowered. When spatial frequency information was intact, the cost of response inhibition increased (longer SSRT), while a correct response was produced faster (shorter reaction times) and with more certainty (decreased errors). When we manipulated the motivation to respond with a deadline (i.e., be fast or accurate), removal of spatial frequency information slowed response times only when instructions emphasized accuracy. However, the slowing of response times did not improve error rates, when compared to fast instruction trials. These behavioral studies suggest that the removal of spatial frequency information differentially affects the speed of response initiation, inhibition, and the efficiency to balance fast or accurate responses. More generally, the present results indicate a task-independent influence of basic sensory information on strategic adjustments in action control.

The relationship between grey-matter and ASD and ADHD traits in typical adults

We tested whether in 85 healthy adults (18-29 years) there is a relationship between grey-matter (GM) volume and autism and ADHD symptom severity. The structural MRI findings and autism and ADHD self-reports revealed that autism and ADHD symptom severity was correlated with GM volume in the left inferior frontal gyrus. Autism symptom-severity was correlated with the left posterior cingulate, ADHD with the right parietal lobe, right temporal frontal cortex, bilateral thalamus, and left hippocampus/amygdala complex. Symptom severity of both disorders form a continuum extending into the general population, but it seems to be an oversimplification to typify psychiatric disorders such as autism and ADHD solely as extremes of brain structure abnormalities.

Does Perceptual Learning Require Consciousness or Attention?

It has been proposed that visual attention and consciousness are separate [Koch, C., & Tsuchiya, N. Attention and consciousness: Two distinct brain processes. Trends in Cognitive Sciences, 11, 16–22, 2007] and possibly even orthogonal processes [Lamme, V. A. F. Why visual attention and awareness are different. Trends in Cognitive Sciences, 7, 12–18, 2003]. Attention and consciousness converge when conscious visual percepts are attended and hence become available for conscious report. In such a view, a lack of reportability can have two causes: the absence of attention or the absence of a conscious percept. This raises an important question in the field of perceptual learning. It is known that learning can occur in the absence of reportability [Gutnisky, D. A., Hansen, B. J., Iliescu, B. F., & Dragoi, V. Attention alters visual plasticity during exposure-based learning. Current Biology, 19, 555–560, 2009; Seitz, A. R., Kim, D., & Watanabe, T. Rewards evoke learning of unconsciously processed visual stimuli in adult humans. Neuron, 61, 700–707, 2009; Seitz, A. R., & Watanabe, T. Is subliminal learning really passive? Nature, 422, 36, 2003; Watanabe, T., Náñez, J. E., & Sasaki, Y. Perceptual learning without perception. Nature, 413, 844–848, 2001], but it is unclear which of the two ingredients—consciousness or attention—is not necessary for learning. We presented textured figure-ground stimuli and manipulated reportability either by masking (which only interferes with consciousness) or with an inattention paradigm (which only interferes with attention). During the second session (24 hr later), learning was assessed neurally and behaviorally, via differences in figure-ground ERPs and via a detection task. Behavioral and neural learning effects were found for stimuli presented in the inattention paradigm and not for masked stimuli. Interestingly, the behavioral learning effect only became apparent when performance feedback was given on the task to measure learning, suggesting that the memory trace that is formed during inattention is latent until accessed. The results suggest that learning requires consciousness, and not attention, and further strengthen the idea that consciousness is separate from attention.

Neural basis of limb ownership in individuals with body integrity identity disorder

Our body feels like it is ours. However, individuals with body integrity identity disorder (BIID) lack this feeling of ownership for distinct limbs and desire amputation of perfectly healthy body parts. This extremely rare condition provides us with an opportunity to study the neural basis underlying the feeling of limb ownership, since these individuals have a feeling of disownership for a limb in the absence of apparent brain damage. Here we directly compared brain activation between limbs that do and do not feel as part of the body using functional MRI during separate tactile stimulation and motor execution experiments. In comparison to matched controls, individuals with BIID showed heightened responsivity of a large somatosensory network including the parietal cortex and right insula during tactile stimulation, regardless of whether the stimulated leg felt owned or alienated. Importantly, activity in the ventral premotor cortex depended on the feeling of ownership and was reduced during stimulation of the alienated compared to the owned leg. In contrast, no significant differences between groups were observed during the performance of motor actions. These results suggest that altered somatosensory processing in the premotor cortex is associated with the feeling of disownership in BIID, which may be related to altered integration of somatosensory and proprioceptive information.

GABA shapes the dynamics of bistable perception

Sometimes, perception fluctuates spontaneously between two distinct interpretations of a constant sensory input. These bistable perceptual phenomena provide a unique window into the neural mechanisms that create the contents of conscious perception. Models of bistable perception posit that mutual inhibition between stimulus-selective neural populations in visual cortex plays a key role in these spontaneous perceptual fluctuations. However, a direct link between neural inhibition and bistable perception has not yet been established experimentally. Here, we link perceptual dynamics in three distinct bistable visual illusions (binocular rivalry, motion-induced blindness, and structure from motion) to measurements of gamma-aminobutyric acid (GABA) concentrations in human visual cortex (as measured with magnetic resonance spectroscopy) and to pharmacological stimulation of the GABAA receptor by means of lorazepam. As predicted by a model of neural interactions underlying bistability, both higher GABA concentrations in visual cortex and lorazepam administration induced slower perceptual dynamics, as reflected in a reduced number of perceptual switches and a lengthening of percept durations. Thus, we show that GABA, the main inhibitory neurotransmitter, shapes the dynamics of bistable perception. These results pave the way for future studies into the competitive neural interactions across the visual cortical hierarchy that elicit conscious perception.