Decoding the motion aftereffect in human visual cortex

Dissociating representations of affect and motion in visual cortices

While a delicious dessert being presented to us may elicit strong feelings of happiness and excitement, the same treat falling slowly away can lead to sadness and disappointment. Our emotional response to the item depends on its visual motion direction. Despite this importance, it remains unclear whether (and how) cortical areas devoted to decoding motion direction represents or integrates emotion with perceived motion direction. Motion-selective visual area V5/MT+ sits, both functionally and anatomically, at the nexus of dorsal and ventral visual streams. These pathways, however, differ in how they are modulated by emotional cues. The current study was designed to disentangle how emotion and motion perception interact, as well as use emotion-dependent modulation of visual cortices to understand the relation of V5/MT+ to canonical processing streams. During functional magnetic resonance imaging (fMRI), approaching, receding, or static motion after-effects (MAEs) were induced on stationary positive, negative, and neutral stimuli. An independent localizer scan was conducted to identify the visual-motion area V5/MT+. Through univariate and multivariate analyses, we demonstrated that emotion representations in V5/MT+ share a more similar response profile to that observed in ventral visual than dorsal, visual structures. Specifically, V5/MT+ and ventral structures were sensitive to the emotional content of visual stimuli, whereas dorsal visual structures were not. Overall, this work highlights the critical role of V5/MT+ in the representation and processing of visually acquired emotional content. It further suggests a role for this region in utilizing affectively salient visual information to augment motion perception of biologically relevant stimuli.

Lack of orientation specific adaptation to vertically oriented Glass patterns in human visual cortex: an fMRI adaptation investigation

The perception of coherent form configurations in natural scenes relies on the activity of early visual areas that respond to local orientation cues. Subsequently, high-level visual areas pool these local signals to construct a global representation of the initial visual input. However, it is still debated whether neurons in the early visual cortex respond also to global form features. Glass patterns (GPs) are visual stimuli employed to investigate local and global form processing and consist of randomly distributed dots pairs called dipoles arranged to form specific global configurations. In the current study, we used GPs and functional magnetic resonance imaging (fMRI) adaptation to reveal the visual areas that subserve the processing of oriented GPs. Specifically, we adapted participants to vertically oriented GP, then we presented test GPs having either the same or different orientations with respect to the adapting GP. We hypothesized that if local form features are processed exclusively by early visual areas and global form by higher-order visual areas, then the effect of visual adaptation should be more pronounced in higher tier visual areas as it requires global processing of the pattern. Contrary to this expectation, our results revealed that adaptation to GPs is robust in early visual areas (V1, V2, and V3), but not in higher tier visual areas (V3AB and V4v), suggesting that form cues in oriented GPs are primarily derived from local-processing mechanisms that originate in V1. Finally, adaptation to vertically oriented GPs causes a modification in the BOLD response within early visual areas, regardless of the relative orientations of the adapting and test stimuli, indicating a lack of orientation selectivity.

Visuo-vestibular conflicts within the roll plane modulate multisensory verticality perception

The integration of visuo-vestibular information is crucial when interacting with the external environment. Under normal circumstances, vision and vestibular signals provide corroborating information, for example regarding the direction and speed of self-motion. However, conflicts in visuo-vestibular signalling, such as optic flow presented to a stationary observer, can change subsequent processing in either modality. While previous studies have demonstrated the impact of sensory conflict on unisensory visual or vestibular percepts, here we investigated whether visuo-vestibular conflicts impact sensitivity to multisensory percepts, specifically verticality. Participants were exposed to a visuo-vestibular conflicting or non-conflicting motion adaptor before completing a Vertical Detection Task. Sensitivity to vertical stimuli was reduced following visuo-vestibular conflict. No significant differences in criterion were found. Our findings suggest that visuo-vestibular conflicts not only modulate processing in unimodal channels, but also broader multisensory percepts, which may have implications for higher-level processing dependent on the integration of visual and vestibular signals.

Rotating One's Head Modulates the Perceived Velocity of Motion Aftereffect

As a prominent illusion, the motion aftereffect (MAE) has traditionally been considered a visual phenomenon. Recent neuroimaging work has revealed increased activities in MT+ and decreased activities in vestibular regions during the MAE, supporting the notion of visual-vestibular interaction on the MAE. Since the head had to remain stationary in fMRI experiments, vestibular self-motion signals were absent in those studies. Accordingly, more direct evidence is still lacking in terms of whether and how vestibular signals modulate the MAE. By developing a virtual reality approach, the present study for the first time demonstrates that horizontal head rotation affects the perceived velocity of the MAE. We found that the MAE was predominantly perceived as moving faster when its direction was opposite to the direction of head rotation than when its direction was the same as head rotation. The magnitude of this effect was positively correlated with the velocity of head rotation. Similar result patterns were not observed for the real motion stimuli. Our findings support a 'cross-modal bias' hypothesis that after living in a multisensory environment long-term the brain develops a strong association between signals from the visual and vestibular pathways. Consequently, weak biasing visual signals in the associated direction can spontaneously emerge with the input of vestibular signals in the multisensory brain areas, substantially modulating the illusory visual motion represented in those areas as well. The hypothesis can also be used to explain other multisensory integration phenomena.

Sensory processing and categorization in cortical and deep neural networks

Many recent advances in artificial intelligence (AI) are rooted in visual neuroscience. However, ideas from more complicated paradigms like decision-making are less used. Although automated decision-making systems are ubiquitous (driverless cars, pilot support systems, medical diagnosis algorithms etc.), achieving human-level performance in decision making tasks is still a challenge. At the same time, these tasks that are hard for AI are easy for humans. Thus, understanding human brain dynamics during these decision-making tasks and modeling them using deep neural networks could improve AI performance. Here we modelled some of the complex neural interactions during a sensorimotor decision making task. We investigated how brain dynamics flexibly represented and distinguished between sensory processing and categorization in two sensory domains: motion direction and color. We used two different approaches for understanding neural representations. We compared brain responses to 1) the geometry of a sensory or category domain (domain selectivity) and 2) predictions from deep neural networks (computation selectivity). Both approaches gave us similar results. This confirmed the validity of our analyses. Using the first approach, we found that neural representations changed depending on context. We then trained deep recurrent neural networks to perform the same tasks as the animals. Using the second approach, we found that computations in different brain areas also changed flexibly depending on context. Color computations appeared to rely more on sensory processing, while motion computations more on abstract categories. Overall, our results shed light to the biological basis of categorization and differences in selectivity and computations in different brain areas. They also suggest a way for studying sensory and categorical representations in the brain: compare brain responses to both a behavioral model and a deep neural network and test if they give similar results.

The Decision Decoding ToolBOX (DDTBOX) - A Multivariate Pattern Analysis Toolbox for Event-Related Potentials

In recent years, neuroimaging research in cognitive neuroscience has increasingly used multivariate pattern analysis (MVPA) to investigate higher cognitive functions. Here we present DDTBOX, an open-source MVPA toolbox for electroencephalography (EEG) data. DDTBOX runs under MATLAB and is well integrated with the EEGLAB/ERPLAB and Fieldtrip toolboxes (Delorme and Makeig 2004; Lopez-Calderon and Luck 2014; Oostenveld et al. 2011). It trains support vector machines (SVMs) on patterns of event-related potential (ERP) amplitude data, following or preceding an event of interest, for classification or regression of experimental variables. These amplitude patterns can be extracted across space/electrodes (spatial decoding), time (temporal decoding), or both (spatiotemporal decoding). DDTBOX can also extract SVM feature weights, generate empirical chance distributions based on shuffled-labels decoding for group-level statistical testing, provide estimates of the prevalence of decodable information in the population, and perform a variety of corrections for multiple comparisons. It also includes plotting functions for single subject and group results. DDTBOX complements conventional analyses of ERP components, as subtle multivariate patterns can be detected that would be overlooked in standard analyses. It further allows for a more explorative search for information when no ERP component is known to be specifically linked to a cognitive process of interest. In summary, DDTBOX is an easy-to-use and open-source toolbox that allows for characterising the time-course of information related to various perceptual and cognitive processes. It can be applied to data from a large number of experimental paradigms and could therefore be a valuable tool for the neuroimaging community.

Direction selective habituation of motion adaptation

Both adaptation and perceptual learning can change how we perceive the visual environment, reflecting the plasticity of the visual system. Our previous work has investigated the interaction between the two aspects of visual plasticity. One of the main findings is that multiple days of repeated motion adaptation attenuates motion aftereffect, which is explained by habituation of motion adaptation. Interestingly, there was almost no transfer of the effect to the untrained adapter, which differed from the trained adapter in the features including retinotopic location, spatiotopic location, and motion direction. Given that the reference frame of motion aftereffect is proposed to be retinotopic, it remains unclear whether the effect we refer to as habituation effect of motion adaptation is more like a special type of motion adaptation or not. Therefore, in three experiments, we examined the role of retinotopic location, spatiotopic location, and motion direction on the transfer of habituation, respectively. In each experiment, only one of the features was kept the same for the trained and untrained conditions. We found that the habituation effect transferred across both the retinotopic and spatiotopic locations as long as the adapting direction remained the same. The findings indicate that the effect we refer to as habituation of motion adaptation is anchored neither in eye-centered (retinotopic) nor world-centered (spatiotopic) coordinates. Rather, it is specific to the direction of the adapter. Therefore, the habituation effect of motion adaptation cannot be ascribed to a variant of motion adaptation.

The direction after-effect is a global motion phenomenon

Prior experience influences visual perception. For example, extended viewing of a moving stimulus results in the misperception of a subsequent stimulus's motion direction-the direction after-effect (DAE). There has been an ongoing debate regarding the locus of the neural mechanisms underlying the DAE. We know the mechanisms are cortical, but there is uncertainty about where in the visual cortex they are located-at relatively early local motion processing stages, or at later global motion stages. We used a unikinetic plaid as an adapting stimulus, then measured the DAE experienced with a drifting random dot test stimulus. A unikinetic plaid comprises a static grating superimposed on a drifting grating of a different orientation. Observers cannot see the true motion direction of the moving component; instead they see pattern motion running parallel to the static component. The pattern motion of unikinetic plaids is encoded at the global processing level-specifically, in cortical areas MT and MST-and the local motion component is encoded earlier. We measured the direction after-effect as a function of the plaid's local and pattern motion directions. The DAE was induced by the plaid's pattern motion, but not by its component motion. This points to the neural mechanisms underlying the DAE being located at the global motion processing level, and no earlier than area MT.

Functional correlate and delineated connectivity pattern of human motion aftereffect responses substantiate a subjacent visual-vestibular interaction

The visual motion aftereffect (MAE) is the most prominent aftereffect in the visual system. Regarding its function, psychophysical studies suggest its function to be a form of sensory error correction, possibly also triggered by incongruent visual-vestibular stimulation. Several observational imaging experiments have deducted an essential role for region MT+ in the perception of a visual MAE but not provided conclusive evidence. Potential confounders with the MAE such as ocular motor performance, attention, and vection sensations have also never been controlled for. Aim of this neuroimaging study was to delineate the neural correlates of MAE and its subjacent functional connectivity pattern. A rotational MAE (n = 22) was induced using differing visual stimuli whilst modulating ocular motor parameters in a 3T scanner. Data was analyzed with SPM12. Eye movements as a response to the same stimuli were studied by means of high-resolution videooculography. Analysis for all stimuli gave bilateral activations along the dorsal visual stream with an emphasis on area MT. The onset of a visual MAE revealed an additional response in the right medial superior temporal area (MST) and a concurrent deactivation of vestibular hub region OP2. There was no correlation for the BOLD effects during the MAE with either ocular motor or attention parameters. The functional correlate of a visual MAE in humans may be represented in the interaction between region MT and area MST. This MAE representation is independent of a potential afternystagmus, attention and the presence of egomotion sensations. Connectivity analyses showed that in the event of conflicting visual-vestibular motion information (here MAE) area MST and area OP2 may act as the relevant mediating network hubs.

Evidence for distinct levels of neural adaptation to both coherent and incoherently moving visual surfaces in visual area hMT

Visual adaptation describes the processes by which the visual system alters its operating properties in response to changes in the environment. It is one of the mechanisms controlling visual perceptual bistability - when two perceptual solutions are available - by controlling the duration of each percept. Moving plaids are an example of such ambiguity. They can be perceived as two surfaces sliding incoherently over each other or as a single coherent surface. Here, we investigated, using fMRI, whether activity in the human motion complex (hMT+), a region tightly related to the perceptual integration of visual motion, is modulated by distinct forms of visual adaptation to coherent or incoherent perception of moving plaids. Our hypothesis is that exposure to global coherent or incoherent moving stimuli leads to different levels of measurable adaptation, reflected in hMT+ activity. We found that the strength of the measured visual adaptation effect depended on whether subjects integrated (coherent percept) or segregated (incoherent percept) surface motion signals. Visual motion adaptation was significant both for coherent motion and globally incoherent surface motion. Although not as strong as to the coherent percept, visual adaptation due to the incoherent percept also affects hMT+. This shows that adaptation can contribute to regulate percept duration during visual bistability, with distinct weights, depending on the type of percept. Our findings suggest a link between bistability and adaptation mechanisms, both due to coherent and incoherent motion percepts, but in an asymmetric manner. These asymmetric adaptation weights have strong implications in models of perceptual decision and may explain asymmetry of perceptual interpretation periods.

Global Motion Processing in Human Visual Cortical Areas V2 and V3

Global motion perception entails the ability to extract the central direction tendency from an extended area of visual space containing widely disparate local directions. A substantial body of evidence suggests that local motion signals generated in primary visual cortex (V1) are spatially integrated to provide perception of global motion, beginning in the middle temporal area (MT) in macaques and its counterpart in humans, hMT. However, V2 and V3 also contain motion-sensitive neurons that have larger receptive fields than those found in V1, giving the potential for spatial integration of motion signals. Despite this, V2 and V3 have been overlooked as sites of global motion processing. To test, free of local-global confounds, whether human V2 and V3 are important for encoding global motion, we developed a visual stimulus that yields a global direction yet includes all possible local directions and is perfectly balanced at the local motion level. We then attempted to decode global motion direction in such stimuli with multivariate pattern classification of fMRI data. We found strong sensitivity to global motion in hMT, as expected, and also in several higher visual areas known to encode optic flow. Crucially, we found that global motion direction could be decoded in human V2 and, particularly, in V3. The results suggest the surprising conclusion that global motion processing is a key function of cortical visual areas V2 and V3. A possible purpose is to provide global motion signals to V6.

SIGNIFICANCE STATEMENT

Humans can readily detect the overall direction of movement in a flock of birds despite large differences in the directions of individual birds at a given moment. This ability to combine disparate motion signals across space underlies many aspects of visual motion perception and has therefore received considerable research attention. The received wisdom is that spatial integration of motion signals occurs in the cortical motion complex MT+ in both human and nonhuman primates. We show here that areas V2 and V3 in humans are also able to perform this function. We suggest that different cortical areas integrate motion signals in different ways for different purposes.

Opposite effects of high- and low-frequency transcranial random noise stimulation probed with visual motion adaptation

Transcranial random noise stimulation (tRNS) is a recent neuro-modulation technique whose effects at both behavioural and neural level are still debated. Here we employed the well-known phenomenon of motion after-effect (MAE) in order to investigate the effects of high- vs. low-frequency tRNS on motion adaptation and recovery. Participants were asked to estimate the MAE duration following prolonged adaptation (20 s) to a complex moving pattern, while being stimulated with either sham or tRNS across different blocks. Different groups were administered with either high- or low-frequency tRNS. Stimulation sites were either bilateral human MT complex (hMT⁺) or frontal areas. The results showed that, whereas no effects on MAE duration were induced by stimulating frontal areas, when applied to the bilateral hMT⁺, high-frequency tRNS caused a significant decrease in MAE duration whereas low-frequency tRNS caused a significant corresponding increase in MAE duration. These findings indicate that high- and low-frequency tRNS have opposed effects on the adaptation-dependent unbalance between neurons tuned to opposite motion directions, and thus on neuronal excitability.

Auditory Motion Elicits a Visual Motion Aftereffect

The visual motion aftereffect is a visual illusion in which exposure to continuous motion in one direction leads to a subsequent illusion of visual motion in the opposite direction. Previous findings have been mixed with regard to whether this visual illusion can be induced cross-modally by auditory stimuli. Based on research on multisensory perception demonstrating the profound influence auditory perception can have on the interpretation and perceived motion of visual stimuli, we hypothesized that exposure to auditory stimuli with strong directional motion cues should induce a visual motion aftereffect. Here, we demonstrate that horizontally moving auditory stimuli induced a significant visual motion aftereffect—an effect that was driven primarily by a change in visual motion perception following exposure to leftward moving auditory stimuli. This finding is consistent with the notion that visual and auditory motion perception rely on at least partially overlapping neural substrates.

From sensation to perception: Using multivariate classification of visual illusions to identify neural correlates of conscious awareness in space and time

An important goal of cognitive neuroscience is understanding the neural underpinnings of conscious awareness. Although the low-level processing of sensory input is well understood in most modalities, it remains a challenge to understand how the brain translates such input into conscious awareness. Here, I argue that the application of multivariate pattern classification techniques to neuroimaging data acquired while observers experience perceptual illusions provides a unique way to dissociate sensory mechanisms from mechanisms underlying conscious awareness. Using this approach, it is possible to directly compare patterns of neural activity that correspond to the contents of awareness, independent from changes in sensory input, and to track these neural representations over time at high temporal resolution. I highlight five recent studies using this approach, and provide practical considerations and limitations for future implementations.

Visual duration aftereffect is position invariant

Adaptation to relatively long or short sensory events leads to a negative aftereffect, such that the durations of the subsequent events within a certain range appear to be contracted or expanded. The distortion in perceived duration is presumed to arise from the adaptation of duration detectors. Here, we focus on the positional sensitivity of those visual duration detectors by exploring whether the duration aftereffect may be constrained by the visual location of stimuli. We adopted two different paradigms, one that tests for transfer across visual hemifields, and the other that tests for simultaneous selectivity between visual hemifields. By employing these experimental designs, we show that the duration aftereffect strongly transfers across visual hemifields and is not contingent on them. The lack of position specificity suggests that duration detectors in the visual system may operate at a relatively later stage of sensory processing.