Attention to motion enhances processing of both visual and auditory stimuli: an event-related potential study

Attentional modulations of audiovisual interactions in apparent motion: Temporal ventriloquism effects on perceived visual speed

The timing of brief stationary sounds has been shown to alter different aspects of visual motion, such as speed estimation. These effects of auditory timing have been explained by temporal ventriloquism and auditory dominance over visual information in the temporal domain. Although previous studies provide unprecedented evidence for the multisensory nature of speed estimation, how attention is involved in these audiovisual interactions remains unclear. Here, we aimed to understand the effects of spatial attention on these audiovisual interactions in time. We utilized a set of audiovisual stimuli that elicit temporal ventriloquism in visual apparent motion and asked participants to perform a speed comparison task. We manipulated attention either in the visual or auditory domain and systematically changed the number of moving objects in the visual field. When attention was diverted to a stationary object in the visual field via a secondary task, the temporal ventriloquism effects on perceived speed decreased. On the other hand, focusing attention on the auditory stimuli facilitated these effects consistently across different difficulty levels of secondary auditory task. Moreover, the effects of auditory timing on perceived speed did not change with the number of moving objects and existed in all the experimental conditions. Taken together, our findings revealed differential effects of allocating attentional resources in the visual and auditory domains. These behavioral results also demonstrate that reliable temporal ventriloquism effects on visual motion can be induced even in the presence of multiple moving objects in the visual field and under different perceptual load conditions.

Realistic (3D) looming of emotional visual stimuli: Attentional effects at neural and behavioral levels

Previous research shows that endogenous attention (the controlled selection of certain aspects of our environment) is enhanced toward emotional stimuli due to its biological relevance. Although looming affective stimuli such as threat seem even more critical for survival, little is known about their effect on endogenous attention. Here, we recorded neural (event-related potentials, ERPs) and behavioral responses (errors and reaction times) to explore the combined effect of emotion and looming motion. 3D-recreated static and moving animals assessed as emotionally positive, negative, and neutral, were presented to participants (n = 71), who performed an indirect categorization task (vertebrate vs. invertebrate). Behavioral results showed better task performance, as reflected by lower number of errors and reaction times, in response to threatening stimuli. Neural indices revealed significant early (P1p, 150 milliseconds), intermediate (P2p, 240), and late (LPP, 450) effects, the latter being more intensely associated with behavior, as revealed by regression analyses. In general, neural indexes of attention to both static and dynamic stimuli showed a positivity offset in early stages and a negativity bias in subsequent phases. However, and importantly, the progressive inclusion of negative stimuli in the attentional focus is produced earlier in the case of dynamic (at P2p latency) than in static versions (at LPP). These results point to an enhancement of attention, particularly in temporal terms, toward stimuli combining motion and biological significance.

Significance and Novelty effects in single-trial ERP components and autonomic responses

The phasic orienting reflex (OR) was investigated in two counterbalanced blocks of an auditory dishabituation paradigm differing in stimulus Significance (operationalised as tone counting). Twelve tones were presented at very long, randomly-varying interstimulus intervals (ISIs). Novelty and Significance were varied within subjects. Stimulus-response patterns were assessed to find ERP matches for autonomic measures. The phasic OR index was represented by the skin conductance response (SCR). SCR decremented over 10 standard trials, showed recovery on trial 11 (change trial), enhancement to re-presentation of the standard tone (trial 12: dishabituation), and a main effect of Significance over the first 10 trials - demonstrating the formal criteria for an OR index. The evoked cardiac response (HR) subcomponents ECR1 (deceleration) and ECR2 (acceleration) showed no trial effects, but ECR2 showed a Significance effect. Respiratory pause (RP) decreased linearly over trials, and showed recovery, but no dishabituation or Significance effect. Temporal PCA was applied to single-trial EOG-corrected data. Ten ERP components were extracted: P1, N1-3, N1-1, PN, P2, P3a, P3b, HabP3, a Frontal Slow Wave (FSW), and the Classic SW. The dependent measures showed 4 distinct patterns. Pattern 1: No trial or Significance effects (ECR1, P1, N1-3, P3a, FSW); Pattern 2: No trial effect but a Significance effect (ECR2, N1-1, P2); Pattern 3: Trial but not Significance effects (RP, PN, P3b, HabP3); Pattern 4: Both trial and Significance effects (SCR and Classic SW). The evidenced fractionation of autonomic and central measures is compatible with Preliminary Process Theory (PPT), contrary to the notion of a unitary OR.

Trials and intensity effects in single-trial ERP components and autonomic responses in a dishabituation paradigm with very long ISIs

The phasic orienting reflex (OR) was investigated using single-trial data collected concurrently from 4 autonomic measures and event-related potentials (ERPs). In an auditory dishabituation paradigm, twelve indifferent tones of two intensities (60 or 80 dB, intensity change on trial 11, counterbalanced between subjects) were presented at very long interstimulus intervals (ISIs). Novelty and intensity based stimulus-response patterns were examined seeking ERP analogues of autonomic measures representing pre-OR and OR processing. Skin conductance response (SCR) represented the phasic OR index. EOG-corrected ERP data for 16 subjects were decomposed by a temporal Principal Components Analysis (PCA). SCR diminished over 10 standard trials, recovered on change trial 11, dishabituated to the re-presentation of the standard tone on trial 12, and showed intensity effects at the change - formal requirements for an OR index. The evoked cardiac response (HR) showed no trial or intensity effects. Respiratory pause (RP) decreased linearly over trials and showed recovery but no dishabituation or intensity effect. Peripheral vasoconstriction (PVC) failed to decrement but exhibited an intensity effect. Ten identifiable ERP components were extracted: Na, P1, N1-1, PN, P2, P3a, P3b, a novelty-sensitive HabP3, an intensity-sensitive IntP3, and the Slow Wave (SW). Pattern 1 showed no trial or intensity effects (HR, P1, PN, P2); Pattern 2 showed no trial effect but an intensity effect (PVC, Na, N1-1, P3a); and Pattern 3 demonstrated habituation and an intensity effect (SCR, RP, P3b, HabP3, IntP3, SW). The observed fractionation of autonomic and central measures is consistent with Preliminary Process Theory (PPT) rather than the notion of a unitary OR.

Trial effects in single-trial ERP components and autonomic responses at very long ISIs

Single-trial data from autonomic and ERP measures were used to capture the rapidly decreasing initial responses characteristic of the orienting reflex (OR) to repeated stimuli. Stimulus-response patterns were compared to determine central analogues of autonomic indices of processes leading to the OR, and the OR itself. Participants were presented with 12 indifferent tones in an auditory dishabituation paradigm. Temporal principal component analysis (PCA) decomposed EOG-corrected ERP data for 16 subjects. Response patterns of ERPs, cardiac, and respiratory responses were compared to the phasic skin conductance response (SCR). SCR decremented over trials, recovered on the change trial, and dishabituated to the representation of the standard, meeting the formal definition of habituation required of the OR. The evoked cardiac response showed no trial effects. Respiratory pause (RP) decreased linearly over trials, recovering marginally on the change trial. Nine identifiable ERP components were extracted: P1, N1-3, N1-1, processing negativity (PN), P2, P3a, P3b, a novelty-sensitive P3 component (labelled HabP3), and the slow wave (SW). P3b and SW showed decrement over trials, but with no recovery, HabP3 showed decrement and increased response on the change trial, while the P1, N1 subcomponents, P2 and P3a were insensitive to novelty. Stimulus-response patterns of the RP and HabP3 suggest sensitivity to novelty processing, while the P1, N1-3, N-1, PN, P2, P3a and cardiac deceleration appear to mark processing prior to novelty, such as stimulus transient detection (cardiac deceleration) and/or intensity processing. This study supports predictions of preliminary process theory, demonstrating fractionation of 3 autonomic and 9 ERP components to novelty, and disconfirming the unitary nature of the OR.

Visual evoked potentials to change in coloration of a moving bar

In our previous study we found that it takes less time to detect coloration change in a moving object compared to coloration change in a stationary one (Kreegipuu etal., 2006). Here, we replicated the experiment, but in addition to reaction times (RTs) we measured visual evoked potentials (VEPs), to see whether this effect of motion is revealed at the cortical level of information processing. We asked our subjects to detect changes in coloration of stationary (0(°)/s) and moving bars (4.4 and 17.6(°)/s). Psychophysical results replicate the findings from the previous study showing decreased RTs to coloration changes with increase of velocity of the color changing stimulus. The effect of velocity on VEPs was opposite to the one found on RTs. Except for component N1, the amplitudes of VEPs elicited by the coloration change of faster moving objects were reduced than those elicited by the coloration change of slower moving or stationary objects. The only significant effect of velocity on latency of peaks was found for P2 in frontal region. The results are discussed in the light of change-to-change interval and the two methods reflecting different processing mechanisms.

A P300-based brain-computer interface with stimuli on moving objects: four-session single-trial and triple-trial tests with a game-like task design

Brain-computer interfaces (BCIs) are tools for controlling computers and other devices without using muscular activity, employing user-controlled variations in signals recorded from the user's brain. One of the most efficient noninvasive BCIs is based on the P300 wave of the brain's response to stimuli and is therefore referred to as the P300 BCI. Many modifications of this BCI have been proposed to further improve the BCI's characteristics or to better adapt the BCI to various applications. However, in the original P300 BCI and in all of its modifications, the spatial positions of stimuli were fixed relative to each other, which can impose constraints on designing applications controlled by this BCI. We designed and tested a P300 BCI with stimuli presented on objects that were freely moving on a screen at a speed of 5.4°/s. Healthy participants practiced a game-like task with this BCI in either single-trial or triple-trial mode within four sessions. At each step, the participants were required to select one of nine moving objects. The mean online accuracy of BCI-based selection was 81% in the triple-trial mode and 65% in the single-trial mode. A relatively high P300 amplitude was observed in response to targets in most participants. Self-rated interest in the task was high and stable over the four sessions (the medians in the 1st/4th sessions were 79/84% and 76/71% in the groups practicing in the single-trial and triple-trial modes, respectively). We conclude that the movement of stimulus positions relative to each other may not prevent the efficient use of the P300 BCI by people controlling their gaze, e.g., in robotic devices and in video games.

Spatiotemporal dynamics of feature-based attention spread: evidence from combined electroencephalographic and magnetoencephalographic recordings

Attentional selection on the basis of nonspatial stimulus features induces a sensory gain enhancement by increasing the firing-rate of individual neurons tuned to the attended feature, while responses of neurons tuned to opposite feature-values are suppressed. Here we recorded event-related potentials (ERPs) and magnetic fields (ERMFs) in human observers to investigate the underlying neural correlates of feature-based attention at the population level. During the task subjects attended to a moving transparent surface presented in the left visual field, while task-irrelevant probe stimuli executing brief movements into varying directions were presented in the opposite visual field. ERP and ERMF amplitudes elicited by the unattended task-irrelevant probes were modulated as a function of the similarity between their movement direction and the task-relevant movement direction in the attended visual field. These activity modulations reflecting globally enhanced processing of the attended feature were observed to start not before 200 ms poststimulus and were localized to the motion-sensitive area hMT. The current results indicate that feature-based attention operates in a global manner but needs time to spread and provide strong support for the feature-similarity gain model.

Multisensory perceptual learning reshapes both fast and slow mechanisms of crossmodal processing

Previous research has shown that sounds facilitate perception of visual patterns appearing immediately after the sound but impair perception of patterns appearing after some delay. Here we examined the spatial gradient of the fast crossmodal facilitation effect and the slow inhibition effect in order to test whether they reflect separate mechanisms. We found that crossmodal facilitation is only observed at visual field locations overlapping with the sound, whereas crossmodal inhibition affects the whole hemifield. Furthermore, we tested whether multisensory perceptual learning with misaligned audio-visual stimuli reshapes crossmodal facilitation and inhibition. We found that training shifts crossmodal facilitation towards the trained location without changing its range. By contrast, training narrows the range of inhibition without shifting its position. Our results suggest that crossmodal facilitation and inhibition reflect separate mechanisms that can both be reshaped by multisensory experience even in adult humans. Multisensory links seem to be more plastic than previously thought.

Influence of early attentional modulation on working memory

It is now established that attention influences working memory (WM) at multiple processing stages. This liaison between attention and WM poses several interesting empirical questions. Notably, does attention impact WM via its influences on early perceptual processing? If so, what are the critical factors at play in this attention-perception-WM interaction. I review recent data from our laboratory utilizing a variety of techniques (electroencephalography (EEG), functional MRI (fMRI) and transcranial magnetic stimulation (TMS)), stimuli (features and complex objects), novel experimental paradigms, and research populations (younger and older adults), which converge to support the conclusion that top-down modulation of visual cortical activity at early perceptual processing stages (100-200 ms after stimulus onset) impacts subsequent WM performance. Factors that affect attentional control at this stage include cognitive load, task practice, perceptual training, and aging. These developments highlight the complex and dynamic relationships among perception, attention, and memory.

3D surface perception from motion involves a temporal-parietal network

Previous research has suggested that three-dimensional (3D) structure-from-motion (SFM) perception in humans involves several motion-sensitive occipital and parietal brain areas. By contrast, SFM perception in nonhuman primates seems to involve the temporal lobe including areas MT, MST and FST. The present functional magnetic resonance imaging study compared several motion-sensitive regions of interest including the superior temporal sulcus (STS) while human observers viewed horizontally moving dots that defined either a 3D corrugated surface or a 3D random volume. Low-level stimulus features such as dot density and velocity vectors as well as attention were tightly controlled. Consistent with previous research we found that 3D corrugated surfaces elicited stronger responses than random motion in occipital and parietal brain areas including area V3A, the ventral and dorsal intraparietal sulcus, the lateral occipital sulcus and the fusiform gyrus. Additionally, 3D corrugated surfaces elicited stronger activity in area MT and the STS but not in area MST. Brain activity in the STS but not in area MT correlated with interindividual differences in 3D surface perception. Our findings suggest that area MT is involved in the analysis of optic flow patterns such as speed gradients and that the STS in humans plays a greater role in the analysis of 3D SFM than previously thought.

Neural suppression of irrelevant information underlies optimal working memory performance

Our ability to focus attention on task-relevant information and ignore distractions is reflected by differential enhancement and suppression of neural activity in sensory cortex (i.e., top-down modulation). Such selective, goal-directed modulation of activity may be intimately related to memory, such that the focus of attention biases the likelihood of successfully maintaining relevant information by limiting interference from irrelevant stimuli. Despite recent studies elucidating the mechanistic overlap between attention and memory, the relationship between top-down modulation of visual processing during working memory (WM) encoding, and subsequent recognition performance has not yet been established. Here, we provide neurophysiological evidence in healthy, young adults that top-down modulation of early visual processing (< 200 ms from stimulus onset) is intimately related to subsequent WM performance, such that the likelihood of successfully remembering relevant information is associated with limiting interference from irrelevant stimuli. The consequences of a failure to ignore distractors on recognition performance was replicated for two types of feature-based memory, motion direction and color. Moreover, attention to irrelevant stimuli was reflected neurally during the WM maintenance period as an increased memory load. These results suggest that neural enhancement of relevant information is not the primary determinant of high-level performance, but rather optimal WM performance is dependent on effectively filtering irrelevant information through neural suppression to prevent overloading a limited memory capacity.

Spatio-temporal Analysis of Feature-Based Attention

The cortical mechanisms of feature-selective attention to color and motion cues were studied in humans using combined electrophysiological, magnetoencephalographic, and hemodynamic (functional magnetic resonance imaging) measures of brain activity. Subjects viewed a display of random dots that periodically either changed color or moved coherently. When attention was directed to the color change it elicited enhanced neural activity in visual area V4v, previously shown to be specialized for processing color information. In contrast, when dot movement was attended it produced enhanced activity in the motion-specialized area human MT. Parallel recordings of event-related electrophysiological and magnetoencephalographic responses indicated that the attention-related facilitation of neural activity in these specialized cortical areas occurred rapidly, beginning as early as 90-120 ms after stimulus onset. We conclude that selection of an entire feature dimension (motion or color) boosts neural activity in its specialized cortical module much more rapidly than does selection of one feature value from another (e.g., one color from another), as reported in previous electrophysiological studies. By combining methods with high spatial and temporal resolution it is possible to analyze the precise time course of feature-selective processing in specialized cortical areas.

Neural Correlates of Coherent Audiovisual Motion Perception

Real-life moving objects are often detected by multisensory cues. We investigated the cortical activity associated with coherent visual motion perception in the presence of a stationary or moving auditory noise source using functional magnetic resonance imaging. Twelve subjects judged episodes of 5-s random-dot motion containing either no (0%) or abundant (16%) coherent direction information. Auditory noise was presented with the displayed visual motion that was moving in phase, was moving out-of-phase, or was stationary. Subjects judged whether visual coherent motion was present, and if so, whether the auditory noise source was moving in phase, was moving out-of-phase, or was not moving. Performance was greatest for a moving sound source that was in phase with the visual coherent dot motion compared with when it was in antiphase. A random-effects analysis revealed that auditory motion activated extended regions in both cerebral hemispheres in the superior temporal gyrus (STG), with a right-hemispheric preponderance. Combined audiovisual motion led to activation clusters in the STG, the supramarginal gyrus, the superior parietal lobule, and the cerebellum. The size of the activated regions was substantially larger than that evoked by either visual or auditory motion alone. The congruent audiovisual motion evoked the most extensive activation pattern, exhibiting several exclusively activated subregions.

Attending to visual or auditory motion affects perception within and across modalities: an event-related potential study

The present event-related potential (ERP) study examined the role of dynamic features in multisensory binding. It was tested whether endogenous attention to the direction of motion affects processing of visual and auditory stimuli within and across modalities. Human participants perceived horizontally moving dot patterns and sounds that were presented either continuously (standards) or briefly interrupted (infrequent deviants). Their task was to detect deviants moving in a particular direction within a primary modality, but to detect all deviants irrespective of their motion direction within the secondary modality. Attending to the direction of visual motion resulted in a broad selection negativity (SN) starting at about 200 ms post-stimulus onset, and attending to the direction of auditory motion resulted in a positive difference wave at 150 ms that was followed by a broad negativity starting at about 200 ms (unimodal effects). Moreover, dot patterns moving in a direction that was attended within audition were detected faster and more accurately than oppositely moving stimuli and elicited a cross-modal SN wave. Corresponding cross-modal behavioural and ERP results were obtained for sounds moving in a direction that was attended within vision. Unimodal and cross-modal ERP attention effects partially differed in their scalp topography. The present study shows that dynamic features (direction of motion) may be used to link input across modalities and demonstrates for the first time that these multisensory interactions take place as early as about 200 ms after stimulus onset.