Multivariate EEG analyses support high-resolution tracking of feature-based attentional selection

Automatic processing of unattended mean emotion: Evidence from visual mismatch responses

The mean emotion from multiple facial expressions can be extracted rapidly and precisely. However, it remains debated whether mean emotion processing is automatic which can occur under no attention. To address this question, we used a passive oddball paradigm and recorded event-related brain potentials when participants discriminated the changes in the central fixation while a set of four faces was presented in the periphery. The face set consisted of one happy and three angry expressions (mean negative) or one angry and three happy expressions (mean positive), and the mean negative and mean positive face sets were shown with a probability of 20% (deviant) and 80% (standard) respectively in the sequence, or the vice versa. The cluster-based permutation analyses showed that the visual mismatch negativity started early at around 92 ms and was also observed in later time windows when the mean emotion was negative, while a mismatch positivity was observed at around 168-266 ms when the mean emotion was positive. The results suggest that there might be different mechanisms underlying the processing of mean negative and mean positive emotions. More importantly, the brain can detect the changes in the mean emotion automatically, and ensemble coding for multiple facial expressions can occur in an automatic fashion without attention.

Decoding cognition in neurodevelopmental, psychiatric and neurological conditions with multivariate pattern analysis of EEG data

Multivariate pattern analysis (MVPA) of electroencephalographic (EEG) data represents a revolutionary approach to investigate how the brain encodes information. By considering complex interactions among spatio-temporal features at the individual level, MVPA overcomes the limitations of univariate techniques, which often fail to account for the significant inter- and intra-individual neural variability. This is particularly relevant when studying clinical populations, and therefore MVPA of EEG data has recently started to be employed as a tool to study cognition in brain disorders. Here, we review the insights offered by this methodology in the study of anomalous patterns of neural activity in conditions such as autism, ADHD, schizophrenia, dyslexia, neurological and neurodegenerative disorders, within different cognitive domains (perception, attention, memory, consciousness). Despite potential drawbacks that should be attentively addressed, these studies reveal a peculiar sensitivity of MVPA in unveiling dysfunctional and compensatory neurocognitive dynamics of information processing, which often remain blind to traditional univariate approaches. Such higher sensitivity in characterizing individual neurocognitive profiles can provide unique opportunities to optimise assessment and promote personalised interventions.

An early effect of the parafoveal preview on post-saccadic processing of English words

A key aspect of efficient visual processing is to use current and previous information to make predictions about what we will see next. In natural viewing, and when looking at words, there is typically an indication of forthcoming visual information from extrafoveal areas of the visual field before we make an eye movement to an object or word of interest. This "preview effect" has been studied for many years in the word reading literature and, more recently, in object perception. Here, we integrated methods from word recognition and object perception to investigate the timing of the preview on neural measures of word recognition. Through a combined use of EEG and eye-tracking, a group of multilingual participants took part in a gaze-contingent, single-shot saccade experiment in which words appeared in their parafoveal visual field. In valid preview trials, the same word was presented during the preview and after the saccade, while in the invalid condition, the saccade target was a number string that turned into a word during the saccade. As hypothesized, the valid preview greatly reduced the fixation-related evoked response. Interestingly, multivariate decoding analyses revealed much earlier preview effects than previously reported for words, and individual decoding performance correlated with participant reading scores. These results demonstrate that a parafoveal preview can influence relatively early aspects of post-saccadic word processing and help to resolve some discrepancies between the word and object literatures.

Neural activity related to productive vocabulary knowledge effects during second language comprehension

Second language learners and educators often believe that improving one's listening ability hinges on acquiring an extensive vocabulary and engaging in thorough listening practice. Our previous study suggested that listening comprehension is also impacted by the ability to produce vocabulary. Nevertheless, it remained uncertain whether quick comprehension could be attributed to a simple acceleration of processing or to changes in neural activity. To identify neural activity changes during sentence listening comprehension according to different levels of lexical knowledge (productive, only comprehensive, uncomprehensive), we measured participants' electrical activity in the brain via electroencephalography (EEG) and conducted a time-frequency-based EEG power analysis. Additionally, we employed a decoding model to verify the predictability of vocabulary knowledge levels based on neural activity. The decoding results showed that EEG activity could discriminate between listening to sentences containing phrases that include productive knowledge and ones without. The positive impact of productive vocabulary knowledge on sentence comprehension, driven by distinctive neural processing during sentence comprehension, was unequivocally evident. Our study emphasizes the importance of productive vocabulary knowledge acquisition to enhance the process of second language listening comprehension.

The neural dynamics of conflict adaptation induced by conflict observation: Evidence from univariate and multivariate analysis

Conflict adaptation can be expressed as greater performance (shorter response time and lower error rate) after incongruent trials when compared to congruent trials. It has been observed in designs that minimize confounding factors, i.e., feature integration, contingency learning, and temporal learning. Our current study aimed to further elucidate the temporal evolution mechanisms of conflict adaptation. To address this issue, the current study employed a combination of behavioral, univariate, and multivariate analysis (MVPA) methods in a modified color-word Stroop task, where half of the trials required button presses (DO trials), and the other half only required observation (LOOK trials). Both behavioral and the ERP results (N450 and SP) in the LOOK-DO transition trials revealed significant conflict adaptation without feature integration, contingency learning, and temporal learning, providing support for the conflict monitoring theory. Furthermore, during the LOOK trials, significant Stroop effect in the N450 and SP components were observed, indicating that conflict monitoring occurred at the stimulus level and triggered reactive control adjustments. The MVPA results decoded the congruent-incongruent and incongruent-incongruent conditions during the conflict adjustment phase but not during the conflict monitoring phase, emphasizing the unique contribution of conflict adjustment to conflict adaptation. The current research findings provided more compelling supporting evidence for the conflict monitoring theory, while also indicating that future studies should employ the present design to elucidate the specific processes of conflict adaptation.

Spatial Decoding for Gaze Independent Brain-Computer Interface Based on Covert Visual Attention Shift Using Electroencephalography

The gaze-independent brain-computer interface (BCI) device is used to re-establish interaction for individuals who have abnormal eye movement. It may be possible to control the BCI by shifting your attention spatially. However, spatial attention is rarely employed to increase the effectiveness of target detection and is typically used to provide a simple "yes" or "no" response to the target recognition inquiry. To improve the effectiveness of detecting target, it is crucial to take advantage of the possible advantages of spatial attention. N2-posterior-contralateral (N2pc) component reflects correlates of visual spatial attention and is used to determine target position. In this study, a long-short-term memory (LSTM) network is used to answer "yes/no" questions by decoding covert spatial attention based on N2pc characteristics using EEG signals. The proposed LSTM-based model's average decoding accuracy is 92.79%. The target detection efficiency was successfully increased by about 4% when compared to conventional machine learning algorithms. The proposed model is tested on the independent dataset to validate its performance. The results of this work show that N2pc characteristics can be employed in gaze-independent BCIs for tracking covert attention shifts, which may help persons with poor eye mobility to connect with their environment.

Prolonged neural encoding of visual information in autism

Autism spectrum disorder (ASD) is associated with a hyper-focused visual attentional style, impacting higher-order social and affective domains. The understanding of such peculiarity can benefit from the use of multivariate pattern analysis (MVPA) of high-resolution electroencephalography (EEG) data, which has proved to be a powerful technique to investigate the hidden neural dynamics orchestrating sensory and cognitive processes. Here, we recorded EEG in typically developing (TD) children and in children with ASD during a visuo-spatial attentional task where attention was exogenously captured by a small (zoom-in) or large (zoom-out) cue in the visual field before the appearance of a target at different eccentricities. MVPA was performed both in the cue-locked period, to reveal potential differences in the modulation of the attentional focus, and in the target-locked period, to reveal potential cascade effects on stimulus processing. Cue-locked MVPA revealed that while in the TD group the pattern of neural activity contained information about the cue mainly before the target appearance, the ASD group showed a temporally sustained and topographically diffuse significant decoding of the cue neural response even after the target onset, suggesting a delayed extinction of cue-related neural activity. Crucially, this delayed extinction positively correlated with behavioral measures of attentional hyperfocusing. Results of target-locked MVPA were coherent with a hyper-focused attentional profile, highlighting an earlier and stronger decoding of target neural responses in small cue trials in the ASD group. The present findings document a spatially and temporally overrepresented encoding of visual information in ASD, which can constitute one of the main reasons behind their peculiar cognitive style.

The time course of category-based attentional template pre-activation depends on the category framework

When searching for a target defined by a set of objects, attention can be directed toward task-relevant objects by creating a category-based attentional template (CAT). Previous studies have found that CAT can be activated before the onset of the target. However, the time course of CAT pre-activation and whether the category framework (prototypical or semantic) can modulate it remain unclear. To explore the time course of CAT pre-activation, we employed a rapid serial probe presentation paradigm (RSPP) with event-related potentials (ERPs). To investigate the effect of the category framework on the time course of CAT pre-activation, the target category was defined as the prototypical category (Experiment 1) or the semantic category (Experiment 2). The results showed that the prototype-based CAT was pre-activated 300 ms prior to the target, whereas the semantics-based CAT was pre-activated 1500 ms before the onset of the target. The difference in the time course of pre-activation between the two CAT types indicates that the category framework can modulate the time course of CAT pre-activation. Additionally, during the attentional selection phase, an overall comparison of the target revealed that a larger N2pc was elicited by the prototype-based CAT than by the semantics-based CAT, suggesting that the prototype-based CAT could capture more attention than the semantics-based CAT. The findings on the difference between the two CAT frameworks in the preparatory and attentional selection phases provide more evidence for categorical information in visual search and extend our understanding of the mechanism of categorical attentional selection.

Is covert attention necessary for programming accurate saccades? Evidence from saccade-locked event-related potentials

For decades, researchers have assumed that shifts of covert attention mandatorily occur prior to eye movements to improve perceptual processing of objects before they are fixated. However, recent research suggests that the N2pc component-a neural measure of covert attentional allocation-does not always precede eye movements. The current study investigated whether the N2pc component mandatorily precedes eye movements and assessed its role in the accuracy of gaze control. In three experiments, participants searched for a letter of a specific color (e.g., red) and directed gaze to it as a response. Electroencephalograms and eye movements were coregistered to determine whether neural markers of covert attention preceded the initial shift of gaze. The results showed that the presaccadic N2pc only occurred under limited conditions: when there were many potential target locations and distractors. Crucially, there was no evidence that the presence or magnitude of the presaccadic N2pc was associated with improved eye movement accuracy in any of the experiments. Interestingly, ERP decoding analyses were able to classify the target location well before the eyes started to move, which likely reflects a presaccadic cognitive process that is distinct from the attentional process measured by the N2pc. Ultimately, we conclude that the covert attentional mechanism indexed by the N2pc is not necessary for precise gaze control.

Enhancing precision in human neuroscience

Human neuroscience has always been pushing the boundary of what is measurable. During the last decade, concerns about statistical power and replicability - in science in general, but also specifically in human neuroscience - have fueled an extensive debate. One important insight from this discourse is the need for larger samples, which naturally increases statistical power. An alternative is to increase the precision of measurements, which is the focus of this review. This option is often overlooked, even though statistical power benefits from increasing precision as much as from increasing sample size. Nonetheless, precision has always been at the heart of good scientific practice in human neuroscience, with researchers relying on lab traditions or rules of thumb to ensure sufficient precision for their studies. In this review, we encourage a more systematic approach to precision. We start by introducing measurement precision and its importance for well-powered studies in human neuroscience. Then, determinants for precision in a range of neuroscientific methods (MRI, M/EEG, EDA, Eye-Tracking, and Endocrinology) are elaborated. We end by discussing how a more systematic evaluation of precision and the application of respective insights can lead to an increase in reproducibility in human neuroscience.

What makes somatosensory short-term memory maintenance effective? An EEG study comparing contralateral delay activity between sighted participants and participants who are blind

Somatosensory short-term memory is essential for object recognition, sensorimotor learning, and, especially, Braille reading for people who are blind. This study examined how visual sensory deprivation and a compensatory focus on somatosensory information influences memory processes in this domain. We measured slow cortical negativity developing during short-term tactile memory maintenance (tactile contralateral delay activity, tCDA) in frontal and somatosensory areas while a sample of 24 sighted participants and 22 participants who are blind completed a tactile change-detection task where varying loads of Braille pin patterns served as stimuli. Auditory cues, appearing at varying latencies between sample arrays, could be used to reduce memory demands during maintenance. Participants who are blind (trained Braille readers) outperformed sighted participants behaviorally. In addition, while task-related frontal activation featured in both groups, participants who are blind uniquely showed higher tCDA amplitudes specifically over somatosensory areas. The site specificity of this component's functional relevance in short-term memory maintenance was further supported by somatosensory tCDA amplitudes first correlating across the whole sample with behavioral performance, and secondly showing sensitivity to varying memory load. The results substantiate sensory recruitment models and provide new insights into the effects of visual sensory deprivation on tactile processing. Between-group differences in the interplay between frontal and somatosensory areas during somatosensory maintenance also suggest that efficient maintenance of complex tactile stimuli in short-term memory is primarily facilitated by lateralized activity in somatosensory cortex.

Spatial localization in target detection based on decoding N2pc component

BACKGROUND

The Gaze-independent BCI system is used to restore communication in patients with eye movement disorders. One available control mechanism is the utilization of spatial attention. However, spatial information is mostly used to simply answer the "True/False" target recognition question and is seldom used to improve the efficiency of target detection. Therefore, it is necessary to utilize the potential advantages of spatial attention to improving the target detection efficiency.

NEW METHOD

We found that N2pc could be used to assess spatial attention shift and determine target position. It was a negative wave in the posterior brain on the contralateral target stimulus. From this, we designed a novel spatial coding paradigm to achieve two main purposes at each stimulus presentation: target recognition and spatial localization.

COMPARISON WITH EXISTING METHODS

We used a two-step classification framework to decode the P300 and N2pc components.

RESULTS

The average decoding accuracy of fourteen subjects was 84.43% (σ = 1.14%), and the classification accuracy of six subjects was more than 85%. The information transfer rate of the spatial coding paradigm could reach 60.52 bits/min. Compared with the single stimulus paradigm, the target detection efficiency was successfully improved by approximately 10%.

CONCLUSIONS

The spatial coding paradigm proposed in this paper answered both "True/False" and "Left/Right" questions by decoding spatial attention information. This method could significantly improve image detection efficiencies, such as visual search tasks, Internet image screening, or military target determination.

Decoding Object-Based Auditory Attention from Source-Reconstructed MEG Alpha Oscillations

How do we attend to relevant auditory information in complex naturalistic scenes? Much research has focused on detecting which information is attended, without regarding underlying top-down control mechanisms. Studies investigating attentional control generally manipulate and cue specific features in simple stimuli. However, in naturalistic scenes it is impossible to dissociate relevant from irrelevant information based on low-level features. Instead, the brain has to parse and select auditory objects of interest. The neural underpinnings of object-based auditory attention remain not well understood. Here we recorded MEG while 15 healthy human subjects (9 female) prepared for the repetition of an auditory object presented in one of two overlapping naturalistic auditory streams. The stream containing the repetition was prospectively cued with 70% validity. Crucially, this task could not be solved by attending low-level features, but only by processing the objects fully. We trained a linear classifier on the cortical distribution of source-reconstructed oscillatory activity to distinguish which auditory stream was attended. We could successfully classify the attended stream from alpha (8-14 Hz) activity in anticipation of repetition onset. Importantly, attention could only be classified from trials in which subjects subsequently detected the repetition, but not from miss trials. Behavioral relevance was further supported by a correlation between classification accuracy and detection performance. Decodability was not sustained throughout stimulus presentation, but peaked shortly before repetition onset, suggesting that attention acted transiently according to temporal expectations. We thus demonstrate anticipatory alpha oscillations to underlie top-down control of object-based auditory attention in complex naturalistic scenes.SIGNIFICANCE STATEMENT In everyday life, we often find ourselves bombarded with auditory information, from which we need to select what is relevant to our current goals. Previous research has highlighted how we attend to specific highly controlled aspects of the auditory input. Although invaluable, it is still unclear how this relates to attentional control in naturalistic auditory scenes. Here we used the high precision of magnetoencephalography in space and time to investigate the brain mechanisms underlying top-down control of object-based attention in ecologically valid sound scenes. We show that rhythmic activity in auditory association cortex at a frequency of ∼10 Hz (alpha waves) controls attention to currently relevant segments within the auditory scene and predicts whether these segments are subsequently detected.

Spatially Guided Distractor Suppression during Visual Search

Past work has demonstrated that active suppression of salient distractors is a critical part of visual selection. Evidence for goal-driven suppression includes below-baseline visual encoding at the position of salient distractors (Gaspelin and Luck, 2018) and neural signals such as the distractor positivity (Pd) that track how many distractors are presented in a given hemifield (Feldmann-Wüstefeld and Vogel, 2019). One basic question regarding distractor suppression is whether it is inherently spatial or nonspatial in character. Indeed, past work has shown that distractors evoke both spatial (Theeuwes, 1992) and nonspatial forms of interference (Folk and Remington, 1998), motivating a direct examination of whether space is integral to goal-driven distractor suppression. Here, we use behavioral and EEG data from adult humans (male and female) to provide clear evidence for a spatial gradient of suppression surrounding salient singleton distractors. Replicating past work, both reaction time and neural indices of target selection improved monotonically as the distance between target and distractor increased. Importantly, these target selection effects were paralleled by a monotonic decline in the amplitude of the Pd, an electrophysiological index of distractor suppression. Moreover, multivariate analyses revealed spatially selective activity in the θ-band that tracked the position of the target and, critically, revealed suppressed activity at spatial channels centered on distractor positions. Thus, goal-driven selection of relevant over irrelevant information benefits from a spatial gradient of suppression surrounding salient distractors.SIGNIFICANCE STATEMENT Past work has shown that distractor suppression is an important part of goal-driven attentional selection, but has not yet revealed whether suppression is spatially directed. Using behavioral data, event-related potentials (ERPs) of the EEG signal [N2pc and distractor positivity (Pd) component], as well as a multivariate model of EEG data [channel tuning functions (CTF)], we show that suppression-related neural activity increases monotonically as the distance between targets and distractors decreases, and that spatially-selective activity in the θ-band reveals depressed activity in spatial channels that index distractor positions. Thus, we provide robust evidence for spatially-guided distractor suppression, a result that has important implications for models of goal-driven attentional control.

High-pass filtering artifacts in multivariate classification of neural time series data

BACKGROUND

Traditionally, EEG/MEG data are high-pass filtered and baseline-corrected to remove slow drifts. Minor deleterious effects of high-pass filtering in traditional time-series analysis have been well-documented, including temporal displacements. However, its effects on time-resolved multivariate pattern classification analyses (MVPA) are largely unknown.

NEW METHOD

To prevent potential displacement effects, we extend an alternative method of removing slow drift noise - robust detrending - with a procedure in which we mask out all cortical events from each trial. We refer to this method as trial-masked robust detrending.

RESULTS

In both real and simulated EEG data of a working memory experiment, we show that both high-pass filtering and standard robust detrending create artifacts that result in the displacement of multivariate patterns into activity silent periods, particularly apparent in temporal generalization analyses, and especially in combination with baseline correction. We show that trial-masked robust detrending is free from such displacements.

COMPARISON WITH EXISTING METHOD(S)

Temporal displacement may emerge even with modest filter cut-off settings such as 0.05 Hz, and even in regular robust detrending. However, trial-masked robust detrending results in artifact-free decoding without displacements. Baseline correction may unwittingly obfuscate spurious decoding effects and displace them to the rest of the trial.

CONCLUSIONS

Decoding analyses benefit from trial-masked robust detrending, without the unwanted side effects introduced by filtering or regular robust detrending. However, for sufficiently clean data sets and sufficiently strong signals, no filtering or detrending at all may work adequately. Implications for other types of data are discussed, followed by a number of recommendations.

Impact of Stimulus Features on the Performance of a Gaze-Independent Brain-Computer Interface Based on Covert Spatial Attention Shifts

Regaining communication abilities in patients who are unable to speak or move is one of the main goals in decoding brain waves for brain-computer interface (BCI) control. Many BCI approaches designed for communication rely on attention to visual stimuli, commonly applying an oddball paradigm, and require both eye movements and adequate visual acuity. These abilities may, however, be absent in patients who depend on BCI communication. We have therefore developed a response-based communication BCI, which is independent of gaze shifts but utilizes covert shifts of attention to the left or right visual field. We recorded the electroencephalogram (EEG) from 29 channels and coregistered the vertical and horizontal electrooculogram. Data-driven decoding of small attention-based differences between the hemispheres, also known as N2pc, was performed using 14 posterior channels, which are expected to reflect correlates of visual spatial attention. Eighteen healthy participants responded to 120 statements by covertly directing attention to one of two colored symbols (green and red crosses for "yes" and "no," respectively), presented in the user's left and right visual field, respectively, while maintaining central gaze fixation. On average across participants, 88.5% (std: 7.8%) of responses were correctly decoded online. In order to investigate the potential influence of stimulus features on accuracy, we presented the symbols with different visual angles, by altering symbol size and eccentricity. The offline analysis revealed that stimulus features have a minimal impact on the controllability of the BCI. Hence, we show with our novel approach that spatial attention to a colored symbol is a robust method with which to control a BCI, which has the potential to support severely paralyzed people with impaired eye movements and low visual acuity in communicating with their environment.

Multivariate analysis of EEG activity indexes contingent attentional capture

An extensive body of work has shown that attentional capture is contingent on the goals of the observer: Capture is strongly reduced or even eliminated when an irrelevant singleton stimulus does not match the target-defining properties (Folk et al., 1992). There has been a long-standing debate on whether attentional capture can be explained by goal-driven and/or stimulus-driven accounts. Here, we shed further light on this matter by using EEG activity (raw EEG and alpha power) to provide a time-resolved index of attentional orienting towards salient stimuli that either matched or did not match target-defining properties. A search display containing the target stimulus was preceded by a spatially uninformative singleton cue that either matched the color of the upcoming target (contingent cues), or that appeared in an irrelevant color (non-contingent cues). Multivariate analysis of raw EEG and alpha power revealed preferential tuning to the location of both contingent and non-contingent cues, with a stronger bias towards contingent than non-contingent cues. The time course of these effects, however, depended on the neural signal. Raw EEG data revealed attentional orienting towards the contingent cue early on in the trial (>156 ms), while alpha power revealed sustained spatial selection in the cued locations at a later moment in the trial (>250 ms). Moreover, while raw EEG showed stronger capture by contingent cues during this early time window, an advantage for contingent cues arose during a later time window in alpha band activity. Thus, our findings suggest that raw EEG activity and alpha-band power tap into distinct neural processes that index separate aspects of covert spatial attention.

Multivariate analysis reveals a generalizable human electrophysiological signature of working memory load

Working memory (WM) is an online memory system that is critical for holding information in a rapidly accessible state during ongoing cognitive processing. Thus, there is strong value in methods that provide a temporally resolved index of WM load. While univariate EEG signals have been identified that vary with WM load, recent advances in multivariate analytic approaches suggest that there may be rich sources of information that do not generate reliable univariate signatures. Here, using data from four published studies (n = 286 and >250,000 trials), we demonstrate that multivariate analysis of EEG voltage topography provides a sensitive index of the number of items stored in WM that generalizes to novel human observers. Moreover, multivariate load detection ("mvLoad") can provide robust information at the single-trial level, exceeding the sensitivity of extant univariate approaches. We show that this method tracks WM load in a manner that is (1) independent of the spatial position of the memoranda, (2) precise enough to differentiate item-by-item increments in the number of stored items, (3) generalizable across distinct tasks and stimulus displays, and (4) correlated with individual differences in WM behavior. Thus, this approach provides a powerful complement to univariate analytic approaches, enabling temporally resolved tracking of online memory storage in humans.

Decoding the covert shift of spatial attention from electroencephalographic signals permits reliable control of a brain-computer interface

OBJECTIVE

One of the main goals of brain-computer interfaces (BCI) is to restore communication abilities in patients. BCIs often use event-related potentials (ERPs) like the P300 which signals the presence of a target in a stream of stimuli. The P300 and related approaches, however, are inherently limited, as they require many stimulus presentations to obtain a usable control signal. Many approaches depend on gaze direction to focus the target, which is also not a viable approach in many cases, because eye movements might be impaired in potential users. Here we report on a BCI that avoids both shortcomings by decoding spatial target information, independent of gaze shifts.

APPROACH

We present a new method to decode from the electroencephalogram (EEG) covert shifts of attention to one out of four targets simultaneously presented in the left and right visual field. The task is designed to evoke the N2pc component-a hemisphere lateralized response, elicited over the occipital scalp contralateral to the attended target. The decoding approach involves decoding of the N2pc based on data-driven estimation of spatial filters and a correlation measure.

MAIN RESULTS

Despite variability of decoding performance across subjects, 22 out of 24 subjects performed well above chance level. Six subjects even exceeded 80% (cross-validated: 89%) correct predictions in a four-class discrimination task. Hence, the single-trial N2pc proves to be a component that allows for reliable BCI control. An offline analysis of the EEG data with respect to their dependence on stimulation time and number of classes demonstrates that the present method is also a workable approach for two-class tasks.

SIGNIFICANCE

Our method extends the range of strategies for gaze-independent BCI control. The proposed decoding approach has the potential to be efficient in similar applications intended to decode ERPs.

Decoding the Temporal Dynamics of Covert Spatial Attention Using Multivariate EEG Analysis: Contributions of Raw Amplitude and Alpha Power

Attention can be oriented in space covertly without the need of eye movements. We used multivariate pattern classification analyses (MVPA) to investigate whether the time course of the deployment of covert spatial attention leading up to the observer’s perceptual decision can be decoded from both EEG alpha power and raw activity traces. Decoding attention from these signals can help determine whether raw EEG signals and alpha power reflect the same or distinct features of attentional selection. Using a classical cueing task, we showed that the orientation of covert spatial attention can be decoded by both signals. However, raw activity and alpha power may reflect different features of spatial attention, with alpha power more associated with the orientation of covert attention in space and raw activity with the influence of attention on perceptual processes.

Distilling the distinct contralateral and ipsilateral attentional responses to lateral stimuli and the bilateral response to midline stimuli for upper and lower visual hemifield locations

A contralateral posterior negativity elicited by lateral oddballs (N2pc) and a bilateral posterior negativity elicited by vertical midline oddballs (bilateral N2) are ERP components reflecting attentional deployment that have been rarely compared. In different tasks, we explored to what extent they reflect similar underlying mechanisms of attention. We used a multiple-frame procedure to present pop-out color oddballs among distractors. A homogeneous condition contained only distractors (0 oddballs) and served as a control condition that was subtracted from oddball-present conditions to isolate attention effects. The number of oddballs and the vertical hemifield containing them (upper vs. lower) were two critical factors. For the lower hemifield, the signal amplitude increased with the number of oddballs, otherwise had similar effects and scalp distributions, suggesting the bilateral N2 acted as a bilateral N2pc and likely reflected similar underlying generators. For the upper hemifield, component amplitude also increased with the number of oddballs, but the scalp distributions were positive and more centered, suggesting inverted generators across the two vertical hemifields. An ipsilateral positivity occurred about 50 ms after a contralateral positivity, similar in magnitude, producing a biphasic contra-minus-ipsi difference wave. Previously reported smaller negative N2pc components for upper hemifield oddballs likely reflected a negative lobe artificially created by the subtraction of a lagged positive ipsilateral response. The results compel us to argue for a systematic separation of data for upper versus lower hemifields in studies of visuo-spatial attention, and the use of an experimental design permitting the separate estimation of contralateral and ipsilateral responses.

Assessing the information content of ERP signals in schizophrenia using multivariate decoding methods

•

This study took multivariate decoding methods that are widely used to assess the nature of neural representations in neurotypical people and applied them to a comparison of people with schizophrenia and matched control subjects.

•

Participants performed a visual working memory task that required remembering 1–5 items from one side of the display and ignoring an equal number of items on the other side of the display.

•

We attempted to decode which side was being held in working memory from the scalp distribution of the ERP activity during the delay period of the working memory task, and we found greater decoding accuracy in people with schizophrenia than in control subjects when a single item was being held in memory.

•

These results support the hyperfocusing hypothesis of cognitive dysfunction in schizophrenia, and they provide an important proof of concept for applying multivariate decoding methods to comparisons of neural representations in psychiatric and non-psychiatric populations.

Multivariate pattern classification (decoding) methods are commonly employed to study mechanisms of neurocognitive processing in typical individuals, where they can be used to quantify the information that is present in single-participant neural signals. These decoding methods are also potentially valuable in determining how the representation of information differs between psychiatric and non-psychiatric populations. Here, we examined ERPs from people with schizophrenia (PSZ) and healthy control subjects (HCS) in a working memory task that involved remembering 1, 3, or 5 items from one side of the display and ignoring the other side. We used the spatial pattern of ERPs to decode which side of the display was being held in working memory. One might expect that decoding accuracy would be inevitably lower in PSZ as a result of increased noise (i.e., greater trial-to-trial variability). However, we found that decoding accuracy was greater in PSZ than in HCS at memory load 1, consistent with previous research in which memory-related ERP signals were larger in PSZ than in HCS at memory load 1. We also observed that decoding accuracy was strongly related to the ratio of the memory-related ERP activity and the noise level. In addition, we found similar noise levels in PSZ and HCS, counter to the expectation that PSZ would exhibit greater trial-to-trial variability. Together, these results demonstrate that multivariate decoding methods can be validly applied at the individual-participant level to understand the nature of impaired cognitive function in a psychiatric population.

Spatial specificity of alpha oscillations in the human visual system

Alpha oscillations are strongly modulated by spatial attention. To what extent, the generators of cortical alpha oscillations are spatially distributed and have selectivity that can be related to retinotopic organization is a matter of continuous scientific debate. In the present report, neuromagnetic activity was quantified by means of spatial location tuning functions from 30 participants engaged in a visuospatial attention task. A cue presented briefly in one of 16 locations directing covert spatial attention resulted in a robust modulation of posterior alpha oscillations. The distribution of the alpha sources approximated the retinotopic organization of the human visual system known from hemodynamic studies. Better performance in terms of target identification was associated with a more spatially constrained alpha modulation. The present findings demonstrate that the generators of posterior alpha oscillations are retinotopically organized when modulated by spatial attention.

The time-course of component processes of selective attention

Attentional selection shapes human perception, enhancing relevant information, according to behavioral goals. While many studies have investigated individual neural signatures of attention, here we used multivariate decoding of electrophysiological brain responses (MEG/EEG) to track and compare multiple component processes of selective attention. Auditory cues instructed participants to select a particular visual target, embedded within a subsequent stream of displays. Combining single and multi-item displays with different types of distractors allowed multiple aspects of information content to be decoded, distinguishing distinct components of attention, as the selection process evolved. Although the task required comparison of items to an attentional "template" held in memory, signals consistent with such a template were largely undetectable throughout the preparatory period but re-emerged after presentation of a non-target choice display. Choice displays evoked strong neural representation of multiple target features, evolving over different timescales. We quantified five distinct processing operations with different time-courses. First, visual properties of the stimulus were strongly represented. Second, the candidate target was rapidly identified and localized in multi-item displays, providing the earliest evidence of modulation by behavioral relevance. Third, the identity of the target continued to be enhanced, relative to distractors. Fourth, only later was the behavioral significance of the target explicitly represented in single-item displays. Finally, if the target was not identified and search was to be resumed, then an attentional template was weakly reactivated. The observation that an item's behavioral relevance directs attention in multi-item displays prior to explicit representation of target/non-target status in single-item displays is consistent with two-stage models of attention.

Humans can efficiently look for but not select multiple visual objects

The human brain recurrently prioritizes task-relevant over task-irrelevant visual information. A central question is whether multiple objects can be prioritized simultaneously. To answer this, we let observers search for two colored targets among distractors. Crucially, we independently varied the number of target colors that observers anticipated, and the number of target colors actually used to distinguish the targets in the display. This enabled us to dissociate the preparation of selection mechanisms from the actual engagement of such mechanisms. Multivariate classification of electroencephalographic activity allowed us to track selection of each target separately across time. The results revealed only small neural and behavioral costs associated with preparing for selecting two objects, but substantial costs when engaging in selection. Further analyses suggest this cost is the consequence of neural competition resulting in limited parallel processing, rather than a serial bottleneck. The findings bridge diverging theoretical perspectives on capacity limitations of feature-based attention.

Learning What Is Irrelevant or Relevant: Expectations Facilitate Distractor Inhibition and Target Facilitation through Distinct Neural Mechanisms

It is well known that attention can facilitate performance by top-down biasing processing of task-relevant information in advance. Recent findings from behavioral studies suggest that distractor inhibition is not under similar direct control but strongly dependent on expectations derived from previous experience. Yet, how expectations about distracting information influence distractor inhibition at the neural level remains unclear. The current study addressed this outstanding question in three experiments in which search displays with repeating distractor or target locations across trials allowed human observers (male and female) to learn which location to selectively suppress or boost. Behavioral findings demonstrated that both distractor and target location learning resulted in more efficient search, as indexed by faster response times. Crucially, distractor learning benefits were observed without target location foreknowledge, unaffected by the number of possible target locations, and could not be explained by priming alone. To determine how distractor location expectations facilitated performance, we applied a spatial encoding model to EEG data to reconstruct activity in neural populations tuned to distractor or target locations. Target location learning increased neural tuning to target locations in advance, indicative of preparatory biasing. This sensitivity increased after target presentation. By contrast, distractor expectations did not change preparatory spatial tuning. Instead, distractor expectations reduced distractor-specific processing, as reflected in the disappearance of the Pd event-related potential component, a neural marker of distractor inhibition, and decreased decoding accuracy. These findings suggest that the brain may no longer process expected distractors as distractors, once it has learned they can safely be ignored.SIGNIFICANCE STATEMENT We constantly try hard to ignore conspicuous events that distract us from our current goals. Surprisingly, and in contrast to dominant attention theories, ignoring distracting, but irrelevant, events does not seem to be as flexible as is focusing our attention on those same aspects. Instead, distractor suppression appears to strongly rely on learned, context-dependent expectations. Here, we investigated how learning about upcoming distractors changes distractor processing and directly contrasted the underlying neural dynamics to target learning. We show that, while target learning enhanced anticipatory sensory tuning, distractor learning only modulated reactive suppressive processing. These results suggest that expected distractors may no longer be considered distractors by the brain once it has learned that they can safely be ignored.

Reactivation of Previous Experiences in a Working Memory Task

Recent experiences influence the processing of new information even when those experiences are irrelevant to the current task. Does this reflect the indirect effects of a passively maintained representation of the previous experience, or is this representation reactivated when a new event occurs? To answer this question, we attempted to decode the orientation of the stimulus on the previous trial from the electroencephalogram on the current trial in a working memory task. Behavioral data confirmed that the previous-trial stimulus orientation influenced the reported orientation on the current trial, even though the previous-trial orientation was now task irrelevant. In two independent experiments, we found that the previous-trial orientation could be decoded from the current-trial electroencephalogram, indicating that the current-trial stimulus reactivated or boosted the representation of the previous-trial orientation. These results suggest that the effects of recent experiences on behavior are driven, in part, by a reactivation of those experiences and not solely by the indirect effects of passive memory traces.

Estimating the Timing of Cognitive Operations With MEG/EEG Latency Measures: A Primer, a Brief Tutorial, and an Implementation of Various Methods

The major advantage of MEG/EEG over other neuroimaging methods is its high temporal resolution. Examining the latency of well-studied components can provide a window into the dynamics of cognitive operations beyond traditional response-time (RT) measurements. While RTs reflect the cumulative duration of all time-consuming cognitive operations involved in a task, component latencies can partition this time into cognitively meaningful sub-steps. Surprisingly, most MEG/EEG studies neglect this advantage and restrict analyses to component amplitudes without considering latencies. The major reasons for this neglect might be that, first, the most easily accessible latency measure (peak latency) is often unreliable and that, second, more complex measures are difficult to conceive, implement, and parametrize. The present article illustrates the key advantages and disadvantages of the three main types of latency-measures (peak latency, onset latency, and percent-area latency), introduces a MATLAB function that extracts all these measures and is compatible with common analysis tools, discusses the most important parameter choices for different research questions and components of interest, and demonstrates its use by various group analyses on one planar gradiometer pair of the publicly available Wakeman and Henson (2015) data. The introduced function can extract from group data not only single-subject latencies, but also grand-average and jackknife latencies. Furthermore, it gives the choice between different approaches to automatically set baselines and anchor points for latency estimation, approaches that were partly developed by me and that capitalize on the informational richness of MEG/EEG data. Although the function comes with a wide range of customization parameters, the default parameters are set so that even beginners get reasonable results. Graphical depictions of latency estimates, baselines, and anchor points overlaid on individual averages further support learning, understanding and trouble-shooting. Once extracted, latency estimates can be submitted to any analysis also available for (averaged) RTs, including tests for mean differences, correlational approaches and cognitive modeling.

Neural Signatures of the Configural Superiority Effect and Fundamental Emergent Features in Human Vision

The concepts of grouping, emergence, and superadditivity (when a whole is qualitatively different from the sum of its parts) are critical in Gestalt psychology and essential to properly understand the information processing mechanisms underlying visual perception. However, very little is known about the neural processes behind these phenomena (particularly in terms of their generality vs. specificity and their time-course). Here, we used the configural superiority effect as a way to define "emergence" and "emergent features" operationally, employing an approach that can isolate different emergent features and compare them on a common scale. By assessing well-established event related potentials in a HD-EEG system, we found that the critical processes behind configural superiority and superadditive Gestalt phenomena are present in the window between 100 and 200 ms after stimulus onset and that these effects seem to be driven by specific attentional selection mechanisms. Also, some emergent features seem to be differentially processed in different brain hemispheres. These results shed new light on the issues of the generality vs. specificity of the neural correlates of different Gestalt principles, the hemispheric asymmetries in the processing of hierarchical image structure and the role of the N1 ERP component in reflecting feature selective mechanisms.

Decoding motion direction using the topography of sustained ERPs and alpha oscillations

The present study sought to determine whether scalp electroencephalogram (EEG) signals contain decodable information about the direction of motion in random dot kinematograms (RDKs), in which the motion information is spatially distributed and mixed with random noise. Any direction of motion from 0 to 360° was possible, and observers reported the precise direction of motion at the end of a 1500-ms stimulus display. We decoded the direction of motion separately during the motion period (during which motion information was being accumulated) and the report period (during which a shift of attention was necessary to make a fine-tuned direction report). Machine learning was used to decode the precise direction of motion (within ±11.25°) from the scalp distribution of either alpha-band EEG activity or sustained event-related potentials (ERPs). We found that ERP-based decoding was above chance (1/16) during both the stimulus and the report periods, whereas alpha-based decoding was above chance only during the report period. Thus, sustained ERPs contain information about spatially distributed direction-of-motion, providing a new method for observing the accumulation of sensory information with high temporal resolution. By contrast, the scalp topography of alpha-band EEG activity appeared to mainly reflect spatially focused attentional processes rather than sensory information.

Dynamics of neural representations when searching for exemplars and categories of human and non-human faces

Face perception abilities in humans exhibit a marked expertise in distinguishing individual human faces at the expense of individual faces from other species (the other-species effect). In particular, one behavioural effect of such specialization is that human adults search for and find categories of non-human faces faster and more accurately than a specific non-human face, and vice versa for human faces. However, a recent visual search study showed that neural responses (event-related potentials, ERPs) were identical when finding either a non-human or human face. We used time-resolved multivariate pattern analysis of the EEG data from that study to investigate the dynamics of neural representations during a visual search for own-species (human) or other-species (non-human ape) faces, with greater sensitivity than traditional ERP analyses. The location of each target (i.e., right or left) could be decoded from the EEG, with similar accuracy for human and non-human faces. However, the neural patterns associated with searching for an exemplar versus a category target differed for human faces compared to non-human faces: Exemplar representations could be more reliably distinguished from category representations for human than non-human faces. These findings suggest that the other-species effect modulates the nature of representations, but preserves the attentional selection of target items based on these representations.

From ERPs to MVPA Using the Amsterdam Decoding and Modeling Toolbox (ADAM)

In recent years, time-resolved multivariate pattern analysis (MVPA) has gained much popularity in the analysis of electroencephalography (EEG) and magnetoencephalography (MEG) data. However, MVPA may appear daunting to those who have been applying traditional analyses using event-related potentials (ERPs) or event-related fields (ERFs). To ease this transition, we recently developed the Amsterdam Decoding and Modeling (ADAM) toolbox in MATLAB. ADAM is an entry-level toolbox that allows a direct comparison of ERP/ERF results to MVPA results using any dataset in standard EEGLAB or Fieldtrip format. The toolbox performs and visualizes multiple-comparison corrected group decoding and forward encoding results in a variety of ways, such as classifier performance across time, temporal generalization (time-by-time) matrices of classifier performance, channel tuning functions (CTFs) and topographical maps of (forward-transformed) classifier weights. All analyses can be performed directly on raw data or can be preceded by a time-frequency decomposition of the data in which case the analyses are performed separately on different frequency bands. The figures ADAM produces are publication-ready. In the current manuscript, we provide a cookbook in which we apply a decoding analysis to a publicly available MEG/EEG dataset involving the perception of famous, non-famous and scrambled faces. The manuscript covers the steps involved in single subject analysis and shows how to perform and visualize a subsequent group-level statistical analysis. The processing pipeline covers computation and visualization of group ERPs, ERP difference waves, as well as MVPA decoding results. It ends with a comparison of the differences and similarities between EEG and MEG decoding results. The manuscript has a level of description that allows application of these analyses to any dataset in EEGLAB or Fieldtrip format.

Dissociable Decoding of Spatial Attention and Working Memory from EEG Oscillations and Sustained Potentials

In human scalp EEG recordings, both sustained potentials and alpha-band oscillations are present during the delay period of working memory tasks and may therefore reflect the representation of information in working memory. However, these signals may instead reflect support mechanisms rather than the actual contents of memory. In particular, alpha-band oscillations have been tightly tied to spatial attention and may not reflect location-independent memory representations per se. To determine how sustained and oscillating EEG signals are related to attention and working memory, we attempted to decode which of 16 orientations was being held in working memory by human observers (both women and men). We found that sustained EEG activity could be used to decode the remembered orientation of a stimulus, even when the orientation of the stimulus varied independently of its location. Alpha-band oscillations also carried clear information about the location of the stimulus, but they provided little or no information about orientation independently of location. Thus, sustained potentials contain information about the object properties being maintained in working memory, consistent with previous evidence of a tight link between these potentials and working memory capacity. In contrast, alpha-band oscillations primarily carry location information, consistent with their link to spatial attention.SIGNIFICANCE STATEMENT Working memory plays a key role in cognition, and working memory is impaired in several neurological and psychiatric disorders. Previous research has suggested that human scalp EEG recordings contain signals that reflect the neural representation of information in working memory. However, to conclude that a neural signal actually represents the object being remembered, it is necessary to show that the signal contains fine-grained information about that object. Here, we show that sustained voltages in human EEG recordings contain fine-grained information about the orientation of an object being held in memory, consistent with a memory storage signal.

Spatially Selective Alpha Oscillations Reveal Moment-by-Moment Trade-offs between Working Memory and Attention

Current theories assume a functional role for covert attention in the maintenance of spatial information in working memory. Consistent with this view, both the locus of attention and positions stored in working memory can be decoded based on the topography of oscillatory alpha-band (8-12 Hz) activity on the scalp. Thus far, however, alpha modulation has been studied in isolation for covert attention and working memory tasks. Here, we applied an inverted spatial encoding model in combination with EEG to study the temporal dynamics of spatially specific alpha activity during a task that required observers to visually select a target location while maintaining another independently varying location in working memory. During the memory delay period, alpha-based spatial tuning functions shifted from the position stored in working memory to the covertly attended position and back again after the attention task was completed. The findings provide further evidence for a common oscillatory mechanism in both the selection and the maintenance of relevant spatial visual information and demonstrate the dynamic trade-off in prioritization between two spatial tasks.