Impaired semantic processing during task-set switching: evidence from the N400 in rapid serial visual presentation

The N400 component reflecting semantic and repetition priming of visual scenes is suppressed during the attentional blink

In the attentional blink paradigm, participants attempt to identify two targets appearing in a rapidly presented stream of distractors. Report accuracy is typically high for the first target (T1) while identification of the second target (T2) is impaired when it follows within about 200-400 ms of T1. An important question is whether T2 is processed to a semantic level even when participants are unaware of its identity. We examined this issue in three studies that used natural scenes as stimuli and the N400 component of the event-related potential (ERP) as a measure of semantic priming. In the first experiment, the prime (e.g., a doghouse in a yard) was presented at the beginning of the trial and a test picture that was related (e.g., a dog standing in the kitchen) or unrelated (e.g., a coffee mug on a table) appeared as T2. In the second experiment, the prime was presented as T2 and the test picture appeared at the end of the picture sequence. In both experiments, we found robust semantic priming when participants were aware of the identity of the blinked picture and an absence of priming when they were unaware. In Experiment 3, we used identity priming to assess whether earlier representations preceding semantics were preserved, and again found that priming critically depended on awareness of the prime's identity. These results suggest that semantic priming in scenes, as measured with the N400, is a higher-level process that critically depends on attention and awareness.

Attentional attenuation (rather than attentional boost) through task switching leads to a selective long-term memory decline

Allocating attention determines what we remember later. Attentional demands vary in a task-switching paradigm, with greater demands for switch than for repeat trials. This also results in lower subsequent memory performance for switch compared to repeat trials. The main goal of the present study was to investigate the consequences of task switching after a long study-test interval and to examine the contributions of the two memory components, recollection and familiarity. In the study phase, the participants performed a task-switching procedure in which they had to switch between two classifications tasks with pictures. After a short vs. a long study-test interval of a week, the participants performed a surprise memory test for the pictures and gave remember/know judgements. The results showed that recognition memory declined after 1 week and this was mainly due to a decrease in "remember" responses. The results also showed that the task-switching effect on memory was enduring. Whereas the results of the immediate test were mixed, the results of the delayed tests showed that the task-switching effect was based on recollection, expressed in more "remember" responses for repeat than for switch trials. As recollection is more sensitive to attention manipulations than familiarity, the results align with the notion that attentional requirements at study determine what we remember, in particular after a long study-test interval.

How Task Set and Task Switching Modulate Perceptual Processes: Is Recognition of Facial Emotion an Exception?

In Part 1 we review task-switching and other studies showing that, even with time for preparation, participants' ability to shift attention to a relevant attribute or object before the stimulus onset is limited: there is a 'residual cost'. In particular, several brain potential markers of perceptual encoding are delayed on task-switch trials, compared to task-repeat trials that require attention to the same attribute as before. Such effects have been documented even for a process often considered 'automatic' - visual word recognition: ERP markers of word frequency and word/nonword status are (1) delayed when the word recognition task follows a judgement of a perceptual property compared to repeating the lexical task, and (2) strongly attenuated during the perceptual judgements. Thus, even lexical access seems influenced by the task/attentional set. In Part 2, we report in detail a demonstration of what seems to be a special case, where task-set and a task switch have no such effect on perceptual encoding. Participants saw an outline letter superimposed on a face expressing neutral or negative emotion, and were auditorily cued to categorise the letter as vowel/consonant, or the face as emotional/neutral. ERPs exhibited a robust emotional-neutral difference (Emotional Expression Effect) no smaller or later when switching to the face task than when repeating it; in the first half of its time-course it did not vary with the task at all. The initial encoding of the valence of a fixated facial emotional expression appears to be involuntary and invariant, whatever the endogenous task/attentional set.

What processes are disrupted during the attentional blink? An integrative review of event-related potential research

Reporting the second of two targets is impaired when these appear in close succession, a phenomenon known as the attentional blink (AB). Despite decades of research, what factors limit our ability to process multiple sequentially presented events remains unclear. Specifically, two central issues remain open: does failure to report the second target (T2) reflect a structural limitation in working memory (WM) encoding or a disruption to attentional processes? And is perceptual processing of the stimulus that we fail to report impaired, or only processes that occur after this stimulus is identified? We address these questions by reviewing event-related potential (ERP) studies of the AB, after providing a brief overview of the theoretical landscape relevant to these debates and clarifying key concepts essential for interpreting ERP studies. We show that failure to report the second target is most often associated with disrupted attentional engagement (associated with a smaller and delayed N2pc component). This disruption occurs after early processing of T2 (associated with an intact P1 component), weakens its semantic processing (typically associated with a smaller N400 component), and prevents its encoding into WM (associated with absent P3b). However, failure to encode T2 in WM can occur despite intact attentional engagement and semantic processing. We conclude that the AB phenomenon, which reflects our limited ability to process sequential events, emerges from the disruption of both attentional engagement and WM encoding.

Dual task interference on early perceptual processing

When two tasks, Task 1 and Task 2, are conducted in close temporal proximity and a separate speeded response is required for each target (T1 and T2), T2 report performance decreases as a function of its temporal proximity to T1. This so-called psychological refractory period (PRP) effect on T2 processing is largely assumed to reflect interference from T1 response selection on T2 response selection. However, interference on early perceptual processing of T2 has been observed in a modified paradigm, which required changes in visual-spatial attention, sensory modality, task modality, and response modality across targets. The goal of the present study was to investigate the possibility of early perceptual interference by systematically and iteratively removing each of these possible non perceptual confounds, in a series of four experiments. To assess T2 visual memory consolidation success, T2 was presented for a varying duration and immediately masked. T2 report accuracy, which was taken as a measure of perceptual-encoding or consolidation-success, decreased across all experimental control conditions as T1-T2 onset proximity increased. We argue that our results, in light of previous studies, show that central processing of a first target, responsible for the classical PRP effect, also interferes with early perceptual processing of a second target. We end with a discussion of broader implications for psychological refractory period and attentional blink effects.

The perceptual wink model of non-switching attentional blink tasks

The attentional blink (AB) is a temporary deficit for a second target (T2) when that target appears after a first target (T1). Although sophisticated models have been developed to explain the substantial AB literature in isolation, the current study considers how the AB relates to perceptual dynamics more broadly. We show that the time-course of the AB is closely related to the time course of the transition from positive to negative repetition priming effects in perceptual identification. Many AB tasks involve a switch between a T1 defined in one manner and a T2 defined in a different manner. Other AB tasks are non-switching, with all targets belonging to the same well-known category (e.g., letter targets versus number distractors) or sharing the same perceptual feature. We propose that these non-switching AB tasks reflect perceptual habituation for the target-defining attribute; thus, a 'perceptual wink', with perception of one attribute (target identity) undisturbed while perception of another (target detection) is impaired. On this account, the immediate benefit following T1 (lag-1 sparing) reflects positive repetition priming and the subsequent deficit (the blink) reflects negative repetition priming for the realization that a target occurred. In developing the perceptual wink model, we extended the nROUSE model of perceptual priming to explain the results of two new experiments combining the AB and identity repetitions. This establishes important connections between non-switching AB tasks and perceptual dynamics.

Semantic Relations between Visual Objects Can Be Unconsciously Processed but Not Reported under Change Blindness

Change blindness—the failure to detect changes in visual scenes—has often been interpreted as a result of impoverished visual information encoding or as a failure to compare the prechange and postchange scene. In the present electroencephalography study, we investigated whether semantic features of prechange and postchange information are processed unconsciously, even when observers are unaware that a change has occurred. We presented scenes composed of natural objects in which one object changed from one presentation to the next. Object changes were either semantically related (e.g., rail car changed to rail) or unrelated (e.g., rail car changed to sausage). Observers were first asked to detect whether any change had occurred and then to judge the semantic relation of the two objects involved in the change. We found a semantic mismatch ERP effect, that is, a more negative-going ERP for semantically unrelated compared to related changes, originating from a cortical network including the left middle temporal gyrus and occipital cortex and resembling the N400 effect, albeit at longer latencies. Importantly, this semantic mismatch effect persisted even when observers were unaware of the change and the semantic relationship of prechange and postchange object. This finding implies that change blindness does not preclude the encoding of the prechange and postchange objects' identities and possibly even the comparison of their semantic content. Thus, change blindness cannot be interpreted as resulting from impoverished or volatile visual representations or as a failure to process the prechange and postchange object. Instead, change detection appears to be limited at a later, postperceptual stage.

Is Semantic Processing During Sentence Reading Autonomous or Controlled? Evidence from the N400 Component in a Dual Task Paradigm

The present study contributes to the discussion on the automaticity of semantic processing. Whereas most previous research investigated semantic processing at word level, the present study addressed semantic processing during sentence reading. A dual task paradigm was combined with the recording of event-related brain potentials. Previous research at word level processing reported different patterns of interference with the N400 by additional tasks: attenuation of amplitude or delay of latency. In the present study, we presented Spanish sentences that were semantically correct or contained a semantic violation in a critical word. At different intervals preceding the critical word a tone was presented that required a high-priority choice response. At short intervals/high temporal overlap between the tasks mean amplitude of the N400 was reduced relative to long intervals/low temporal overlap, but there were no shifts of peak latency. We propose that processing at sentence level exerts a protective effect against the additional task. This is in accord with the attentional sensitization model (Kiefer & Martens, 2010), which suggests that semantic processing is an automatic process that can be enhanced by the currently activated task set. The present experimental sentences also induced a P600, which is taken as an index of integrative processing. Additional task effects are comparable to those in the N400 time window and are briefly discussed.

Distractor suppression when attention fails: behavioral evidence for a flexible selective attention mechanism

Despite consistent evidence showing that attention is a multifaceted mechanism that can operate at multiple levels of processing depending on the structure and demands of the task, investigations of the attentional blink phenomenon have consistently shown that the impairment in reporting the second of two targets typically occurs at a late, or post-perceptual, stage of processing. This suggests that the attentional blink phenomenon may represent the operation of a unique attentional mechanism that is not as flexible as other attentional mechanisms. To test whether the attentional blink is a fixed or flexible phenomenon, we manipulated first target task demands (i.e., difficulty) and measured the influence this had on processing a subsequently presented distractor and the second target. If the attentional blink represents a mechanism that is fixed and consistently fails at a single stage of processing, then manipulations of task difficulty should not affect distractor processing. However, if the attentional blink represents a more multifaceted and flexible mechanism, then task difficulty should modulate distractor processing. The results revealed that distractor processing during the AB was attenuated under high task difficulty. In addition, unlike previous studies, we failed to find a correlation between distractor processing and the severity of the attentional blink. Using a simulation, we demonstrate that the previously reported correlations may have been spurious and due to using variables that were not independent. Overall, the present results support the conclusion that the selectivity of attention during the AB is flexible and depends on the structure and demands of the task.

Failure of temporal selectivity: Electrophysiological evidence for (mis)selection of distractors during the attentional blink

The attentional blink (AB) refers to the impairment in accurate report of a second target (T2) when presented shortly after a first target (T1) in a rapid serial visual presentation of distractors. The goal of the present study was to determine whether the AB is caused by a failure in early selection processes, which leads to the selection and consolidation of the wrong item in working memory, by measuring the frequency-related P3 to T2 and to T2 + 1. During the AB, an attenuation of the P3 to T2 was observed, as well as an increase in the amplitude of the P3 to T2 + 1. Whereas the P3 to T2 was observed only when T2 was correctly reported, the P3 to T2 + 1 was observed only in trials where T2 was incorrectly reported, and its amplitude was correlated to individual differences in misselection rate. These results support the claim that failure of temporal selection underlies the AB.

Task switching mediates direct interference of intertarget distractors in the attentional blink: An event-related potential study

The attentional blink (AB) refers to the difficulty in reporting a second target (T2) presented shortly after a first target (T1) in a stream of distractors. The goal of the present study was to investigate distractor-based interference in the AB by recording the P3 component of the event-related potential to both targets. An intertarget distractor was presented at lag 1 (T1+1), at lag 2 (T1+2), or at neither of these two lags (no distractor). T2 was always presented at lag 3, as the last item in the stream. In two experiments, the P3 from T1 was attenuated in the T1+1 condition compared to the two other distractor conditions. In absence of a task switch (Experiment 1), the P3 from T2 was delayed in both the T1+1 and T1+2 conditions. In the presence of a task switch (Experiment 2), the P3 from T2 was delayed only in the T1+1 condition. Results demonstrate that intertarget distractors can modulate the AB directly but only in absence of a task switch.

Post-perceptual processing during the attentional blink is modulated by inter-trial task expectancies

The selective processing of goal-relevant information depends on an attention system that can flexibly adapt to changing task demands and expectations. Evidence from visual search tasks indicates that the perceptual selectivity of attention increases when the bottom-up demands of the task increase and when the expectations about task demands engendered by trial history are violated. Evidence from studies of the attentional blink (AB), which measures the temporal dynamics of attention, also indicates that perceptual selectivity during the AB is increased if the bottom-up task demands are increased. The present work tested whether expectations about task demands engendered by trial history also modulate perceptual selectivity during the AB. Two experiments tested the extent to which inter-trial switches in task demands reduced post-perceptual processing of targets presented during the AB. Experiment 1 indexed post-perceptual processing using the event-related potential (ERP) technique to isolate the context sensitive N400 ERP component evoked by words presented during the AB. Experiment 2 indexed post-perceptual processing using behavioral performance to determine the extent to which personal names survive the AB. The results of both experiments revealed that both electrophysiological (Exp. 1) and behavioral (Exp. 2) indices of post-perceptual processing were attenuated when consecutive trials differed in terms of their perceptual demands. The results are consistent with the notion that the selectivity of attention during the AB is modulated not only by within-trial task demands, but also can be flexibly determined by trial-by-trial expectations.

Disruption of visual awareness during the attentional blink is reflected by selective disruption of late-stage neural processing

Any information represented in the brain holds the potential to influence behavior. It is therefore of broad interest to determine the extent and quality of neural processing of stimulus input that occurs with and without awareness. The attentional blink is a useful tool for dissociating neural and behavioral measures of perceptual visual processing across conditions of awareness. The extent of higher-order visual information beyond basic sensory signaling that is processed during the attentional blink remains controversial. To determine what neural processing at the level of visual-object categorization occurs in the absence of awareness, electrophysiological responses to images of faces and houses were recorded both within and outside the attentional blink period during a rapid serial visual presentation stream. Electrophysiological results were sorted according to behavioral performance (correctly identified targets vs. missed targets) within these blink and nonblink periods. An early index of face-specific processing (the N170, 140- to 220-msec poststimulus) was observed regardless of whether the participant demonstrated awareness of the stimulus, whereas a later face-specific effect with the same topographic distribution (500- to 700-msec poststimulus) was only seen for accurate behavioral discrimination of the stimulus content. The present findings suggest a multistage process of object-category processing, with only the later phase being associated with explicit visual awareness.

Nonexplicit change detection in complex dynamic settings: what eye movements reveal

OBJECTIVE

We employed a computer-controlled command-and-control (C2) simulation and recorded eye movements to examine the extent and nature of the inability to detect critical changes in dynamic displays when change detection is implicit (i.e., requires no explicit report) to the operator's task.

BACKGROUND

Change blindness-the failure to notice significant changes to a visual scene-may have dire consequences on performance in C2 and surveillance operations.

METHOD

Participants performed a radar-based risk-assessment task involving multiple subtasks. Although participants were not required to explicitly report critical changes to the operational display, change detection was critical in informing decision making. Participants' eye movements were used as an index of visual attention across the display.

RESULTS

Nonfixated (i.e., unattended) changes were more likely to be missed than were fixated (i.e., attended) changes, supporting the idea that focused attention is necessary for conscious change detection. The finding of significant pupil dilation for changes undetected but fixated suggests that attended changes can nonetheless be missed because of a failure of attentional processes.

CONCLUSION

Change blindness in complex dynamic displays takes the form of failures in establishing task-appropriate patterns of attentional allocation.

APPLICATION

These findings have implications in the design of change-detection support tools for dynamic displays and work procedure in C2 and surveillance.

Event-related potential evidence for two functionally dissociable sources of semantic effects in the attentional blink

Three target words (T1, T2, and T3) were embedded in a rapid serial visual presentation (RSVP) stream of non-word distractors, and participants were required to report the targets at the end of each RSVP stream. T2 and T3 were semantically related words in half of the RSVP streams, and semantically unrelated words in the other half of the RSVP streams. Using an identical design, a recent study reported distinct reflections of the T2–T3 semantic relationship on the P2 and N400 components of event-related potentials (ERPs) time-locked to T3, suggesting an early, automatic, source of P2 semantic effects and a late, controlled, source of N400 semantic effects. Here, P2 and N400 semantic effects were examined by manipulating list-wide context. Relative to participants performing in a semantically unbiased context, participants over-exposed to filler RSVP streams always including semantically related T2/T3 words reported a dilution of T3-locked P2 semantic effects and a magnification of T3-locked N400 semantic effects. Opposite effects on P2 and N400 ERP components of list-wide semantic context are discussed in relation to recent proposals on the representational status of RSVP targets at processing stages prior to consolidation in visual short-term memory.

Exploring the additive effects of stimulus quality and word frequency: the influence of local and list-wide prime relatedness

Stimulus quality and word frequency produce additive effects in lexical decision performance, whereas the semantic priming effect interacts with both stimulus quality and word frequency effects. This pattern places important constraints on models of visual word recognition. In Experiment 1, all three variables were investigated within a single speeded pronunciation study. The results indicated that the joint effects of stimulus quality and word frequency were dependent upon prime relatedness. In particular, an additive effect of stimulus quality and word frequency was found after related primes, and an interactive effect was found after unrelated primes. It was hypothesized that this pattern reflects an adaptive reliance on related prime information within the experimental context. In Experiment 2, related primes were eliminated from the list, and the interactive effects of stimulus quality and word frequency found following unrelated primes in Experiment 1 reverted to additive effects for the same unrelated prime conditions. The results are supportive of a flexible lexical processor that adapts to both local prime information and global list-wide context.

On the automaticity of semantic processing during task switching

There is growing evidence that processes formerly believed to be automatic are, in fact, strongly modulated by top-down influences. The purpose of the present work was to investigate how cognitive control can affect the purported automaticity of word processing by examining the impact of task switching on semantic processing using the ERP technique. In the context of the psychological refractory period dual-task paradigm, two experiments contrasted the context-sensitive N400 ERP elicited by the second of two target words under conditions that involved either a task switch or no-task switch. Although the N400 was not affected by SOA in the absence of switching, it was strongly attenuated at short SOAs when the psychological refractory period procedure involved a switch from a perceptual to a semantic task (Experiment 1) or a switch between two different semantic tasks (Experiment 2). These findings suggest that semantic processing cannot be performed in parallel with task switching and illustrate limitations in the ability of the cognitive system to adapt flexibly to the dynamically changing challenges of the environment according to task demands and behavioral goals.

A shared cortical bottleneck underlying Attentional Blink and Psychological Refractory Period

Doing two things at once is difficult. When two tasks have to be performed within a short interval, the second is sharply delayed, an effect called the Psychological Refractory Period (PRP). Similarly, when two successive visual targets are briefly flashed, people may fail to detect the second target (Attentional Blink or AB). Although AB and PRP are typically studied in very different paradigms, a recent detailed neuromimetic model suggests that both might arise from the same serial stage during which stimuli gain access to consciousness and, as a result, can be arbitrarily routed to any other appropriate processor. Here, in agreement with this model, we demonstrate that AB and PRP can be obtained on alternate trials of the same cross-modal paradigm and result from limitations in the same brain mechanisms. We asked participants to respond as fast as possible to an auditory target T1 and then to a visual target T2 embedded in a series of distractors, while brain activity was recorded with magneto-encephalography (MEG). For identical stimuli, we observed a mixture of blinked trials, where T2 was entirely missed, and PRP trials, where T2 processing was delayed. MEG recordings showed that PRP and blinked trials underwent identical sensory processing in visual occipito-temporal cortices, even including the non-conscious separation of targets from distractors. However, late activations in frontal cortex (>350 ms), strongly influenced by the speed of task-1 execution, were delayed in PRP trials and absent in blinked trials. Our findings suggest that PRP and AB arise from similar cortical stages, can occur with the same exact stimuli, and are merely distinguished by trial-by-trial fluctuations in task processing.

ERP Evidence for Ultra-Fast Semantic Processing in the Picture-Word Interference Paradigm

We used the event-related potential (ERP) approach combined with a subtraction technique to explore the timecourse of activation of semantic and phonological representations in the picture–word interference paradigm. Subjects were exposed to to-be-named pictures superimposed on to-be-ignored semantically related, phonologically related, or unrelated words, and distinct ERP waveforms were generated time-locked to these different classes of stimuli. Difference ERP waveforms were generated in the semantic condition and in the phonological condition by subtracting ERP activity associated with unrelated picture–word stimuli from ERP activity associated with related picture–word stimuli. We measured both latency and amplitude of these difference ERP waveforms in a pre-articulatory time-window. The behavioral results showed standard interference effects in the semantic condition, and facilitatory effects in the phonological condition. The ERP results indicated a bimodal distribution of semantic effects, characterized by the extremely rapid onset (at about 100 ms) of a primary component followed by a later, distinct, component. Phonological effects in ERPs were characterized by components with later onsets and distinct scalp topography of ERP sources relative to semantic ERP components. Regression analyses revealed a covariation between semantic and phonological behavioral effect sizes and ERP component amplitudes, and no covariation between the behavioral effects and ERP component latency. The early effect of semantic distractors is thought to reflect very fast access to semantic representations from picture stimuli modulating on-going orthographic processing of distractor words.