Separate and combined effects of action relevance and motivational value on visual working memory

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Better Controlled, Better Maintained: Sense of Agency Facilitates Working Memory

The cognitive effects of sense of agency (SoA) attract increasing attention. It is unclear how SoA influences working memory (WM). In the present study, participants first moved several boxes. One of the boxes was more or less controllable than the majority. After boxes stopped moving, to-be-remembered items appeared. Memory performance and SoA over them were subsequently tested. In Experiments 1a and 1b, the majority of boxes, serving as context, were under low and high control respectively. To further examine whether the maintenance stage was influenced, the effect of selective encoding was minimized in Experiments 2a and 2b. Experiment 2b further eliminated the impact of agency judgments and tested the relationship among SoA, WM, and preference. Memory was better for items with strong SoA in the high and low control contexts. The effect partly stems from the modulation of the maintenance stage in WM, in which reward-based processes could be engaged.

Planning the Potential Future during Multi-item Visual Working Memory

Working memory allows us to retain visual information to guide upcoming future behavior. In line with this future-oriented purpose of working memory, recent studies have shown that action planning occurs during encoding and retention of a single visual item, for which the upcoming action is certain. We asked whether and how this extends to multi-item visual working memory, when visual representations serve the potential future. Human participants performed a visual working-memory task with a memory-load manipulation (one/two/four items) and a delayed orientation-reproduction report (of one item). We measured EEG to track 15- to 25-Hz beta activity in electrodes contralateral to the required response hand-a canonical marker of action planning. We show an attenuation of beta activity, not only in Load 1 (with one certain future action) but also in Load 2 (with two potential future actions), compared with Load 4 (with low prospective-action certainty). Moreover, in Load 2, potential action planning occurs regardless whether both visual items afford similar or dissimilar manual responses, and it predicts the speed of ensuing memory-guided behavior. This shows that potential action planning occurs during multi-item visual working memory and brings the perspective that working memory helps us prepare for the potential future.

Less motor (re-)planning requires fewer working memory resources

In the current study, we asked if less motor re-planning requires fewer resources in working memory (WM). To this end, participants executed a spatial WM task in parallel to different sequential motor tasks: (1) a randomised task with a high amount of motor re-planning and (2) an ordered task with a lower amount of motor re-planning. Recall performance in the spatial WM task was measured as the dependent variable. Hand posture was used to calculate the percentage of motor re-planning and, thus, to validate the experimental manipulation. The percentage of motor re-planning was lower in the ordered task, while spatial WM performance was higher. This indicates that WM resources depleted by the motor task scale with the amount of motor re-planning. Results further showed a significant recency effect (i.e. better recall of late items) in the spatial WM task. As previous studies found that recency effects in a verbal WM task are disrupted by a concurrent motor task, the presence of recency in the current study indicates a differential interference of a concurrent motor task on verbal vs. spatial recall, which has important implications for several current models of WM.

Reward Expectation Differentially Modulates Global and Local Spatial Working Memory Accuracy

Although it has been suggested that reward expectation affects the performance of spatial working memory tasks, controversial results have been found in previous experiments. Hence, it is still unclear to what extent reward expectation has an effect on working memory. To clarify this question, a memory-guided saccade task was applied, in which participants were instructed to retain and reconstruct a temporospatial sequence of four locations by moving their eyes in each trial. The global- and local-level spatial working memory accuracies were calculated to determine the reward effect on the global and local level of processing in spatial working memory tasks. Although high reward expectation enhanced the encoding of spatial information, the percentage of trials in which the cued location was correctly fixated decreased with increment of reward expectation. The reconstruction of the global temporospatial sequence was enhanced by reward expectation, whereas the local reconstruction performance was not affected by reward. Furthermore, the improvements in local representations of uncued locations and local sequences were at the cost of the representation of cued locations. The results suggest that the reward effect on spatial working memory is modulated by the level of processing, which supports the flexible resource theory during maintenance.

Neural measures of working memory in a bilateral change detection task

The change detection task is a widely used paradigm to examine visual working memory processes. Participants memorize a set of items and then, try to detect changes in the set after a retention period. The negative slow wave (NSW) and contralateral delay activity (CDA) are event-related potentials in the EEG signal that are commonly used in change detection tasks to track working memory load, as both increase with the number of items maintained in working memory (set size). While the CDA was argued to more purely reflect the memory-specific neural activity than the NSW, it also requires a lateralized design and attention shifts prior to memoranda onset, imposing more restrictions on the task than the NSW. The present study proposes a novel change detection task in which both CDA and NSW can be measured at the same time. Memory items were presented bilaterally, but their distribution in the left and right hemifield varied, inducing a target imbalance or "net load." NSW increased with set size, whereas CDA increased with net load. In addition, a multivariate linear classifier was able to decode the set size and net load from the EEG signal. CDA, NSW, and decoding accuracy predicted an individual's working memory capacity. In line with the notion of a bilateral advantage in working memory, accuracy, and CDA data suggest that participants tended to encode items relatively balanced. In sum, this novel change detection task offers a basis to make use of converging neural measures of working memory in a comprehensive paradigm.

Bold moves: Inevitable saccadic selection in visual short-term memory

Selection for visual short-term memory (vstm) provides a basis for many cognitive functions. Saccadic eye movements sway this selection in favor of stimuli previously seen at locations congruent with their target. In three experiments, we provide converging evidence that this saccadic selection is implemented as a fundamental, inevitable selection process, rather than a top-down strategy. In particular, benefits for congruent over incongruent items were largely constant across set sizes ranging from two to eight items (Experiment 1), showing that saccadic selection imposes priorities on vstm irrespective of memory load and is effective even when only few representations need to be maintained. Moreover, a decrement in performance for incongruent items occurred reliably, whether the congruent location contained a task-relevant item or an irrelevant noise patch (Experiment 2). Finally, saccadic selection was immune to a strong manipulation of the observer's attentional priorities (Experiment 3). Given the prevalence of saccades in natural vision, our results demonstrate a fundamental and ecologically relevant selection mechanism for vstm: Saccades systematically eliminate information seen at non-target locations, while information at the saccade target remains available to recall. This simple heuristic is effective in the absence of informative cues and may incapacitate voluntary selection mechanisms that are incongruent with ongoing movement plans.

The spatial and temporal properties of attentional selectivity for saccades and reaches

The preparation and execution of saccades and goal-directed movements elicits an accompanying shift in attention at the locus of the impending movement. However, some key aspects of the spatiotemporal profile of this attentional shift between eye and hand movements are not resolved. While there is evidence that attention is improved at the target location when making a reach, it is not clear how attention shifts over space and time around the movement target as a saccade and a reach are made to that target. Determining this spread of attention is an important aspect in understanding how attentional resources are used in relation to movement planning and guidance in real world tasks. We compared performance on a perceptual discrimination paradigm during a saccade-alone task, reach-alone task, and a saccade-plus-reach task to map the temporal profile of the premotor attentional shift at the goal of the movement and at three surrounding locations. We measured performance relative to a valid baseline level to determine whether motor planning induces additional attentional facilitation compared to mere covert attention. Sensitivity increased relative to movement onset at the target and at the surrounding locations, for both the saccade-alone and saccade-plus-reach conditions. The results suggest that the temporal profile of the attentional shift is similar for the two tasks involving saccades (saccade-alone and saccade-plus-reach tasks), but is very different when the influence of the saccade is removed. In this case, performance in the saccade-plus-reach task reflects the lower sensitivity observed when a reach-alone task is being conducted. In addition, the spatial profile of this spread of attention is not symmetrical around the target. This suggests that when a saccade and reach are being planned together, the saccade drives the attentional shift, and the reach-alone carries little attentional weight.

Evidence accumulation is biased by motivation: A computational account

To make good judgments people gather information. An important problem an agent needs to solve is when to continue sampling data and when to stop gathering evidence. We examine whether and how the desire to hold a certain belief influences the amount of information participants require to form that belief. Participants completed a sequential sampling task in which they were incentivized to accurately judge whether they were in a desirable state, which was associated with greater rewards than losses, or an undesirable state, which was associated with greater losses than rewards. While one state was better than the other, participants had no control over which they were in, and to maximize rewards they had to maximize accuracy. Results show that participants’ judgments were biased towards believing they were in the desirable state. They required a smaller proportion of supporting evidence to reach that conclusion and ceased gathering samples earlier when reaching the desirable conclusion. The findings were replicated in an additional sample of participants. To examine how this behavior was generated we modeled the data using a drift-diffusion model. This enabled us to assess two potential mechanisms which could be underlying the behavior: (i) a valence-dependent response bias and/or (ii) a valence-dependent process bias. We found that a valence-dependent model, with both a response bias and a process bias, fit the data better than a range of other alternatives, including valence-independent models and models with only a response or process bias. Moreover, the valence-dependent model provided better out-of-sample prediction accuracy than the valence-independent model. Our results provide an account for how the motivation to hold a certain belief decreases the need for supporting evidence. The findings also highlight the advantage of incorporating valence into evidence accumulation models to better explain and predict behavior.

People tend to gather information before making judgments. As information is often unlimited a decision has to be made as to when the data is sufficient to reach a conclusion. Here, we show that the decision to stop gathering data is influenced by whether the data points towards the desired conclusion. Importantly, we characterize the factors that generate this behaviour using a valence-dependent evidence accumulation model. In a sequential sampling task participants sampled less evidence before reaching a desirable than undesirable conclusion. Despite being incentivized for accuracy, participants’judgments were biased towards believing they were in a desirable state. Fitting the data to an evidence accumulation model revealed this behavior was due both to the starting point and rate of evidence accumulation being biased towards desirable beliefs. Our results show that evidence accumulation is altered by what people want to believe and provide an account for how this modulation is generated.

The possibility to make choices modulates feature-based effects of reward

When making decisions, humans can maximize the positive outcome of their actions by choosing the option associated with the highest reward. We have recently shown that choices modulate effects of reward via a bias in spatial attention: Locations associated with a lower reward are anticipatorily suppressed, as indicated by delayed responses to low-reward targets and increased parieto-occipital alpha power. Here, we investigated whether this inhibition also occurs when reward is not coupled to location but to a nonspatial feature (color). We analyzed reaction times to single targets associated with a low or high reward as a function of whether a second trial type, choice-trials, were interleaved. In choice-trials, participants could choose either one of two targets to obtain the associated reward. Indeed, responses to low-reward targets were slower when choice-trials were present, magnifying the influence of reward, and this delay was more pronounced in trials immediately following a choice. No corresponding changes in parieto-occipital alpha power were observed, but the behavioral findings suggest that choices modulate a reward-related bias in feature-based attention in a similar manner as for spatial attention, and support the idea that reward primarily affects behaviour when it is of immediate relevance.

Explicit knowledge of sensory non-redundancy can reduce the strength of multisensory integration

The brain integrates incoming sensory signals to a degree that depends on the signals' redundancy. Redundancy-which is commonly high when signals originate from a common physical object or event-is estimated by the brain from the signals' spatial and/or temporal correspondence. Here we tested whether verbally instructed knowledge of non-redundancy can also be used to reduce the strength of the sensory integration. We used a cursor-control task in which cursor motions in the frontoparallel plane were controlled by hand movements in the horizontal plane, yet with a small and randomly varying visuomotor rotation that created spatial discrepancies between hand and cursor positions. Consistent with previous studies, we found mutual biases in the hand and cursor position judgments, indicating partial sensory integration. The integration was reduced in strength, but not eliminated, after participants were verbally informed about the non-redundancy (i.e., the spatial discrepancies) in the hand and cursor positions. Comparisons with model predictions excluded confounding bottom-up effects of the non-redundancy instruction. Our findings thus show that participants have top-down control over the degree to which they integrate sensory information. Additionally, we found that the magnitude of this top-down modulatory capability is a reliable individual trait. A comparison between participants with and without video-gaming experience tentatively suggested a relation between top-down modulation of integration strength and attentional control.