Separate neural correlates for the mnemonic components of successive discrimination and working memory tasks

Perceptual Learning beyond Perception: Mnemonic Representation in Early Visual Cortex and Intraparietal Sulcus

The ability to discriminate between stimuli relies on a chain of neural operations associated with perception, memory and decision-making. Accumulating studies show learning-dependent plasticity in perception or decision-making, yet whether perceptual learning modifies mnemonic processing remains unclear. Here, we trained human participants of both sexes in an orientation discrimination task, while using functional magnetic resonance imaging (fMRI) and transcranial magnetic stimulation (TMS) to separately examine training-induced changes in working memory (WM) representation. fMRI decoding revealed orientation-specific neural patterns during the delay period in primary visual cortex (V1) before, but not after, training, whereas neurodisruption of V1 during the delay period led to behavioral deficits in both phases. In contrast, both fMRI decoding and disruptive effect of TMS showed that intraparietal sulcus (IPS) represented WM content after, but not before, training. These results suggest that training does not affect the necessity of sensory area in representing WM information, consistent with the sensory recruitment hypothesis in WM, but likely alters the coding format of the stored stimulus in this region. On the other hand, training can render WM content to be maintained in higher-order parietal areas, complementing sensory area to support more robust maintenance of information.SIGNIFICANCE STATEMENT There has been accumulating progresses regarding experience-dependent plasticity in perception or decision-making, yet how perceptual experience moulds mnemonic processing of visual information remains less explored. Here, we provide novel findings that learning-dependent improvement of discriminability accompanies altered WM representation at different cortical levels. Critically, we suggest a role of training in modulating cortical locus of WM representation, providing a plausible explanation to reconcile the discrepant findings between human and animal studies regarding the recruitment of sensory or higher-order areas in WM.

Single-exposure visual memory judgments are reflected in inferotemporal cortex

Our visual memory percepts of whether we have encountered specific objects or scenes before are hypothesized to manifest as decrements in neural responses in inferotemporal cortex (IT) with stimulus repetition. To evaluate this proposal, we recorded IT neural responses as two monkeys performed a single-exposure visual memory task designed to measure the rates of forgetting with time. We found that a weighted linear read-out of IT was a better predictor of the monkeys’ forgetting rates and reaction time patterns than a strict instantiation of the repetition suppression hypothesis, expressed as a total spike count scheme. Behavioral predictions could be attributed to visual memory signals that were reflected as repetition suppression and were intermingled with visual selectivity, but only when combined across the most sensitive neurons.

As we go about our daily lives, we store visual memories of the objects and scenes that we encounter. This type of memory, known as visual recognition memory, can be remarkably powerful. Imagine viewing thousands of images for only a few seconds each, for example. Several days later, you will still be able to distinguish most of those images from previously unseen ones. How does the brain do this?

Visual information travels from the eyes to an area of the brain called visual cortex. Neurons in a region of visual cortex called inferotemporal cortex fire in a particular pattern to reflect what is being seen. These neurons also reflect memories of whether those things have been seen before, by firing more when things are new and less when they are viewed again. This decrease in firing, known as repetition suppression, may be the signal in the brain responsible for the sense of remembering.

Meyer and Rust have now tested this idea by training macaque monkeys to report whether images on a screen were new or familiar. The monkeys were very good at remembering the images they had seen more recently, although they tended to forget some of the images with time. Then, the rate at which the monkeys forgot the images was compared with the rate at which repetition suppression disappeared in inferotemporal cortex. The results showed that the total number of firing events in this region was not a great predictor of how long the monkeys remembered images. However, a decrease in the number of firing events for a particular subset of the neurons did predict the remembering and forgetting. Repetition suppression in certain inferotemporal cortex neurons can thus account for visual recognition memory.

Brain disorders and aging can both give rise to memory deficits. Identifying the mechanisms underlying memory may lead to new treatments for memory-related disorders. Visual recognition memory may be a good place to start because of our existing knowledge of how the brain processes visual information. Understanding visual recognition memory could help us understand the mechanisms of memory more broadly.

Linear and nonlinear prefrontal and parietal activity during multiple-item working memory

Most parts of the prefrontal and posterior parietal cortices show sustained activity during short-term maintenance of visual information and their activity increases with increasing memory set size. To investigate the interplay of feature selectivity, memory load and inter-item interaction (or interference) on sustained activity, we compared and contrasted fMRI signal during the retention of two items from the same or different visual feature categories (e.g., two line orientations versus a line and a color) relative to the retention of single items. Data from 16 young adults revealed three types of activation patterns in the prefrontal and posterior parietal cortices. First, among the prefrontal and posterior parietal areas that showed preferential responses to line orientations, some exhibited linear increases in sustained activity whereas others exhibited nonlinear increases in correspondence to the number of lines in the memory set. Second, the right lateral prefrontal and ventral posterior parietal areas, albeit not showing differential sustained activity relative to lines or colors, were disproportionately more active during holding two lines in comparison to holding a line and a color. Third, the left posterior intraparietal sulcus showed a weak effect of memory set size regardless of the items' visual features. These observations suggest that rather than number of items, a combination of factors such as visual feature and memory-set homogeneity may have the greater influence on prefrontal and parietal activity during multiple-item working memory. This is consistent with the view that working memory capacity is influenced by the level of interaction or interference between visual stimuli, which is stronger between items from the same feature category.

Transcranial magnetic stimulation over posterior parietal cortex disrupts transsaccadic memory of multiple objects

The posterior parietal cortex (PPC) plays a role in spatial updating of goals for eye and arm movements across saccades, but less is known about its role in updating perceptual memory. We reported previously that transsaccadic memory has a capacity for storing the orientations of three to four Gabor patches either within a single fixation (fixation task) or between separate fixations (saccade task). Here, we tested the role of the PPC in transsaccadic memory in eight subjects by simultaneously applying single-pulse transcranial magnetic stimulation (TMS) over the right and left PPC, over several control sites, and comparing these to behavioral controls with no TMS. In TMS trials, we randomly delivered pulses at one of three different time intervals around the time of the saccade, or at an equivalent time in the fixation task. Controls confirmed that subjects could normally retain at least three visual features. TMS over the left PPC and a control site had no significant effect on this performance. However, TMS over the right PPC disrupted memory performance in both tasks. This TMS-induced effect was most disruptive in the saccade task, in particular when stimulation coincided more closely with saccade timing. Here, the capacity to compare presaccadic and postsaccadic features was reduced to one object, as expected if the spatial aspect of memory was disrupted. This finding suggests that right PPC plays a role in the spatial processing involved in transsaccadic memory of visual features. We propose that this process uses saccade-related feedback signals similar to those observed in spatial updating.

Delayed discrimination of spatial frequency for gratings of different orientation: behavioral and fMRI evidence for low-level perceptual memory stores in early visual cortex

The concept of perceptual memory refers to the neural and cognitive processes underlying the storage of specific stimulus features such as spatial frequency, orientation, shape, contrast, and color. Psychophysical studies of perceptual memory indicate that observers can retain visual information about the spatial frequency of Gabor patterns independent of the orientation with which they are presented. Compared to discrimination of gratings with the same orientation, reaction times to orthogonally oriented gratings, however, increase suggesting additional processing. Using event-related fMRI we examined the pattern of neural activation evoked when subjects discriminated the spatial frequency of Gabors presented with the same or orthogonal orientation. Blood-oxygen level dependent BOLD fMRI revealed significantly elevated bilateral activity in visual areas (V1, V2) when the gratings to be compared had an orthogonal orientation, compared to when they had the same orientation. These findings suggest that a change in an irrelevant stimulus dimension requires additional processing in primary and secondary visual areas. The finding that the task-irrelevant stimulus property (orientation) had no significant effect on the prefrontal and intraparietal cortex supports a model of working memory in which discrimination and retention of basic stimulus dimensions is based on low-level perceptual memory stores that are located at an early stage in the visual process. Our findings suggest that accessing different stores requires time and has higher metabolic costs.

Dissociation of neural correlates of verbal and non-verbal visual working memory with different delays

Background

Dorsolateral prefrontal cortex (DLPFC), posterior parietal cortex, and regions in the occipital cortex have been identified as neural sites for visual working memory (WM). The exact involvement of the DLPFC in verbal and non-verbal working memory processes, and how these processes depend on the time-span for retention, remains disputed.

Methods

We used functional MRI to explore the neural correlates of the delayed discrimination of Gabor stimuli differing in orientation. Twelve subjects were instructed to code the relative orientation either verbally or non-verbally with memory delays of short (2 s) or long (8 s) duration.

Results

Blood-oxygen level dependent (BOLD) 3-Tesla fMRI revealed significantly more activity for the short verbal condition compared to the short non-verbal condition in bilateral superior temporal gyrus, insula and supramarginal gyrus. Activity in the long verbal condition was greater than in the long non-verbal condition in left language-associated areas (STG) and bilateral posterior parietal areas, including precuneus. Interestingly, right DLPFC and bilateral superior frontal gyrus was more active in the non-verbal long delay condition than in the long verbal condition.

Conclusion

The results point to a dissociation between the cortical sites involved in verbal and non-verbal WM for long and short delays. Right DLPFC seems to be engaged in non-verbal WM tasks especially for long delays. Furthermore, the results indicate that even slightly different memory maintenance intervals engage largely differing networks and that this novel finding may explain differing results in previous verbal/non-verbal WM studies.

Cortical oscillatory activity during spatial echoic memory

In human magnetoencephalogram, we have found gamma-band activity (GBA), a putative measure of cortical network synchronization, during both bottom-up and top-down auditory processing. When sound positions had to be retained in short-term memory for 800 ms, enhanced GBA was detected over posterior parietal cortex, possibly reflecting the activation of higher sensory storage systems along the hypothesized auditory dorsal space processing stream. Additional prefrontal GBA increases suggested an involvement of central executive networks in stimulus maintenance. The present study assessed spatial echoic memory with the same stimuli but a shorter memorization interval of 200 ms. Statistical probability mapping revealed posterior parietal GBA increases at 80 Hz near the end of the memory phase and both gamma and theta enhancements in response to the test stimulus. In contrast to the previous short-term memory study, no prefrontal gamma or theta enhancements were detected. This suggests that spatial echoic memory is performed by networks along the putative auditory dorsal stream, without requiring an involvement of prefrontal executive regions.

Attention to 3-D shape, 3-D motion, and texture in 3-D structure from motion displays

We used fMRI to directly compare the neural substrates of three-dimensional (3-D) shape and motion processing for realistic textured objects rotating in depth. Subjects made judgments about several different attributes of these objects, including 3-D shape, the 3-D motion, and the scale of surface texture. For all of these tasks, we equated visual input, motor output, and task difficulty, and we controlled for differences in spatial attention. Judgments about 3-D shape from motion involve both parietal and occipito-temporal regions. The processing of 3-D shape is associated with the analysis of 3-D motion in parietal regions and the analysis of surface texture in occipito-temporal regions, which is consistent with the different behavioral roles that are typically attributed to the dorsal and ventral processing streams.

Neuroimaging studies of working memory: a meta-analysis

We performed meta-analyses on 60 neuroimaging (PET and fMRI) studies of working memory (WM), considering three types of storage material (spatial, verbal, and object), three types of executive function (continuous updating of WM, memory for temporal order, and manipulation of information in WM), and interactions between material and executive function. Analyses of material type showed the expected dorsal-ventral dissociation between spatial and nonspatial storage in the posterior cortex, but not in the frontal cortex. Some support was found for left frontal dominance in verbal WM, but only for tasks with low executive demand. Executive demand increased right lateralization in the frontal cortex for spatial WM. Tasks requiring executive processing generally produce more dorsal frontal activations than do storage-only tasks, but not all executive processes show this pattern. Brodmann's areas (BAs) 6, 8, and 9, in the superior frontal cortex, respond most when WM must be continuously updated and when memory for temporal order must be maintained. Right BAs 10 and 47, in the ventral frontal cortex, respond more frequently with demand for manipulation (including dual-task requirements or mental operations). BA 7, in the posterior parietal cortex, is involved in all types of executive function. Finally, we consider a potential fourth executive function: selective attention to features of a stimulus to be stored in WM, which leads to increased probability of activating the medial prefrontal cortex (BA 32) in storage tasks.

Involvement of multiple functionally distinct cerebellar regions in visual discrimination: a human functional imaging study

We investigated the contribution of the human cerebellum to cerebral function during visual discrimination using PET and fMRI. The cognitive task was a successive discrimination of shades of brown with a parametric variation of the stimulus presentation rate and a constant task difficulty. The successive color discrimination task was contrasted to a dimming detection control task, with identical retinal input but with double the number of motor responses. Three sets of activated cerebellar and cerebral regions were observed: rate-dependent and rate-independent color discrimination networks and a motor-and-detection network. The rate-dependent color discrimination network included both an anterior and a posterior activation site in lobule-VI of the two lateral cerebellar hemispheres, whereas the rate-independent network involved a bilateral activation site in lateral Crus-I. Cerebellar sites of the motor-and-detection network were located in medial lobule-V bilaterally, in the vermis, and in posterior left Crus-I and right Crus-II. An additional fMRI study was performed to control for differences in motor output and response timing between the tasks. In this control study, the cerebellar activation sites of the rate-dependent and rate-independent color discrimination networks remained unaltered. The motor-and-detection network included cerebellar activations in posterior left Crus-I and right Crus-II, but none in lobule-V or the vermis. Thus, cerebellar activation sites of the motor-and-detection network could be subdivided into those related to a motor network and those belonging to a dimming detection network. We conclude that successive color discrimination activates multiple, functionally distinct cerebellar regions.

Cognitive processing in the primary visual cortex: from perception to memory

The primary visual cortex is the first cortical area of the visual system that receives information from the external visual world. Based on the receptive field characteristics of the neurons in this area, it has been assumed that the primary visual cortex is a pure sensory area extracting basic elements of the visual scene. This information is then subsequently further processed upstream in the higher-order visual areas and provides us with perception and storage of the visual environment. However, recent findings show that such neural implementations are observed in the primary visual cortex. These neural correlates are expressed by the modulated activity of the late response of a neuron to a stimulus, and most likely depend on recurrent interactions between several areas of the visual system. This favors the concept of a distributed nature of visual processing in perceptual organization.

Large capacity storage of integrated objects before change blindness

Normal people have a strikingly low ability to detect changes in a visual scene. This has been taken as evidence that the brain represents only a few objects at a time, namely those currently in the focus of attention. In the present study, subjects were asked to detect changes in the orientation of rectangular figures in a textured display across a 1600 ms gray interval. In the first experiment, change detection improved when the location of a possible change was cued during the interval. The cue remained effective during the entire interval, but after the interval, it was ineffective, suggesting that an initially large representation was overwritten by the post-change display. To control for an effect of light intensity during the interval on the decay of the representation, we compared performance with a gray or a white interval screen in a second experiment. We found no difference between these conditions. In the third experiment, attention was occasionally misdirected during the interval by first cueing the wrong figure, before cueing the correct figure. This did not compromise performance compared to a single cue, indicating that when an item is attentionally selected, the representation of yet unchosen items remains available. In the fourth experiment, the cue was shown to be effective when changes in figure size and orientation were randomly mixed. At the time the cue appeared, subjects could not know whether size or orientation would change, therefore these results suggest that the representation contains features in their 'bound' state. Together, these findings indicate that change blindness involves overwriting of a large capacity representation by the post-change display.

Human brain regions involved in visual categorization

Categorization of dot patterns is a frequently used paradigm in the behavioral study of natural categorization. To determine the human brain regions involved in categorization, we used Positron Emission Tomography to compare regional Cerebral Blood Flow patterns in two tasks employing patterns that consisted of nine dots. In the categorization task, subjects categorized novel exemplars of two categories, generated by distorting two prototypes, and other random dot patterns. In the control task, subjects judged the position of similarly distorted patterns. Each task was presented at two matched levels of difficulty. Fixation of the fixation target served as baseline condition. The categorization task differentially activated the orbitofrontal cortex and two dorsolateral prefrontal regions. These three prefrontal regions were equally weakly active in the position discrimination task and the baseline condition. The intraparietal sulcus was activated in both tasks, albeit significantly less in the position discrimination than in the categorization task. A similar activation pattern was present in the neostriatum. Task difficulty had no effect. These functional imaging results show that the dot-pattern categorization task strongly engages prefrontal and parietal cortical areas. The activation of prefrontal cortex during visual categorization in humans agrees with the recent finding of category-related responses in macaque prefrontal neurons.

Task-dependent transfer of perceptual to memory representations during delayed spatial frequency discrimination

Discrimination thresholds were obtained using a delayed spatial frequency discrimination task. In Experiment 1, we found that presentation of a mask 3 s before onset of a reference Gabor patch caused a selective, spatial frequency dependent interference in a subsequent discrimination task. However, a 10 s interval abolished this masking effect. In Experiment 2, the mask was associated with a second spatial frequency discrimination task so that a representation of the mask had to be coded into short-term perceptual memory. This experiment was performed to assess whether absence of masking in the 10 s condition of Experiment 1 might be due to decay of the mask information in the perceptual or the memory representational domain. The presence of this second discrimination task now caused similar interference effects on the primary discrimination task at both the 3 s and 10 s interstimulus intervals (ISI) conditions. Finally, to test the robustness of the masking effect, the nature of the secondary masking task was changed from a spatial frequency discrimination task to an orientation discrimination task in Experiment 3. The masking effect was now abolished in both the 3 and 10 s ISI conditions. Together, the results from these experiments are consistent with the idea of a two-level perceptual memory mechanism. The results also suggest that stimulus representations during a perceptual discrimination task are shared between the perceptual and memory representation domains in a task-dependent manner.

The quantitative nature of a visual task differentiates between ventral and dorsal stream

The aim of the present positron emission tomography (PET) study was to investigate how visual processing in dorsal and ventral streams depends on the quantitative nature of the task. In the same-different task, participants identified the presence of an orientation difference between two gratings, presented centrally in succession. In the quantification task, participants estimated the magnitude of the difference and compared it to a fixed standard. Detection of dimming of the fixation point was used as a control task. Visual input, motor responses, and performance were equated across tasks. Subtracting same-different from quantification yielded significant activation in the left superior parietal lobule and left ventral premotor cortex, consistent with results obtained in number-processing tasks. The reverse subtraction yielded activation in the right inferior temporal gyrus, in agreement with earlier results. These results demonstrate that a single attribute can be processed either in the ventral or dorsal stream, depending on the cognitive operations required by the tasks.

The neural substrate of orientation short-term memory and resistance to distractor items

We used Positron Emission Tomography to map the neural substrate of human short-term memory for orientation, defined as retaining a single orientation in memory over a long delay, by comparing a successive discrimination task with a 6-s delay to the same task with a brief 0.3 s delay and to an identification control task. Short-term memory engaged the superior parietal lobe bilaterally, the middle occipital gyrus bilaterally and the left dorsolateral prefrontal cortex. In addition, we studied the resistance to a distractor item by comparing the successive discrimination task with long delay, with and without an intervening distractor stimulus. This manipulative process engaged left ventral premotor cortex and left dorsolateral prefrontal cortex. The activation of left dorsolateral prefrontal cortex is interpreted as reflecting co-ordination between task components. These results, combined with those of two previous studies using an identical reduction strategy, underscore the functional heterogeneity in the prefrontal cortex during short-term and working memory.

The neural substrate of orientation working memory

We have used positron emission tomography (PET) to identify the neural substrate of two major cognitive components of working memory (WM), maintenance and manipulation of a single elementary visual attribute, i.e., the orientation of a grating presented in central vision. This approach allowed us to equate difficulty across tasks and prevented subjects from using verbal strategies or vestibular cues. Maintenance of orientations involved a distributed fronto-parietal network, that is, left and right lateral superior frontal sulcus (SFSl), bilateral ventrolateral prefrontal cortex (VLPFC), bilateral precuneus, and right superior parietal lobe (SPL). A more medial superior frontal sulcus region (SFSm) was identified as being instrumental in the manipulative operation of updating orientations retained in the WM. Functional connectivity analysis revealed that orientation WM relies on a coordinated interaction between frontal and parietal regions. In general, the current findings confirm the distinction between maintenance and manipulative processes, highlight the functional heterogeneity in the prefrontal cortex (PFC), and suggest a more dynamic view of WM as a process requiring the coordinated interaction of anatomically distinct brain areas.

Orientation discrimination of objects and gratings compared: an fMRI study

We used functional magnetic resonance imaging to compare the human brain regions involved in orientation discrimination of two-dimensional (2D) objects and gratings. The orientation discrimination tasks, identification and successive discrimination, were contrasted to a dimming detection control condition with identical retinal input. Regions involved in orientation discrimination were very similar for the two types of tasks and for the two types of stimuli and both belonged to the dorsal and ventral visual pathways. They included posterior occipital, lingual, posterior fusiform, inferior temporal, dorsal intraparietal and medial parietal regions. The main difference between the two types of stimuli was a larger activation of precuneus when 2D objects were used compared to gratings. The main difference between discrimination tasks was an enhanced activity, at the group level, in superior frontal sulcus in identification compared to successive discrimination, and at least at the single subject level, a larger activity in right fusiform cortex in successive discriminations compared to identification. Thus, in contradiction to generally accepted views, orientation discrimination of gratings and objects involve largely similar networks including both ventral and dorsal visual regions.