Decoding complex flow-field patterns in visual working memory

Cortical and white matter substrates supporting visuospatial working memory

OBJECTIVE

In tasks involving new visuospatial information, we rely on working memory, supported by a distributed brain network. We investigated the dynamic interplay between brain regions, including cortical and white matter structures, to understand how neural interactions change with different memory loads and trials, and their subsequent impact on working memory performance.

METHODS

Patients undertook a task of immediate spatial recall during intracranial EEG monitoring. We charted the dynamics of cortical high-gamma activity and associated functional connectivity modulations in white matter tracts.

RESULTS

Elevated memory loads were linked to enhanced functional connectivity via occipital longitudinal tracts, yet decreased through arcuate, uncinate, and superior-longitudinal fasciculi. As task familiarity grew, there was increased high-gamma activity in the posterior inferior-frontal gyrus (pIFG) and diminished functional connectivity across a network encompassing frontal, parietal, and temporal lobes. Early pIFG high-gamma activity was predictive of successful recall. Including this metric in a logistic regression model yielded an accuracy of 0.76.

CONCLUSIONS

Optimizing visuospatial working memory through practice is tied to early pIFG activation and decreased dependence on irrelevant neural pathways.

SIGNIFICANCE

This study expands our knowledge of human adaptation for visuospatial working memory, showing the spatiotemporal dynamics of cortical network modulations through white matter tracts.

Nonfrontal Control of Working Memory

Items held in visual working memory can be quickly updated, replaced, removed, and even manipulated in accordance with current behavioral goals. Here, we use multivariate pattern analyses to identify the patterns of neuronal activity that realize the executive control processes supervising these flexible stores. We find that portions of the middle temporal gyrus and the intraparietal sulcus represent what item is cued for continued memorization independently of representations of the item itself. Importantly, this selection-specific activity could not be explained by sensory representations of the cue and is only present when control is exerted. Our results suggest that the selection of memorized items might be controlled in a distributed and decentralized fashion. This evidence provides an alternative perspective to the notion of "domain general" central executive control over memory function.

Working memory signals in early visual cortex are present in weak and strong imagers

It has been suggested that visual images are memorized across brief periods of time by vividly imagining them as if they were still there. In line with this, the contents of both working memory and visual imagery are known to be encoded already in early visual cortex. If these signals in early visual areas were indeed to reflect a combined imagery and memory code, one would predict them to be weaker for individuals with reduced visual imagery vividness. Here, we systematically investigated this question in two groups of participants. Strong and weak imagers were asked to remember images across brief delay periods. We were able to reliably reconstruct the memorized stimuli from early visual cortex during the delay. Importantly, in contrast to the prediction, the quality of reconstruction was equally accurate for both strong and weak imagers. The decodable information also closely reflected behavioral precision in both groups, suggesting it could contribute to behavioral performance, even in the extreme case of completely aphantasic individuals. Our data thus suggest that working memory signals in early visual cortex can be present even in the (near) absence of phenomenal imagery.

Working Memory Maintenance of Visual and Auditory Spatial Information Relies on Supramodal Neural Codes in the Dorsal Frontoparietal Cortex

The frontoparietal attention network plays a pivotal role during working memory (WM) maintenance, especially under high-load conditions. Nevertheless, there is ongoing debate regarding whether this network relies on supramodal or modality-specific neural signatures. In this study, we used multi-voxel pattern analysis (MVPA) to evaluate the neural representation of visual versus auditory information during WM maintenance. During fMRI scanning, participants maintained small or large spatial configurations (low- or high-load trials) of either colour shades or sound pitches in WM for later retrieval. Participants were less accurate in retrieving high- vs. low-load trials, demonstrating an effective manipulation of WM load, irrespective of the sensory modality. The frontoparietal regions involved in maintaining high- vs. low-load spatial maps in either sensory modality were highlighted using a conjunction analysis. Widespread activity was found across the dorsal frontoparietal network, peaking on the frontal eye fields and the superior parietal lobule, bilaterally. Within these regions, MVPAs were performed to quantify the pattern of distinctness of visual vs. auditory neural codes during WM maintenance. These analyses failed to reveal distinguishable patterns in the dorsal frontoparietal regions, thus providing support for a common, supramodal neural code associated with the retention of either visual or auditory spatial configurations.

Spectrotemporal content of human auditory working memory represented in functional connectivity patterns

Recent research suggests that working memory (WM), the mental sketchpad underlying thinking and communication, is maintained by multiple regions throughout the brain. Whether parts of a stable WM representation could be distributed across these brain regions is, however, an open question. We addressed this question by examining the content-specificity of connectivity-pattern matrices between subparts of cortical regions-of-interest (ROI). These connectivity patterns were calculated from functional MRI obtained during a ripple-sound auditory WM task. Statistical significance was assessed by comparing the decoding results to a null distribution derived from a permutation test considering all comparable two- to four-ROI connectivity patterns. Maintained WM items could be decoded from connectivity patterns across ROIs in frontal, parietal, and superior temporal cortices. All functional connectivity patterns that were specific to maintained sound content extended from early auditory to frontoparietal cortices. Our results demonstrate that WM maintenance is supported by content-specific patterns of functional connectivity across different levels of cortical hierarchy.

Bimodal sensory integration in migraine: A study of the effect of visual stimulation on somatosensory evoked cortical responses

BACKGROUND

Merging of sensory information is a crucial process for adapting the behaviour to the environment in all species. It is not known if this multisensory integration might be dysfunctioning interictally in migraine without aura, where sensory stimuli of various modalities are processed abnormally when delivered separately. To investigate this question, we compared the effects of a concomitant visual stimulation on conventional low-frequency somatosensory evoked potentials and embedded high-frequency oscillations between migraine patients and healthy volunteers.

METHODS

We recorded somatosensory evoked potentials in 19 healthy volunteers and in 19 interictal migraine without aura patients before, during, and 5 min after (T2) simultaneous synchronous pattern-reversal visual stimulation. At each time point, we measured amplitude and habituation of the N20-P25 low-frequency-somatosensory evoked potentials component and maximal peak-to-peak amplitude of early and late bursts of high-frequency oscillations.

RESULTS

In healthy volunteers, the bimodal stimulation significantly reduced low-frequency-somatosensory evoked potentials habituation and tended to reduce early high-frequency oscillations that reflect thalamocortical activity. By contrast, in migraine without aura patients, bimodal stimulation significantly increased low-frequency-somatosensory evoked potentials habituation and early high-frequency oscillations. At T2, all visual stimulation-induced changes of somatosensory processing had vanished.

CONCLUSION

These results suggest a malfunctioning multisensory integration process, which could be favoured by an abnormal excitability level of thalamo-cortical loops.

Behavioral Prioritization Enhances Working Memory Precision and Neural Population Gain

Humans allocate visual working memory (WM) resource according to behavioral relevance, resulting in more precise memories for more important items. Theoretically, items may be maintained by feature-tuned neural populations, where the relative gain of the populations encoding each item determines precision. To test this hypothesis, we compared the amplitudes of delay period activity in the different parts of retinotopic maps representing each of several WM items, predicting the amplitudes would track behavioral priority. Using fMRI, we scanned participants while they remembered the location of multiple items over a WM delay and then reported the location of one probed item using a memory-guided saccade. Importantly, items were not equally probable to be probed (0.6, 0.3, 0.1, 0.0), which was indicated with a precue. We analyzed fMRI activity in 10 visual field maps in occipital, parietal, and frontal cortex known to be important for visual WM. In early visual cortex, but not association cortex, the amplitude of BOLD activation within voxels corresponding to the retinotopic location of visual WM items increased with the priority of the item. Interestingly, these results were contrasted with a common finding that higher-level brain regions had greater delay period activity, demonstrating a dissociation between the absolute amount of activity in a brain area and the activity of different spatially selective populations within it. These results suggest that the distribution of WM resources according to priority sculpts the relative gains of neural populations that encode items, offering a neural mechanism for how prioritization impacts memory precision.

Disrupting Short-Term Memory Maintenance in Premotor Cortex Affects Serial Dependence in Visuomotor Integration

Human behavior is biased by past experience. For example, when intercepting a moving target, the speed of previous targets will bias responses in future trials. Neural mechanisms underlying this so-called serial dependence are still under debate. Here, we tested the hypothesis that the previous trial leaves a neural trace in brain regions associated with encoding task-relevant information in visual and/or motor regions. We reasoned that injecting noise by means of transcranial magnetic stimulation (TMS) over premotor and visual areas would degrade such memory traces and hence reduce serial dependence. To test this hypothesis, we applied bursts of TMS pulses to right visual motion processing region hV5/MT+ and to left dorsal premotor cortex (PMd) during intertrial intervals of a coincident timing task performed by twenty healthy human participants (15 female). Without TMS, participants presented a bias toward the speed of the previous trial when intercepting moving targets. TMS over PMd decreased serial dependence in comparison to the control Vertex stimulation, whereas TMS applied over hV5/MT+ did not. In addition, TMS seems to have specifically affected the memory trace that leads to serial dependence, as we found no evidence that participants' behavior worsened after applying TMS. These results provide causal evidence that an implicit short-term memory mechanism in premotor cortex keeps information from one trial to the next, and that this information is blended with current trial information so that it biases behavior in a visuomotor integration task with moving objects.SIGNIFICANCE STATEMENT Human perception and action are biased by the recent past. The origin of such serial bias is still not fully understood, but a few components seem to be fundamental for its emergence: the brain needs to keep previous trial information in short-term memory and blend it with incoming information. Here, we present evidence that a premotor area has a potential role in storing previous trial information in short-term memory in a visuomotor task and that this information is responsible for biasing ongoing behavior. These results corroborate the perspective that areas associated with processing information of a stimulus or task also participate in maintaining that information in short-term memory even when this information is no longer relevant for current behavior.

Persistent Activity During Working Memory From Front to Back

Working memory (WM) extends the duration over which information is available for processing. Given its importance in supporting a wide-array of high level cognitive abilities, uncovering the neural mechanisms that underlie WM has been a primary goal of neuroscience research over the past century. Here, we critically review what we consider the two major "arcs" of inquiry, with a specific focus on findings that were theoretically transformative. For the first arc, we briefly review classic studies that led to the canonical WM theory that cast the prefrontal cortex (PFC) as a central player utilizing persistent activity of neurons as a mechanism for memory storage. We then consider recent challenges to the theory regarding the role of persistent neural activity. The second arc, which evolved over the last decade, stemmed from sophisticated computational neuroimaging approaches enabling researchers to decode the contents of WM from the patterns of neural activity in many parts of the brain including early visual cortex. We summarize key findings from these studies, their implications for WM theory, and finally the challenges these findings pose. Our goal in doing so is to identify barriers to developing a comprehensive theory of WM that will require a unification of these two "arcs" of research.

Decoding Spatial Versus Non-spatial Processing in Auditory Working Memory

Objective

Research on visual working memory has shown that individual stimulus features are processed in both specialized sensory regions and higher cortical areas. Much less evidence exists for auditory working memory. Here, a main distinction has been proposed between the processing of spatial and non-spatial sound features. Our aim was to examine feature-specific activation patterns in auditory working memory.

Methods

We collected fMRI data while 28 healthy adults performed an auditory delayed match-to-sample task. Stimuli were abstract sounds characterized by both spatial and non-spatial information, i.e., interaural time delay and central frequency, respectively. In separate recording blocks, subjects had to memorize either the spatial or non-spatial feature, which had to be compared with a probe sound presented after a short delay. We performed both univariate and multivariate comparisons between spatial and non-spatial task blocks.

Results

Processing of spatial sound features elicited a higher activity in a small cluster in the superior parietal lobe than did sound pattern processing, whereas there was no significant activation difference for the opposite contrast. The multivariate analysis was applied using a whole-brain searchlight approach to identify feature-selective processing. The task-relevant auditory feature could be decoded from multiple brain regions including the auditory cortex, posterior temporal cortex, middle occipital gyrus, and extended parietal and frontal regions.

Conclusion

In summary, the lack of large univariate activation differences between spatial and non-spatial processing could be attributable to the identical stimulation in both tasks. In contrast, the whole-brain multivariate analysis identified feature-specific activation patterns in widespread cortical regions. This suggests that areas beyond the auditory dorsal and ventral streams contribute to working memory processing of auditory stimulus features.

Attention Biases Competition for Visual Representation via Dissociable Influences from Frontal and Parietal Cortex

What mechanisms underlie the prioritization of neural representations of visually perceived information to guide behavior? We assessed the dynamics whereby attention biases competition for representation of visual stimuli by enhancing representations of relevant information and suppressing the irrelevant. Multivariate pattern analysis (MVPA) classifiers were trained to discriminate patterns of fMRI activity associated with each of three stimuli, within several predefined ROIs. Participants performed a change-detection task wherein two of three presented items flashed at 1 Hz, one to each side of central fixation. Both flashing stimuli would unpredictably change state, but participants covertly counted the number of changes only for the cued item. In the ventral occipito-temporal ROI, MVPA evidence (a proxy for representational fidelity) was dynamically enhanced for attended stimuli and suppressed for unattended stimuli, consistent with a mechanism of biased competition between stimulus representations. Frontal and parietal ROIs displayed a qualitatively distinct, more "source-like" profile, wherein MVPA evidence for only the attended stimulus could be observed above baseline levels. To assess how attentional modulation of ventral occipito-temporal representations might relate to signals originating in the frontal and/or parietal ROIs, we analyzed informational connectivity (IC), which indexes time-varying covariation between regional levels of MVPA evidence. Parietal-posterior IC was elevated during the task, but did not differ for cued versus uncued items. Frontal-posterior IC, in contrast, was sensitive to an item's priority status. Thus, although regions of frontal and parietal cortex act as sources of top-down attentional control, their precise functions likely differ.

Consequence of stroke for feature recall and binding in visual working memory

Visual memory for objects involves the integration, or binding, of individual features into a coherent representation. We used a novel approach to assess feature binding, using a delayed-reproduction task in combination with computational modeling and lesion analysis. We assessed stroke patients and neurotypical controls on a visual working memory task in which spatial arrays of colored disks were presented. After a brief delay, participants either had to report the color of one disk cued by its location or the location of one disk cued by its color. Our results demonstrate that, in the controls, report imprecision and swap errors (non-target reports) can be explained by a single source of variability. Stroke patients showed an overall decrease in memory precision for both color and location, with only limited evidence for deviations from the predicted relationship between report precision and swap errors. These deviations were primarily deficits in reporting items rather than selecting items based on the cue. Atlas-based lesion-symptom mapping showed that selection and reporting deficits, precision in reporting color, and precision in reporting location were associated with different lesion profiles. Deficits in binding are associated with lesions in the left somatosensory cortex, deficits in the precision of reporting color with bilateral fronto-parietal regions, and no anatomical substrates were identified for precision in reporting location. Our results converge with previous reports that working memory representations are widely distributed in the brain and can be found across sensory, parietal, temporal, and prefrontal cortices. Stroke patients demonstrate mostly subtle impairments in visual working memory, perhaps because representations from different areas in the brain can partly compensate for impaired encoding in lesioned areas. These findings contribute to understanding of the relation between memorizing features and their bound representations.

Decoding verbal working memory representations of Chinese characters from Broca's area

Representations of sensory working memory can be found across the entire neocortex. But how are verbal working memory (VWM) contents retained in the human brain? Here we used fMRI and multi-voxel pattern analyses to study Chinese native speakers (15 males, 13 females) memorizing Chinese characters. Chinese characters are uniquely suitable to study VWM because verbal encoding is encouraged by their complex visual appearance and monosyllabic pronunciation. We found that activity patterns in Broca's area and left premotor cortex carried information about the memorized characters. These language-related areas carried (1) significantly more information about cued characters than those not cued for memorization, (2) significantly more information on the left than the right hemisphere and (3) significantly more information about Chinese symbols than complex visual patterns which are hard to verbalize. In contrast, early visual cortex carries a comparable amount of information about cued and uncued stimuli and is thus unlikely to be involved in memory retention. This study provides evidence for verbal working memory maintenance in a distributed network of language-related brain regions, consistent with distributed accounts of WM. The results also suggest that Broca's area and left premotor cortex form the articulatory network which serves articulatory rehearsal in the retention of verbal working memory contents.

Cross-modal involvement of the primary somatosensory cortex in visual working memory: A repetitive TMS study

Recent literature suggests that the primary somatosensory cortex (S1), once thought to be a low-level area only modality-specific, is also involved in higher-level, cross-modal, cognitive functions. In particular, electrophysiological studies have highlighted that the cross-modal activation of this area may also extend to visual Working Memory (WM), being part of a mnemonic network specific for the temporary storage and manipulation of visual information concerning bodies and body-related actions. However, the causal recruitment of S1 in the WM network remains speculation. In the present study, by taking advantage of repetitive Transcranial Magnetic Stimulation (rTMS), we look for causal evidence that S1 is implicated in the retention of visual stimuli that are salient for this cortical area. To this purpose, in a first experiment, high-frequency (10 Hz) rTMS was delivered over S1 of the right hemisphere, and over two control sites, the right lateral occipital cortex (LOC) and the right dorsolateral prefrontal cortex (dlPFC), during the maintenance phase of a high-load delayed match-to-sample task in which body-related visual stimuli (non-symbolic hand gestures) have to be retained. In a second experiment, the specificity of S1 recruitment was deepened by using a version of the delayed match-to-sample task in which visual stimuli depict geometrical shapes (non-body related stimuli). Results show that rTMS perturbation of S1 activity leads to an enhancement of participants' performance that is selective for body-related visual stimuli; instead, the stimulation of the right LOC and dlPFC does not affect the temporary storage of body-related visual stimuli. These findings suggest that S1 may be recruited in visual WM when information to store (and recall) is salient for this area, corroborating models which suggest the existence of a dedicated mnemonic system for body-related information in which also somatosensory cortices play a key role, likely thanks to their cross-modal (visuo-tactile) properties.

Revisit once more the sensory storage account of visual working memory

Recent work has highlighted the role of early visual areas in visual working memory (VWM) storage and put forward a sensory storage account of VWM. Using a distractor interference paradigm, however, we previolsy showed that the contribution of early visual areas to VWM storage may not be essential. Instead, higher cortical regions such as the posterior parietal cortex may play a more significant role in VWM storage. This is consistent with reviews of other available behavioral, neuroimaging and neurophysiology results. Recently, a number of studies brought forward new evidence regarding this debate. Here I review these new pieces of evidence in detail and show that there is still no strong and definitive evidence supporting an essential role of the early visual areas in VWM storage. Instead, converging evidence suggests that early visual areas may contribute to the decision stage of a VWM task by facilitating target and probe comparison. Aside from further clarifying this debate, it is also important to note that whether or not VWM storage uses a sensory code depends on how it is defined, and that behavioral interactions between VWM and perception tasks do not necessarily support the involvement of sensory regions in VWM storage.

Decoding stimulus identity in occipital, parietal and inferotemporal cortices during visual mental imagery

In the absence of input from the external world, humans are still able to generate vivid mental images. This cognitive process, known as visual mental imagery, involves a network of prefrontal, parietal, inferotemporal, and occipital regions. Using multivariate pattern analysis (MVPA), previous studies were able to distinguish between the different orientations of imagined gratings, but not between more complex imagined stimuli, such as common objects, in early visual cortex (V1). Here we asked whether letters, simple shapes, and objects can be decoded in early visual areas during visual mental imagery. In a delayed spatial judgment task, we asked participants to observe or imagine stimuli. To examine whether it is possible to discriminate between neural patterns during perception and visual mental imagery, we performed ROI-based and whole-brain searchlight-based MVPA. We were able to decode imagined stimuli in early visual (V1, V2), parietal (SPL, IPL, aIPS), inferotemporal (LOC) and prefrontal (PMd) areas. In a subset of these areas (i.e., V1, V2, LOC, SPL, IPL and aIPS), we also obtained significant cross-decoding across visual imagery and perception. Moreover, we observed a linear relationship between behavioral accuracy and the amplitude of the BOLD signal in parietal and inferotemporal cortices, but not in early visual cortex, in line with the view that these areas contribute to the ability to perform visual imagery. Together, our results suggest that in the absence of bottom-up visual inputs, patterns of functional activation in early visual cortex allow distinguishing between different imagined stimulus exemplars, most likely mediated by signals from parietal and inferotemporal areas.

Parametric Representation of Tactile Numerosity in Working Memory

Estimated numerosity perception is processed in an approximate number system (ANS) that resembles the perception of a continuous magnitude. The ANS consists of a right lateralized frontoparietal network comprising the lateral prefrontal cortex (LPFC) and the intraparietal sulcus. Although the ANS has been extensively investigated, only a few studies have focused on the mental representation of retained numerosity estimates. Specifically, the underlying mechanisms of estimated numerosity working memory (WM) is unclear. Besides numerosities, as another form of abstract quantity, vibrotactile WM studies provide initial evidence that the right LPFC takes a central role in maintaining magnitudes. In the present fMRI multivariate pattern analysis study, we designed a delayed match-to-numerosity paradigm to test what brain regions retain approximate numerosity memoranda. In line with parametric WM results, our study found numerosity-specific WM representations in the right LPFC as well as in the supplementary motor area and the left premotor cortex extending into the superior frontal gyrus, thus bridging the gap in abstract quantity WM literature.

View-Independent Working Memory Representations of Artificial Shapes in Prefrontal and Posterior Regions of the Human Brain

Traditional views of visual working memory postulate that memorized contents are stored in dorsolateral prefrontal cortex using an adaptive and flexible code. In contrast, recent studies proposed that contents are maintained by posterior brain areas using codes akin to perceptual representations. An important question is whether this reflects a difference in the level of abstraction between posterior and prefrontal representations. Here, we investigated whether neural representations of visual working memory contents are view-independent, as indicated by rotation-invariance. Using functional magnetic resonance imaging and multivariate pattern analyses, we show that when subjects memorize complex shapes, both posterior and frontal brain regions maintain the memorized contents using a rotation-invariant code. Importantly, we found the representations in frontal cortex to be localized to the frontal eye fields rather than dorsolateral prefrontal cortices. Thus, our results give evidence for the view-independent storage of complex shapes in distributed representations across posterior and frontal brain regions.

Recall of facial expressions and simple orientations reveals competition for resources at multiple levels of the visual hierarchy

Many studies of visual working memory have tested humans' ability to reproduce primary visual features of simple objects, such as the orientation of a grating or the hue of a color patch, following a delay. A consistent finding of such studies is that precision of responses declines as the number of items in memory increases. Here we compared visual working memory for primary features and high-level objects. We presented participants with memory arrays consisting of oriented gratings, facial expressions, or a mixture of both. Precision of reproduction for all facial expressions declined steadily as the memory load was increased from one to five faces. For primary features, this decline and the specific distributions of error observed, have been parsimoniously explained in terms of neural population codes. We adapted the population coding model for circular variables to the non-circular and bounded parameter space used for expression estimation. Total population activity was held constant according to the principle of normalization and the intensity of expression was decoded by drawing samples from the Bayesian posterior distribution. The model fit the data well, showing that principles of population coding can be applied to model memory representations at multiple levels of the visual hierarchy. When both gratings and faces had to be remembered, an asymmetry was observed. Increasing the number of faces decreased precision of orientation recall, but increasing the number of gratings did not affect recall of expression, suggesting that memorizing faces involves the automatic encoding of low-level features, in addition to higher-level expression information.

The Posterior Parietal Cortex in Adaptive Visual Processing

Although the primate posterior parietal cortex (PPC) has been largely associated with space, attention, and action-related processing, a growing number of studies have reported the direct representation of a diverse array of action-independent nonspatial visual information in the PPC during both perception and visual working memory. By describing the distinctions and the close interactions of visual representation with space, attention, and action-related processing in the PPC, here I propose that we may understand these diverse PPC functions together through the unique contribution of the PPC to adaptive visual processing and form a more integrated and structured view of the role of the PPC in vision, cognition, and action.

Content-specific codes of parametric auditory working memory in humans

Brain activity in frontal regions has been found to represent frequency information with a parametric code during working memory delay phases. The mental representation of frequencies has furthermore been shown to be modality independent in non-human primate electrophysiology and human EEG studies, suggesting frontal regions encoding quantitative information in a supramodal manner. A recent fMRI study using multivariate pattern analysis (MVPA) supports an overlapping multimodal network for the maintenance of visual and tactile frequency information over frontal and parietal brain regions. The present study extends the investigation of working memory representation of frequency information to the auditory domain. To this aim, we used MVPA on fMRI data recorded during an auditory frequency maintenance task. A support vector regression analysis revealed working memory information in auditory association areas and, consistent with earlier findings of parametric working memory, in a frontoparietal network. A direct comparison to an analogous dataset of vibrotactile parametric working memory revealed an overlap of information coding in prefrontal regions, particularly in the right inferior frontal gyrus. Therefore, our findings indicate that the prefrontal cortex represents frequency-specific working memory content irrespective of the modality as has been now also revealed for the auditory modality.

A Tale of Two Visual Systems: Invariant and Adaptive Visual Information Representations in the Primate Brain

Visual information processing contains two opposite needs. There is both a need to comprehend the richness of the visual world and a need to extract only pertinent visual information to guide thoughts and behavior at a given moment. I argue that these two aspects of visual processing are mediated by two complementary visual systems in the primate brain-specifically, the occipitotemporal cortex (OTC) and the posterior parietal cortex (PPC). The role of OTC in visual processing has been documented extensively by decades of neuroscience research. I review here recent evidence from human imaging and monkey neurophysiology studies to highlight the role of PPC in adaptive visual processing. I first document the diverse array of visual representations found in PPC. I then describe the adaptive nature of visual representation in PPC by contrasting visual processing in OTC and PPC and by showing that visual representations in PPC largely originate from OTC.

Dorsolateral prefrontal cortex bridges bilateral primary somatosensory cortices during cross-modal working memory

Neural activity in the dorsolateral prefrontal cortex (DLPFC) has been suggested to integrate information from distinct sensory areas. However, how the DLPFC interacts with the bilateral primary somatosensory cortices (SIs) in tactile-visual cross-modal working memory has not yet been established. In the present study, we applied single-pulse transcranial magnetic stimulation (sp-TMS) over the contralateral DLPFC and bilateral SIs of human participants at various time points, while they performed a tactile-visual delayed matching-to-sample task with a 2-second delay. sp-TMS over the contralateral DLPFC or the contralateral SI at either an sensory encoding stage [i.e. 100 ms after the onset of a vibrotactile sample stimulus (200-ms duration)] or an early maintenance stage (i.e. 300 ms after the onset), significantly impaired the accuracy of task performance; sp-TMS over the contralateral DLPFC or the ipsilateral SI at a late maintenance stage (1600 ms and 1900 ms) also significantly disrupted the performance. Furthermore, at 300 ms after the onset of the vibrotactile sample stimulus, there was a significant correlation between the deteriorating effects of sp-TMS over the contralateral SI and the contralateral DLPFC. These results imply that the DLPFC and the bilateral SIs play causal roles at distinctive stages during cross-modal working memory, while the contralateral DLPFC communicates with the contralateral SI in the early delay, and cooperates with the ipsilateral SI in the late delay.

Selective attention on representations in working memory: cognitive and neural mechanisms

Selective attention and working memory are inter-dependent core cognitive functions. It is critical to allocate attention on selected targets during the capacity-limited working memory processes to fulfill the goal-directed behavior. The trends of research on both topics are increasing exponentially in recent years, and it is considered that selective attention and working memory share similar underlying neural mechanisms. Different types of attention orientation in working memory are introduced by distinctive cues, and the means using retrospective cues are strengthened currently as it is manipulating the representation in memory, instead of the perceptual representation. The cognitive and neural mechanisms of the retro-cue effects are further reviewed, as well as the potential molecular mechanism. The frontal-parietal network that is involved in both attention and working memory is also the neural candidate for attention orientation during working memory. Neural oscillations in the gamma and alpha/beta oscillations may respectively be employed for the feedforward and feedback information transfer between the sensory cortices and the association cortices. Dopamine and serotonin systems might interact with each other subserving the communication between memory and attention. In conclusion, representations which attention shifts towards are strengthened, while representations which attention moves away from are degraded. Studies on attention orientation during working memory indicates the flexibility of the processes of working memory, and the beneficial way that overcome the limited capacity of working memory.

Cortical specialization for attended versus unattended working memory

Items held in working memory can be either attended or not, depending on their current behavioral relevance. It has been suggested that unattended contents might be solely retained in an activity-silent form. Instead, we demonstrate here that encoding unattended contents involves a division of labor. While visual cortex only maintains attended items, intraparietal areas and the frontal eye fields represent both attended and unattended items.

Content-Specific Codes of Parametric Vibrotactile Working Memory in Humans

To understand how the brain handles mentally represented information flexibly in the absence of sensory stimulation, working memory (WM) studies have been essential. A seminal finding in monkey research is that neurons in the prefrontal cortex (PFC) retain stimulus-specific information when vibrotactile frequencies were memorized. A direct mapping between monkey studies and human research is still controversial. Although oscillatory signatures, in terms of frequency-dependent parametric beta-band modulation, have been observed recently in human EEG studies, the content specificity of these representations in terms of multivariate pattern analysis has not yet been shown. Here, we used fMRI in combination with multivariate classification techniques to determine which brain regions retain information during WM. In a retro-cue delayed-match-to-sample task, human subjects memorized the frequency of vibrotactile stimulation over a 12 s delay phase. Using an assumption-free whole-brain searchlight approach, we tested with support vector regression which brain regions exhibited multivariate parametric WM codes of the maintained frequencies during the WM delay. Interestingly, our analysis revealed an overlap with regions previously identified in monkeys composed of bilateral premotor cortices, supplementary motor area, and the right inferior frontal gyrus as part of the PFC. Therefore, our results establish a link between the WM codes found in monkeys and those in humans and emphasize the importance of the PFC for information maintenance during WM also in humans.SIGNIFICANCE STATEMENT Working memory (WM) research in monkeys has identified a network of regions, including prefrontal regions, to code stimulus-specific information when vibrotactile frequencies are memorized. Here, we performed an fMRI study during which human subjects had to memorize vibratory frequencies in parallel to previous monkey research. Using an assumption-free, whole-brain searchlight decoding approach, we identified for the first time regions in the human brain that exhibit multivariate patterns of activity to code the vibratory frequency parametrically during WM. Our results parallel previous monkey findings and show that the supplementary motor area, premotor, and the right prefrontal cortex are involved in vibrotactile WM coding in humans.

Reevaluating the Sensory Account of Visual Working Memory Storage

Recent human fMRI pattern-decoding studies have highlighted the involvement of sensory areas in visual working memory (VWM) tasks and argue for a sensory account of VWM storage. In this review, evidence is examined from human behavior, fMRI decoding, and transcranial magnetic stimulation (TMS) studies, as well as from monkey neurophysiology studies. Contrary to the prevalent view, the available evidence provides little support for the sensory account of VWM storage. Instead, when the ability to resist distraction and the existence of top-down feedback are taken into account, VWM-related activities in sensory areas seem to reflect feedback signals indicative of VWM storage elsewhere in the brain. Collectively, the evidence shows that prefrontal and parietal regions, rather than sensory areas, play more significant roles in VWM storage.

Reconstructing Perceived and Retrieved Faces from Activity Patterns in Lateral Parietal Cortex

Recent findings suggest that the contents of memory encoding and retrieval can be decoded from the angular gyrus (ANG), a subregion of posterior lateral parietal cortex. However, typical decoding approaches provide little insight into the nature of ANG content representations. Here, we tested whether complex, multidimensional stimuli (faces) could be reconstructed from ANG by predicting underlying face components from fMRI activity patterns in humans. Using an approach inspired by computer vision methods for face recognition, we applied principal component analysis to a large set of face images to generate eigenfaces. We then modeled relationships between eigenface values and patterns of fMRI activity. Activity patterns evoked by individual faces were then used to generate predicted eigenface values, which could be transformed into reconstructions of individual faces. We show that visually perceived faces were reliably reconstructed from activity patterns in occipitotemporal cortex and several lateral parietal subregions, including ANG. Subjective assessment of reconstructed faces revealed specific sources of information (e.g., affect and skin color) that were successfully reconstructed in ANG. Strikingly, we also found that a model trained on ANG activity patterns during face perception was able to successfully reconstruct an independent set of face images that were held in memory. Together, these findings provide compelling evidence that ANG forms complex, stimulus-specific representations that are reflected in activity patterns evoked during perception and remembering.

SIGNIFICANCE STATEMENT

Neuroimaging studies have consistently implicated lateral parietal cortex in episodic remembering, but the functional contributions of lateral parietal cortex to memory remain a topic of debate. Here, we used an innovative form of fMRI pattern analysis to test whether lateral parietal cortex actively represents the contents of memory. Using a large set of human face images, we first extracted latent face components (eigenfaces). We then used machine learning algorithms to predict face components from fMRI activity patterns and, ultimately, to reconstruct images of individual faces. We show that activity patterns in a subregion of lateral parietal cortex, the angular gyrus, supported successful reconstruction of perceived and remembered faces, confirming a role for this region in actively representing remembered content.

Alterations in visual cortical activation and connectivity with prefrontal cortex during working memory updating in major depressive disorder

The present study examined the impacts of major depressive disorder (MDD) on visual and prefrontal cortical activity as well as their connectivity during visual working memory updating and related them to the core clinical features of the disorder. Impairment in working memory updating is typically associated with the retention of irrelevant negative information which can lead to persistent depressive mood and abnormal affect. However, performance deficits have been observed in MDD on tasks involving little or no demand on emotion processing, suggesting dysfunctions may also occur at the more basic level of information processing. Yet, it is unclear how various regions in the visual working memory circuit contribute to behavioral changes in MDD. We acquired functional magnetic resonance imaging data from 18 unmedicated participants with MDD and 21 age-matched healthy controls (CTL) while they performed a visual delayed recognition task with neutral faces and scenes as task stimuli. Selective working memory updating was manipulated by inserting a cue in the delay period to indicate which one or both of the two memorized stimuli (a face and a scene) would remain relevant for the recognition test. Our results revealed several key findings. Relative to the CTL group, the MDD group showed weaker postcue activations in visual association areas during selective maintenance of face and scene working memory. Across the MDD subjects, greater rumination and depressive symptoms were associated with more persistent activation and connectivity related to no-longer-relevant task information. Classification of postcue spatial activation patterns of the scene-related areas was also less consistent in the MDD subjects compared to the healthy controls. Such abnormalities appeared to result from a lack of updating effects in postcue functional connectivity between prefrontal and scene-related areas in the MDD group. In sum, disrupted working memory updating in MDD was revealed by alterations in activity patterns of the visual association areas, their connectivity with the prefrontal cortex, and their relationship with core clinical characteristics. These results highlight the role of information updating deficits in the cognitive control and symptomatology of depression.

•

Unmedicated individuals with major depressive disorder showed several forms of deficits during visual working memory updating.

•

Impaired visual working memory updating performance.

•

Diminished category-specific response patterns in visual association areas, particularly those involved in scene processing.

•

Loss of frontal functional connectivity with the scene-selective visual region during updating of scene information.

•

Rumination is associated with heightened activity and functional connectivity for obsolete information in working memory.

The Distributed Nature of Working Memory

Studies in humans and non-human primates have provided evidence for storage of working memory contents in multiple regions ranging from sensory to parietal and prefrontal cortex. We discuss potential explanations for these distributed representations: (i) features in sensory regions versus prefrontal cortex differ in the level of abstractness and generalizability; and (ii) features in prefrontal cortex reflect representations that are transformed for guidance of upcoming behavioral actions. We propose that the propensity to produce persistent activity is a general feature of cortical networks. Future studies may have to shift focus from asking where working memory can be observed in the brain to how a range of specialized brain areas together transform sensory information into a delayed behavioral response.

Neural mechanisms of information storage in visual short-term memory

The capacity to briefly memorize fleeting sensory information supports visual search and behavioral interactions with relevant stimuli in the environment. Traditionally, studies investigating the neural basis of visual short term memory (STM) have focused on the role of prefrontal cortex (PFC) in exerting executive control over what information is stored and how it is adaptively used to guide behavior. However, the neural substrates that support the actual storage of content-specific information in STM are more controversial, with some attributing this function to PFC and others to the specialized areas of early visual cortex that initially encode incoming sensory stimuli. In contrast to these traditional views, I will review evidence suggesting that content-specific information can be flexibly maintained in areas across the cortical hierarchy ranging from early visual cortex to PFC. While the factors that determine exactly where content-specific information is represented are not yet entirely clear, recognizing the importance of task-demands and better understanding the operation of non-spiking neural codes may help to constrain new theories about how memories are maintained at different resolutions, across different timescales, and in the presence of distracting information.

Comparing the Effects of 10-Hz Repetitive TMS on Tasks of Visual STM and Attention

Numerous studies have demonstrated that visual STM (VSTM) and attention are tightly linked processes that share a number of neuroanatomical substrates. Here, we used repetitive TMS (rTMS) along with simultaneous EEG to examine the causal relationship between intraparietal sulcus functioning and performance on tasks of attention and VSTM. Participants performed two tasks in which they were required to attend to or remember colored items over a brief interval, with 10-Hz rTMS applied on some of the trials. Although no overall behavioral changes were observed across either task, rTMS did affect individual performance on both the attention and VSTM tasks in a manner that was predicted by individual differences in baseline performance. Furthermore, rTMS also affected ongoing oscillations in the alpha and beta bands, and these changes were related to the observed change in behavioral performance. The results reveal a causal relationship between intraparietal sulcus activity and tasks measuring both visual attention and VSTM.

The Brain Is Faster than the Hand in Split-Second Intentions to Respond to an Impending Hazard: A Simulation of Neuroadaptive Automation to Speed Recovery to Perturbation in Flight Attitude

The goal of this research is to test the potential for neuroadaptive automation to improve response speed to a hazardous event by using a brain-computer interface (BCI) to decode perceptual-motor intention. Seven participants underwent four experimental sessions while measuring brain activity with magnetoencephalograpy. The first three sessions were of a simple constrained task in which the participant was to pull back on the control stick to recover from a perturbation in attitude in one condition and to passively observe the perturbation in the other condition. The fourth session consisted of having to recover from a perturbation in attitude while piloting the plane through the Grand Canyon constantly maneuvering to track over the river below. Independent component analysis was used on the first two sessions to extract artifacts and find an event related component associated with the onset of the perturbation. These two sessions were used to train a decoder to classify trials in which the participant recovered from the perturbation (motor intention) vs. just passively viewing the perturbation. The BCI-decoder was tested on the third session of the same simple task and found to be able to significantly distinguish motor intention trials from passive viewing trials (mean = 69.8%). The same BCI-decoder was then used to test the fourth session on the complex task. The BCI-decoder significantly classified perturbation from no perturbation trials (73.3%) with a significant time savings of 72.3 ms (Original response time of 425.0-352.7 ms for BCI-decoder). The BCI-decoder model of the best subject was shown to generalize for both performance and time savings to the other subjects. The results of our off-line open loop simulation demonstrate that BCI based neuroadaptive automation has the potential to decode motor intention faster than manual control in response to a hazardous perturbation in flight attitude while ignoring ongoing motor and visual induced activity related to piloting the airplane.

Simultaneous EEG-fMRI for working memory of the human brain

Memory plays an important role in human life. Memory can be divided into two categories, i.e., long term memory and short term memory (STM). STM or working memory (WM) stores information for a short span of time and it is used for information manipulations and fast response activities. WM is generally involved in the higher cognitive functions of the brain. Different studies have been carried out by researchers to understand the WM process. Most of these studies were based on neuroimaging modalities like fMRI, EEG, MEG etc., which use standalone processes. Each neuroimaging modality has some pros and cons. For example, EEG gives high temporal resolution but poor spatial resolution. On the other hand, the fMRI results have a high spatial resolution but poor temporal resolution. For a more in depth understanding and insight of what is happening inside the human brain during the WM process or during cognitive tasks, high spatial as well as high temporal resolution is desirable. Over the past decade, researchers have been working to combine different modalities to achieve a high spatial and temporal resolution at the same time. Developments of MRI compatible EEG equipment in recent times have enabled researchers to combine EEG-fMRI successfully. The research publications in simultaneous EEG-fMRI have been increasing tremendously. This review is focused on the WM research involving simultaneous EEG-fMRI data acquisition and analysis. We have covered the simultaneous EEG-fMRI application in WM and data processing. Also, it adds to potential fusion methods which can be used for simultaneous EEG-fMRI for WM and cognitive tasks.

Multi-Voxel Decoding and the Topography of Maintained Information During Visual Working Memory

The ability to maintain representations in the absence of external sensory stimulation, such as in working memory, is critical for guiding human behavior. Human functional brain imaging studies suggest that visual working memory can recruit a network of brain regions from visual to parietal to prefrontal cortex. In this review, we focus on the maintenance of representations during visual working memory and discuss factors determining the topography of those representations. In particular, we review recent studies employing multi-voxel pattern analysis (MVPA) that demonstrate decoding of the maintained content in visual cortex, providing support for a “sensory recruitment” model of visual working memory. However, there is some evidence that maintained content can also be decoded in areas outside of visual cortex, including parietal and frontal cortex. We suggest that the ability to maintain representations during working memory is a general property of cortex, not restricted to specific areas, and argue that it is important to consider the nature of the information that must be maintained. Such information-content is critically determined by the task and the recruitment of specific regions during visual working memory will be both task- and stimulus-dependent. Thus, the common finding of maintained information in visual, but not parietal or prefrontal, cortex may be more of a reflection of the need to maintain specific types of visual information and not of a privileged role of visual cortex in maintenance.

Prefrontal cortex and sensory cortices during working memory: quantity and quality

The activity in sensory cortices and the prefrontal cortex (PFC) throughout the delay interval of working memory (WM) tasks reflect two aspects of WM-quality and quantity, respectively. The delay activity in sensory cortices is fine-tuned to sensory information and forms the neural basis of the precision of WM storage, while the delay activity in the PFC appears to represent behavioral goals and filters out irrelevant distractions, forming the neural basis of the quantity of task-relevant information in WM. The PFC and sensory cortices interact through different frequency bands of neuronal oscillation (theta, alpha, and gamma) to fulfill goal-directed behaviors.

The cognitive neuroscience of visual short-term memory

Our understanding of the neural bases of visual short-term memory (STM), the ability to mentally retain information over short periods of time, is being reshaped by two important developments: the application of methods from statistical machine learning, often a variant of multivariate pattern analysis (MVPA), to functional magnetic resonance imaging (fMRI) and electroencephalographic (EEG) data sets; and advances in our understanding of the physiology and functions of neuronal oscillations. One consequence is that many commonly observed physiological "signatures" that have previously been interpreted as directly related to the retention of information in visual STM may require reinterpretation as more general, state-related changes that can accompany cognitive-task performance. Another is important refinements of theoretical models of visual STM.

The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data

The multivariate analysis of brain signals has recently sparked a great amount of interest, yet accessible and versatile tools to carry out decoding analyses are scarce. Here we introduce The Decoding Toolbox (TDT) which represents a user-friendly, powerful and flexible package for multivariate analysis of functional brain imaging data. TDT is written in Matlab and equipped with an interface to the widely used brain data analysis package SPM. The toolbox allows running fast whole-brain analyses, region-of-interest analyses and searchlight analyses, using machine learning classifiers, pattern correlation analysis, or representational similarity analysis. It offers automatic creation and visualization of diverse cross-validation schemes, feature scaling, nested parameter selection, a variety of feature selection methods, multiclass capabilities, and pattern reconstruction from classifier weights. While basic users can implement a generic analysis in one line of code, advanced users can extend the toolbox to their needs or exploit the structure to combine it with external high-performance classification toolboxes. The toolbox comes with an example data set which can be used to try out the various analysis methods. Taken together, TDT offers a promising option for researchers who want to employ multivariate analyses of brain activity patterns.