Perceptual Learning beyond Perception: Mnemonic Representation in Early Visual Cortex and Intraparietal Sulcus

The causal involvement of the visual cortex in visual working memory remains uncertain

The role of the early visual cortex in visual working memory (VWM) is a matter of current debate. Neuroimaging studies have consistently shown that visual areas encode the content of working memory, while transcranial magnetic stimulation (TMS) studies have presented incongruent results. Thus, we lack conclusive evidence supporting the causal role of early visual areas in VWM. In a recent registered report, Phylactou et al. (Phylactou P, Shimi A, Konstantinou N 2023 R. Soc. Open Sci. 10, 230321 (doi:10.1098/rsos.230321)) sought to tackle this controversy via two well-powered TMS experiments, designed to correct possible methodological issues of previous attempts identified in a preceding systematic review and meta-analysis (Phylactou P, Traikapi A, Papadatou-Pastou M, Konstantinou N 2022 Psychon. Bull. Rev. 29, 1594-1624 (doi:10.3758/s13423-022-02107-y)). However, a key part of their critique and experimental design was based on a misunderstanding of the visual system. They disregarded two important anatomical facts, namely that early visual areas of each hemisphere represent the contralateral visual hemifield, and that each hemisphere receives equally strong input from each eye-both leading to confounded conditions and artefactual effects in their studies. Here, we explain the correct anatomy, describe why their experiments failed to address current issues in the literature and perform a thorough reanalysis of their TMS data revealing important null results. We conclude that the causal role of the visual cortex in VWM remains uncertain.

Divergent roles of early visual cortex and inferior frontal junction in visual working memory

BACKGROUND

Recent studies indicate that both prefrontal and visual regions play critical roles in visual working memory (VWM), with prefrontal regions mainly associated with executive functions, and visual cortices linked to representations of memory contents. VWM involves the selective filtering of irrelevant information, yet the specific contributions of the prefrontal regions and visual cortex in this process remain unclear.

OBJECTIVE

To understand the dynamic causal roles of prefrontal and visual regions in VWM.

METHODS

The differentiation of VWM components was achieved using a computational model that incorporated a swap rate for non-target stimuli. Single-pulse magnetic transcranial stimulation (spTMS) was delivered to the early visual cortex (EVC) and the inferior frontal junction (IFJ) across different phases of an orientation recall task that with or without distractors.

RESULTS

Our results indicate that spTMS over the EVC and IFJ influences VWM particularly when distractors are present. VWM precision can be impacted by spTMS applied to either region during the early retention, while spTMS effect is especially prominent when EVC is stimulated during the late retention phase and when directed at the ipsilateral EVC. Conversely, the probability of accurately recalling the target exhibited comparable patterns when spTMS was administered to either the EVC or IFJ.

CONCLUSIONS

We highlight the "sensory recruitment" of VWM characterized by critical involvement of EVC particularly in the information-filtering process within VWM. The maintenance of memory content representations necessitates ongoing communication between the EVC and IFJ throughout the entirety of the VWM process in a dynamic pattern.

Top-down modulation of DLPFC in visual search: a study based on fMRI and TMS

Effective visual search is essential for daily life, and attention orientation as well as inhibition of return play a significant role in visual search. Researches have established the involvement of dorsolateral prefrontal cortex in cognitive control during selective attention. However, neural evidence regarding dorsolateral prefrontal cortex modulates inhibition of return in visual search is still insufficient. In this study, we employed event-related functional magnetic resonance imaging and dynamic causal modeling to develop modulation models for two types of visual search tasks. In the region of interest analyses, we found that the right dorsolateral prefrontal cortex and temporoparietal junction were selectively activated in the main effect of search type. Dynamic causal modeling results indicated that temporoparietal junction received sensory inputs and only dorsolateral prefrontal cortex →temporoparietal junction connection was modulated in serial search. Such neural modulation presents a significant positive correlation with behavioral reaction time. Furthermore, theta burst stimulation via transcranial magnetic stimulation was utilized to modulate the dorsolateral prefrontal cortex region, resulting in the disappearance of the inhibition of return effect during serial search after receiving continuous theta burst stimulation. Our findings provide a new line of causal evidence that the top-down modulation by dorsolateral prefrontal cortex influences the inhibition of return effect during serial search possibly through the retention of inhibitory tagging via working memory storage.

Electrophysiological hallmarks for event relations and event roles in working memory

The ability to maintain events (i.e., interactions between/among objects) in working memory is crucial for our everyday cognition, yet the format of this representation is poorly understood. The current ERP study was designed to answer two questions: How is maintaining events (e.g., the tiger hit the lion) neurally different from maintaining item coordinations (e.g., the tiger and the lion)? That is, how is the event relation (present in events but not coordinations) represented? And how is the agent, or initiator of the event encoded differently from the patient, or receiver of the event during maintenance? We used a novel picture-sentence match-across-delay approach in which the working memory representation was "pinged" during the delay, replicated across two ERP experiments with Chinese and English materials. We found that maintenance of events elicited a long-lasting late sustained difference in posterior-occipital electrodes relative to non-events. This effect resembled the negative slow wave reported in previous studies of working memory, suggesting that the maintenance of events in working memory may impose a higher cost compared to coordinations. Although we did not observe significant ERP differences associated with pinging the agent vs. the patient during the delay, we did find that the ping appeared to dampen the ongoing sustained difference, suggesting a shift from sustained activity to activity silent mechanisms. These results suggest a new method by which ERPs can be used to elucidate the format of neural representation for events in working memory.

Ultra-High Field Imaging of Human Visual Cognition

Functional magnetic resonance imaging (fMRI), the key methodology for mapping the functions of the human brain in a noninvasive manner, is limited by low temporal and spatial resolution. Recent advances in ultra-high field (UHF) fMRI provide a mesoscopic (i.e., submillimeter resolution) tool that allows us to probe laminar and columnar circuits, distinguish bottom-up versus top-down pathways, and map small subcortical areas. We review recent work demonstrating that UHF fMRI provides a robust methodology for imaging the brain across cortical depths and columns that provides insights into the brain's organization and functions at unprecedented spatial resolution, advancing our understanding of the fine-scale computations and interareal communication that support visual cognition.

Reactivation-induced memory integration prevents proactive interference in perceptual learning

We acquire perceptual skills through experience to adapt ourselves to the changing environment. Accomplishing an effective skill acquisition is a main purpose of perceptual learning research. Given the often observed learning effect specificity, multiple perceptual learnings with shared parameters could serve to improve the generalization of the learning effect. However, the interference between the overlapping memory traces of different learnings may impede this effort. Here, we trained human participants on an orientation discrimination task. We observed a proactive interference effect that the first training blocked the second training at its untrained location. This was a more pronounced effect than the well-known location specificity in perceptual learning. We introduced a short reactivation of the first training before the second training and successfully eliminated the proactive interference when the second training was inside the reconsolidation time window of the reactivated first training. Interestingly, we found that practicing an irrelevant task at the location of the second training immediately after the reactivation of the first training could also restore the effect of the second training but in a smaller magnitude, even if the second training was conducted outside of the reconsolidation window. We proposed a two-level mechanism of reactivation-induced memory integration to account for these results. The reactivation-based procedure could integrate either the previously trained and untrained locations or the two trainings at these locations, depending on the activated representations during the reconsolidation process. The findings provide us with new insight into the roles of long-term memory mechanisms in perceptual learning.

Causal evidence for the role of the sensory visual cortex in visual short-term memory maintenance

The role of the sensory visual cortex during visual short-term memory (VSTM) remains controversial. This controversy is possibly due to methodological issues in previous attempts to investigate the effects of transcranial magnetic stimulation (TMS) on VSTM. The aim of this study was to use TMS, while covering previous methodological deficits. Sixty-four young adults were recruited to participate in two experiments (Experiment 1: n = 36; Experiment 2: n = 28) using a VSTM orientation change-detection task under TMS. Monocular vision was ensured using red-blue goggles combined with red-blue stimuli. Double-pulse TMS was delivered at different times (Experiment 1: 0, 200 or 1000 ms; Experiment 2: 200, 1000 ms) during a 2 s maintenance phase, on one side of the occipital hemisphere. In Experiment 2, a sham TMS condition was introduced. Decreased detection sensitivity (d') in the ipsilateral occipital hemisphere to visual hemifield, and in the real TMS (compared with sham TMS) condition indicated inhibitory TMS effects, and thus, a causal involvement of the sensory visual cortex during early (200 ms) and late (1000 ms) maintenance in VSTM. These findings are aligned with sensory recruitment, which proposes that both perceptual and memory processes rely upon the same neural substrates in the sensory visual cortex. The methods used in this study were preregistered and had received in-principle acceptance on 6 June 2022 (Stage 1 protocol can be found in: https://doi.org/10.17605/OSF.IO/EMPDT).

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Simple contextual cueing prevents retroactive interference in short-term perceptual training of orientation detection tasks

Perceptual training of multiple tasks suffers from interference between the trained tasks. Here, we conducted five psychophysical experiments with separate groups of participants to investigate the possibility of preventing the interference in short-term perceptual training. We trained the participants to detect two orientations of Gabor stimuli in two adjacent days at the same retinal location and examined the interference of training effects between the two orientations. The results showed significant retroactive interference from the second orientation to the first orientation (Experiment 1 and Experiment 2). Introducing a 6-h interval between the pre-test and training of the second orientation did not eliminate the interference effect, excluding the interpretation of disrupted reconsolidation as the pre-test of the second orientation may reactivate and destabilize the representation of the first orientation (Experiment 3). Finally, the training of the two orientations was accompanied by fixations in two colors, each serving as a contextual cue for one orientation. The results showed that the retroactive interference was not evident if the participants passively perceived contextual cues during the training and test sessions (Experiment 4). Importantly, this facilitation effect could be observed if the contextual cues appeared only during the training, demonstrating the robustness of the effect (Experiment 5). Our findings suggest that the retroactive interference effect in short-term perceptual training of orientation detection tasks was likely the result of higher-level factors such as shared contextual cues embedded in the tasks. The efficiency of multiple perceptual trainings could be facilitated by associating the trained tasks with different contextual cues.

Sensory recruitment in visual short-term memory: A systematic review and meta-analysis of sensory visual cortex interference using transcranial magnetic stimulation

Sensory visual areas are involved in encoding information in visual short-term memory (VSTM). Yet it remains unclear whether sensory visual cortex is a necessary component of the brain network for maintenance of information in VSTM. Here, we aimed to systematically review studies that have investigated the role of the sensory visual cortex in VSTM using transcranial magnetic stimulation (TMS) and to quantitatively explore these effects using meta-analyses. Fourteen studies were identified and reviewed. Eight studies provided sufficient data for meta-analysis. Two meta-analyses, one regarding the VSTM encoding phase (17 effect sizes) and one regarding the VSTM maintenance phase (15 effect sizes), two meta-regressions (32 effect sizes in each), and one exploratory meta-analysis were conducted. Our results indicate that the sensory visual cortex is similarly involved in both the encoding and maintenance VSTM phase. We suggest that some cases where evidence did not show significant TMS effects was due to low memory or perceptual task demands. Overall, these findings support the idea that sensory visual areas are part of the brain network responsible for successfully maintaining information in VSTM.

Sensory Recruitment Revisited: Ipsilateral V1 Involved in Visual Working Memory

The "sensory recruitment hypothesis" posits an essential role of sensory cortices in working memory, beyond the well-accepted frontoparietal areas. Yet, this hypothesis has recently been challenged. In the present study, participants performed a delayed orientation recall task while high-spatial-resolution 3 T functional magnetic resonance imaging (fMRI) signals were measured in posterior cortices. A multivariate inverted encoding model approach was used to decode remembered orientations based on blood oxygen level-dependent fMRI signals from visual cortices during the delay period. We found that not only did activity in the contralateral primary visual cortex (V1) retain high-fidelity representations of the visual stimuli, but activity in the ipsilateral V1 also contained such orientation tuning. Moreover, although the encoded tuning was faded in the contralateral V1 during the late delay period, tuning information in the ipsilateral V1 remained sustained. Furthermore, the ipsilateral representation was presented in secondary visual cortex (V2) as well, but not in other higher-level visual areas. These results thus supported the sensory recruitment hypothesis and extended it to the ipsilateral sensory areas, which indicated the distributed involvement of visual areas in visual working memory.