The Neural Codes Underlying Internally Generated Representations in Visual Working Memory

Spatiotemporal dynamics of self-generated imagery reveal a reverse cortical hierarchy from cue-induced imagery

Visual imagery allows for the construction of rich internal experience in our mental world. However, it has remained poorly understood how imagery experience derives volitionally as opposed to being cue driven. Here, using electroencephalography and functional magnetic resonance imaging, we systematically investigate the spatiotemporal dynamics of self-generated imagery by having participants volitionally imagining one of the orientations from a learned pool. We contrast self-generated imagery with cue-induced imagery, where participants imagined line orientations based on associative cues acquired previously. Our results reveal overlapping neural signatures of cue-induced and self-generated imagery. Yet, these neural signatures display substantially differential sensitivities to the two types of imagery: self-generated imagery is supported by an enhanced involvement of the anterior cortex in representing imagery contents. By contrast, cue-induced imagery is supported by enhanced imagery representations in the posterior visual cortex. These results jointly support a reverse cortical hierarchy in generating and maintaining imagery contents in self-generated versus externally cued imagery.

Neural retrieval processes occur more rapidly for visual mental images that were previously encoded with high-vividness

Visual mental imagery refers to our ability to experience visual images in the absence of sensory stimulation. Studies have shown that visual mental imagery can improve episodic memory. However, we have limited understanding of the neural mechanisms underlying this improvement. Using electroencephalography, we examined the neural processes associated with the retrieval of previously generated visual mental images, focusing on how the vividness at generation can modulate retrieval processes. Participants viewed word stimuli referring to common objects, forming a visual mental image of each word and rating the vividness of the mental image. This was followed by a surprise old/new recognition task. We compared retrieval performance for items rated as high- versus low-vividness at encoding. High-vividness items were retrieved with faster reaction times and higher confidence ratings in the memory judgment. While controlling for confidence, neural measures indicated that high-vividness items produced an earlier decrease in alpha-band activity at retrieval compared with low-vividness items, suggesting an earlier memory reinstatement. Even when low-vividness items were remembered with high confidence, they were not retrieved as quickly as high-vividness items. These results indicate that when highly vivid mental images are encoded, the speed of their retrieval occurs more rapidly, relative to low-vivid items.

Neural Representations in Visual and Parietal Cortex Differentiate between Imagined, Perceived, and Illusory Experiences

Humans constantly receive massive amounts of information, both perceived from the external environment and imagined from the internal world. To function properly, the brain needs to correctly identify the origin of information being processed. Recent work has suggested common neural substrates for perception and imagery. However, it has remained unclear how the brain differentiates between external and internal experiences with shared neural codes. Here we tested this question in human participants (male and female) by systematically investigating the neural processes underlying the generation and maintenance of visual information from voluntary imagery, veridical perception, and illusion. The inclusion of illusion allowed us to differentiate between objective and subjective internality: while illusion has an objectively internal origin and can be viewed as involuntary imagery, it is also subjectively perceived as having an external origin like perception. Combining fMRI, eye-tracking, multivariate decoding, and encoding approaches, we observed superior orientation representations in parietal cortex during imagery compared with perception, and conversely in early visual cortex. This imagery dominance gradually developed along a posterior-to-anterior cortical hierarchy from early visual to parietal cortex, emerged in the early epoch of imagery and sustained into the delay epoch, and persisted across varied imagined contents. Moreover, representational strength of illusion was more comparable to imagery in early visual cortex, but more comparable to perception in parietal cortex, suggesting content-specific representations in parietal cortex differentiate between subjectively internal and external experiences, as opposed to early visual cortex. These findings together support a domain-general engagement of parietal cortex in internally generated experience.SIGNIFICANCE STATEMENT How does the brain differentiate between imagined and perceived experiences? Combining fMRI, eye-tracking, multivariate decoding, and encoding approaches, the current study revealed enhanced stimulus-specific representations in visual imagery originating from parietal cortex, supporting the subjective experience of imagery. This neural principle was further validated by evidence from visual illusion, wherein illusion resembled perception and imagery at different levels of cortical hierarchy. Our findings provide direct evidence for the critical role of parietal cortex as a domain-general region for content-specific imagery, and offer new insights into the neural mechanisms underlying the differentiation between subjectively internal and external experiences.

jsQuestPlus: A JavaScript implementation of the QUEST+ method for estimating psychometric function parameters in online experiments

The two Bayesian adaptive psychometric methods named QUEST (Watson & Pelli, 1983) and QUEST+ (Watson, 2017) are widely used to estimate psychometric parameters, especially the threshold, in laboratory-based psychophysical experiments. Considering the increase of online psychophysical experiments in recent years, there is a growing need to have the QUEST and QUEST+ methods available online as well. We developed JavaScript libraries for both, with this article introducing one of them: jsQuestPlus. We offer integrations with online experimental tools such as jsPsych (de Leeuw, 2015), PsychoPy/JS (Peirce et al., 2019), and lab.js (Henninger et al., 2021). We measured the computation time required by jsQuestPlus under four conditions. Our simulations on 37 browser-computer combinations showed that the mean initialization time was 461.08 ms, 95% CI [328.29, 593.87], the mean computation time required to determine the stimulus parameters for the next trial was less than 1 ms, and the mean update time was 79.39 ms, 95% CI [46.22, 112.55] even in extremely demanding conditions. Additionally, psychometric parameters were estimated as accurately as the original QUEST+ method did. We conclude that jsQuestPlus is fast and accurate enough to conduct online psychophysical experiments despite the complexity of the matrix calculations. The latest version of jsQuestPlus can be downloaded freely from https://github.com/kurokida/jsQuestPlus under the MIT license.

Preparatory attention to visual features primarily relies on non-sensory representation

Prior knowledge of behaviorally relevant information promotes preparatory attention before the appearance of stimuli. A key question is how our brain represents the attended information during preparation. A sensory template hypothesis assumes that preparatory signals evoke neural activity patterns that resembled the perception of the attended stimuli, whereas a non-sensory, abstract template hypothesis assumes that preparatory signals reflect the abstraction of attended stimuli. To test these hypotheses, we used fMRI and multivariate analysis to characterize neural activity patterns when human participants were prepared to attend a feature and then select it from a compound stimulus. In an fMRI experiment using basic visual feature (motion direction), we observed reliable decoding of the to-be-attended feature from the preparatory activity in both visual and frontoparietal areas. However, while the neural patterns constructed by a single feature from a baseline task generalized to the activity patterns during stimulus selection, they could not generalize to the activity patterns during preparation. Our findings thus suggest that neural signals during attentional preparation are predominantly non-sensory in nature that may reflect an abstraction of the attended feature. Such a representation could provide efficient and stable guidance of attention.

Sensory Recruitment Revisited: Ipsilateral V1 Involved in Visual Working Memory

The "sensory recruitment hypothesis" posits an essential role of sensory cortices in working memory, beyond the well-accepted frontoparietal areas. Yet, this hypothesis has recently been challenged. In the present study, participants performed a delayed orientation recall task while high-spatial-resolution 3 T functional magnetic resonance imaging (fMRI) signals were measured in posterior cortices. A multivariate inverted encoding model approach was used to decode remembered orientations based on blood oxygen level-dependent fMRI signals from visual cortices during the delay period. We found that not only did activity in the contralateral primary visual cortex (V1) retain high-fidelity representations of the visual stimuli, but activity in the ipsilateral V1 also contained such orientation tuning. Moreover, although the encoded tuning was faded in the contralateral V1 during the late delay period, tuning information in the ipsilateral V1 remained sustained. Furthermore, the ipsilateral representation was presented in secondary visual cortex (V2) as well, but not in other higher-level visual areas. These results thus supported the sensory recruitment hypothesis and extended it to the ipsilateral sensory areas, which indicated the distributed involvement of visual areas in visual working memory.

Working memory representations in visual cortex mediate distraction effects

Although the contents of working memory can be decoded from visual cortex activity, these representations may play a limited role if they are not robust to distraction. We used model-based fMRI to estimate the impact of distracting visual tasks on working memory representations in several visual field maps in visual and frontoparietal association cortex. Here, we show distraction causes the fidelity of working memory representations to briefly dip when both the memorandum and distractor are jointly encoded by the population activities. Distraction induces small biases in memory errors which can be predicted by biases in neural decoding in early visual cortex, but not other regions. Although distraction briefly disrupts working memory representations, the widespread redundancy with which working memory information is encoded may protect against catastrophic loss. In early visual cortex, the neural representation of information in working memory and behavioral performance are intertwined, solidifying its importance in visual memory.

The relative roles of visual, parietal, and frontal cortex in working memory have been actively debated. Here, the authors show that distraction impacts visual working memory representations in primary visual areas, indicating that these regions play a key role in the maintenance of working memory.