Why do imagery and perception look and feel so different?

How workload and availability of spatial reference shape eye movement coupling in visuospatial working memory

Eyes are active in memory recall and visual imagination, yet our grasp of the underlying qualities and factors of these internally coupled eye movements is limited. To explore this, we studied 50 participants, examining how workload, spatial reference availability, and imagined movement direction influence internal coupling of eye movements. We designed a visuospatial working memory task in which participants mentally moved a black patch along a path within a matrix and each trial involved one step along this path (presented via speakers: up, down, left, or right). We varied workload by adjusting matrix size (3 × 3 vs. 5 × 5), manipulated availability of a spatial frame of reference by presenting either a blank screen (requiring participants to rely solely on their mental representation of the matrix) or spatial reference in the form of an empty matrix, and contrasted active task performance to two control conditions involving only active or passive listening. Our findings show that eye movements consistently matched the imagined movement of the patch in the matrix, not driven solely by auditory or semantic cues. While workload influenced pupil diameter, perceived demand, and performance, it had no observable impact on internal coupling. The availability of spatial reference enhanced coupling of eye movements, leading more frequent, precise, and resilient saccades against noise and bias. The absence of workload effects on coupled saccades in our study, in combination with the relatively high degree of coupling observed even in the invisible matrix condition, indicates that eye movements align with shifts in attention across both visually and internally represented information. This suggests that coupled eye movements are not merely strategic efforts to reduce workload, but rather a natural response to where attention is directed.

Laminar dissociation of feedforward and feedback in high-level ventral visual cortex during imagery and perception

Visual imagery and perception share neural machinery but rely on different information flow. While perception is driven by the integration of sensory feedforward and internally generated feedback information, imagery relies on feedback only. This suggests that although imagery and perception may activate overlapping brain regions, they do so in informationally distinctive ways. Using lamina-resolved MRI at 7 T, we measured the neural activity during imagery and perception of faces and scenes in high-level ventral visual cortex at the mesoscale of laminar organization that distinguishes feedforward from feedback signals. We found distinctive laminar profiles for imagery and perception of scenes and faces in the parahippocampal place area and the fusiform face area, respectively. Our findings provide insight into the neural basis of the phenomenology of visual imagery versus perception and shed new light into the mesoscale organization of feedforward and feedback information flow in high-level ventral visual cortex.

Consciousness and sleep

Sleep is a universal, essential biological process. It is also an invaluable window on consciousness. It tells us that consciousness can be lost but also that it can be regained, in all its richness, when we are disconnected from the environment and unable to reflect. By considering the neurophysiological differences between dreaming and dreamless sleep, we can learn about the substrate of consciousness and understand why it vanishes. We also learn that the ongoing state of the substrate of consciousness determines the way each experience feels regardless of how it is triggered-endogenously or exogenously. Dreaming consciousness is also a window on sleep and its functions. Dreams tell us that the sleeping brain is remarkably lively, recombining intrinsic activation patterns from a vast repertoire, freed from the requirements of ongoing behavior and cognitive control.

Uncovering the Role of the Early Visual Cortex in Visual Mental Imagery

The question of whether the early visual cortex (EVC) is involved in visual mental imagery remains a topic of debate. In this paper, I propose that the inconsistency in findings can be explained by the unique challenges associated with investigating EVC activity during imagery. During perception, the EVC processes low-level features, which means that activity is highly sensitive to variation in visual details. If the EVC has the same role during visual mental imagery, any change in the visual details of the mental image would lead to corresponding changes in EVC activity. Within this context, the question should not be whether the EVC is 'active' during imagery but how its activity relates to specific imagery properties. Studies using methods that are sensitive to variation in low-level features reveal that imagery can recruit the EVC in similar ways as perception. However, not all mental images contain a high level of visual details. Therefore, I end by considering a more nuanced view, which states that imagery can recruit the EVC, but that does not mean that it always does so.

Confidence ratings do not distinguish imagination from reality

Perceptual reality monitoring refers to the ability to distinguish internally triggered imagination from externally triggered reality. Such monitoring can take place at perceptual or cognitive levels-for example, in lucid dreaming, perceptual experience feels real but is accompanied by a cognitive insight that it is not real. We recently developed a paradigm to reveal perceptual reality monitoring errors during wakefulness in the general population, showing that imagined signals can be erroneously attributed to perception during a perceptual detection task. In the current study, we set out to investigate whether people have insight into perceptual reality monitoring errors by additionally measuring perceptual confidence. We used hierarchical Bayesian modeling of confidence criteria to characterize metacognitive insight into the effects of imagery on detection. Over two experiments, we found that confidence criteria moved in tandem with the decision criterion shift, indicating a failure of reality monitoring not only at a perceptual but also at a metacognitive level. These results further show that such failures have a perceptual rather than a decisional origin. Interestingly, offline queries at the end of the experiment revealed global, task-level insight, which was uncorrelated with local, trial-level insight as measured with confidence ratings. Taken together, our results demonstrate that confidence ratings do not distinguish imagination from reality during perceptual detection. Future research should further explore the different cognitive dimensions of insight into reality judgments and how they are related.

Multisensory subtypes of aphantasia: Mental imagery as supramodal perception in reverse

Cognitive neuroscience research on mental imagery has largely focused on the visual imagery modality in unimodal task contexts. Recent studies have uncovered striking individual differences in visual imagery capacity, with some individuals reporting a subjective absence of conscious visual imagery ability altogether ("aphantasia"). However, naturalistic mental imagery is often multi-sensory, and preliminary findings suggest that many individuals with aphantasia also report a subjective lack of mental imagery in other sensory domains (such as auditory or olfactory imagery). In this paper, we perform a series of cluster analyses on the multi-sensory imagery questionnaire scores of two large groups of aphantasic subjects, defining latent sub-groups in this sample population. We demonstrate that aphantasia is a heterogenous phenomenon characterised by dominant sub-groups of individuals with visual aphantasia (those who report selective visual imagery absence) and multi-sensory aphantasia (those who report an inability to generate conscious mental imagery in any sensory modality). We replicate our findings in a second large sample and show that more unique aphantasia sub-types also exist, such as individuals with selectively preserved mental imagery in only one sensory modality (e.g. intact auditory imagery). We outline the implications of our findings for network theories of mental imagery, discussing how unique aphantasia aetiologies with distinct self-report patterns might reveal alterations to various levels of the sensory processing hierarchy implicated in mental imagery.

Investigating the different mechanisms in related neural activities: a focus on auditory perception and imagery

Neuroimaging studies have shown that the neural representation of imagery is closely related to the perception modality; however, the undeniable different experiences between perception and imagery indicate that there are obvious neural mechanism differences between them, which cannot be explained by the simple theory that imagery is a form of weak perception. Considering the importance of functional integration of brain regions in neural activities, we conducted correlation analysis of neural activity in brain regions jointly activated by auditory imagery and perception, and then brain functional connectivity (FC) networks were obtained with a consistent structure. However, the connection values between the areas in the superior temporal gyrus and the right precentral cortex were significantly higher in auditory perception than in the imagery modality. In addition, the modality decoding based on FC patterns showed that the FC network of auditory imagery and perception can be significantly distinguishable. Subsequently, voxel-level FC analysis further verified the distribution regions of voxels with significant connectivity differences between the 2 modalities. This study complemented the correlation and difference between auditory imagery and perception in terms of brain information interaction, and it provided a new perspective for investigating the neural mechanisms of different modal information representations.

Different Mechanisms for Supporting Mental Imagery and Perceptual Representations: Modulation Versus Excitation

Recent research suggests imagery is functionally equivalent to a weak form of visual perception. Here we report evidence across five independent experiments on adults that perception and imagery are supported by fundamentally different mechanisms: Whereas perceptual representations are largely formed via increases in excitatory activity, imagery representations are largely supported by modulating nonimagined content. We developed two behavioral techniques that allowed us to first put the visual system into a state of adaptation and then probe the additivity of perception and imagery. If imagery drives similar excitatory visual activity to perception, pairing imagery with perceptual adapters should increase the state of adaptation. Whereas pairing weak perception with adapters increased measures of adaptation, pairing imagery reversed their effects. Further experiments demonstrated that these nonadditive effects were due to imagery weakening representations of nonimagined content. Together these data provide empirical evidence that the brain uses categorically different mechanisms to represent imagery and perception.

Spatiotemporal dynamics of self-generated imagery reveal a reverse cortical hierarchy from cue-induced imagery

Visual imagery allows for the construction of rich internal experience in our mental world. However, it has remained poorly understood how imagery experience derives volitionally as opposed to being cue driven. Here, using electroencephalography and functional magnetic resonance imaging, we systematically investigate the spatiotemporal dynamics of self-generated imagery by having participants volitionally imagining one of the orientations from a learned pool. We contrast self-generated imagery with cue-induced imagery, where participants imagined line orientations based on associative cues acquired previously. Our results reveal overlapping neural signatures of cue-induced and self-generated imagery. Yet, these neural signatures display substantially differential sensitivities to the two types of imagery: self-generated imagery is supported by an enhanced involvement of the anterior cortex in representing imagery contents. By contrast, cue-induced imagery is supported by enhanced imagery representations in the posterior visual cortex. These results jointly support a reverse cortical hierarchy in generating and maintaining imagery contents in self-generated versus externally cued imagery.

Real-world implications of aphantasia: episodic recall of eyewitnesses with aphantasia is less complete but no less accurate than typical imagers

Individuals with aphantasia report an inability to voluntarily visually image and reduced episodic memory, yet episodic accounts provided by witnesses and victims are fundamental for criminal justice. Using the mock-witness paradigm, we investigated eyewitness memory of individuals with aphantasia versus typical imagers. Participants viewed a mock crime and 48 hours later were interviewed about the event, randomly allocated to one of three conditions. Two interview conditions included techniques designed to support episodic retrieval mode, namely (i) Mental Reinstatement of Context (MRC) and (ii) Sketch Reinstatement of Context (Sketch-RC). A third Control condition did not include retrieval support. Aphantasic mock-eyewitnesses recalled 30% less correct information and accounts were less complete, but they made no more errors and were as accurate as typical imagers. Interaction effects revealed reduced correct recall and less complete accounts for aphantasic participants in MRC interviews versus Sketch-RC and Control. Aphantaisic participants in the Control outperformed those in both the Sketch-RC and MRC, although Sketch-RC improved completeness by 15% versus MRC. Our pattern of results indicates reduced mental imagery ability might be compensated for by alternative self-initiated cognitive strategies. Findings offer novel insights into episodic recall performance in information gathering interviews when ability to voluntarily visualize is impoverished.

Neural Representations in Visual and Parietal Cortex Differentiate between Imagined, Perceived, and Illusory Experiences

Humans constantly receive massive amounts of information, both perceived from the external environment and imagined from the internal world. To function properly, the brain needs to correctly identify the origin of information being processed. Recent work has suggested common neural substrates for perception and imagery. However, it has remained unclear how the brain differentiates between external and internal experiences with shared neural codes. Here we tested this question in human participants (male and female) by systematically investigating the neural processes underlying the generation and maintenance of visual information from voluntary imagery, veridical perception, and illusion. The inclusion of illusion allowed us to differentiate between objective and subjective internality: while illusion has an objectively internal origin and can be viewed as involuntary imagery, it is also subjectively perceived as having an external origin like perception. Combining fMRI, eye-tracking, multivariate decoding, and encoding approaches, we observed superior orientation representations in parietal cortex during imagery compared with perception, and conversely in early visual cortex. This imagery dominance gradually developed along a posterior-to-anterior cortical hierarchy from early visual to parietal cortex, emerged in the early epoch of imagery and sustained into the delay epoch, and persisted across varied imagined contents. Moreover, representational strength of illusion was more comparable to imagery in early visual cortex, but more comparable to perception in parietal cortex, suggesting content-specific representations in parietal cortex differentiate between subjectively internal and external experiences, as opposed to early visual cortex. These findings together support a domain-general engagement of parietal cortex in internally generated experience.SIGNIFICANCE STATEMENT How does the brain differentiate between imagined and perceived experiences? Combining fMRI, eye-tracking, multivariate decoding, and encoding approaches, the current study revealed enhanced stimulus-specific representations in visual imagery originating from parietal cortex, supporting the subjective experience of imagery. This neural principle was further validated by evidence from visual illusion, wherein illusion resembled perception and imagery at different levels of cortical hierarchy. Our findings provide direct evidence for the critical role of parietal cortex as a domain-general region for content-specific imagery, and offer new insights into the neural mechanisms underlying the differentiation between subjectively internal and external experiences.

Modelling perception as a hierarchical competition differentiates imagined, veridical, and hallucinated percepts

Mental imagery is a process by which thoughts become experienced with sensory characteristics. Yet, it is not clear why mental images appear diminished compared to veridical images, nor how mental images are phenomenologically distinct from hallucinations, another type of non-veridical sensory experience. Current evidence suggests that imagination and veridical perception share neural resources. If so, we argue that considering how neural representations of externally generated stimuli (i.e. sensory input) and internally generated stimuli (i.e. thoughts) might interfere with one another can sufficiently differentiate between veridical, imaginary, and hallucinatory perception. We here use a simple computational model of a serially connected, hierarchical network with bidirectional information flow to emulate the primate visual system. We show that modelling even first approximations of neural competition can more coherently explain imagery phenomenology than non-competitive models. Our simulations predict that, without competing sensory input, imagined stimuli should ubiquitously dominate hierarchical representations. However, with competition, imagination should dominate high-level representations but largely fail to outcompete sensory inputs at lower processing levels. To interpret our findings, we assume that low-level stimulus information (e.g. in early visual cortices) contributes most to the sensory aspects of perceptual experience, while high-level stimulus information (e.g. towards temporal regions) contributes most to its abstract aspects. Our findings therefore suggest that ongoing bottom-up inputs during waking life may prevent imagination from overriding veridical sensory experience. In contrast, internally generated stimuli may be hallucinated when sensory input is dampened or eradicated. Our approach can explain individual differences in imagery, along with aspects of daydreaming, hallucinations, and non-visual mental imagery.

Emotion in the mind's eye: Imagination for adaptive cognition

In a complex world, we are constantly faced with environmental stimuli that shape our moment-to-moment experiences. But just as rich and complex as the external world is the internal milieu-our imagination. Imagination offers a powerful vehicle for playing out hypothetical experiences in the mind's eye. It allows us to mentally time travel to behold what the future might bring, including our greatest desires or fears. Indeed, imagined experiences tend to be emotion-laden. How and why are humans capable of this remarkable feat? Based on psychological findings, we highlight the importance of imagination for emotional aspects of cognition and behavior, namely in the generation and regulation of emotions. Based on recent cognitive neuroscience work, we identify putative neural networks that are most critical for emotional imagination, with a major focus on the default mode network. Finally, we briefly highlight the possible functional implications of individual differences in imagination. Overall, we hope to address why humans have the capacity to simulate hypothetical emotional experiences and how this ability can be harnessed in adaptive (and sometimes maladaptive) ways. We end by discussing open questions.

Rumination as a Transdiagnostic Phenomenon in the 21st Century: The Flow Model of Rumination

Rumination and its related mental phenomena share associated impairments in cognition, such as executive functions and attentional processes across different clinical conditions (e.g., in psychotic disorders). In recent decades, however, the notion of rumination has been increasingly narrowed to the "self-focused" type in depressive disorders. A closer review of the literature shows that rumination may be construed as a broader process characterized by repetitive thoughts about certain mental contents that interfere with one's daily activities, not only limited to those related to "self". A further examination of the construct of rumination beyond the narrowly focused depressive rumination would help expand intervention opportunities for mental disorders in today's context. We first review the development of the clinical construct of rumination with regard to its historical roots and its roles in psychopathology. This builds the foundation for the introduction of the "Flow Model of Rumination (FMR)", which conceptualizes rumination as a disruption of a smooth flow of mental contents in conscious experience that depends on the coordinated interactions between intention, memory, affect, and external events. The conceptual review concludes with a discussion of the impact of rapid technological advances (such as smartphones) on rumination. Particularly in contemporary societies today, a broader consideration of rumination not only from a cognition viewpoint, but also incorporating a human-device interaction perspective, is necessitated. The implications of the FMR in contemporary mental health practice are discussed.

Subjective signal strength distinguishes reality from imagination

Humans are voracious imaginers, with internal simulations supporting memory, planning and decision-making. Because the neural mechanisms supporting imagery overlap with those supporting perception, a foundational question is how reality and imagination are kept apart. One possibility is that the intention to imagine is used to identify and discount self-generated signals during imagery. Alternatively, because internally generated signals are generally weaker, sensory strength is used to index reality. Traditional psychology experiments struggle to investigate this issue as subjects can rapidly learn that real stimuli are in play. Here, we combined one-trial-per-participant psychophysics with computational modelling and neuroimaging to show that imagined and perceived signals are in fact intermixed, with judgments of reality being determined by whether this intermixed signal is strong enough to cross a reality threshold. A consequence of this account is that when virtual or imagined signals are strong enough, they become subjectively indistinguishable from reality.

Synchronous motor imagery and visual feedback of finger movement elicit the moving rubber hand illusion, at least in illusion-susceptible individuals

Recent evidence suggests that imagined auditory and visual sensory stimuli can be integrated with real sensory information from a different sensory modality to change the perception of external events via cross-modal multisensory integration mechanisms. Here, we explored whether imagined voluntary movements can integrate visual and proprioceptive cues to change how we perceive our own limbs in space. Participants viewed a robotic hand wearing a glove repetitively moving its right index finger up and down at a frequency of 1 Hz, while they imagined executing the corresponding movements synchronously or asynchronously (kinesthetic-motor imagery); electromyography (EMG) from the participants' right index flexor muscle confirmed that the participants kept their hand relaxed while imagining the movements. The questionnaire results revealed that the synchronously imagined movements elicited illusory ownership and a sense of agency over the moving robotic hand-the moving rubber hand illusion-compared with asynchronously imagined movements; individuals who affirmed experiencing the illusion with real synchronous movement also did so with synchronous imagined movements. The results from a proprioceptive drift task further demonstrated a shift in the perceived location of the participants' real hand toward the robotic hand in the synchronous versus the asynchronous motor imagery condition. These results suggest that kinesthetic motor imagery can be used to replace veridical congruent somatosensory feedback from a moving finger in the moving rubber hand illusion to trigger illusory body ownership and agency, but only if the temporal congruence rule of the illusion is obeyed. This observation extends previous studies on the integration of mental imagery and sensory perception to the case of multisensory bodily awareness, which has potentially important implications for research into embodiment of brain-computer interface controlled robotic prostheses and computer-generated limbs in virtual reality.

Brain activation and individual differences of emotional perception and imagery in healthy adults: A functional near-infrared spectroscopy (fNIRS) study

The current study investigated the brain activation and individual differences in perception and imagery of sad pictures versus happy and neutral pictures. Sixty-eight healthy adults were instructed to view and visualize sad, happy, and neutral pictures during 64-channel functional near-infrared spectroscopy (fNIRS) recording. The results indicated that emotional perception evoked increased occipital activation, while emotional imagery involved increased activation in the bilateral prefrontal and parietal cortex. Sad pictures evoked decreased brain activation in the occipital and prefrontal cortex than happy and neutral pictures. For women, imagery activation was greater than perception activation in the right parietal cortex. Additionally, participants' self-rated imagery vividness was positively correlated with the occipital activation during happy imagery and trait rumination was negatively correlated with the occipital activation during perception. The findings suggest that emotional perception may involve the bottom-up sensory input, while emotional imagery may involve the top-down cognitive processes. Healthy individuals engage decreased cognitive resources for sad perception and imagery. Moreover, our observation could provide useful information to establish fNIRS assessment as an objective tool to monitor the emotional status on an individual trait basis.

The clear and not so clear signatures of perceptual reality in the Bayesian brain

In a Bayesian brain, every perceptual decision will take into account internal priors as well as new incoming evidence. A reality monitoring system-eventually providing the agent us with a subjective sense of reality avoids us them being confused about whether our experience is perceptual or imagined. Yet not all confusions we experience mean that we wonder wonder whether we may be imagining: some confused experiences feel clearly perceptual but still feel not right. What happens in such confused perceptions, and can the Bayesian brain explain this kind of confusion? In this paper, we offer a characterisation of perceptual confusion and argue that it requires our subjective sense of reality to be a composite of several subjective markers, including a categorical one that can clearly identify an experience as perceptual and connecting us to reality. Our composite account makes new predictions regarding the robustness, the non-linear development and the possible breakdowns of the sense of reality in perception.

Perceptual reality monitoring: Neural mechanisms dissociating imagination from reality

There is increasing evidence that imagination relies on similar neural mechanisms as externally triggered perception. This overlap presents a challenge for perceptual reality monitoring: deciding what is real and what is imagined. Here, we explore how perceptual reality monitoring might be implemented in the brain. We first describe sensory and cognitive factors that could dissociate imagery and perception and conclude that no single factor unambiguously signals whether an experience is internally or externally generated. We suggest that reality monitoring is implemented by higher-level cortical circuits that evaluate first-order sensory and cognitive factors to determine the source of sensory signals. According to this interpretation, perceptual reality monitoring shares core computations with metacognition. This multi-level architecture might explain several types of source confusion as well as dissociations between simply knowing whether something is real and actually experiencing it as real. We discuss avenues for future research to further our understanding of perceptual reality monitoring, an endeavour that has important implications for our understanding of clinical symptoms as well as general cognitive function.

Photographs of Actions: What Makes Them Special Cues to Social Perception

I have reviewed studies on neural responses to pictured actions in the action observation network (AON) and the cognitive functions of these responses. Based on this review, I have analyzed the specific representational characteristics of action photographs. There has been consensus that AON responses provide viewers with knowledge of observed or pictured actions, but there has been controversy about the properties of this knowledge. Is this knowledge causally provided by AON activities or is it dependent on conceptual processing? What elements of actions does it refer to, and how generalized or specific is it? The answers to these questions have come from studies that used transcranial magnetic stimulation (TMS) to stimulate motor or somatosensory cortices. In conjunction with electromyography (EMG), TMS allows researchers to examine changes of the excitability in the corticospinal tract and muscles of people viewing pictured actions. The timing of these changes and muscle specificity enable inferences to be drawn about the cognitive products of processing pictured actions in the AON. Based on a review of studies using TMS and other neuroscience methods, I have proposed a novel hypothetical account that describes the characteristics of action photographs that make them effective cues to social perception. This account includes predictions that can be tested experimentally.

V1 as an egocentric cognitive map

We typically distinguish between V1 as an egocentric perceptual map and the hippocampus as an allocentric cognitive map. In this article, we argue that V1 also functions as a post-perceptual egocentric cognitive map. We argue that three well-documented functions of V1, namely (i) the estimation of distance, (ii) the estimation of size, and (iii) multisensory integration, are better understood as post-perceptual cognitive inferences. This argument has two important implications. First, we argue that V1 must function as the neural correlates of the visual perception/cognition distinction and suggest how this can be accommodated by V1's laminar structure. Second, we use this insight to propose a low-level account of visual consciousness in contrast to mid-level accounts (recurrent processing theory; integrated information theory) and higher-level accounts (higher-order thought; global workspace theory). Detection thresholds have been traditionally used to rule out such an approach, but we explain why it is a mistake to equate visibility (and therefore the presence/absence of visual experience) with detection thresholds.

Beyond imagination: Hypnotic visual hallucination induces greater lateralised brain activity than visual mental imagery

Hypnotic suggestions can produce a broad range of perceptual experiences, including hallucinations. Visual hypnotic hallucinations differ in many ways from regular mental images. For example, they are usually experienced as automatic, vivid, and real images, typically compromising the sense of reality. While both hypnotic hallucination and mental imagery are believed to mainly rely on the activation of the visual cortex via top-down mechanisms, it is unknown how they differ in the neural processes they engage. Here we used an adaptation paradigm to test and compare top-down processing between hypnotic hallucination, mental imagery, and visual perception in very highly hypnotisable individuals whose ability to hallucinate was assessed. By measuring the N170/VPP event-related complex and using multivariate decoding analysis, we found that hypnotic hallucination of faces involves greater top-down activation of sensory processing through lateralised neural mechanisms in the right hemisphere compared to mental imagery. Our findings suggest that the neural signatures that distinguish hypnotically hallucinated faces from imagined faces lie in the right brain hemisphere.

Offline perception: an introduction

Experiences that are self-generated and independent of sensory stimulations permeate our whole life. This theme issue examines their similarities and differences, systematizes the literature from an integrative perspective, critically discusses state-of-the-art empirical findings and proposes new theoretical approaches. The aim of the theme issue is to foster interaction between the different disciplines and research directions involved and to explore the prospects of a unificatory account of offline perception in general. This article is part of the theme issue 'Offline perception: voluntary and spontaneous perceptual experiences without matching external stimulation'.