A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes

Mental rotation task specifically modulates functional connectivity strength of intrinsic brain activity in low frequency domains: A maximum uncertainty linear discriminant analysis

Neuroimaging studies have highlighted that intrinsic brain activity is modified to implement task demands. However, the relation between mental rotation and intrinsic brain activity remains unclear. To answer this question, we collected functional MRI (fMRI) data from 30 healthy participants in two mental rotation task periods (1st-task state, 2nd-task state) and two rest periods before (pre-task resting state) and after the task (post-task resting state) respectively. By combining the spatial independent component analysis (ICA) and voxel-wise functional connectivity strength (FCS), we identified FCS maps of 10 brain resting state networks (RSNs) within six different bands (i.e., 0-0.05, 0.05-0.1, 0.1-0.15, 0.15-0.2, 0.2-0.25, and 0.01-0.08Hz) corresponding to the four states for each subject. The maximum uncertainty linear discriminant analysis (MLDA) method showed that the FCS within the low frequency bandwidth of 0.05-0.1Hz could effectively classify the mental rotation task state from pre-/post-task resting states but failed to discriminate the pre- and post-task resting states. Discriminative FCSs were observed in the cognitive executive-control network (central executive and attention) and the imagery-based internal mental manipulation network (default mode, primary sensorimotor, and primary visual). Imagery manipulation is a stable mental element of mental rotation, and the involvement of executive control is dependent on the degree of task familiarity. Together, the present study provides evidence that mental rotation task specifically modifies intrinsic brain activity to complement cognitive demands, which provides further insight into the neural basis of mental rotation manipulation.

Individual differences shape the content of visual representations

Visually perceiving a stimulus activates a pictorial representation of that item in the brain, but how pictorial is the representation of a stimulus in the absence of visual stimulation? Here I address this question with a review of the literatures on visual imagery (VI), visual working memory (VWM), and visual preparatory templates, all of which require activating visual information in the absence of sensory stimulation. These processes have historically been studied separately, but I propose that they can provide complimentary evidence for the pictorial nature of their contents. One major challenge in studying the contents of visual representations is the discrepant findings concerning the extent of overlap (both cortical and behavioral) between externally and internally sourced visual representations. I argue that these discrepancies may in large part be due to individual differences in VI vividness and precision, the specific representative abilities required to perform a task, appropriateness of visual preparatory strategies, visual cortex anatomy, and level of expertise with a particular object category. Individual differences in visual representative abilities greatly impact task performance and may influence the likelihood of experiences such as intrusive VI and hallucinations, but research still predominantly focuses on uniformities in visual experience across individuals. In this paper I review the evidence for the pictorial content of visual representations activated for VI, VWM, and preparatory templates, and highlight the importance of accounting for various individual differences in conducting research on this topic.

Mental Imagery: Functional Mechanisms and Clinical Applications

Mental imagery research has weathered both disbelief of the phenomenon and inherent methodological limitations. Here we review recent behavioral, brain imaging, and clinical research that has reshaped our understanding of mental imagery. Research supports the claim that visual mental imagery is a depictive internal representation that functions like a weak form of perception. Brain imaging work has demonstrated that neural representations of mental and perceptual images resemble one another as early as the primary visual cortex (V1). Activity patterns in V1 encode mental images and perceptual images via a common set of low-level depictive visual features. Recent translational and clinical research reveals the pivotal role that imagery plays in many mental disorders and suggests how clinicians can utilize imagery in treatment.

Resolving Ambiguities of MVPA Using Explicit Models of Representation

We advocate a shift in emphasis within cognitive neuroscience from multivariate pattern analysis (MVPA) to the design and testing of explicit models of neural representation. With such models, it becomes possible to identify the specific representations encoded in patterns of brain activity and to map them across the brain.

Spontaneous ocular positioning during visual imagery in patients with hemianopia and/or hemineglect

Spontaneous eye movements during imagery are not random and can be used to study and reveal mental visualization processes (Fourtassi et al., 2013; Johansson et al. 2006). For example, we previously showed that during memory recall of French towns via imagery healthy individuals looks straight ahead when recalling Paris and their subsequent gaze positions are significantly correlated with the real GPS coordinates of the recalled towns. This correlation suggests that memory retrieval is done via depictive representations as it is never found when the towns are recalled using verbal fluency. In the present paper we added to this finding by showing that the mental image is spontaneously centered on the head or body midline. In order to investigate the capacities of visual imagery in patients, and by extension, the role of primary visual cortex and fronto-parietal cortex in spatial visual imagery, we recorded gaze positions during memory recall of French towns in an imagery task, a non-imagery task (verbal fluency), and a visually-guided task in five patients with left or right hemianopia and in four patients with hemineglect (two with left hemianopia and two without). The correlation between gaze position and real GPS coordinates of the recalled towns was significant in all hemianopic patients, but in patients with hemineglect this was only the case for towns located on the right half of the map of France. This suggests hemianopic patients can perform spatially consistent mental imagery despite direct or indirect unilateral lesions of the primary visual cortex. In contrast, the left-sided towns recalled by hemineglect patients, revealed that they have some spatial inconsistency or representational difficulty. Hemianopic patients positioned and maintained their gaze in their contralesional hemispace, suggesting that their mental map was not centered on their head or body midline. This contralesional gaze positioning appeared to be a general compensation strategy and was not observed in patients with neglect (with or without hemianopia). Instead, neglect patients positioned their gaze in their ipsilesional hemispace and only when performing the visual imagery task. These findings are discussed in the context of the role of occipital and fronto-parietal cortices in the neuroanatomical model of visual imagery developed by Kosslyn et al. (2006).

On Picturing a Candle: The Prehistory of Imagery Science

The past 25 years have seen a rapid growth of knowledge about brain mechanisms involved in visual mental imagery. These advances have largely been made independently of the long history of philosophical - and even psychological - reckoning with imagery and its parent concept 'imagination'. We suggest that the view from these empirical findings can be widened by an appreciation of imagination's intellectual history, and we seek to show how that history both created the conditions for - and presents challenges to - the scientific endeavor. We focus on the neuroscientific literature's most commonly used task - imagining a concrete object - and, after sketching what is known of the neurobiological mechanisms involved, we examine the same basic act of imagining from the perspective of several key positions in the history of philosophy and psychology. We present positions that, firstly, contextualize and inform the neuroscientific account, and secondly, pose conceptual and methodological challenges to the scientific analysis of imagery. We conclude by reflecting on the intellectual history of visualization in the light of contemporary science, and the extent to which such science may resolve long-standing theoretical debates.

Linear Discriminant Analysis Achieves High Classification Accuracy for the BOLD fMRI Response to Naturalistic Movie Stimuli

Naturalistic stimuli like movies evoke complex perceptual processes, which are of great interest in the study of human cognition by functional MRI (fMRI). However, conventional fMRI analysis based on statistical parametric mapping (SPM) and the general linear model (GLM) is hampered by a lack of accurate parametric models of the BOLD response to complex stimuli. In this situation, statistical machine-learning methods, a.k.a. multivariate pattern analysis (MVPA), have received growing attention for their ability to generate stimulus response models in a data-driven fashion. However, machine-learning methods typically require large amounts of training data as well as computational resources. In the past, this has largely limited their application to fMRI experiments involving small sets of stimulus categories and small regions of interest in the brain. By contrast, the present study compares several classification algorithms known as Nearest Neighbor (NN), Gaussian Naïve Bayes (GNB), and (regularized) Linear Discriminant Analysis (LDA) in terms of their classification accuracy in discriminating the global fMRI response patterns evoked by a large number of naturalistic visual stimuli presented as a movie. Results show that LDA regularized by principal component analysis (PCA) achieved high classification accuracies, above 90% on average for single fMRI volumes acquired 2 s apart during a 300 s movie (chance level 0.7% = 2 s/300 s). The largest source of classification errors were autocorrelations in the BOLD signal compounded by the similarity of consecutive stimuli. All classifiers performed best when given input features from a large region of interest comprising around 25% of the voxels that responded significantly to the visual stimulus. Consistent with this, the most informative principal components represented widespread distributions of co-activated brain regions that were similar between subjects and may represent functional networks. In light of these results, the combination of naturalistic movie stimuli and classification analysis in fMRI experiments may prove to be a sensitive tool for the assessment of changes in natural cognitive processes under experimental manipulation.

Multi-Voxel Decoding and the Topography of Maintained Information During Visual Working Memory

The ability to maintain representations in the absence of external sensory stimulation, such as in working memory, is critical for guiding human behavior. Human functional brain imaging studies suggest that visual working memory can recruit a network of brain regions from visual to parietal to prefrontal cortex. In this review, we focus on the maintenance of representations during visual working memory and discuss factors determining the topography of those representations. In particular, we review recent studies employing multi-voxel pattern analysis (MVPA) that demonstrate decoding of the maintained content in visual cortex, providing support for a “sensory recruitment” model of visual working memory. However, there is some evidence that maintained content can also be decoded in areas outside of visual cortex, including parietal and frontal cortex. We suggest that the ability to maintain representations during working memory is a general property of cortex, not restricted to specific areas, and argue that it is important to consider the nature of the information that must be maintained. Such information-content is critically determined by the task and the recruitment of specific regions during visual working memory will be both task- and stimulus-dependent. Thus, the common finding of maintained information in visual, but not parietal or prefrontal, cortex may be more of a reflection of the need to maintain specific types of visual information and not of a privileged role of visual cortex in maintenance.

Modality-independent representations of small quantities based on brain activation patterns

Machine learning or MVPA (Multi Voxel Pattern Analysis) studies have shown that the neural representation of quantities of objects can be decoded from fMRI patterns, in cases where the quantities were visually displayed. Here we apply these techniques to investigate whether neural representations of quantities depicted in one modality (say, visual) can be decoded from brain activation patterns evoked by quantities depicted in the other modality (say, auditory). The main finding demonstrated, for the first time, that quantities of dots were decodable by a classifier that was trained on the neural patterns evoked by quantities of auditory tones, and vice-versa. The representations that were common across modalities were mainly right-lateralized in frontal and parietal regions. A second finding was that the neural patterns in parietal cortex that represent quantities were common across participants. These findings demonstrate a common neuronal foundation for the representation of quantities across sensory modalities and participants and provide insight into the role of parietal cortex in the representation of quantity information.

Hallucinations and mental imagery demonstrate top-down effects on visual perception

In this commentary, we present two examples where perception is not only influenced by, but also in fact driven by, top-down effects: hallucinations and mental imagery. Crucially, both examples avoid all six of the potential confounds that Firestone & Scholl (F&S) raised as arguments against previous studies claiming to demonstrate the influence of top-down effects on perception.

Reconstructing representations of dynamic visual objects in early visual cortex

As raw sensory data are partial, our visual system extensively fills in missing details, creating enriched percepts based on incomplete bottom-up information. Despite evidence for internally generated representations at early stages of cortical processing, it is not known whether these representations include missing information of dynamically transforming objects. Long-range apparent motion (AM) provides a unique test case because objects in AM can undergo changes both in position and in features. Using fMRI and encoding methods, we found that the "intermediate" orientation of an apparently rotating grating, never presented in the retinal input but interpolated during AM, is reconstructed in population-level, feature-selective tuning responses in the region of early visual cortex (V1) that corresponds to the retinotopic location of the AM path. This neural representation is absent when AM inducers are presented simultaneously and when AM is visually imagined. Our results demonstrate dynamic filling-in in V1 for object features that are interpolated during kinetic transformations.

Distributed Patterns of Reactivation Predict Vividness of Recollection

According to the principle of reactivation, memory retrieval evokes patterns of brain activity that resemble those instantiated when an event was first experienced. Intuitively, one would expect neural reactivation to contribute to recollection (i.e., the vivid impression of reliving past events), but evidence of a direct relationship between the subjective quality of recollection and multiregional reactivation of item-specific neural patterns is lacking. The current study assessed this relationship using fMRI to measure brain activity as participants viewed and mentally replayed a set of short videos. We used multivoxel pattern analysis to train a classifier to identify individual videos based on brain activity evoked during perception and tested how accurately the classifier could distinguish among videos during mental replay. Classification accuracy correlated positively with memory vividness, indicating that the specificity of multivariate brain patterns observed during memory retrieval was related to the subjective quality of a memory. In addition, we identified a set of brain regions whose univariate activity during retrieval predicted both memory vividness and the strength of the classifier's prediction irrespective of the particular video that was retrieved. Our results establish distributed patterns of neural reactivation as a valid and objective marker of the quality of recollection.

The heterogeneity of mental representation: Ending the imagery debate

The possible ways that information can be represented mentally have been discussed often over the past thousand years. However, this issue could not be addressed rigorously until late in the 20th century. Initial empirical findings spurred a debate about the heterogeneity of mental representation: Is all information stored in propositional, language-like, symbolic internal representations, or can humans use at least two different types of representations (and possibly many more)? Here, in historical context, we describe recent evidence that humans do not always rely on propositional internal representations but, instead, can also rely on at least one other format: depictive representation. We propose that the debate should now move on to characterizing all of the different forms of human mental representation.