Predicting the stream of consciousness from activity in human visual cortex

How-tests for consciousness and direct neurophenomenal structuralism

Despite recent criticism, the search for neural correlates of consciousness (NCCs) is still at the core of a contemporary neuroscience of consciousness. One common aim is to distinguish merely statistical correlates from "NCCs proper", i.e., NCCs that are uniquely associated with a conscious experience and lend themselves to a metaphysical interpretation. We should then distinguish between NCCs as data and NCCs as hypotheses, where the first is just recorded data while the second goes beyond any set of recorded data. Still, such NCC-hypotheses ought to be testable. Here, I present a framework for so-called "sufficiency tests." We can distinguish four different classes of such tests, depending on whether they predict creature consciousness (which systems are conscious), state consciousness (when a system is conscious), phenomenal content (what a system is conscious of), or phenomenal character (how a system experiences). For each kind of test, I provide examples from the empirical literature. I also argue that tests for phenomenal character (How-Tests) are preferable because they bracket problematic aspects of the other kinds of tests. However, How-Tests imply a metaphysical tie between the neural and phenomenal domain that is stronger than supervenience, delivers explanations but does not close the explanatory gap, uses first-person methods to test hypotheses, and thereby relies on a form of direct neurophenomenal structuralism.

Have you been there before? Decoding recognition of spatial scenes from fMRI signals in precuneus

One potential application of forensic "brain reading" is to test whether a suspect has previously experienced a crime scene. Here, we investigated whether it is possible to decode real life autobiographic exposure to spatial locations using fMRI. In the first session, participants visited four out of eight possible rooms on a university campus. During a subsequent scanning session, subjects passively viewed pictures and videos from these eight possible rooms (four old, four novel) without giving any responses. A multivariate searchlight analysis was employed that trained a classifier to distinguish between "seen" versus "unseen" stimuli from a subset of six rooms. We found that bilateral precuneus encoded information that can be used to distinguish between previously seen and unseen rooms and that also generalized to the two stimuli left out from training. We conclude that activity in bilateral precuneus is associated with the memory of previously visited rooms, irrespective of the identity of the room, thus supporting a parietal contribution to episodic memory for spatial locations. Importantly, we could decode whether a room was visited in real life without the need of explicit judgments about the rooms. This suggests that recognition is an automatic response that can be decoded from fMRI data, thus potentially supporting forensic applications of concealed information tests for crime scene recognition.

An Accumulating Neural Signal Underlying Binocular Rivalry Dynamics

During binocular rivalry, conflicting images are presented one to each eye and perception alternates stochastically between them. Despite stable percepts between alternations, modeling suggests that neural signals representing the two images change gradually, and that the duration of stable percepts are determined by the time required for these signals to reach a threshold that triggers an alternation. However, direct physiological evidence for such signals has been lacking. Here, we identify a neural signal in the human visual cortex that shows these predicted properties. We measured steady-state visual evoked potentials (SSVEPs) in 84 human participants (62 females, 22 males) who were presented with orthogonal gratings, one to each eye, flickering at different frequencies. Participants indicated their percept while EEG data were collected. The time courses of the SSVEP amplitudes at the two frequencies were then compared across different percept durations, within participants. For all durations, the amplitude of signals corresponding to the suppressed stimulus increased and the amplitude corresponding to the dominant stimulus decreased throughout the percept. Critically, longer percepts were characterized by more gradual increases in the suppressed signal and more gradual decreases of the dominant signal. Changes in signals were similar and rapid at the end of all percepts, presumably reflecting perceptual transitions. These features of the SSVEP time courses are well predicted by a model in which perceptual transitions are produced by the accumulation of noisy signals. Identification of this signal underlying binocular rivalry should allow strong tests of neural models of rivalry, bistable perception, and neural suppression.SIGNIFICANCE STATEMENT During binocular rivalry, two conflicting images are presented to the two eyes and perception alternates between them, with switches occurring at seemingly random times. Rivalry is an important and longstanding model system in neuroscience, used for understanding neural suppression, intrinsic neural dynamics, and even the neural correlates of consciousness. All models of rivalry propose that it depends on gradually changing neural activity that on reaching some threshold triggers the perceptual switches. This manuscript reports the first physiological measurement of neural signals with that set of properties in human participants. The signals, measured with EEG in human observers, closely match the predictions of recent models of rivalry, and should pave the way for much future work.

Variational relevance evaluation of individual fMRI data enables deconstruction of task-dependent neural dynamics

In neuroimaging research, univariate analysis has always been used to localize "representations" at the microscale, whereas network approaches have been applied to characterize transregional "operations". How are representations and operations linked through dynamic interactions? We developed the variational relevance evaluation (VRE) method to analyze individual task fMRI data, which selects informative voxels during model training to localize the "representation", and quantifies the dynamic contributions of single voxels across the whole-brain to different cognitive functions to characterize the "operation". Using 15 individual fMRI data files for higher visual area localizers, we evaluated the characterization of selected voxel positions of VRE and revealed different object-selective regions functioning in similar dynamics. Using another 15 individual fMRI data files for memory retrieval after offline learning, we found similar task-related regions working in different neural dynamics for tasks with diverse familiarities. VRE demonstrates a promising horizon in individual fMRI research.

Flexible sensory-motor mapping rules manifest in correlated variability of stimulus and action codes across the brain

Humans and non-human primates can flexibly switch between different arbitrary mappings from sensation to action to solve a cognitive task. It has remained unknown how the brain implements such flexible sensory-motor mapping rules. Here, we uncovered a dynamic reconfiguration of task-specific correlated variability between sensory and motor brain regions. Human participants switched between two rules for reporting visual orientation judgments during fMRI recordings. Rule switches were either signaled explicitly or inferred by the participants from ambiguous cues. We used behavioral modeling to reconstruct the time course of their belief about the active rule. In both contexts, the patterns of correlations between ongoing fluctuations in stimulus- and action-selective activity across visual- and action-related brain regions tracked participants' belief about the active rule. The rule-specific correlation patterns broke down around the time of behavioral errors. We conclude that internal beliefs about task state are instantiated in brain-wide, selective patterns of correlated variability.

Neural fatigue by passive induction: repeated stimulus exposure results in cognitive fatigue and altered representations in task-relevant networks

Cognitive fatigue is defined by a reduced capacity to perform mental tasks. Despite its pervasiveness, the underlying neural mechanisms remain elusive. Specifically, it is unclear whether prolonged effort affects performance through alterations in over-worked task-relevant neuronal assemblies. Our paradigm based on repeated passive visual stimulation discerns fatigue effects from the influence of motivation, skill and boredom. We induced performance loss and observed parallel alterations in the neural blueprint of the task, by mirroring behavioral performance with multivariate neuroimaging techniques (MVPA) that afford a subject-specific approach. Crucially, functional areas that responded the most to repeated stimulation were also the most affected. Finally, univariate analysis revealed clusters displaying significant disruption within the extrastriate visual cortex. In sum, here we show that repeated stimulation impacts the implicated brain areas' activity and causes tangible behavioral repercussions, providing evidence that cognitive fatigue can result from local, functional, disruptions in the neural signal induced by protracted recruitment.

Exploring the role of structuralist methodology in the neuroscience of consciousness: a defense and analysis

Traditional contrastive analysis has been the foundation of consciousness science, but its limitations due to the lack of a reliable method for measuring states of consciousness have prompted the exploration of alternative approaches. Structuralist theories have gained attention as an alternative that focuses on the structural properties of phenomenal experience and seeks to identify their neural encoding via structural similarities between quality spaces and neural state spaces. However, the intertwining of philosophical assumptions about structuralism and structuralist methodology may pose a challenge to those who are skeptical of the former. In this paper, I offer an analysis and defense of structuralism as a methodological approach in consciousness science, which is partly independent of structuralist assumptions on the nature of consciousness. By doing so, I aim to make structuralist methodology more accessible to a broader scientific and philosophical audience. I situate methodological structuralism in the context of questions concerning mental representation, psychophysical measurement, holism, and functional relevance of neural processes. At last, I analyze the relationship between the structural approach and the distinction between conscious and unconscious states.

Can neurofeedback provide evidence of direct brain-behavior causality?

Neurofeedback is a procedure that measures brain activity in real-time and presents it as feedback to an individual, thus allowing them to self-regulate brain activity with effects on cognitive processes inferred from behavior. One common argument is that neurofeedback studies can reveal how the measured brain activity causes a particular cognitive process. The causal claim is often made regarding the measured brain activity being manipulated as an independent variable, similar to brain stimulation studies. However, this causal inference is vulnerable to the argument that other upstream brain activities change concurrently and cause changes in the brain activity from which feedback is derived. In this paper, we outline the inference that neurofeedback may causally affect cognition by indirect means. We further argue that researchers should remain open to the idea that the trained brain activity could be part of a "causal network" that collectively affects cognition rather than being necessarily causally primary. This particular inference may provide a better translation of evidence from neurofeedback studies to the rest of neuroscience. We argue that the recent advent of multivariate pattern analysis, when combined with implicit neurofeedback, currently comprises the strongest case for causality. Our perspective is that although the burden of inferring direct causality is difficult, it may be triangulated using a collection of various methods in neuroscience. Finally, we argue that the neurofeedback methodology provides unique advantages compared to other methods for revealing changes in the brain and cognitive processes but that researchers should remain mindful of indirect causal effects.

Inattentive Perception, Time, and the Incomprehensibility of Consciousness

Cerebral energy supply is insufficient to support continuous neuronal processing of the plethora of time-constant objects that we are aware of. As a result, the brain is forced to limit processing resources to (the most relevant) cases of change. The neuronally generated world is thus temporally discontinuous. This parallels the fact that, in all relevant microscopic fundamental equations of nature, temporal change plays a dominant role. When a scientist calculates a "solution" to such an equation, integration over time is an essential step. The present Hypothesis expresses that the step from neuronal activity to phenomenal content of consciousness is reflective of a (phenomenal) "solution:" the main source of the incomprehensibility of consciousness is proposed to result from the introduction of phenomenal time-constant entities. These are "filled-in" via integration, even though neuronal data only exists for changes to these entities. In this way, a temporally continuous picture of the world phenomenally appears. Qualia are "initial conditions," which are required for integration and cannot be deduced from present data. Phenomenal "identity" (vs. "high similarity") is related to qualia. Inattentive visual perception, which is only rarely investigated, offers insights into these relationships. Introspectively, unattended vision appears rich because percepts are cumulated over long time spans, whereas attentive perception relies purely on present neuronal signals. The present Hypothesis is that a brief neuronal activity can signify long-lasting and constant phenomenal content of consciousness. Experimental support is presented that comes from discrepancies between neuronal activity and perception: transient neuronal responses to sustained stimuli, "filling-in," change blindness, identity vs. close resemblance.

Non-stimulated regions in early visual cortex encode the contents of conscious visual perception

Predictions shape our perception. The theory of predictive processing poses that our brains make sense of incoming sensory input by generating predictions, which are sent back from higher to lower levels of the processing hierarchy. These predictions are based on our internal model of the world and enable inferences about the hidden causes of the sensory input data. It has been proposed that conscious perception corresponds to the currently most probable internal model of the world. Accordingly, predictions influencing conscious perception should be fed back from higher to lower levels of the processing hierarchy. Here, we used functional magnetic resonance imaging and multivoxel pattern analysis to show that non‐stimulated regions of early visual areas contain information about the conscious perception of an ambiguous visual stimulus. These results indicate that early sensory cortices in the human brain receive predictive feedback signals that reflect the current contents of conscious perception.

Wrist and finger motor representations embedded in the cerebral and cerebellar resting-state activation

Several functional magnetic resonance imaging (fMRI) studies have demonstrated that resting-state brain activity consists of multiple components, each corresponding to the spatial pattern of brain activity induced by performing a task. Especially in a movement task, such components have been shown to correspond to the brain activity pattern of the relevant anatomical region, meaning that the voxels of pattern that are cooperatively activated while using a body part (e.g., foot, hand, and tongue) also behave cooperatively in the resting state. However, it is unclear whether the components involved in resting-state brain activity correspond to those induced by the movement of discrete body parts. To address this issue, in the present study, we focused on wrist and finger movements in the hand, and a cross-decoding technique trained to discriminate between the multi-voxel patterns induced by wrist and finger movement was applied to the resting-state fMRI. We found that the multi-voxel pattern in resting-state brain activity corresponds to either wrist or finger movements in the motor-related areas of each hemisphere of the cerebrum and cerebellum. These results suggest that resting-state brain activity in the motor-related areas consists of the components corresponding to the elementary movements of individual body parts. Therefore, the resting-state brain activity possibly has a finer structure than considered previously.

Frontal activation as a key for deciphering context congruity and valence during visual perception: An electrical neuroimaging study

The object-context associations and the valence are two important stimulus attributes that influence visual perception. The current study investigates the neural sources associated with schema congruent and incongruent object-context associations within positive, negative, and neutral valence during an intermittent binocular rivalry task with simultaneous high-density EEG recording. Cortical sourceswere calculated using the sLORETA algorithm in 150 ms after stimulus onset (Stim + 150) and 400 ms before response (Resp-400) time windows. No significant difference in source activity was found between congruent and incongruent associations in any of the valence categories in the Stim + 150 ms window indicating that immediately after stimulus presentation the basic visual processing remains the same for both. In the Resp-400 ms window, different frontal regions showed higher activity for incongruent associations with different valence such as the superior frontal gyrus showed significantly higher activations for negative while the middle and medial frontal gyrus showed higher activations for neutral and finally, the inferior frontal gyrus showed higher activations for positive valence. Besides replicating the previous knowledge of frontal activations in response to context congruity, the current study provides further evidence for the sensitivity of the frontal lobe to the valence associated with the incongruent stimuli.

Zooming-in on higher-level vision: High-resolution fMRI for understanding visual perception and awareness

One of the central questions in visual neuroscience is how the sparse retinal signals leaving our eyes are transformed into a rich subjective visual experience of the world. Invasive physiology studies, which offers the highest spatial resolution, have revealed many facts about the processing of simple visual features like contrast, color, and orientation, focusing on the early visual areas. At the same time, standard human fMRI studies with comparably coarser spatial resolution have revealed more complex, functionally specialized, and category-selective responses in higher visual areas. Although the visual system is the best understood among the sensory modalities, these two areas of research remain largely segregated. High-resolution fMRI opens up a possibility for linking them. On the one hand, it allows studying how the higher-level visual functions affect the fine-scale activity in early visual areas. On the other hand, it allows discovering the fine-scale functional organization of higher visual areas and exploring their functional connectivity with visual areas lower in the hierarchy. In this review, I will discuss examples of successful work undertaken in these directions using high-resolution fMRI and discuss where this method could be applied in the future to advance our understanding of the complexity of higher-level visual processing.

Ultra-high field fMRI reveals origins of feedforward and feedback activity within laminae of human ocular dominance columns

Ultra-high field MRI can functionally image the cerebral cortex of human subjects at the submillimeter scale of cortical columns and laminae. Here, we investigate both in concert, by imaging ocular dominance columns (ODCs) in primary visual cortex (V1) across different cortical depths. We ensured that putative ODC patterns in V1 (a) are stable across runs, sessions, and scanners located in different continents, (b) have a width (~1.3 mm) expected from post-mortem and animal work and (c) are absent at the retinotopic location of the blind spot. We then dissociated the effects of bottom-up thalamo-cortical input and attentional feedback processes on activity in V1 across cortical depth. Importantly, the separation of bottom-up information flows into ODCs allowed us to validly compare attentional conditions while keeping the stimulus identical throughout the experiment. We find that, when correcting for draining vein effects and using both model-based and model-free approaches, the effect of monocular stimulation is largest at deep and middle cortical depths. Conversely, spatial attention influences BOLD activity exclusively near the pial surface. Our findings show that simultaneous interrogation of columnar and laminar dimensions of the cortical fold can dissociate thalamocortical inputs from top-down processing, and allow the investigation of their interactions without any stimulus manipulation.

Identification of internet gaming disorder individuals based on ventral tegmental area resting-state functional connectivity

Objective neuroimaging markers are imminently in need for more accurate clinical diagnosis of Internet gaming disorder (IGD). Recent neuroimaging evidence suggested that IGD is associated with abnormalities in the mesolimbic dopamine (DA) system. As the key nodes of the DA pathways, ventral tegmental area (VTA) and substantia nigra (SN) and their connected brain regions may serve as potential markers to identify IGD. Therefore, we aimed to develop optimal classifiers to identify IGD individuals by using VTA and bilateral SN resting-state functional connectivity (RSFC) patterns. A dataset including 146 adolescents (66 IGDs and 80 healthy controls (HCs)) was used to build classification models and another independent dataset including 28 subjects (14 IGDs and 14 HCs) was employed to validate the generalization ability of the models. Multi-voxel pattern analysis (MVPA) with linear support vector machine (SVM) was used to select the features. Our results demonstrated that the VTA RSFC circuits successfully identified IGD individuals (mean accuracy: 86.1%, mean sensitivity: 84.5%, mean specificity: 86.6%, the mean area under the receiver operating characteristic curve: 0.91). Furthermore, the independent generalization ability of the VTA RSFC classifier model was also satisfied (accuracy = 78.5%, sensitivity = 71.4%, specificity = 85.8%). The VTA connectivity circuits that were selected as distinguishing features were mainly included bilateral thalamus, right hippocampus, right pallidum, right temporal pole superior gyrus and bilateral temporal superior gyrus. These findings demonstrated that the potential of the resting-state neuroimaging features of VTA RSFC as objective biomarkers for the IGD clinical diagnosis in the future.

Breaking down the cocktail party: Attentional modulation of cerebral audiovisual speech processing

Recent studies utilizing electrophysiological speech envelope reconstruction have sparked renewed interest in the cocktail party effect by showing that auditory neurons entrain to selectively attended speech. Yet, the neural networks of attention to speech in naturalistic audiovisual settings with multiple sound sources remain poorly understood. We collected functional brain imaging data while participants viewed audiovisual video clips of lifelike dialogues with concurrent distracting speech in the background. Dialogues were presented in a full-factorial design, comprising task (listen to the dialogues vs. ignore them), audiovisual quality and semantic predictability. We used univariate analyses in combination with multivariate pattern analysis (MVPA) to study modulations of brain activity related to attentive processing of audiovisual speech. We found attentive speech processing to cause distinct spatiotemporal modulation profiles in distributed cortical areas including sensory and frontal-control networks. Semantic coherence modulated attention-related activation patterns in the earliest stages of auditory cortical processing, suggesting that the auditory cortex is involved in high-level speech processing. Our results corroborate views that emphasize the dynamic nature of attention, with task-specificity and context as cornerstones of the underlying neuro-cognitive mechanisms.

Intracranial Recordings Reveal Unique Shape and Timing of Responses in Human Visual Cortex during Illusory Visual Events

During binocular rivalry, perception spontaneously changes without any alteration to the visual stimulus. What neural events bring about this illusion that a constant stimulus is changing? We recorded from intracranial electrodes placed on the occipital and posterior temporal cortex of two patients with epilepsy while they experienced illusory changes of a face-house binocular-rivalry stimulus or observed a control stimulus that physically changed. We performed within-patient comparisons of broadband high-frequency responses, focusing on single epochs recorded along the ventral processing stream. We found transient face- and house-selective responses localized to the same electrodes for illusory and physical changes, but the temporal characteristics of these responses markedly differed. In comparison with physical changes, responses to illusory changes were longer lasting, in particular exhibiting a characteristic slow rise. Furthermore, the temporal order of responses across the visual hierarchy was reversed for illusory as compared to physical changes: for illusory changes, higher order fusiform and parahippocampal regions responded before lower order occipital regions. Our tentative interpretation of these findings is that two stages underlie the initiation of illusory changes: a destabilization stage in which activity associated with the impending change gradually accumulates across the visual hierarchy, ultimately graduating in a top-down cascade of activity that may stabilize the new perceptual interpretation of the stimulus.

Neural representations of perceptual color experience in the human ventral visual pathway

Color is a perceptual construct that arises from neural processing in hierarchically organized cortical visual areas. Previous research, however, often failed to distinguish between neural responses driven by stimulus chromaticity versus perceptual color experience. An unsolved question is whether the neural responses at each stage of cortical processing represent a physical stimulus or a color we see. The present study dissociated the perceptual domain of color experience from the physical domain of chromatic stimulation at each stage of cortical processing by using a switch rivalry paradigm that caused the color percept to vary over time without changing the retinal stimulation. Using functional MRI (fMRI) and a model-based encoding approach, we found that neural representations in higher visual areas, such as V4 and VO1, corresponded to the perceived color, whereas responses in early visual areas V1 and V2 were modulated by the chromatic light stimulus rather than color perception. Our findings support a transition in the ascending human ventral visual pathway, from a representation of the chromatic stimulus at the retina in early visual areas to responses that correspond to perceptually experienced colors in higher visual areas.

A hierarchical model of perceptual multistability involving interocular grouping

Ambiguous visual images can generate dynamic and stochastic switches in perceptual interpretation known as perceptual rivalry. Such dynamics have primarily been studied in the context of rivalry between two percepts, but there is growing interest in the neural mechanisms that drive rivalry between more than two percepts. In recent experiments, we showed that split images presented to each eye lead to subjects perceiving four stochastically alternating percepts (Jacot-Guillarmod et al. Vision research, 133, 37-46, 2017): two single eye images and two interocularly grouped images. Here we propose a hierarchical neural network model that exhibits dynamics consistent with our experimental observations. The model consists of two levels, with the first representing monocular activity, and the second representing activity in higher visual areas. The model produces stochastically switching solutions, whose dependence on task parameters is consistent with four generalized Levelt Propositions, and with experiments. Moreover, dynamics restricted to invariant subspaces of the model demonstrate simpler forms of bistable rivalry. Thus, our hierarchical model generalizes past, validated models of binocular rivalry. This neuromechanistic model also allows us to probe the roles of interactions between populations at the network level. Generalized Levelt's Propositions hold as long as feedback from the higher to lower visual areas is weak, and the adaptation and mutual inhibition at the higher level is not too strong. Our results suggest constraints on the architecture of the visual system and show that complex visual stimuli can be used in perceptual rivalry experiments to develop more detailed mechanistic models of perceptual processing.

Characterizing the Structural Pattern of Heavy Smokers Using Multivoxel Pattern Analysis

Background: Smoking addiction is a major public health issue which causes a series of chronic diseases and mortalities worldwide. We aimed to explore the most discriminative gray matter regions between heavy smokers and healthy controls with a data-driven multivoxel pattern analysis technique, and to explore the methodological differences between multivoxel pattern analysis and voxel-based morphometry. Methods: Traditional voxel-based morphometry has continuously contributed to finding smoking addiction-related regions on structural magnetic resonance imaging. However, voxel-based morphometry has its inherent limitations. In this study, a multivoxel pattern analysis using a searchlight algorithm and support vector machine was applied on structural magnetic resonance imaging to identify the spatial pattern of gray matter volume in heavy smokers. Results: Our proposed method yielded a voxel-wise accuracy of at least 81% for classifying heavy smokers from healthy controls. The identified regions were primarily located at the temporal cortex and prefrontal cortex, occipital cortex, thalamus (bilateral), insula (left), anterior and median cingulate gyri, and precuneus (left). Conclusions: Our results suggested that several regions, which were seldomly reported in voxel-based morphometry analysis, might be latently correlated with smoking addiction. Such findings might provide insights for understanding the mechanism of chronic smoking and the creation of effective cessation treatment. Multivoxel pattern analysis can be efficient in locating brain discriminative regions which were neglected by voxel-based morphometry.

View-Independent Working Memory Representations of Artificial Shapes in Prefrontal and Posterior Regions of the Human Brain

Traditional views of visual working memory postulate that memorized contents are stored in dorsolateral prefrontal cortex using an adaptive and flexible code. In contrast, recent studies proposed that contents are maintained by posterior brain areas using codes akin to perceptual representations. An important question is whether this reflects a difference in the level of abstraction between posterior and prefrontal representations. Here, we investigated whether neural representations of visual working memory contents are view-independent, as indicated by rotation-invariance. Using functional magnetic resonance imaging and multivariate pattern analyses, we show that when subjects memorize complex shapes, both posterior and frontal brain regions maintain the memorized contents using a rotation-invariant code. Importantly, we found the representations in frontal cortex to be localized to the frontal eye fields rather than dorsolateral prefrontal cortices. Thus, our results give evidence for the view-independent storage of complex shapes in distributed representations across posterior and frontal brain regions.

The spatiotemporal neural dynamics underlying perceived similarity for real-world objects

The degree to which we perceive real-world objects as similar or dissimilar structures our perception and guides categorization behavior. Here, we investigated the neural representations enabling perceived similarity using behavioral judgments, fMRI and MEG. As different object dimensions co-occur and partly correlate, to understand the relationship between perceived similarity and brain activity it is necessary to assess the unique role of multiple object dimensions. We thus behaviorally assessed perceived object similarity in relation to shape, function, color and background. We then used representational similarity analyses to relate these behavioral judgments to brain activity. We observed a link between each object dimension and representations in visual cortex. These representations emerged rapidly within 200 ms of stimulus onset. Assessing the unique role of each object dimension revealed partly overlapping and distributed representations: while color-related representations distinctly preceded shape-related representations both in the processing hierarchy of the ventral visual pathway and in time, several dimensions were linked to high-level ventral visual cortex. Further analysis singled out the shape dimension as neither fully accounted for by supra-category membership, nor a deep neural network trained on object categorization. Together our results comprehensively characterize the relationship between perceived similarity of key object dimensions and neural activity.

Evidence for model-based encoding of Pavlovian contingencies in the human brain

Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavlovian conditioning. We therefore performed multivoxel pattern analysis of high-resolution functional MRI data in human participants to test for the encoding of stimulus-stimulus associations that could support model-based computations during Pavlovian conditioning. We found that dissociable sub-regions of the striatum encode predictions of stimulus-stimulus associations and predictive value, in a manner that is directly related to learning performance. Activity patterns in the orbitofrontal cortex were also found to be related to stimulus-stimulus as well as value encoding. These results suggest that the brain encodes model-based representations during Pavlovian conditioning, and that these representations are utilized in the service of behavior.

Pattern classification differentiates decision of intertemporal choices using multi-voxel pattern analysis

In daily life, individuals frequently make trade-offs between the small-but-immediate benefits and large-but-delayed profits. This type of decision is known as intertemporal choice. Previous studies have uncovered the neurobiological mechanism of the intertemporal choice, but it still remains unclear how the patterns of brain activity predict the decisions of intertemporal choices. To fill this gap, we used functional magnetic resonance imaging (fMRI), in conjunction with the machine learning technique of multi-voxel pattern analysis (MVPA), to ascertain the predictive capability of the neuronal pattern for classifying individuals' intertemporal decisions across two independent samples. To further probe how this neuronal pattern worked in predicting individual intertemporal decision, we drew on the Power Atlas to examine the accuracies of classification within each regional mask as well. Classification findings showed that the pattern of neuronal activity over the whole-brain can correctly classify the accuracies of individual decisions up to 84.3%. Encouragingly, further analysis shows that the neuronal information encoded in three brain functional networks can predict individuals' decisions with significant discriminative power in cross-samples, namely the valuation network (e.g., striatum), the cognitive control network (e.g., dorsolateral prefrontal cortex) and the episodic prospection network (e.g., amygdala, parahippocampus gyrus, insula). Collectively, these findings advance our comprehension of the neuronal mechanism of human intertemporal decisions, and substantially reshape our understanding for this cardinal behaviour from behavioural-brain scheme to brain-behavioural configuration.

Existence of Initial Dip for BCI: An Illusion or Reality

A tight coupling between the neuronal activity and the cerebral blood flow (CBF) is the motivation of many hemodynamic response (HR)-based neuroimaging modalities. The increase in neuronal activity causes the increase in CBF that is indirectly measured by HR modalities. Upon functional stimulation, the HR is mainly categorized in three durations: (i) initial dip, (ii) conventional HR (i.e., positive increase in HR caused by an increase in the CBF), and (iii) undershoot. The initial dip is a change in oxygenation prior to any subsequent increase in CBF and spatially more specific to the site of neuronal activity. Despite additional evidence from various HR modalities on the presence of initial dip in human and animal species (i.e., cat, rat, and monkey); the existence/occurrence of an initial dip in HR is still under debate. This article reviews the existence and elusive nature of the initial dip duration of HR in intrinsic signal optical imaging (ISOI), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The advent of initial dip and its elusiveness factors in ISOI and fMRI studies are briefly discussed. Furthermore, the detection of initial dip and its role in brain-computer interface using fNIRS is examined in detail. The best possible application for the initial dip utilization and its future implications using fNIRS are provided.

Revisiting individual differences in the time course of binocular rivalry

Simultaneously showing an observer two incompatible displays, one to each eye, causes binocular rivalry, during which the observer regularly switches between perceiving one eye's display and perceiving the other. Observers differ in the rate of this perceptual cycle, and these individual differences have been reported to correlate with differences in the perceptual switch rate for other bistable perception phenomena. Identifying which psychological or neural factors explain this variability can help clarify the mechanisms underlying binocular rivalry and of bistable perception generally. Motivated by the prominent theory that perceptual switches during binocular rivalry are brought about by neural adaptation, we investigated whether perceptual switch rates are correlated with the strength of neural adaptation, indexed by visual aftereffects. We found no compelling evidence for such correlations. Moreover, we did not corroborate previous findings that switch rates are correlated between binocular rivalry and other forms of bistable perception. This latter nonreplication prompted us to perform a meta-analysis of existing research into correlations among forms of bistable perception, which revealed that evidence for such correlations is much weaker than is generally believed. By showing no common factor linking individual differences in binocular rivalry and in our other paradigms, these results fit well with other work that has shown such common factors to be rare among visual phenomena generally.

Brains in dialogue: decoding neural preparation of speaking to a conversational partner

In dialogue, language processing is adapted to the conversational partner. We hypothesize that the brain facilitates partner-adapted language processing through preparatory neural configurations (task sets) that are tailored to the conversational partner. In this experiment, we measured neural activity with functional magnetic resonance imaging (fMRI) while healthy participants in the scanner (a) engaged in a verbal communication task with a conversational partner outside of the scanner, or (b) spoke outside of a conversational context (to test the microphone). Using multivariate searchlight analysis, we identify cortical regions that represent information on whether speakers plan to speak to a conversational partner or without having a partner. Most notably a region that has been associated with processing social-affective information and perspective taking, the ventromedial prefrontal cortex, as well as regions that have been associated with prospective task representation, the bilateral ventral prefrontal cortex, are involved in encoding the speaking condition. Our results suggest that speakers prepare, in advance of speaking, for the social context in which they will speak.

Differentiating aversive conditioning in bistable perception: Avoidance of a percept vs. salience of a stimulus

Alternating conscious visual perception of bistable stimuli is influenced by several factors. In order to understand the effect of negative valence, we tested the effect of two types of aversive conditioning on dominance durations in binocular rivalry. Participants received either aversive classical conditioning of the stimuli shown alone between rivalry blocks, or aversive percept conditioning of one of the two possible perceptual choices during rivalry. Both groups showed successful aversive conditioning according to skin conductance responses and affective valence ratings. However, while classical conditioning led to an immediate but short-lived increase in dominance durations of the conditioned stimulus, percept conditioning yielded no significant immediate effect but tended to decrease durations of the conditioned percept during extinction. These results show dissociable effects of value learning on perceptual inference in situations of perceptual conflict, depending on whether learning relates to the decision between conflicting perceptual choices or the sensory stimuli per se.

Cortical specialization for attended versus unattended working memory

Items held in working memory can be either attended or not, depending on their current behavioral relevance. It has been suggested that unattended contents might be solely retained in an activity-silent form. Instead, we demonstrate here that encoding unattended contents involves a division of labor. While visual cortex only maintains attended items, intraparietal areas and the frontal eye fields represent both attended and unattended items.

On the primacy and irreducible nature of first-person versus third-person information

In this essay, we will support the claim that at the current level of scientific advancement a) some first-person accounts cannot be reduced to their third-person neural and psychophysiological correlates and b) that these first-person accounts are the only information to reckon when it is necessary to analyse qualia contents. Consequently, for many phenomena, first-person accounts are the only reliable source of information available and the knowledge of their neural and psychophysical correlates don't offer any additional information about them.

Combined fMRI- and eye movement-based decoding of bistable plaid motion perception

The phenomenon of bistable perception, in which perception alternates spontaneously despite constant sensory stimulation, has been particularly useful in probing the neural bases of conscious perception. The study of such bistability requires access to the observer's perceptual dynamics, which is usually achieved via active report. This report, however, constitutes a confounding factor in the study of conscious perception and can also be biased in the context of certain experimental manipulations. One approach to circumvent these problems is to track perceptual alternations using signals from the eyes or the brain instead of observers' reports. Here we aimed to optimize such decoding of perceptual alternations by combining eye and brain signals. Eye-tracking and functional magnetic resonance imaging (fMRI) was performed in twenty participants while they viewed a bistable visual plaid motion stimulus and reported perceptual alternations. Multivoxel pattern analysis (MVPA) for fMRI was combined with eye-tracking in a Support vector machine to decode participants' perceptual time courses from fMRI and eye-movement signals. While both measures individually already yielded high decoding accuracies (on average 86% and 88% correct, respectively) classification based on the two measures together further improved the accuracy (91% correct). These findings show that leveraging on both fMRI and eye movement data may pave the way for optimized no-report paradigms through improved decodability of bistable motion perception and hence for a better understanding of the neural correlates of consciousness.

A Hierarchical Bayesian Model for the Identification of PET Markers Associated to the Prediction of Surgical Outcome after Anterior Temporal Lobe Resection

We develop an integrative Bayesian predictive modeling framework that identifies individual pathological brain states based on the selection of fluoro-deoxyglucose positron emission tomography (PET) imaging biomarkers and evaluates the association of those states with a clinical outcome. We consider data from a study on temporal lobe epilepsy (TLE) patients who subsequently underwent anterior temporal lobe resection. Our modeling framework looks at the observed profiles of regional glucose metabolism in PET as the phenotypic manifestation of a latent individual pathologic state, which is assumed to vary across the population. The modeling strategy we adopt allows the identification of patient subgroups characterized by latent pathologies differentially associated to the clinical outcome of interest. It also identifies imaging biomarkers characterizing the pathological states of the subjects. In the data application, we identify a subgroup of TLE patients at high risk for post-surgical seizure recurrence after anterior temporal lobe resection, together with a set of discriminatory brain regions that can be used to distinguish the latent subgroups. We show that the proposed method achieves high cross-validated accuracy in predicting post-surgical seizure recurrence.

Dissociable Electroencephalograph Correlates of Visual Awareness and Feature-Based Attention

Background: The relationship between awareness and attention is complex and controversial. A growing body of literature has shown that the neural bases of consciousness and endogenous attention (voluntary attention) are independent. The important role of exogenous attention (reflexive attention) on conscious experience has been noted in several studies. However, exogenous attention can also modulate subliminal processing, suggesting independence between the two processes. The question of whether visual awareness and exogenous attention rely on independent mechanisms under certain circumstances remains unanswered.

Methods: In the current study, electroencephalograph recordings were conducted using 64 channels from 16 subjects while subjects attempted to detect faint speed changes of colored rotating dots. Awareness and attention were manipulated throughout trials in order to test whether exogenous attention and visual awareness rely on independent mechanisms.

Results: Neural activity related to consciousness was recorded in the following cue-locked time-windows (event related potential, cluster- based permutation test): 0–50, 150–200, and 750–800 ms. With a more liberal threshold, the inferior occipital lobe was found to be the source of awareness-related activity in the 0–50 ms range. In the later 150–200 ms range, activity in the fusiform and post-central gyrus was related to awareness. Awareness-related activation in the later 750–800 ms range was more widely distributed. This awareness-related activation pattern was quite different from that of attention. Attention-related neural activity was emphasized in the 750–800 ms time window and the main source of attention-related activity was localized to the right angular gyrus. These results suggest that exogenous attention and visual consciousness correspond to different and relatively independent neural mechanisms and are distinct processes under certain conditions.

Category Selectivity of Human Visual Cortex in Perception of Rubin Face-Vase Illusion

When viewing the Rubin face-vase illusion, our conscious perception spontaneously alternates between the face and the vase; this illusion has been widely used to explore bistable perception. Previous functional magnetic resonance imaging (fMRI) studies have studied the neural mechanisms underlying bistable perception through univariate and multivariate pattern analyses; however, no studies have investigated the issue of category selectivity. Here, we used fMRI to investigate the neural mechanisms underlying the Rubin face-vase illusion by introducing univariate amplitude and multivariate pattern analyses. The results from the amplitude analysis suggested that the activity in the fusiform face area was likely related to the subjective face perception. Furthermore, the pattern analysis results showed that the early visual cortex (EVC) and the face-selective cortex could discriminate the activity patterns of the face and vase perceptions. However, further analysis of the activity patterns showed that only the face-selective cortex contains the face information. These findings indicated that although the EVC and face-selective cortex activities could discriminate the visual information, only the activity and activity pattern in the face-selective areas contained the category information of face perception in the Rubin face-vase illusion.

Multistable Perception and the Role of the Frontoparietal Cortex in Perceptual Inference

A given pattern of optical stimulation can arise from countless possible real-world sources, creating a dilemma for vision: What in the world actually gives rise to the current pattern? This dilemma was pointed out centuries ago by the astronomer and mathematician Ibn Al-Haytham and was forcefully restated 150 years ago when von Helmholtz characterized perception as unconscious inference. To buttress his contention, von Helmholtz cited multistable perception: recurring changes in perception despite unchanging sensory input. Recent neuroscientific studies have exploited multistable perception to identify brain areas uniquely activated in association with these perceptual changes, but the specific roles of those activations remain controversial. This article provides an overview of theoretical models of multistable perception, a review of recent neuroimaging and brain stimulation studies focused on mechanisms associated with these perceptual changes, and a synthesis of available evidence within the context of current notions about Bayesian inference that find their historical roots in von Helmholtz's work.

Mining Time-Resolved Functional Brain Graphs to an EEG-Based Chronnectomic Brain Aged Index (CBAI)

The brain at rest consists of spatially and temporal distributed but functionally connected regions that called intrinsic connectivity networks (ICNs). Resting state electroencephalography (rs-EEG) is a way to characterize brain networks without confounds associated with task EEG such as task difficulty and performance. A novel framework of how to study dynamic functional connectivity under the notion of functional connectivity microstates (FCμstates) and symbolic dynamics is further discussed. Furthermore, we introduced a way to construct a single integrated dynamic functional connectivity graph (IDFCG) that preserves both the strength of the connections between every pair of sensors but also the type of dominant intrinsic coupling modes (DICM). The whole methodology is demonstrated in a significant and unexplored task for EEG which is the definition of an objective Chronnectomic Brain Aged index (CBAI) extracted from resting-state data (N = 94 subjects) with both eyes-open and eyes-closed conditions. Novel features have been defined based on symbolic dynamics and the notion of DICM and FCμstates. The transition rate of FCμstates, the symbolic dynamics based on the evolution of FCμstates (the Markovian Entropy, the complexity index), the probability distribution of DICM, the novel Flexibility Index that captures the dynamic reconfiguration of DICM per pair of EEG sensors and the relative signal power constitute a valuable pool of features that can build the proposed CBAI. Here we applied a feature selection technique and Extreme Learning Machine (ELM) classifier to discriminate young adults from middle-aged and a Support Vector Regressor to build a linear model of the actual age based on EEG-based spatio-temporal features. The most significant type of features for both prediction of age and discrimination of young vs. adults age groups was the dynamic reconfiguration of dominant coupling modes derived from a subset of EEG sensor pairs. Specifically, our results revealed a very high prediction of age for eyes-open (R² = 0.60; y = 0.79x + 8.03) and lower for eyes-closed (R² = 0.48; y = 0.71x + 10.91) while we succeeded to correctly classify young vs. middle-age group with 97.8% accuracy in eyes-open and 87.2% for eyes-closed. Our results were reproduced also in a second dataset for further external validation of the whole analysis. The proposed methodology proved valuable for the characterization of the intrinsic properties of dynamic functional connectivity through the age untangling developmental differences using EEG resting-state recordings.

Pre-coincidence brain activity predicts the perceptual outcome of streaming/bouncing motion display

When two identical visual discs move toward each other on a two-dimensional visual display, they can be perceived as either "streaming through" or "bouncing off" each other after their coincidence. Previous studies have observed a strong bias toward the streaming percept. Additionally, the incidence of the bouncing percept in this ambiguous display could be increased by various factors, such as a brief sound at the moment of coincidence and a momentary pause of the two discs. The streaming/bouncing bistable motion phenomenon has been studied intensively since its discovery. However, little is known regarding the neural basis underling the perceptual ambiguity in the classic version of the streaming/bouncing motion display. The present study investigated the neural basis of the perception disambiguating underling the processing of the streaming/bouncing bistable motion display using event-related potential (ERP) recordings. Surprisingly, the amplitude of frontal central P2 (220-260 ms) that was elicited by the moving discs ~200 ms before the coincidence of the two discs was observed to be predictive of subsequent streaming or bouncing percept. A larger P2 amplitude was observed for streaming percept than the bouncing percept. These findings suggest that the streaming/bouncing bistable perception may have been disambiguated unconsciously ~200 ms before the coincidence of the two discs.

Decoding fMRI events in sensorimotor motor network using sparse paradigm free mapping and activation likelihood estimates

Most functional MRI (fMRI) studies map task-driven brain activity using a block or event-related paradigm. Sparse paradigm free mapping (SPFM) can detect the onset and spatial distribution of BOLD events in the brain without prior timing information, but relating the detected events to brain function remains a challenge. In this study, we developed a decoding method for SPFM using a coordinate-based meta-analysis method of activation likelihood estimation (ALE). We defined meta-maps of statistically significant ALE values that correspond to types of events and calculated a summation overlap between the normalized meta-maps and SPFM maps. As a proof of concept, this framework was applied to relate SPFM-detected events in the sensorimotor network (SMN) to six motor functions (left/right fingers, left/right toes, swallowing, and eye blinks). We validated the framework using simultaneous electromyography (EMG)-fMRI experiments and motor tasks with short and long duration, and random interstimulus interval. The decoding scores were considerably lower for eye movements relative to other movement types tested. The average successful rate for short and long motor events were 77 ± 13% and 74 ± 16%, respectively, excluding eye movements. We found good agreement between the decoding results and EMG for most events and subjects, with a range in sensitivity between 55% and 100%, excluding eye movements. The proposed method was then used to classify the movement types of spontaneous single-trial events in the SMN during resting state, which produced an average successful rate of 22 ± 12%. Finally, this article discusses methodological implications and improvements to increase the decoding performance. Hum Brain Mapp 38:5778-5794, 2017. © 2017 Wiley Periodicals, Inc.

Potential for false positive results from multi-voxel pattern analysis on functional imaging data

BACKGROUND

Multi-voxel pattern analysis (MVPA) provides a powerful tool to investigate neural mechanisms for various cognitive processes under functional brain imaging. However, the high sensitivity of the MVPA method could bring about false positive results, which has been overlooked by previous research.

OBJECTIVE

We investigated the potential for obtaining false positives from the MVPA method.

METHODS

We conducted MVPA on a public functional MRI dataset on the neural encoding of various object categories. Different scenarios for pattern classification were involved by varying the number of voxels for each region of interest (ROI) and the number of object categories.

RESULTS

The classification accuracy became higher with more voxels involved, and false positive results emerged for the primary auditory cortex and even a white matter ROI, where object-related neural processing was not supposed to occur.

CONCLUSIONS

Our results imply that the classification accuracy obtained from MVPA may be inflated due to the high sensitivity of the method. Therefore, we suggest involving control ROIs in future MVPA studies and comparing the classification accuracy for a target ROI with that for a control ROI, instead of comparing the obtained accuracy with the chance-level accuracy.

Intracranial Recordings of Occipital Cortex Responses to Illusory Visual Events

Ambiguous visual stimuli elicit different perceptual interpretations over time, creating the illusion that a constant stimulus is changing. We investigate whether such spontaneous changes in visual perception involve occipital brain regions specialized for processing visual information, despite the absence of concomitant changes in stimulation. Spontaneous perceptual changes observed while viewing a binocular rivalry stimulus or an ambiguous structure-from-motion stimulus were compared with stimulus-induced perceptual changes that occurred in response to an actual stimulus change. Intracranial recordings from human occipital cortex revealed that spontaneous and stimulus-induced perceptual changes were both associated with an early transient increase in high-frequency power that was more spatially confined than a later transient decrease in low-frequency power. We suggest that the observed high-frequency and low-frequency modulations relate to initiation and maintenance of a percept, respectively. Our results are compatible with the idea that spontaneous changes in perception originate from competitive interactions within visual neural networks.

SIGNIFICANCE STATEMENT

Ambiguous visual stimuli elicit different perceptual interpretations over time, creating the illusion that a constant stimulus is changing. The literature on the neural correlates of conscious visual perception remains inconclusive regarding the extent to which such spontaneous changes in perception involve sensory brain regions. In an attempt to bridge the gap between existing animal and human studies, we recorded from intracranial electrodes placed on the human occipital lobe. We compared two different kinds of ambiguous stimuli, binocular rivalry and the phenomenon of ambiguous structure-from-motion, enabling generalization of our findings across different stimuli. Our results indicate that spontaneous and stimulus-induced changes in perception (i.e., "illusory" and "real" changes in the stimulus, respectively) may involve sensory regions to a similar extent.

fMRI-based decoding of reward effects in binocular rivalry

Binocular rivalry is a phenomenon where the simultaneous presentation of two different stimuli to the two eyes leads to alternating perception of the two stimuli. The temporary dominance of one stimulus over the other is influenced by several factors. Here, we studied the influence of reward on binocular rivalry dynamics and its neural representation in visual cortex. Orthogonal rotating grating stimuli were shown continuously, while monetary reward was given during the conscious perception of one stimulus but not the other. Periods of perceptual dominance were assessed both through participants’ subjective report and objectively using functional magnetic resonance imaging and multi-voxel pattern analysis. Results did not confirm previous evidence for an effect of reward on perceptual dominance durations. Exploratory post-hoc analyses indicated that knowledge regarding both the reward contingency and the subjective nature of perceptual alternations may have interfered with potential reward effects on perceptual phase durations, suggesting a moderating role of meta-cognitive awareness in reward-based perceptual inference. Future studies of top-down influences on bistable perception should carefully consider the methodological challenges related to meta-cognitive awareness.

Aerobic Exercise Effects on Ocular Dominance Plasticity with a Phase Combination Task in Human Adults

Several studies have shown that short-term monocular patching can induce ocular dominance plasticity in normal adults, in which the patched eye becomes stronger in binocular viewing. There is a recent study showing that exercise enhances this plasticity effect when assessed with binocular rivalry. We address one question, is this enhancement from exercise a general effect such that it is seen for measures of binocular processing other than that revealed using binocular rivalry? Using a binocular phase combination task in which we directly measure each eye's contribution to the binocularly fused percept, we show no additional effect of exercise after short-term monocular occlusion and argue that the enhancement of ocular dominance plasticity from exercise could not be demonstrated with our approach.

Extending Levelt's Propositions to perceptual multistability involving interocular grouping

Levelt's Propositions are central to understanding a wide range of multistable perceptual phenomena, but it is unclear whether they extend to perceptual multistability involving interocular grouping. We presented split-grating stimuli with complementary halves of the same color (either red or green) to human subjects. The subjects reported four percepts in alternation: the two stimuli presented to each eye (half red and half green), as well as the two single color (all red or all green), interocularly grouped percepts. Increasing color saturation lead to increased reports of the single color percept in most subjects, indicating increased predominance of grouped percepts (Levelt's Proposition I). This increase in predominance was due to a decrease in the average dominance duration of single-eye percepts, with grouped percept dominance largely unaffected. This agrees with a generalization of Levelt's Proposition II, as the average dominance duration of the stronger (in this case single-eye) percept was primarily affected by changes in stimulus strength. Moreover, in agreement with Levelt's Proposition III the alternation rate between percepts increased as the difference in the strength of the percepts decreased.

Decoding eye-of-origin outside of awareness

In the primary visual cortex of many mammals, ocular dominance columns segregate information from the two eyes. Yet under controlled conditions, most human observers are unable to correctly report the eye to which a stimulus has been shown, indicating that this information is lost during subsequent processing. This study investigates whether eye-of-origin information is available in the pattern of electrophysiological activity evoked by visual stimuli, recorded using EEG and decoded using multivariate pattern analysis. Observers (N=24) viewed sine-wave grating and plaid stimuli of different orientations, shown to either the left or right eye (or both). Using a support vector machine, eye-of-origin could be decoded above chance at around 140 and 220ms post stimulus onset, yet observers were at chance for reporting this information. Other stimulus features, such as binocularity, orientation, spatial pattern, and the presence of interocular conflict (i.e. rivalry), could also be decoded using the same techniques, though all of these were perceptually discriminable above chance. A control analysis found no evidence to support the possibility that eye dominance was responsible for the eye-of-origin effects. These results support a structural explanation for multivariate decoding of electrophysiological signals - information organised in cortical columns can be decoded, even when observers are unaware of this information.

Integrating Theoretical Models with Functional Neuroimaging

The development of mathematical models to characterize perceptual and cognitive processes dates back almost to the inception of the field of psychology. Since the 1990s, human functional neuroimaging has provided for rapid empirical and theoretical advances across a variety of domains in cognitive neuroscience. In more recent work, formal modeling and neuroimaging approaches are being successfully combined, often producing models with a level of specificity and rigor that would not have been possible by studying behavior alone. In this review, we highlight examples of recent studies that utilize this combined approach to provide novel insights into the mechanisms underlying human cognition. The studies described here span domains of perception, attention, memory, categorization, and cognitive control, employing a variety of analytic and model-inspired approaches. Across these diverse studies, a common theme is that individually tailored, creative solutions are often needed to establish compelling links between multi-parameter models and complex sets of neural data. We conclude that future developments in model-based cognitive neuroscience will have great potential to advance our theoretical understanding and ability to model both low-level and high-level cognitive processes.