Occipitotemporal representations reflect individual differences in conceptual knowledge

Prototype or Exemplar Representations in the 5/5 Category Learning Task

Theories of category learning have typically focused on how the underlying category structure affects the category representations acquired by learners. However, there is limited research as to how other factors affect what representations are learned and utilized and how representations might change across the time course of learning. We used a novel "5/5" categorization task developed from the well-studied 5/4 task with the addition of one more stimulus to clarify an ambiguity in the 5/4 prototypes. We used multiple methods including computational modeling to identify whether participants categorized on the basis of exemplar or prototype representations. We found that, overall, for the stimuli we used (schematic robot-like stimuli), learning was best characterized by the use of prototypes. Most importantly, we found that relative use of prototype and exemplar strategies changed across learning, with use of exemplar representations decreasing and prototype representations increasing across blocks.

Reconciling category exceptions through representational shifts

Real-world categories often contain exceptions that disobey the perceptual regularities followed by other members. Prominent psychological and neurobiological theories indicate that exception learning relies on the flexible modulation of object representations, but the specific representational shifts key to learning remain poorly understood. Here, we leveraged behavioral and computational approaches to elucidate the representational dynamics during the acquisition of exceptions that violate established regularity knowledge. In our study, participants (n = 42) learned novel categories in which regular and exceptional items were introduced successively; we then fitted a computational model to individuals' categorization performance to infer latent stimulus representations before and after exception learning. We found that in the representational space, exception learning not only drove confusable exceptions to be differentiated from regular items, but also led exceptions within the same category to be integrated based on shared characteristics. These shifts resulted in distinct representational clusters of regular items and exceptions that constituted hierarchically structured category representations, and the distinct clustering of exceptions from regular items was associated with a high ability to generalize and reconcile knowledge of regularities and exceptions. Moreover, by having a second group of participants (n = 42) to judge stimuli's similarity before and after exception learning, we revealed misalignment between representational similarity and behavioral similarity judgments, which further highlights the hierarchical layouts of categories with regularities and exceptions. Altogether, our findings elucidate the representational dynamics giving rise to generalizable category structures that reconcile perceptually inconsistent category members, thereby advancing the understanding of knowledge formation.

Distinct hippocampal mechanisms support concept formation and updating

Learning systems must constantly decide whether to create new representations or update existing ones. For example, a child learning that a bat is a mammal and not a bird would be best served by creating a new representation, whereas updating may be best when encountering a second similar bat. Characterizing the neural dynamics that underlie these complementary memory operations requires identifying the exact moments when each operation occurs. We address this challenge by interrogating fMRI brain activation with a computational learning model that predicts trial-by-trial when memories are created versus updated. We found distinct neural engagement in anterior hippocampus and ventral striatum for model-predicted memory create and update events during early learning. Notably, the degree of this effect in hippocampus, but not ventral striatum, significantly related to learning outcome. Hippocampus additionally showed distinct patterns of functional coactivation with ventromedial prefrontal cortex and angular gyrus during memory creation and premotor cortex during memory updating. These findings suggest that complementary memory functions, as formalized in computational learning models, underlie the rapid formation of novel conceptual knowledge, with the hippocampus and its interactions with frontoparietal circuits playing a crucial role in successful learning.

Significance statement

How do we reconcile new experiences with existing knowledge? Prominent theories suggest that novel information is either captured by creating new memories or leveraged to update existing memories, yet empirical support of how these distinct memory operations unfold during learning is limited. Here, we combine computational modeling of human learning behaviour with functional neuroimaging to identify moments of memory formation and updating and characterize their neural signatures. We find that both hippocampus and ventral striatum are distinctly engaged when memories are created versus updated; however, it is only hippocampus activation that is associated with learning outcomes. Our findings motivate a key theoretical revision that positions hippocampus is a key player in building organized memories from the earliest moments of learning.

Testing cognitive theories with multivariate pattern analysis of neuroimaging data

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

Degrees of algorithmic equivalence between the brain and its DNN models

Deep neural networks (DNNs) have become powerful and increasingly ubiquitous tools to model human cognition, and often produce similar behaviors. For example, with their hierarchical, brain-inspired organization of computations, DNNs apparently categorize real-world images in the same way as humans do. Does this imply that their categorization algorithms are also similar? We have framed the question with three embedded degrees that progressively constrain algorithmic similarity evaluations: equivalence of (i) behavioral/brain responses, which is current practice, (ii) the stimulus features that are processed to produce these outcomes, which is more constraining, and (iii) the algorithms that process these shared features, the ultimate goal. To improve DNNs as models of cognition, we develop for each degree an increasingly constrained benchmark that specifies the epistemological conditions for the considered equivalence.

Distributed Neural Systems Support Flexible Attention Updating during Category Learning

To accurately categorize items, humans learn to selectively attend to stimulus dimensions that are most relevant to the task. Models of category learning describe the interconnected cognitive processes that contribute to attentional tuning as labeled stimuli are progressively observed. The Adaptive Attention Representation Model (AARM), for example, provides an account whereby categorization decisions are based on the perceptual similarity of a new stimulus to stored exemplars, and dimension-wise attention is updated on every trial in the direction of a feedback-based error gradient. As such, attention modulation as described by AARM requires interactions among orienting, visual perception, memory retrieval, prediction error, and goal maintenance to facilitate learning across trials. The current study explored the neural bases of attention mechanisms using quantitative predictions from AARM to analyze behavioral and fMRI data collected while participants learned novel categories. Generalized linear model analyses revealed patterns of BOLD activation in the parietal cortex (orienting), visual cortex (perception), medial temporal lobe (memory retrieval), basal ganglia (prediction error), and pFC (goal maintenance) that covaried with the magnitude of model-predicted attentional tuning. Results are consistent with AARM's specification of attention modulation as a dynamic property of distributed cognitive systems.

Bidirectional influences of information sampling and concept learning

Contemporary models of categorization typically tend to sidestep the problem of how information is initially encoded during decision making. Instead, a focus of this work has been to investigate how, through selective attention, stimulus representations are "contorted" such that behaviorally relevant dimensions are accentuated (or "stretched"), and the representations of irrelevant dimensions are ignored (or "compressed"). In high-dimensional real-world environments, it is computationally infeasible to sample all available information, and human decision makers selectively sample information from sources expected to provide relevant information. To address these and other shortcomings, we develop an active sampling model, Sampling Emergent Attention (SEA), which sequentially and strategically samples information sources until the expected cost of information exceeds the expected benefit. The model specifies the interplay of two components, one involved in determining the expected utility of different information sources and the other in representing knowledge and beliefs about the environment. These two components interact such that knowledge of the world guides information sampling, and what is sampled updates knowledge. Like human decision makers, the model displays strategic sampling behavior, such as terminating information search when sufficient information has been sampled and adaptively adjusting the search path in response to previously sampled information. The model also shows human-like failure modes. For example, when information exploitation is prioritized over exploration, the bidirectional influences between information sampling and learning can lead to the development of beliefs that systematically differ from reality. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Top-Down Attention Guidance Shapes Action Encoding in the pSTS

The posterior superior temporal sulcus (pSTS) is a brain region characterized by perceptual representations of human body actions that promote the understanding of observed behavior. Increasingly, action observation is recognized as being strongly shaped by the expectations of the observer (Kilner 2011; Koster-Hale and Saxe 2013; Patel et al. 2019). Therefore, to characterize top-down influences on action observation, we evaluated the statistical structure of multivariate activation patterns from the action observation network (AON) while observers attended to the different dimensions of action vignettes (the action kinematics, goal, or identity of avatars jumping or crouching). Decoding accuracy varied as a function of attention instruction in the right pSTS and left inferior frontal cortex (IFC), with the right pSTS classifying actions most accurately when observers attended to the action kinematics and the left IFC classifying most accurately when observed attended to the actor's goal. Functional connectivity also increased between the right pSTS and right IFC when observers attended to the actions portrayed in the vignettes. Our findings are evidence that the attentive state of the viewer modulates sensory representations in the pSTS, consistent with proposals that the pSTS occupies an interstitial zone mediating top-down context and bottom-up perceptual cues during action observation.

Identifying the neural dynamics of category decisions with computational model-based functional magnetic resonance imaging

Successful categorization requires a careful coordination of attention, representation, and decision making. Comprehensive theories that span levels of analysis are key to understanding the computational and neural dynamics of categorization. Here, we build on recent work linking neural representations of category learning to computational models to investigate how category decision making is driven by neural signals across the brain. We uniquely combine functional magnetic resonance imaging with drift diffusion and exemplar-based categorization models to show that trial-by-trial fluctuations in neural activation from regions of occipital, cingulate, and lateral prefrontal cortices are linked to category decisions. Notably, only lateral prefrontal cortex activation was associated with exemplar-based model predictions of trial-by-trial category evidence. We propose that these brain regions underlie distinct functions that contribute to successful category learning.

The Costs and Benefits of Goal-Directed Attention in Deep Convolutional Neural Networks

People deploy top-down, goal-directed attention to accomplish tasks, such as finding lost keys. By tuning the visual system to relevant information sources, object recognition can become more efficient (a benefit) and more biased toward the target (a potential cost). Motivated by selective attention in categorisation models, we developed a goal-directed attention mechanism that can process naturalistic (photographic) stimuli. Our attention mechanism can be incorporated into any existing deep convolutional neural networks (DCNNs). The processing stages in DCNNs have been related to ventral visual stream. In that light, our attentional mechanism incorporates top-down influences from prefrontal cortex (PFC) to support goal-directed behaviour. Akin to how attention weights in categorisation models warp representational spaces, we introduce a layer of attention weights to the mid-level of a DCNN that amplify or attenuate activity to further a goal. We evaluated the attentional mechanism using photographic stimuli, varying the attentional target. We found that increasing goal-directed attention has benefits (increasing hit rates) and costs (increasing false alarm rates). At a moderate level, attention improves sensitivity (i.e. increases d ' ) at only a moderate increase in bias for tasks involving standard images, blended images and natural adversarial images chosen to fool DCNNs. These results suggest that goal-directed attention can reconfigure general-purpose DCNNs to better suit the current task goal, much like PFC modulates activity along the ventral stream. In addition to being more parsimonious and brain consistent, the mid-level attention approach performed better than a standard machine learning approach for transfer learning, namely retraining the final network layer to accommodate the new task.

Abstract Neural Representations of Category Membership beyond Information Coding Stimulus or Response

For decades, researchers have debated whether mental representations are symbolic or grounded in sensory inputs and motor programs. Certainly, aspects of mental representations are grounded. However, does the brain also contain abstract concept representations that mediate between perception and action in a flexible manner not tied to the details of sensory inputs and motor programs? Such conceptual pointers would be useful when concepts remain constant despite changes in appearance and associated actions. We evaluated whether human participants acquire such representations using fMRI. Participants completed a probabilistic concept learning task in which sensory, motor, and category variables were not perfectly coupled or entirely independent, making it possible to observe evidence for abstract representations or purely grounded representations. To assess how the learned concept structure is represented in the brain, we examined brain regions implicated in flexible cognition (e.g., pFC and parietal cortex) that are most likely to encode an abstract representation removed from sensory-motor details. We also examined sensory-motor regions that might encode grounded sensory-motor-based representations tuned for categorization. Using a cognitive model to estimate participants' category rule and multivariate pattern analysis of fMRI data, we found the left pFC and MT coded for category in the absence of information coding for stimulus or response. Because category was based on the stimulus, finding an abstract representation of category was not inevitable. Our results suggest that certain brain areas support categorization behavior by constructing concept representations in a format akin to a symbol that differs from stimulus-motor codes.

Subjective value and decision entropy are jointly encoded by aligned gradients across the human brain

Recent work has considered the relationship between value and confidence in both behavioural and neural representation. Here we evaluated whether the brain organises value and confidence signals in a systematic fashion that reflects the overall desirability of options. If so, regions that respond to either increases or decreases in both value and confidence should be widespread. We strongly confirmed these predictions through a model-based fMRI analysis of a mixed gambles task that assessed subjective value (SV) and inverse decision entropy (iDE), which is related to confidence. Purported value areas more strongly signalled iDE than SV, underscoring how intertwined value and confidence are. A gradient tied to the desirability of actions transitioned from positive SV and iDE in ventromedial prefrontal cortex to negative SV and iDE in dorsal medial prefrontal cortex. This alignment of SV and iDE signals could support retrospective evaluation to guide learning and subsequent decisions.

Explicit Categorization Goals Affect Attention-Related Processing of Race and Gender During Person Construal

Faces are categorized by gender and race very quickly, seemingly without regard to perceivers' goals or motivations, suggesting an automaticity to these judgments that has downstream consequences for evaluations, stereotypes, and social interactions. The current study investigated the extent to which early neurocognitive processes involved in the categorization of faces vary when participants' tasks goals were to categorize faces by race or by gender. In contrast to previous findings, task-related differences were found, such that differentiation in the P2 event-related potential (ERP) according to perceived gender was facilitated by having an explicit task goal of categorizing faces by gender; however, the P2 was sensitive to race regardless of task goals. Use of principal components analysis (PCA) revealed two underlying components that comprised the P2 and that were differentially sensitive to the gender and race of the faces, depending on participants' top-down task goals. Results suggest that top-down task demands facilitate discrimination of faces along the attended dimension within less than 200 ms, but that the effect of top-down task demands may not be evident when examining early ERP components that reflect more than one distinct underlying process.

Is an artificial limb embodied as a hand? Brain decoding in prosthetic limb users

The potential ability of the human brain to represent an artificial limb as a body part (embodiment) has been inspiring engineers, clinicians, and scientists as a means to optimise human-machine interfaces. Using functional MRI (fMRI), we studied whether neural embodiment actually occurs in prosthesis users' occipitotemporal cortex (OTC). Compared with controls, different prostheses types were visually represented more similarly to each other, relative to hands and tools, indicating the emergence of a dissociated prosthesis categorisation. Greater daily life prosthesis usage correlated positively with greater prosthesis categorisation. Moreover, when comparing prosthesis users' representation of their own prosthesis to controls' representation of a similar looking prosthesis, prosthesis users represented their own prosthesis more dissimilarly to hands, challenging current views of visual prosthesis embodiment. Our results reveal a use-dependent neural correlate for wearable technology adoption, demonstrating adaptive use-related plasticity within the OTC. Because these neural correlates were independent of the prostheses' appearance and control, our findings offer new opportunities for prosthesis design by lifting restrictions imposed by the embodiment theory for artificial limbs.

Ventromedial prefrontal cortex compression during concept learning

Prefrontal cortex (PFC) is thought to support the ability to focus on goal-relevant information by filtering out irrelevant information, a process akin to dimensionality reduction. Here, we test this dimensionality reduction hypothesis by relating a data-driven approach to characterizing the complexity of neural representation with a theoretically-supported computational model of learning. We find evidence of goal-directed dimensionality reduction within human ventromedial PFC during learning. Importantly, by using computational predictions of each participant’s attentional strategies during learning, we find that that the degree of neural compression predicts an individual’s ability to selectively attend to concept-specific information. These findings suggest a domain-general mechanism of learning through compression in ventromedial PFC.

Efficient learning is akin to goal-directed dimensionality reduction, in which relevant information is highlighted and irrelevant input is ignored. Here, the authors show that ventromedial prefrontal cortex uniquely supports such learning by compressing neural codes to represent goal-specific information.

Measures of Neural Similarity

One fundamental question is what makes two brain states similar. For example, what makes the activity in visual cortex elicited from viewing a robin similar to a sparrow? One common assumption in fMRI analysis is that neural similarity is described by Pearson correlation. However, there are a host of other possibilities, including Minkowski and Mahalanobis measures, with each differing in its mathematical, theoretical, and neural computational assumptions. Moreover, the operable measures may vary across brain regions and tasks. Here, we evaluated which of several competing similarity measures best captured neural similarity. Our technique uses a decoding approach to assess the information present in a brain region, and the similarity measures that best correspond to the classifier’s confusion matrix are preferred. Across two published fMRI datasets, we found the preferred neural similarity measures were common across brain regions but differed across tasks. Moreover, Pearson correlation was consistently surpassed by alternatives.

Occipitotemporal representations reflect individual differences in conceptual knowledge

Through selective attention, decision-makers can learn to ignore behaviorally irrelevant stimulus dimensions. This can improve learning and increase the perceptual discriminability of relevant stimulus information. Across cognitive models of categorization, this is typically accomplished through the inclusion of attentional parameters, which provide information about the importance assigned to each stimulus dimension by each participant. The effect of these parameters on psychological representation is often described geometrically, such that perceptual differences over relevant psychological dimensions are accentuated (or stretched), and differences over irrelevant dimensions are down-weighted (or compressed). In sensory and association cortex, representations of stimulus features are known to covary with their behavioral relevance. Although this implies that neural representational space might closely resemble that hypothesized by formal categorization theory, to date, attentional effects in the brain have been demonstrated through powerful experimental manipulations (e.g., contrasts between relevant and irrelevant features). This approach sidesteps the role of idiosyncratic conceptual knowledge in guiding attention to useful information sources. To bridge this divide, we used formal categorization models, which were fit to behavioral data, to make inferences about the concepts and strategies used by individual participants during decision-making. We found that when greater attentional weight was devoted to a particular visual feature (e.g., "color"), its value (e.g., "red") was more accurately decoded from occipitotemporal cortex. We also found that this effect was sufficiently sensitive to reflect individual differences in conceptual knowledge, indicating that occipitotemporal stimulus representations are embedded within a space closely resembling that formalized by classic categorization theory. (PsycINFO Database Record (c) 2019 APA, all rights reserved).