Dissociating task acquisition from expression during learning reveals latent knowledge

Vagus nerve stimulation during training fails to improve learning in healthy rats

Learning new skills requires neuroplasticity. Vagus nerve stimulation (VNS) during sensory and motor events can increase neuroplasticity in networks related to these events and might therefore serve to facilitate learning on sensory and motor tasks. We tested if VNS could broadly improve learning on a wide variety of tasks across different skill domains in healthy, female adult rats. VNS was paired with presentation of stimuli or on successful trials during training, strategies known to facilitate plasticity and improve recovery in models of neurological disorders. VNS failed to improve either rate of learning or performance for any of the tested tasks, which included skilled forelimb motor control, speech sound discrimination, and paired-associates learning. These results contrast recent findings from multiple labs which found VNS pairing during training produced learning enhancements across motor, auditory, and cognitive domains. We speculate that these contrasting results may be explained by key differences in task designs, training timelines and animal handling approaches, and that while VNS may be able to facilitate rapid and early learning processes in healthy subjects, it does not broadly enhance learning for difficult tasks.

A neural circuit architecture for rapid learning in goal-directed navigation

Anchoring goals to spatial representations enables flexible navigation but is challenging in novel environments when both representations must be acquired simultaneously. We propose a framework for how Drosophila uses internal representations of head direction (HD) to build goal representations upon selective thermal reinforcement. We show that flies use stochastically generated fixations and directed saccades to express heading preferences in an operant visual learning paradigm and that HD neurons are required to modify these preferences based on reinforcement. We used a symmetric visual setting to expose how flies' HD and goal representations co-evolve and how the reliability of these interacting representations impacts behavior. Finally, we describe how rapid learning of new goal headings may rest on a behavioral policy whose parameters are flexible but whose form is genetically encoded in circuit architecture. Such evolutionarily structured architectures, which enable rapidly adaptive behavior driven by internal representations, may be relevant across species.

Sex differences in discrimination behavior and orbitofrontal engagement during context-gated reward prediction

Animals, including humans, rely on contextual information to interpret ambiguous stimuli. Impaired context processing is a hallmark of several neuropsychiatric disorders, including schizophrenia, autism spectrum disorders, post-traumatic stress disorder, and addiction. While sex differences in the prevalence and manifestations of these disorders are well established, potential sex differences in context processing remain uncertain. Here, we examined sex differences in the contextual control over cue-evoked reward seeking and its neural correlates, in rats. Male and female rats were trained in a bidirectional occasion-setting preparation in which the validity of two auditory reward-predictive cues was informed by the presence, or absence, of a visual contextual feature (LIGHT: X+/DARK: X-/LIGHT: Y-/DARK: Y+). Females were significantly slower to acquire contextual control over cue-evoked reward seeking. However, once established, the contextual control over behavior was more robust in female rats; it showed less within-session variability (less influence of prior reward) and greater resistance to acute stress. This superior contextual control achieved by females was accompanied by an increased activation of the orbitofrontal cortex (OFC) compared to males. Critically, these behavioral and neural sex differences were specific to the contextual modulation process and not observed in simple, context-independent, reward prediction tasks. These results indicate a sex-biased trade-off between the speed of acquisition and the robustness of performance in the contextual modulation of cued reward seeking. The different distribution of sexes along the fast learning ↔ steady performance continuum might reflect different levels of engagement of the OFC, and might have implications for our understanding of sex differences in psychiatric disorders.

Rapid emergence of latent knowledge in the sensory cortex drives learning

Rapid learning confers significant advantages to animals in ecological environments. Despite the need for speed, animals appear to only slowly learn to associate rewarded actions with predictive cues1-4. This slow learning is thought to be supported by a gradual expansion of predictive cue representation in the sensory cortex2,5. However, evidence is growing that animals learn more rapidly than classical performance measures suggest6-8, challenging the prevailing model of sensory cortical plasticity. Here, we investigated the relationship between learning and sensory cortical representations. We trained mice on an auditory go/no-go task that dissociated the rapid acquisition of task contingencies (learning) from its slower expression (performance)7. Optogenetic silencing demonstrated that the auditory cortex (AC) drives both rapid learning and slower performance gains but becomes dispensable at expert. Rather than enhancement or expansion of cue representations9, two-photon calcium imaging of AC excitatory neurons throughout learning revealed two higher-order signals that were causal to learning and performance. First, a reward prediction (RP) signal emerged rapidly within tens of trials, was present after action-related errors only early in training, and faded at expert levels. Strikingly, silencing at the time of the RP signal impaired rapid learning, suggesting it serves an associative and teaching role. Second, a distinct cell ensemble encoded and controlled licking suppression that drove the slower performance improvements. These two ensembles were spatially clustered but uncoupled from underlying sensory representations, indicating a higher-order functional segregation within AC. Our results reveal that the sensory cortex manifests higher-order computations that separably drive rapid learning and slower performance improvements, reshaping our understanding of the fundamental role of the sensory cortex.

From innate to instructed: A new look at perceptual decision-making

Understanding how subjects perceive sensory stimuli in their environment and use this information to guide appropriate actions is a major challenge in neuroscience. To study perceptual decision-making in animals, researchers use tasks that either probe spontaneous responses to stimuli (often described as "naturalistic") or train animals to associate stimuli with experimenter-defined responses. Spontaneous decisions rely on animals' pre-existing knowledge, while trained tasks offer greater versatility, albeit often at the cost of extensive training. Here, we review emerging approaches to investigate perceptual decision-making using both spontaneous and trained behaviors, highlighting their strengths and limitations. Additionally, we propose how trained decision-making tasks could be improved to achieve faster learning and a more generalizable understanding of task rules.

Dimensionality reduction beyond neural subspaces with slice tensor component analysis

Recent work has argued that large-scale neural recordings are often well described by patterns of coactivation across neurons. Yet the view that neural variability is constrained to a fixed, low-dimensional subspace may overlook higher-dimensional structure, including stereotyped neural sequences or slowly evolving latent spaces. Here we argue that task-relevant variability in neural data can also cofluctuate over trials or time, defining distinct 'covariability classes' that may co-occur within the same dataset. To demix these covariability classes, we develop sliceTCA (slice tensor component analysis), a new unsupervised dimensionality reduction method for neural data tensors. In three example datasets, including motor cortical activity during a classic reaching task in primates and recent multiregion recordings in mice, we show that sliceTCA can capture more task-relevant structure in neural data using fewer components than traditional methods. Overall, our theoretical framework extends the classic view of low-dimensional population activity by incorporating additional classes of latent variables capturing higher-dimensional structure.

Performance errors during rodent learning reflect a dynamic choice strategy

Humans, even as infants, use cognitive strategies, such as exploration and hypothesis testing, to learn about causal interactions in the environment. In animal learning studies, however, it is challenging to disentangle higher-order behavioral strategies from errors arising from imperfect task knowledge or inherent biases. Here, we trained head-fixed mice on a wheel-based auditory two-choice task and exploited the intra- and inter-animal variability to understand the drivers of errors during learning. During learning, performance errors are dominated by a choice bias, which, despite appearing maladaptive, reflects a dynamic strategy. Early in learning, mice develop an internal model of the task contingencies such that violating their expectation of reward on correct trials (by using short blocks of non-rewarded "probe" trials) leads to an abrupt shift in strategy. During the probe block, mice behave more accurately with less bias, thereby using their learned stimulus-action knowledge to test whether the outcome contingencies have changed. Despite having this knowledge, mice continued to exhibit a strong choice bias during reinforced trials. This choice bias operates on a timescale of tens to hundreds of trials with a dynamic structure, shifting between left, right, and unbiased epochs. Biased epochs also coincided with faster motor kinematics. Although bias decreased across learning, expert mice continued to exhibit short bouts of biased choices interspersed with longer bouts of unbiased choices and higher performance. These findings collectively suggest that during learning, rodents actively probe their environment in a structured manner to refine their decision-making and maintain long-term flexibility.

Signatures of task learning in neural representations

While neural plasticity has long been studied as the basis of learning, the growth of large-scale neural recording techniques provides a unique opportunity to study how learning-induced activity changes are coordinated across neurons within the same circuit. These distributed changes can be understood through an evolution of the geometry of neural manifolds and latent dynamics underlying new computations. In parallel, studies of multi-task and continual learning in artificial neural networks hint at a tradeoff between non-interference and compositionality as guiding principles to understand how neural circuits flexibly support multiple behaviors. In this review, we highlight recent findings from both biological and artificial circuits that together form a new framework for understanding task learning at the population level.

Haploinsufficiency of Shank3 in Mice Selectively Impairs Target Odor Recognition in Novel Background Odors

Individuals with mutations in a single copy of the SHANK3 gene present with social interaction deficits. Although social behavior in mice depends on olfaction, mice with mutations in a single copy of the Shank3 gene do not have olfactory deficits in simple odor identification tasks (Drapeau et al., 2018). Here, we tested olfaction in mice with mutations in a single copy of the Shank3 gene (Peça et al., 2011) using a complex odor task and imaging in awake mice. Average glomerular responses in the olfactory bulb of Shank3B +/- were correlated with WT mice. However, there was increased trial-to-trial variability in the odor responses for Shank3B +/- mice. Simulations demonstrated that this increased variability could affect odor detection in novel environments. To test whether performance was affected by the increased variability, we tested target odor recognition in the presence of novel background odors using a recently developed task (Li et al., 2023). Head-fixed mice were trained to detect target odors in the presence of known background odors. Performance was tested using catch trials where the known background odors were replaced by novel background odors. We compared the performance of eight Shank3B +/- mice (five males, three females) on this task with six WT mice (three males, three females). Performance for known background odors and learning rates were similar between Shank3B +/- and WT mice. However, when tested with novel background odors, the performance of Shank3B +/- mice dropped to almost chance levels. Thus, haploinsufficiency of the Shank3 gene causes a specific deficit in odor detection in novel environments. Our results are discussed in the context of other Shank3 mouse models and have implications for understanding olfactory function in neurodevelopmental disorders.SIGNIFICANCE STATEMENT People and mice with mutations in a single copy in the synaptic gene Shank3 show features seen in autism spectrum disorders, including social interaction deficits. Although mice social behavior uses olfaction, mice with mutations in a single copy of Shank3 have so far not shown olfactory deficits when tested using simple tasks. Here, we used a recently developed task to show that these mice could identify odors in the presence of known background odors as well as wild-type mice. However, their performance fell below that of wild-type mice when challenged with novel background odors. This deficit was also previously reported in the Cntnap2 mouse model of autism, suggesting that odor detection in novel backgrounds is a general deficit across mouse models of autism.

Revealing abrupt transitions from goal-directed to habitual behavior

A fundamental tenet of animal behavior is that decision-making involves multiple 'controllers.' Initially, behavior is goal-directed, driven by desired outcomes, shifting later to habitual control, where cues trigger actions independent of motivational state. Clark Hull's question from 1943 still resonates today: "Is this transition abrupt, or is it gradual and progressive?"1 Despite a century-long belief in gradual transitions, this question remains unanswered2,3 as current methods cannot disambiguate goal-directed versus habitual control in real-time. Here, we introduce a novel 'volitional engagement' approach, motivating animals by palatability rather than biological need. Offering less palatable water in the home cage4,5 reduced motivation to 'work' for plain water in an auditory discrimination task when compared to water-restricted animals. Using quantitative behavior and computational modeling6, we found that palatability-driven animals learned to discriminate as quickly as water-restricted animals but exhibited state-like fluctuations when responding to the reward-predicting cue-reflecting goal-directed behavior. These fluctuations spontaneously and abruptly ceased after thousands of trials, with animals now always responding to the reward-predicting cue. In line with habitual control, post-transition behavior displayed motor automaticity, decreased error sensitivity (assessed via pupillary responses), and insensitivity to outcome devaluation. Bilateral lesions of the habit-related dorsolateral striatum7 blocked transitions to habitual behavior. Thus, 'volitional engagement' reveals spontaneous and abrupt transitions from goal-directed to habitual behavior, suggesting the involvement of a higher-level process that arbitrates between the two.

Prior experience modifies acquisition trajectories via response-strategy sampling

Few studies have considered how signal detection parameters evolve during acquisition periods. We addressed this gap by training mice with differential prior experience in a conditional discrimination, auditory signal detection task. Naïve mice, mice given separate experience with each of the later correct choice options (Correct Choice Response Transfer, CCRT), and mice experienced in conditional discriminations (Conditional Discrimination Transfer, CDT) were trained to detect the presence or absence of a tone in white noise. We analyzed data assuming a two-period model of acquisition: a pre-solution and solution period (Heinemann EG (1983) in The Presolution period and the detection of statistical associations. In: Quantitative analyses of behavior: discrimination processes, vol. 4, pp. 21-36). Ballinger. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.536.1978andrep=rep1andtype=pdf ). The pre-solution period was characterized by a selective sampling of biased response strategies until adoption of a conditional responding strategy in the solution period. Correspondingly, discriminability remained low until the solution period; criterion took excursions reflecting response-strategy sampling. Prior experience affected the length and composition of the pre-solution period. Whereas CCRT and CDT mice had shorter pre-solution periods than naïve mice, CDT and Naïve mice developed substantial criterion biases and acquired asymptotic discriminability faster than CCRT mice. To explain these data, we propose a learning model in which mice selectively sample and test different response-strategies and corresponding task structures until they exit the pre-solution period. Upon exit, mice adopt the conditional responding strategy and task structure, with action values updated via inference and generalization from the other task structures. Simulations of representative mouse data illustrate the viability of this model.

Continual task learning in natural and artificial agents

How do humans and other animals learn new tasks? A wave of brain recording studies has investigated how neural representations change during task learning, with a focus on how tasks can be acquired and coded in ways that minimise mutual interference. We review recent work that has explored the geometry and dimensionality of neural task representations in neocortex, and computational models that have exploited these findings to understand how the brain may partition knowledge between tasks. We discuss how ideas from machine learning, including those that combine supervised and unsupervised learning, are helping neuroscientists understand how natural tasks are learned and coded in biological brains.

Robust odor identification in novel olfactory environments in mice

Relevant odors signaling food, mates, or predators can be masked by unpredictable mixtures of less relevant background odors. Here, we developed a mouse behavioral paradigm to test the role played by the novelty of the background odors. During the task, mice identified target odors in previously learned background odors and were challenged by catch trials with novel background odors, a task similar to visual CAPTCHA. Female wild-type (WT) mice could accurately identify known targets in novel background odors. WT mice performance was higher than linear classifiers and the nearest neighbor classifier trained using olfactory bulb glomerular activation patterns. Performance was more consistent with an odor deconvolution method. We also used our task to investigate the performance of female Cntnap2^-/- mice, which show some autism-like behaviors. Cntnap2^-/- mice had glomerular activation patterns similar to WT mice and matched WT mice target detection for known background odors. However, Cntnap2^-/- mice performance fell almost to chance levels in the presence of novel backgrounds. Our findings suggest that mice use a robust algorithm for detecting odors in novel environments and this computation is impaired in Cntnap2^-/- mice.

Cortical sensory processing across motivational states during goal-directed behavior

Behavioral states can influence performance of goal-directed sensorimotor tasks. Yet, it is unclear how altered neuronal sensory representations in these states relate to task performance and learning. We trained water-restricted mice in a two-whisker discrimination task to study cortical circuits underlying perceptual decision-making under different levels of thirst. We identified somatosensory cortices as well as the premotor cortex as part of the circuit necessary for task execution. Two-photon calcium imaging in these areas identified populations selective to sensory or motor events. Analysis of task performance during individual sessions revealed distinct behavioral states induced by decreasing levels of thirst-related motivation. Learning was better explained by improvements in motivational state control rather than sensorimotor association. Whisker sensory representations in the cortex were altered across behavioral states. In particular, whisker stimuli could be better decoded from neuronal activity during high task performance states, suggesting that state-dependent changes of sensory processing influence decision-making.

Removal of reinforcement improves instrumental performance in humans by decreasing a general action bias rather than unmasking learnt associations

Performance during instrumental learning is commonly believed to reflect the knowledge that has been acquired up to that point. However, recent work in rodents found that instrumental performance was enhanced during periods when reinforcement was withheld, relative to periods when reinforcement was provided. This suggests that reinforcement may mask acquired knowledge and lead to impaired performance. In the present study, we investigated whether such a beneficial effect of removing reinforcement translates to humans. Specifically, we tested whether performance during learning was improved during non-reinforced relative to reinforced task periods using signal detection theory and a computational modelling approach. To this end, 60 healthy volunteers performed a novel visual go/no-go learning task with deterministic reinforcement. To probe acquired knowledge in the absence of reinforcement, we interspersed blocks without feedback. In these non-reinforced task blocks, we found an increased d', indicative of enhanced instrumental performance. However, computational modelling showed that this improvement in performance was not due to an increased sensitivity of decision making to learnt values, but to a more cautious mode of responding, as evidenced by a reduction of a general response bias. Together with an initial tendency to act, this is sufficient to drive differential changes in hit and false alarm rates that jointly lead to an increased d'. To conclude, the improved instrumental performance in the absence of reinforcement observed in studies using asymmetrically reinforced go/no-go tasks may reflect a change in response bias rather than unmasking latent knowledge.

Vocalization categorization behavior explained by a feature-based auditory categorization model

Vocal animals produce multiple categories of calls with high between- and within-subject variability, over which listeners must generalize to accomplish call categorization. The behavioral strategies and neural mechanisms that support this ability to generalize are largely unexplored. We previously proposed a theoretical model that accomplished call categorization by detecting features of intermediate complexity that best contrasted each call category from all other categories. We further demonstrated that some neural responses in the primary auditory cortex were consistent with such a model. Here, we asked whether a feature-based model could predict call categorization behavior. We trained both the model and guinea pigs (GPs) on call categorization tasks using natural calls. We then tested categorization by the model and GPs using temporally and spectrally altered calls. Both the model and GPs were surprisingly resilient to temporal manipulations, but sensitive to moderate frequency shifts. Critically, the model predicted about 50% of the variance in GP behavior. By adopting different model training strategies and examining features that contributed to solving specific tasks, we could gain insight into possible strategies used by animals to categorize calls. Our results validate a model that uses the detection of intermediate-complexity contrastive features to accomplish call categorization.

Training protocol for probabilistic Pavlovian conditioning in mice using an open-source head-fixed setup

High throughput, temporally controlled, reproducible quantitative behavioral assays are important for understanding the neural mechanisms underlying behavior. Here, we provide a step-by-step training protocol for a probabilistic Pavlovian conditioning task, where two auditory cues predict probabilistic outcomes with different contingencies. This protocol allows us to study the differential behavioral and neuronal correlates of expected and surprising outcomes. It has been tested in combination with chronic in vivo electrophysiological recordings and optogenetic manipulations in ChAT-Cre and PV-Cre mouse lines.

For complete details on the use and execution of this protocol, please refer to Hegedüs et al. (2021).

•

We provide a training protocol for a probabilistic Pavlovian conditioning task in mice

•

Two auditory cues predict probabilistic outcomes with different contingencies

•

Possible to combine with chronic in vivo electrophysiology and optogenetics

•

Ideal for testing behavioral and neural correlates of expected and surprising outcomes

High-precision coding in visual cortex

Individual neurons in visual cortex provide the brain with unreliable estimates of visual features. It is not known whether the single-neuron variability is correlated across large neural populations, thus impairing the global encoding of stimuli. We recorded simultaneously from up to 50,000 neurons in mouse primary visual cortex (V1) and in higher order visual areas and measured stimulus discrimination thresholds of 0.35° and 0.37°, respectively, in an orientation decoding task. These neural thresholds were almost 100 times smaller than the behavioral discrimination thresholds reported in mice. This discrepancy could not be explained by stimulus properties or arousal states. Furthermore, behavioral variability during a sensory discrimination task could not be explained by neural variability in V1. Instead, behavior-related neural activity arose dynamically across a network of non-sensory brain areas. These results imply that perceptual discrimination in mice is limited by downstream decoders, not by neural noise in sensory representations.

Probing relationships between reinforcement learning and simple behavioral strategies to understand probabilistic reward learning

BACKGROUND

Reinforcement learning (RL) and win stay/lose shift model accounts of decision making are both widely used to describe how individuals learn about and interact with rewarding environments. Though mutually informative, these accounts are often conceptualized as independent processes and so the potential relationships between win stay/lose shift tendencies and RL parameters have not been explored.

NEW METHOD

We introduce a methodology to directly relate RL parameters to behavioral strategy. Specifically, by calculating a truncated multivariate normal distribution of RL parameters given win stay/lose shift tendencies from simulating these tendencies across the parameter space, we maximize the normal distribution for a given set of win stay/lose shift tendencies to approximate reinforcement learning parameters.

RESULTS

We demonstrate novel relationships between win stay/lose shift tendencies and RL parameters that challenge conventional interpretations of lose shift as a metric of loss sensitivity. Further, we demonstrate in both simulated and empirical data that this method of parameter approximation yields reliable parameter recovery.

COMPARISON WITH EXISTING METHOD

We compare this method against the conventionally used maximum likelihood estimation method for parameter approximation in simulated noisy and empirical data. For simulated noisy data, we show that this method performs similarly to maximum likelihood estimation. For empirical data, however, this method provides a more reliable approximation of reinforcement learning parameters than maximum likelihood estimation.

CONCLUSIONS

We demonstrate the existence of relationships between win stay/lose shift tendencies and RL parameters and introduce a method that leverages these relationships to enable recovery of RL parameters exclusively from win stay/lose shift tendencies.

Temporal chunking as a mechanism for unsupervised learning of task-sets

Depending on environmental demands, humans can learn and exploit multiple concurrent sets of stimulus-response associations. Mechanisms underlying the learning of such task-sets remain unknown. Here we investigate the hypothesis that task-set learning relies on unsupervised chunking of stimulus-response associations that occur in temporal proximity. We examine behavioral and neural data from a task-set learning experiment using a network model. We first show that task-set learning can be achieved provided the timescale of chunking is slower than the timescale of stimulus-response learning. Fitting the model to behavioral data on a subject-by-subject basis confirmed this expectation and led to specific predictions linking chunking and task-set retrieval that were borne out by behavioral performance and reaction times. Comparing the model activity with BOLD signal allowed us to identify neural correlates of task-set retrieval in a functional network involving ventral and dorsal prefrontal cortex, with the dorsal system preferentially engaged when retrievals are used to improve performance.

V1 microcircuits underlying mouse visual behavior

Visual behavior is based on the concerted activity of neurons in visual areas, where sensory signals are integrated with top-down information. In the past decade, the advent of new tools, such as functional imaging of populations of identified single neurons, high-density electrophysiology, virus-assisted circuit mapping, and precisely timed, cell-type specific manipulations, has advanced our understanding of the neuronal microcircuits underlying visual behavior. Studies in head-fixed mice, where such tools can routinely be applied, begin to provide new insights into the neural code of primary visual cortex (V1) underlying visual perception, and the micro-circuits of attention, predictive processing, and learning.