Time-based reward maximization

Integrator dynamics in the cortico-basal ganglia loop underlie flexible motor timing

Flexible control of motor timing is crucial for behavior. Before volitional movement begins, the frontal cortex and striatum exhibit ramping spiking activity, with variable ramp slopes anticipating movement onsets. This activity in the cortico-basal ganglia loop may function as an adjustable 'timer,' triggering actions at the desired timing. However, because the frontal cortex and striatum share similar ramping dynamics and are both necessary for timing behaviors, distinguishing their individual roles in this timer function remains challenging. To address this, we conducted perturbation experiments combined with multi-regional electrophysiology in mice performing a flexible lick-timing task. Following transient silencing of the frontal cortex, cortical and striatal activity swiftly returned to pre-silencing levels and resumed ramping, leading to a shift in lick timing close to the silencing duration. Conversely, briefly inhibiting the striatum caused a gradual decrease in ramping activity in both regions, with ramping resuming from post-inhibition levels, shifting lick timing beyond the inhibition duration. Thus, inhibiting the frontal cortex and striatum effectively paused and rewound the timer, respectively. These findings suggest the striatum is a part of the network that temporally integrates input from the frontal cortex and generates ramping activity that regulates motor timing.

Interval timing relative to response inhibition in the differential reinforcement of low-rate responding in normally developing young adults

With recent proposal suggesting the multifaceted nature of impulsivity, researchers have been intrigued by the question of whether the impulsive behaviour measured in the traditionally psychological paradigms is unitary. One such paradigm, the differential reinforcement of low-rate responding (DRL), has been used to assess response inhibition, but its underlying mechanism has still been debated. In present research, we examined and differentiated the effects of both response inhibition and interval timing on a multisession DRL-10 s (DRL-10 s) in a large sample of normally developing young adults, as well as with three other measures including the stop-signal reaction task (SSRT), time production task-10 s (TPT-10 s), and the Barrett impulsivity scale-11 (BIS-11). The results showed that behavioural changes existed in DRL. As the task sessions progressed, there was an increase in both reinforcement probability and peak time, but a decrease in burst responses. Most importantly, both principal component analysis and generalized multilevel modeling yielded consistent results that as the task progressed, there was an increasing involvement of the TPT in the late sessions of DRL. However, none of the effect of SSRT was found. In sum, the differential degrees of involvement of the timing process, relative to response inhibition, were observed in DRL.

Mice are Near Optimal Timers

Many conventional interval timing tasks do not contain asymmetric cost (loss) functions and thereby favor high temporal accuracy. On the other hand, asymmetric cost functions that differentially penalize/reinforce the early or late responses result in adaptive biases (shift) in timed responses due to timing uncertainty. Consequently, optimal performance in these tasks entails the normative parametrization of adaptive timing biases by the level of timing uncertainty. Differential reinforcement of response duration (DRRD) is one of these tasks that require mice to actively respond (e.g., continuously depressing a lever) for a minimum amount of time to be reinforced. The active production of a time interval by mice in DRRD differentiates this task from the differential reinforcement of low rates of responding (DRL) task as a passive waiting task that was used in earlier studies to investigate the optimality of adaptive biases in timing behavior. We tested 21 Th-Cre male mice (9 weeks old) in a DRRD task with a minimum requirement of 2 s. Mean response durations were positively biased (longer than the minimum requirement), and the extent of bias was predicted by the level of endogenous timing uncertainty. Mice nearly maximized the reward rate in this task. These results contribute to the accumulating evidence supporting optimal temporal risk assessment in non-human animals.

Rodents monitor their error in self-generated duration on a single trial basis

A fundamental question in neuroscience is what type of internal representation leads to complex, adaptive behavior. When faced with a deadline, individuals' behavior suggests that they represent the mean and the uncertainty of an internal timer to make near-optimal, time-dependent decisions. Whether this ability relies on simple trial-and-error adjustments or whether it involves richer representations is unknown. Richer representations suggest a possibility of error monitoring, that is, the ability for an individual to assess its internal representation of the world and estimate discrepancy in the absence of external feedback. While rodents show timing behavior, whether they can represent and report temporal errors in their own produced duration on a single-trial basis is unknown. We designed a paradigm requiring rats to produce a target time interval and, subsequently, evaluate its error. Rats received a reward in a given location depending on the magnitude of their timing errors. During the test trials, rats had to choose a port corresponding to the error magnitude of their just-produced duration to receive a reward. High-choice accuracy demonstrates that rats kept track of the values of the timing variables on which they based their decision. Additionally, the rats kept a representation of the mapping between those timing values and the target value, as well as the history of the reinforcements. These findings demonstrate error-monitoring abilities in evaluating self-generated timing in rodents. Together, these findings suggest an explicit representation of produced duration and the possibility to evaluate its relation to the desired target duration.

The ventral hippocampus is necessary for cue-elicited, but not outcome driven approach-avoidance conflict decisions: a novel operant choice decision-making task

Approach-avoidance conflict is induced when an organism encounters a stimulus that carries both positive and negative attributes. Accumulating evidence implicates the ventral hippocampus (VH) in the detection and resolution of approach-avoidance conflict, largely on the basis of maze-based tasks assaying innate and conditioned responses to situations of conflict. However, its role in discrete trial approach-avoidance decision-making has yet to be elucidated. In this study, we designed a novel cued operant conflict decision-making task in which rats were required to choose and respond for a low reward option or high reward option paired with varying shock intensities on a differential reinforcement of low rates of responding schedule. Post training, the VH was chemogenetically inhibited while animals performed the task with the usual outcomes delivered, and with the presentation of cues associated with the reward vs. conflict options only (extinction condition). We found that VH inhibition led to an avoidance of the conflict option and longer latency to choose this option when decision-making was being made on the basis of cues alone with no outcomes. Consistent with these findings, VH-inhibited animals spent more time in the central component of the elevated plus maze (EPM), indicating a potential deficit in decision-making under innate forms of approach-avoidance conflict. Taken together, these findings implicate the VH in cue-driven approach-avoidance decisions in the face of motivational conflict.

The ventral hippocampus CA3 is critical in regulating timing uncertainty in temporal decision-making

Timing uncertainty is a critical component of temporal decision-making, as it determines the decision strategies that maximize reward rate. However, little is known about the biological substrates of timing uncertainty. In this study, we report that the CA3 subregion of the ventral hippocampus (vCA3), a relatively unexplored area in timing, is critical in regulating timing uncertainty that informs temporal decision making. Using a variant of the differential reinforcement of low rates of responding (DRL) task that incorporates differential levels of approach-avoidance conflict, rats were trained to wait a minimum of 6 s to earn a reward that was paired with varying durations of foot shock. Post-training chemogenetic inhibition of the vCA3 reduced timing uncertainty without affecting mean wait times, irrespective of the level of conflict experienced. Simulations based on the information-processing variant of scalar expectancy theory (SET) revealed that the vCA3 may be important in modulating decision threshold or switch closure latency variability.

Individual optimization of risky decisions in duration and distance estimations

Many everyday decisions require an accurate perception of how much time has passed since a previous event. Although humans estimate time intervals with a high degree of mean accuracy, the precision of estimations varies greatly between individuals. In situations in which accurate timing is rewarded but responding too early is punished, the optimal amount of risk is directly dependent on the precision of the timer. Previously, it was found that humans and rodents displayed near-optimal adjustment of their mean response time based on their individual precision and the level of punishment. It is as of yet unknown whether these strategies of optimality in interval timing are specific to the timing domain, or instead reflect an ability that generalizes to other sensorimotor modalities of decision making. Here, we address this by combining a temporal reproduction experiment and a distance estimation experiment with an identical reward scheme. We found that participants approached optimality in both tasks, but generally underadjusted their responses in the face of high risk. As this individual adjustment was consistent over modalities, these results can best be explained by assuming that the adjustment of behavior towards optimal performance is driven by a modality independent mechanism.

Numerical averaging in mice

Rodents can be trained to associate different durations with different stimuli (e.g., light/sound). When the associated stimuli are presented together, maximal responding is observed around the average of individual durations (akin to averaging). The current study investigated whether mice can also average independently trained numerosities. Mice were initially trained to make 10 or 20 lever presses on a single (run) lever to obtain a reward and each fixed-ratio schedule was signaled either with an auditory or visual stimulus. Then, mice were trained to press another lever to obtain the reward after they responded on the run lever for the minimum number of presses [Fixed Consecutive Number (FCN)-10 or -20 trials] signaled by the corresponding discriminative stimulus. Following this training, FCN trials with the compound stimulus were introduced to test the counting behavior of mice when they encountered conflicting information regarding the number of responses required to obtain the reward. Our results showed that the numbers of responses on these compound test trials were around the average of the number of responses in FCN-10 and FCN-20 trials particularly when the auditory stimulus was associated with a fewer number of required responses. The counting strategy explained the behavior of the majority of the mice in the FCN-Compound test trials (as opposed to the timing strategy). The number of responses in FCN-Compound trials was accounted for equally well by the arithmetic, geometric, and Bayesian averages of the number of responses observed in FCN-10 and FCN-20 trials.

Temporal error monitoring with directional error magnitude judgements: a robust phenomenon with no effect of being watched

A key aspect of metacognition is the ability to monitor performance. A recent line of work has shown that error-monitoring ability captures both the magnitude and direction of timing errors, thereby pointing at the metric composition of error monitoring [e.g., Akdoğan and Balcı (J Exp Psychol https://dx.doi.org/10.1037/xge0000265 , 2017)]. These studies, however, primarily used a composite variable that combined isolated measures of ordinal confidence ratings (as a proxy for error magnitude judgement) and "shorter/longer than the target" judgements. In two experiments we tested temporal error monitoring (TEM) performance with a more direct measure of directional error magnitude rating on a continuum. The second aim of this study is to test if TEM performance is modulated by the feeling of being watched that was previously shown to influence metacognitive-like monitoring processes. We predicted that being watched would improve TEM performance, particularly in participants with high timing precision (a proxy for high task mastery), and disrupt TEM performance in participants with low timing precision (a proxy for low task mastery). In both experiments, we found strong evidence for TEM ability. However, we did not find any reliable effect of the social stimulus on TEM performance. In short, our results demonstrate that metric error monitoring is a robust metacognitive phenomenon, which is not sensitive to social influence.

Discarding optimality: Throwing out the baby with the bathwater?

Rahnev & Denison (R&D) argue against normative theories and in favor of a more descriptive "standard observer model" of perceptual decision making. We agree with the authors in many respects, but we argue that optimality (specifically, reward-rate maximization) has proved demonstrably useful as a hypothesis, contrary to the authors' claims.

A Simplified Model of Communication Between Time Cells: Accounting for the Linearly Increasing Timing Imprecision

Many organisms can time intervals flexibly on average with high accuracy but substantial variability between the trials. One of the core psychophysical features of interval timing functions relates to the signatures of this timing variability; for a given individual, the standard deviation of timed responses/time estimates is nearly proportional to their central tendency (scalar property). Many studies have aimed at elucidating the neural basis of interval timing based on the neurocomputational principles in a fashion that would explain the scalar property. Recent experimental evidence shows that there is indeed a specialized neural system for timekeeping. This system, referred to as the "time cells," is composed of a group of neurons that fire sequentially as a function of elapsed time. Importantly, the time interval between consecutively firing time cell ensembles has been shown to increase with more elapsed time. However, when the subjective time is calculated by adding the distributions of time intervals between these sequentially firing time cell ensembles, the standard deviation would be compressed by the square root function. In light of this information the question becomes, "How should the signaling between the sequentially firing time cell ensembles be for the resulting variability to increase linearly with time as required by the scalar property?" We developed a simplified model of time cells that offers a mechanism for the synaptic communication of the sequentially firing neurons to address this ubiquitous property of interval timing. The model is composed of a single layer of time cells formulated in the form of integrate-and-fire neurons with feed-forward excitatory connections. The resulting behavior is simple neural wave activity. When this model is simulated with noisy conductances, the standard deviation of the time cell spike times increases proportionally to the mean of the spike-times. We demonstrate that this statistical property of the model outcomes is robustly observed even when the values of the key model parameters are varied.

Spontaneous integration of temporal information: implications for representational/computational capacity of animals

How do animals adapt their behaviors to changing conditions? This question relates to the debate between associative versus representational/computational approaches in cognitive science. An influential line of research that has significantly shaped the conceptual development of animal learning over decades has primarily focused on the role of associative dynamics with little-to-no ascription of representational/combinatorial capacities. The common assumption of these models is that behavioral adjustments are incremental and they result from updating of associations based on actions and their outcomes, without encoding the critical information serving as the determinant(s) of such contingencies (e.g., time in interval schedules, number in ratio schedules). On the other hand, an independent line of research provides evidence for behavioral phenomena that cannot be readily accounted for by the conventional associationist approach. In this paper, we will review different sets of findings particularly in the area of interval timing that suggest the ability of animals to make swift spontaneous computations on subjective quantities and incorporate them into their behavior. Findings of these studies constitute empirical challenges for the associationist approaches to behavioral flexibility. We argue that interval timing is a fertile ground for the formulation of critical tests of different theoretical approaches to animal behavior.

The role of multisensory interplay in enabling temporal expectations

Temporal regularities can guide our attention to focus on a particular moment in time and to be especially vigilant just then. Previous research provided evidence for the influence of temporal expectation on perceptual processing in unisensory auditory, visual, and tactile contexts. However, in real life we are often exposed to a complex and continuous stream of multisensory events. Here we tested - in a series of experiments - whether temporal expectations can enhance perception in multisensory contexts and whether this enhancement differs from enhancements in unisensory contexts. Our discrimination paradigm contained near-threshold targets (subject-specific 75% discrimination accuracy) embedded in a sequence of distractors. The likelihood of target occurrence (early or late) was manipulated block-wise. Furthermore, we tested whether spatial and modality-specific target uncertainty (i.e. predictable vs. unpredictable target position or modality) would affect temporal expectation (TE) measured with perceptual sensitivity (d^') and response times (RT). In all our experiments, hidden temporal regularities improved performance for expected multisensory targets. Moreover, multisensory performance was unaffected by spatial and modality-specific uncertainty, whereas unisensory TE effects on d^' but not RT were modulated by spatial and modality-specific uncertainty. Additionally, the size of the temporal expectation effect, i.e. the increase in perceptual sensitivity and decrease of RT, scaled linearly with the likelihood of expected targets. Finally, temporal expectation effects were unaffected by varying target position within the stream. Together, our results strongly suggest that participants quickly adapt to novel temporal contexts, that they benefit from multisensory (relative to unisensory) stimulation and that multisensory benefits are maximal if the stimulus-driven uncertainty is highest. We propose that enhanced informational content (i.e. multisensory stimulation) enables the robust extraction of temporal regularities which in turn boost (uni-)sensory representations.

Mice optimize timed decisions about probabilistic outcomes under deadlines

Optimal performance in temporal decisions requires the integration of timing uncertainty with environmental statistics such as probability or cost functions. Reward maximization under response deadlines constitutes one of the most stringent examples of these problems. The current study investigated whether and how mice can optimize their timing behavior in a complex experimental setting under a response deadline in which reward maximization required the integration of timing uncertainty with a geometrically increasing probability/decreasing cost function. Mice optimized their performance under seconds-long response deadlines when the underlying function was reward probability but approached this level of performance when the underlying function was reward cost, only under the assumption of logarithmically scaled subjective costs. The same subjects were then tested in a timed response inhibition task characterized by response rules that conflicted with the initial task, not responding earlier than a schedule as opposed to not missing the deadline. Irrespective of original test groups, mice optimized the timing of their inhibitory control in the second experiment. These results provide strong support for the ubiquity of optimal temporal risk assessment in mice.

The effects of payoff manipulations on temporal bisection performance

There is growing evidence that alterations in reward rates modify timing behavior demonstrating the role of motivational factors in interval timing behavior. This study aimed to investigate the effects of manipulations of rewards and penalties on temporal bisection performance in humans. Participants were trained to classify experienced time intervals as short or long based on the reference durations. Two groups of participants were tested under three different bias conditions in which either the relative reward magnitude or penalty associated with correct or incorrect categorizations of short and long reference durations was manipulated. Participants adapted their choice behavior (i.e., psychometric functions shifted) based on these payoff manipulations in directions predicted by reward maximization. The signal detection theory-based analysis of the data revealed that payoff contingencies affected the response bias parameter (B″) without altering participants' sensitivity (A') to temporal distances. Finally, the response time (RT) analysis showed that short categorization RTs increased, whereas long categorization RTs decreased as a function of stimulus durations. However, overall RTs did not exhibit any modulation in response to payoff manipulations. Taken together, this study provides additional support for the effects of motivational variables on temporal decision-making.

Mice and rats fail to integrate exogenous timing noise into their time-based decisions

Endogenous timing uncertainty results in variability in time-based judgments. In many timing tasks, animals need to incorporate their level of endogenous timing uncertainty into their decisions in order to maximize the reward rate. Although animals have been shown to adopt such optimal behavioral strategies in time-based decisions, whether they can optimize their behavior under exogenous noise is an open question. In this study, we tested mice and rats in a task that required them to space their responses for a minimum duration (DRL task) in different task conditions. In one condition, the minimum wait time was fixed, whereas in other conditions minimum wait time was a Gaussian random variable. Although reward maximization entailed waiting longer with added exogenous timing variability, results indicated that both mice and rats became more impulsive and deviated from optimality with increasing levels of exogenous noise. We introduce a reward-rate-dependent sampling function to SET to account for optimal performance in noiseless and suboptimal performance in noisy environments.

Stimulus probability effects on temporal bisection performance of mice (Mus musculus)

In the temporal bisection task, participants classify experienced stimulus durations as short or long based on their temporal similarity to previously learned reference durations. Temporal decision making in this task should be influenced by the experienced probabilities of the reference durations for adaptiveness. In this study, we tested the temporal bisection performance of mice (Mus musculus) under different short and long reference duration probability conditions implemented across two experimental phases. In Phase 1, the proportion of reference durations (compared to probe durations) was 0.5, whereas in Phase 2 it was increased to 0.8 to further examine the adjustment of choice behavior with more frequent reference duration presentations (under higher reinforcement rate). Our findings suggest that mice developed adaptive biases in their choice behaviors. These adjustments in choice behavior were nearly optimal as the mice maximized their gain to a great extent which required them to monitor stimulus probabilities as well as the level of variability in their temporal judgments. We further found that short but not long categorization response times were sensitive to stimulus probability manipulations, which in turn suggests an asymmetry between short and long categorizations. Finally, we investigated the latent decision processes underlying the bias manifested in subjects' choice behavior within the diffusion model framework. Our results revealed that probabilistic information influenced the starting point and the rate of evidence accumulation process. Overall, the stimulus probability effects on choice behavior were modulated by the reinforcement rate. Our findings illustrate that mice can adapt their temporal behaviors with respect to the probabilistic contingencies in the environment.

Optimal time discrimination

In the temporal bisection task, participants categorize experienced stimulus durations as short or long based on their similarity to previously acquired reference durations. Reward maximization in this task requires integrating endogenous timing uncertainty as well as exogenous probabilities of the reference durations into temporal judgements. We tested human participants on the temporal bisection task with different short and long reference duration probabilities (exogenous probability) in two separate test sessions. Incorrect categorizations were not penalized in Experiment 1 but were penalized in Experiment 2, leading to different levels of stringency in the reward functions that participants tried to maximize. We evaluated the judgements within the framework of optimality. Our participants adapted their choice behaviour in a nearly optimal fashion and earned nearly the maximum possible expected gain they could attain given their level of endogenous timing uncertainty and exogenous probabilities in both experiments. These results point to the optimality of human temporal risk assessment in the temporal bisection task. The long categorization response times (RTs) were overall faster than short categorization RTs, and short but not long categorization RTs were modulated by reference duration probability manipulations. These observations suggested an asymmetry between short and long categorizations in the temporal bisection task.

Everywhere and everything: The power and ubiquity of time

Anticipatory timing plays a critical role in many aspects of human and non-human animal behavior. Timing has been consistently observed in the range of milliseconds to hours, and demonstrates a powerful influence on the organization of behavior. Anticipatory timing is acquired early in associative learning and appears to guide association formation in important ways. Importantly, timing participates in regulating goal-directed behaviors in many schedules of reinforcements, and plays a critical role in value-based decision making under concurrent schedules. In addition to playing a key role in fundamental learning processes, timing often dominates when temporal cues are available concurrently with other stimulus dimensions. Such control by the passage of time has even been observed when other cues provide more accurate information and can lead to sub-optimal behaviors. The dominance of temporal cues in governing anticipatory behavior suggests that time may be inherently more salient than many other stimulus dimensions. Discussions of the interface of the timing system with other cognitive processes are provided to demonstrate the powerful and primitive nature of time as a stimulus dimension.

Optimal response rates in humans and rats

The analysis of response rates has been highly influential in psychology, giving rise to many prominent theories of learning. There is, however, growing interest in explaining response rates, not as a global response to associations or value, but as a decision about how to space responses in time. Recently, researchers have shown that humans and mice can time a single response optimally; that is, in a way that maximizes reward. Here, we use the well-established differential reinforcement of low rates (DRL) timing task to show that humans and rats come close to optimizing reinforcement rate, but respond systematically faster than they should.

Speed accuracy trade-off under response deadlines

Perceptual decision making has been successfully modeled as a process of evidence accumulation up to a threshold. In order to maximize the rewards earned for correct responses in tasks with response deadlines, participants should collapse decision thresholds dynamically during each trial so that a decision is reached before the deadline. This strategy ensures on-time responding, though at the cost of reduced accuracy, since slower decisions are based on lower thresholds and less net evidence later in a trial (compared to a constant threshold). Frazier and Yu (2008) showed that the normative rate of threshold reduction depends on deadline delays and on participants' uncertainty about these delays. Participants should start collapsing decision thresholds earlier when making decisions under shorter deadlines (for a given level of timing uncertainty) or when timing uncertainty is higher (for a given deadline). We tested these predictions using human participants in a random dot motion discrimination task. Each participant was tested in free-response, short deadline (800 ms), and long deadline conditions (1000 ms). Contrary to optimal-performance predictions, the resulting empirical function relating accuracy to response time (RT) in deadline conditions did not decline to chance level near the deadline; nor did the slight decline we typically observed relate to measures of endogenous timing uncertainty. Further, although this function did decline slightly with increasing RT, the decline was explainable by the best-fitting parameterization of Ratcliff's diffusion model (Ratcliff, 1978), whose parameters are constant within trials. Our findings suggest that at the very least, typical decision durations are too short for participants to adapt decision parameters within trials.