An economic perspective on addiction and matching

Nucleus accumbens and delay discounting in rats: evidence from a new quantitative protocol for analysing inter-temporal choice

RATIONALE

There is evidence that the core of the nucleus accumbens (AcbC) is involved in inter-temporal choice behaviour.

OBJECTIVE

A new behavioural protocol was used to examine the effect of destruction of the AcbC on delay discounting in inter-temporal choice schedules in rats.

METHOD

Rats with excitotoxic lesions of the AcbC or sham lesions made repeated choices on an adjusting-delay schedule between a smaller reinforcer (A) that was delivered immediately and a larger reinforcer (B) that was delivered after a delay which increased or decreased depending on the subject's choices. In two phases of the experiment, reinforcer sizes were selected which enabled theoretical parameters expressing delay discounting and sensitivity to reinforcer size to be estimated from the ratio of the indifference delays (i.e. the quasi-stable values of the adjusting delay seen after extended training) obtained in the two phases.

RESULTS

In both groups, indifference delays were shorter when the sizes of A and B were 14 and 25 μl than when they were 25 and 100 μl of a 0.6 M sucrose solution. Indifference delays were shorter in AcbC-lesioned than in sham-lesioned rats. Estimates of delay discounting rate based on the ratio of the indifference delays were lower in the AcbC-lesioned than in the sham-lesioned rats. The size sensitivity parameter did not differ between the groups. Adjusting delays in successive blocks of trials were analysed using Fourier transform. The period corresponding to the dominant frequency of the power spectrum and power within the dominant frequency band did not differ between the groups.

CONCLUSIONS

Destruction of the AcbC increased the rate of delay discounting.

Temporal-difference reinforcement learning with distributed representations

Temporal-difference (TD) algorithms have been proposed as models of reinforcement learning (RL). We examine two issues of distributed representation in these TD algorithms: distributed representations of belief and distributed discounting factors. Distributed representation of belief allows the believed state of the world to distribute across sets of equivalent states. Distributed exponential discounting factors produce hyperbolic discounting in the behavior of the agent itself. We examine these issues in the context of a TD RL model in which state-belief is distributed over a set of exponentially-discounting "micro-Agents", each of which has a separate discounting factor (gamma). Each microAgent maintains an independent hypothesis about the state of the world, and a separate value-estimate of taking actions within that hypothesized state. The overall agent thus instantiates a flexible representation of an evolving world-state. As with other TD models, the value-error (delta) signal within the model matches dopamine signals recorded from animals in standard conditioning reward-paradigms. The distributed representation of belief provides an explanation for the decrease in dopamine at the conditioned stimulus seen in overtrained animals, for the differences between trace and delay conditioning, and for transient bursts of dopamine seen at movement initiation. Because each microAgent also includes its own exponential discounting factor, the overall agent shows hyperbolic discounting, consistent with behavioral experiments.