Application of Herrnstein's hyperbola to time allocation of naturalistic human behavior maintained by naturalistic social reinforcement

Undermatching Is a Consequence of Policy Compression

The matching law describes the tendency of agents to match the ratio of choices allocated to the ratio of rewards received when choosing among multiple options (Herrnstein, 1961). Perfect matching, however, is infrequently observed. Instead, agents tend to undermatch or bias choices toward the poorer option. Overmatching, or the tendency to bias choices toward the richer option, is rarely observed. Despite the ubiquity of undermatching, it has received an inadequate normative justification. Here, we assume agents not only seek to maximize reward, but also seek to minimize cognitive cost, which we formalize as policy complexity (the mutual information between actions and states of the environment). Policy complexity measures the extent to which the policy of an agent is state dependent. Our theory states that capacity-constrained agents (i.e., agents that must compress their policies to reduce complexity) can only undermatch or perfectly match, but not overmatch, consistent with the empirical evidence. Moreover, using mouse behavioral data (male), we validate a novel prediction about which task conditions exaggerate undermatching. Finally, in patients with Parkinson's disease (male and female), we argue that a reduction in undermatching with higher dopamine levels is consistent with an increased policy complexity.SIGNIFICANCE STATEMENT The matching law describes the tendency of agents to match the ratio of choices allocated to different options to the ratio of reward received. For example, if option a yields twice as much reward as option b, matching states that agents will choose option a twice as much. However, agents typically undermatch: they choose the poorer option more frequently than expected. Here, we assume that agents seek to simultaneously maximize reward and minimize the complexity of their action policies. We show that this theory explains when and why undermatching occurs. Neurally, we show that policy complexity, and by extension undermatching, is controlled by tonic dopamine, consistent with other evidence that dopamine plays an important role in cognitive resource allocation.

Entropy-based metrics for predicting choice behavior based on local response to reward

For decades, behavioral scientists have used the matching law to quantify how animals distribute their choices between multiple options in response to reinforcement they receive. More recently, many reinforcement learning (RL) models have been developed to explain choice by integrating reward feedback over time. Despite reasonable success of RL models in capturing choice on a trial-by-trial basis, these models cannot capture variability in matching behavior. To address this, we developed metrics based on information theory and applied them to choice data from dynamic learning tasks in mice and monkeys. We found that a single entropy-based metric can explain 50% and 41% of variance in matching in mice and monkeys, respectively. We then used limitations of existing RL models in capturing entropy-based metrics to construct more accurate models of choice. Together, our entropy-based metrics provide a model-free tool to predict adaptive choice behavior and reveal underlying neural mechanisms.

In search of a definition of reinforcer value: Some successes and failures of the multiplicative hyperbolic model

The concept of 'value' has enjoyed a central position in many theoretical accounts of choice behaviour. Several definitions of 'value' are contrasted in this paper, and one particular approach is defended, whereby value is defined as a dimensionless intervening variable. This definition is a cornerstone of the multiplicative hyperbolic model of choice (MHM), which was proposed twenty years ago as a modification of Mazur's (1987) hyperbolic model of delay discounting. This paper reviews some of the merits and shortcomings of MHM, and suggests some ways in which MHM might be extended and improved. A formal link between 'value' and the related concept of 'response strength' is suggested, and revisions of the model are proposed which may enable it to accommodate several behavioural phenomena not considered in the original formulation. Broadening the scope of MHM comes at the cost of adding to its burden of free parameters, and it is emphasised that addition of any new parameters needs empirical justification. The status of value as a dimensionless intervening variable is upheld; however it is noted that a growing body of empirical evidence for links between neurobiological phenomena and value suggests that interpretation of value as a hypothetical construct may be warranted.

Human sensitivity to reinforcement: A comment on Kollins, Newland, and Critchfield's (1997) quantitative literature review

In a quantitative review of human operant experiments, Kollins, Newland, and Critchfield (1997) found that humans are less sensitive to reinforcement contingencies than nonhumans are. Human performances were not as consistent with the matching law, and they were more variable from subject to subject. Some of the variables correlated with reduced human sensitivity were surprising. These included collection of the data under more controlled conditions (laboratory rather than naturalistic settings), and inclusions of discriminative stimuli correlated with alternative sources of reinforcement. We discuss these unexpected findings in the light of criticisms that have been leveled against meta-analytic literature reviews (e.g., the wisdom of grouping studies with widely diverse methods), and we suggest ways of improving future analyses of the behavior-analytic literature.

An application of the matching law to social dynamics

Using a procedure similar to the one described by Conger and Killeen (1974), we evaluated levels of attending for 25 college students who participated in either a 20-min (n = 12) or 30-min (n = 13) discussion on juvenile delinquency. Confederates delivered statements of agreement (e.g., "I agree with that point") according to independent variable-interval schedules. Pooled results were evaluated using three generalized formulations of the matching law, and showed that matching was more likely during the first 5 min of the discussion than during the last 5 min. Individual data for 7 of 9 participants were better described by the generalized response-rate matching equation than by the generalized time-allocation matching equation when response allocation was characterized in terms of frequency rather than duration.

Undermatching is an emergent property of selection by consequences

A computational theory of selection by consequences [McDowell, J.J, 2004. A computational model of selection by consequences. J. Exp. Anal. Behav. 81, 297-317] was tested by studying the responding of virtual organisms that were animated by the theory on random interval schedules of reinforcement. The theory generated responding by applying principles of selection, reproduction, and mutation to a population of potential behaviors that evolved in response to the selection pressure exerted by reinforcement. The organisms' equilibrium response rates were well described by the modern version of the Herrnstein hyperbola, which includes an exponent on reinforcement rate. Under strong selection pressure this exponent decreased with increasing mutation rate from a value near 1.0 at 1% mutation to an asymptotic value of 0.83 at mutation rates of 10% and greater. This asymptotic value is consistent with values obtained by fitting the equation to data from live organisms responding on single schedules, and with the value of about 0.80 that is expected on the basis of extensive research with live organisms responding on concurrent schedules. These results show that the computational theory is consistent with the modern theory of matching [McDowell, J.J, 2005. On the classic and modern theories of matching. J. Exp. Anal. Behav. 84, 111-127], and that it is a viable candidate for a mathematical dynamics of behavior.

A computational model of selection by consequences

Darwinian selection by consequences was instantiated in a computational model that consisted of a repertoire of behaviors undergoing selection, reproduction, and mutation over many generations. The model in effect created a digital organism that emitted behavior continuously. The behavior of this digital organism was studied in three series of computational experiments that arranged reinforcement according to random-interval (RI) schedules. The quantitative features of the model were varied over wide ranges in these experiments, and many of the qualitative features of the model also were varied. The digital organism consistently showed a hyperbolic relation between response and reinforcement rates, and this hyperbolic description of the data was consistently better than the description provided by other, similar, function forms. In addition, the parameters of the hyperbola varied systematically with the quantitative, and some of the qualitative, properties of the model in ways that were consistent with findings from biological organisms. These results suggest that the material events responsible for an organism's responding on RI schedules are computationally equivalent to Darwinian selection by consequences. They also suggest that the computational model developed here is worth pursuing further as a possible dynamic account of behavior.

The living legacy of the Harvard Pigeon Lab: quantitative analysis in the wide world

From the Harvard Pigeon Lab of the 1960s arose a behavior-analytic approach that was quantitative and rigorous, rooted in Herrnstein's matching law. Researchers modified the matching law to describe choice behavior in a variety of different settings and examined its relations with other quantitative models. Beginning in the early 1970s, researchers began using the Harvard Pigeon Lab's quantitative framework to study in the laboratory specific aspects of the world outside the laboratory. Much of this work concerned investigations of self-control-choice of a larger, more delayed reinforcer over a smaller, less delayed reinforcer. Experiments using a quantitative framework derived from the matching law have also been conducted outside the laboratory; however, these have been far less frequent. Current and future researchers will benefit the field by devising new, creative ways to investigate the matching law and related quantitative models outside the laboratory. Such research can help to demonstrate the validity of these models as basic principles of behavior, can enhance public opinion of and rewards for such research, and can stimulate further development of the Harvard Pigeon Lab's quantitative approach by using that approach with new variables.

An application of the matching law to severe problem behavior

We evaluated problem behavior and appropriate behavior using the matching equation with 4 individuals with developmental disabilities. Descriptive observations were conducted during interactions between the participants and their primary care providers in either a clinical laboratory environment (3 participants) or the participant's home (1 participant). Data were recorded on potential reinforcers, problem behavior, and appropriate behavior. After identifying the reinforcers that maintained each participant's problem behavior by way of functional analysis, the descriptive data were analyzed retrospectively, based on the matching equation. Results showed that the proportional rate of problem behavior relative to appropriate behavior approximately matched the proportional rate of reinforcement for problem behavior for all participants. The results extend prior research because a functional analysis was conducted and because multiple sources of reinforcement (other than attention) were evaluated. Methodological constraints were identified, which may limit the application of the matching law on both practical and conceptual levels.