Dynamical systems as a level of cognitive analysis of multi-agent learning: Algorithmic foundations of temporal-difference learning dynamics

A dynamical systems perspective on multi-agent learning, based on the link between evolutionary game theory and reinforcement learning, provides an improved, qualitative understanding of the emerging collective learning dynamics. However, confusion exists with respect to how this dynamical systems account of multi-agent learning should be interpreted. In this article, I propose to embed the dynamical systems description of multi-agent learning into different abstraction levels of cognitive analysis. The purpose of this work is to make the connections between these levels explicit in order to gain improved insight into multi-agent learning. I demonstrate the usefulness of this framework with the general and widespread class of temporal-difference reinforcement learning. I find that its deterministic dynamical systems description follows a minimum free-energy principle and unifies a boundedly rational account of game theory with decision-making under uncertainty. I then propose an on-line sample-batch temporal-difference algorithm which is characterized by the combination of applying a memory-batch and separated state-action value estimation. I find that this algorithm serves as a micro-foundation of the deterministic learning equations by showing that its learning trajectories approach the ones of the deterministic learning equations under large batch sizes. Ultimately, this framework of embedding a dynamical systems description into different abstraction levels gives guidance on how to unleash the full potential of the dynamical systems approach to multi-agent learning.

The mechanistic-holistic divide revisited: The case of the lac operon

In this paper, I revisit the development of the repression model of genetic regulation in the lac operon to challenge a common application of a conceptual framework in the history of biology. I take Allen's (1978) account of the changes in the life sciences during the early and mid-twentieth century as an example of a common application of a framework based on the dichotomy between a mechanistic, or reductionist, approach to science and a holistic one. From this conceptual framework, Allen infers two general claims about the process of science and its goals: (1) that "mechanistic materialism" has often presented a more practical way to begin the study of complex phenomena in the life sciences, and (2) that the approach described as "holistic materialism" provides a more complete or accurate description of the natural world. The development of the lac operon model does not fit Allen's generalizations about scientific developments, and it can be used to cast some doubt on the scope of application of that conceptual framework. I argue that a better framework to interpret particular episodes in the history of molecular biology is to consider the ways in which biologists prioritize and track different aspects of the phenomena under study, rather than to focus on whether certain scientific practices are best described as developing from mechanistic to more holistic approaches. I end with some implications for the historiography of science by considering the appropriateness of different conceptual frameworks for different grains of resolution in the history of biology.

Meta-T: TetrisⓇ as an experimental paradigm for cognitive skills research

Studies of human performance in complex tasks using video games are an attractive prospect, but many existing games lack a comprehensive way to modify the game and track performance beyond basic levels of analysis. Meta-T provides experimenters a tool to study behavior in a dynamic task environment with time-stressed decision-making and strong perceptual-motor elements, offering a host of experimental manipulations with a robust and detailed logging system for all user events, system events, and screen objects. Its experimenter-friendly interface provides control over detailed parameters of the task environment without need for programming expertise. Support for eye-tracking and computational cognitive modeling extend the paradigm's scope.

Rational use of cognitive resources: levels of analysis between the computational and the algorithmic

Marr's levels of analysis-computational, algorithmic, and implementation-have served cognitive science well over the last 30 years. But the recent increase in the popularity of the computational level raises a new challenge: How do we begin to relate models at different levels of analysis? We propose that it is possible to define levels of analysis that lie between the computational and the algorithmic, providing a way to build a bridge between computational- and algorithmic-level models. The key idea is to push the notion of rationality, often used in defining computational-level models, deeper toward the algorithmic level. We offer a simple recipe for reverse-engineering the mind's cognitive strategies by deriving optimal algorithms for a series of increasingly more realistic abstract computational architectures, which we call "resource-rational analysis."

Grounding cognitive-level processes in behavior: the view from dynamic systems theory

Marr's seminal work laid out a program of research by specifying key questions for cognitive science at different levels of analysis. Because dynamic systems theory (DST) focuses on time and interdependence of components, DST research programs come to very different conclusions regarding the nature of cognitive change. We review a specific DST approach to cognitive-level processes: dynamic field theory (DFT). We review research applying DFT to several cognitive-level processes: object permanence, naming hierarchical categories, and inferring intent, that demonstrate the difference in understanding of behavior and cognition that results from a DST perspective. These point to a central challenge for cognitive science research as defined by Marr-emergence. We argue that appreciating emergence raises questions about the utility of computational-level analyses and opens the door to insights concerning the origin of novel forms of behavior and thought (e.g., a new chess strategy). We contend this is one of the most fundamental questions about cognition and behavior.

The algorithmic level is the bridge between computation and brain

Every scientist chooses a preferred level of analysis and this choice shapes the research program, even determining what counts as evidence. This contribution revisits Marr's (1982) three levels of analysis (implementation, algorithmic, and computational) and evaluates the prospect of making progress at each individual level. After reviewing limitations of theorizing within a level, two strategies for integration across levels are considered. One is top-down in that it attempts to build a bridge from the computational to algorithmic level. Limitations of this approach include insufficient theoretical constraint at the computation level to provide a foundation for integration, and that people are suboptimal for reasons other than capacity limitations. Instead, an inside-out approach is forwarded in which all three levels of analysis are integrated via the algorithmic level. This approach maximally leverages mutual data constraints at all levels. For example, algorithmic models can be used to interpret brain imaging data, and brain imaging data can be used to select among competing models. Examples of this approach to integration are provided. This merging of levels raises questions about the relevance of Marr's tripartite view.

Meta-T: TetrisⓇ as an experimental paradigm for cognitive skills research

Studies of human performance in complex tasks using video games are an attractive prospect, but many existing games lack a comprehensive way to modify the game and track performance beyond basic levels of analysis. Meta-T provides experimenters a tool to study behavior in a dynamic task environment with time-stressed decision-making and strong perceptual-motor elements, offering a host of experimental manipulations with a robust and detailed logging system for all user events, system events, and screen objects. Its experimenter-friendly interface provides control over detailed parameters of the task environment without need for programming expertise. Support for eye-tracking and computational cognitive modeling extend the paradigm's scope.

Thirty Years After Marr's Vision: Levels of Analysis in Cognitive Science

Thirty years after the publication of Marr's seminal book Vision (Marr, 1982) the papers in this topic consider the contemporary status of his influential conception of three distinct levels of analysis for information-processing systems, and in particular the role of the algorithmic and representational level with its cognitive-level concepts. This level has (either implicitly or explicitly) been downplayed or eliminated both by reductionist neuroscience approaches from below that seek to account for behavior from the implementation level and by Bayesian approaches from above that seek to account for behavior in purely computational-level terms.

Mechanism, function, and computation in neural systems

What constitutes a "mechanism" of behavior? In this tribute to Jerry Hogan we examine how questions of behavioral mechanism can be reframed as causes and consequences of neural circuit activity. Drawing from our work on the hippocampus and the medial prefrontal cortex we discuss the inherent difficulties of characterizing the behavioral functions of circuits that are many synapses away from sensory reception and motor/visceral expression. We briefly review the advantages of reframing a region's functions according to its computations, while also distinguishing those computations from the algorithms by which they are achieved. As an example of how these ideas can be applied, we discuss why the hippocampus and medial prefrontal cortex may have overlapping roles in memory expression in spite of being very different circuits. The present analysis draws inspiration from David Marr, whose framework for describing neural systems can be compared with Aristotle's "causes." This article is part of a Special Issue entitled: In Honor of Jerry Hogan.