1
|
German JS, Jacobs RA. Implications of capacity-limited, generative models for human vision. Behav Brain Sci 2023; 46:e391. [PMID: 38054373 DOI: 10.1017/s0140525x23001772] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
Although discriminative deep neural networks are currently dominant in cognitive modeling, we suggest that capacity-limited, generative models are a promising avenue for future work. Generative models tend to learn both local and global features of stimuli and, when properly constrained, can learn componential representations and response biases found in people's behaviors.
Collapse
Affiliation(s)
- Joseph Scott German
- Department of Cognitive Science, University of California, San Diego, La Jolla, CA, USA
| | - Robert A Jacobs
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, USA https://www2.bcs.rochester.edu/sites/jacobslab/people.html
| |
Collapse
|
2
|
Dubova M, Goldstone RL. Carving joints into nature: reengineering scientific concepts in light of concept-laden evidence. Trends Cogn Sci 2023; 27:656-670. [PMID: 37173157 DOI: 10.1016/j.tics.2023.04.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 04/11/2023] [Accepted: 04/12/2023] [Indexed: 05/15/2023]
Abstract
A new wave of proposals suggests that scientists must reassess scientific concepts in light of accumulated evidence. However, reengineering scientific concepts in light of data is challenging because scientific concepts affect the evidence itself in multiple ways. Among other possible influences, concepts (i) prime scientists to overemphasize within-concept similarities and between-concept differences; (ii) lead scientists to measure conceptually relevant dimensions more accurately; (iii) serve as units of scientific experimentation, communication, and theory-building; and (iv) affect the phenomena themselves. When looking for improved ways to carve nature at its joints, scholars must take the concept-laden nature of evidence into account to avoid entering a vicious circle of concept-evidence mutual substantiation.
Collapse
Affiliation(s)
- Marina Dubova
- Cognitive Science Program, Indiana University, 1101 E. 10th Street, Bloomington, IN 47405, USA.
| | - Robert L Goldstone
- Cognitive Science Program, Indiana University, 1101 E. 10th Street, Bloomington, IN 47405, USA; Department of Psychological and Brain Sciences, Indiana University, 1101 E. 10th Street, Bloomington, IN 47405, USA
| |
Collapse
|
3
|
Schaffner J, Bao SD, Tobler PN, Hare TA, Polania R. Sensory perception relies on fitness-maximizing codes. Nat Hum Behav 2023:10.1038/s41562-023-01584-y. [PMID: 37106080 PMCID: PMC10365992 DOI: 10.1038/s41562-023-01584-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Accepted: 03/09/2023] [Indexed: 04/29/2023]
Abstract
Sensory information encoded by humans and other organisms is generally presumed to be as accurate as their biological limitations allow. However, perhaps counterintuitively, accurate sensory representations may not necessarily maximize the organism's chances of survival. To test this hypothesis, we developed a unified normative framework for fitness-maximizing encoding by combining theoretical insights from neuroscience, computer science, and economics. Behavioural experiments in humans revealed that sensory encoding strategies are flexibly adapted to promote fitness maximization, a result confirmed by deep neural networks with information capacity constraints trained to solve the same task as humans. Moreover, human functional MRI data revealed that novel behavioural goals that rely on object perception induce efficient stimulus representations in early sensory structures. These results suggest that fitness-maximizing rules imposed by the environment are applied at early stages of sensory processing in humans and machines.
Collapse
Affiliation(s)
- Jonathan Schaffner
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
- Neuroscience Center Zurich, Zurich, Switzerland
| | - Sherry Dongqi Bao
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
- Neuroscience Center Zurich, Zurich, Switzerland
| | - Philippe N Tobler
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
- Neuroscience Center Zurich, Zurich, Switzerland
| | - Todd A Hare
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland.
- Neuroscience Center Zurich, Zurich, Switzerland.
| | - Rafael Polania
- Neuroscience Center Zurich, Zurich, Switzerland.
- Decision Neuroscience Lab, Department of Health Sciences and Technology, ETH Zurich, Zurich, Switzerland.
| |
Collapse
|
4
|
Gershman SJ, Burke T. Mental control of uncertainty. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2022:10.3758/s13415-022-01034-8. [PMID: 36168079 DOI: 10.3758/s13415-022-01034-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 09/01/2022] [Indexed: 06/16/2023]
Abstract
Can you reduce uncertainty by thinking? Intuition suggests that this happens through the elusive process of attention: if we expend mental effort, we can increase the reliability of our sensory data. Models based on "rational inattention" formalize this idea in terms of a trade-off between the costs and benefits of attention. This paper surveys the origin of these models in economics, their connection to rate-distortion theory, and some of their recent applications to psychology and neuroscience. We also report new data from a numerosity judgment task in which we manipulate performance incentives. Consistent with rational inattention, people are able to improve performance on this task when incentivized, in part by increasing the reliability of their sensory data.
Collapse
Affiliation(s)
- Samuel J Gershman
- Department of Psychology and Center for Brain Science, Harvard University, MA, Cambridge, USA.
| | - Taylor Burke
- Department of Psychology and Center for Brain Science, Harvard University, MA, Cambridge, USA
| |
Collapse
|
5
|
Abstract
Categorical perception refers to the enhancement of perceptual sensitivity near category boundaries, generally along dimensions that are informative about category membership. However, it remains unclear exactly which dimensions are treated as informative and why. This article reports a series of experiments in which subjects were asked to learn statistically defined categories in a novel, unfamiliar 2D perceptual space of shapes. Perceptual discrimination was tested before and after category learning of various features in the space, each defined by its position and orientation relative to the maximally informative dimension. The results support a remarkably simple generalization: The magnitude of improvement in perceptual discrimination of each feature is proportional to the mutual information between the feature and the category variable. This finding suggests a rational basis for categorical perception in which the precision of perceptual discrimination is tuned to the statistical structure of the environment.
Collapse
Affiliation(s)
- Jacob Feldman
- Department of Psychology, Center for Cognitive Science, Rutgers University
| |
Collapse
|
6
|
Dubova M, Goldstone RL. The Influences of Category Learning on Perceptual Reconstructions. Cogn Sci 2021; 45:e12981. [PMID: 34018243 DOI: 10.1111/cogs.12981] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2020] [Revised: 10/08/2020] [Accepted: 04/02/2021] [Indexed: 11/30/2022]
Abstract
We explore different ways in which the human visual system can adapt for perceiving and categorizing the environment. There are various accounts of supervised (categorical) and unsupervised perceptual learning, and different perspectives on the functional relationship between perception and categorization. We suggest that common experimental designs are insufficient to differentiate between hypothesized perceptual learning mechanisms and reveal their possible interplay. We propose a relatively underutilized way of studying potential categorical effects on perception, and we test the predictions of different perceptual learning models using a two-dimensional, interleaved categorization-plus-reconstruction task. We find evidence that the human visual system adapts its encodings to the feature structure of the environment, uses categorical expectations for robust reconstruction, allocates encoding resources with respect to categorization utility, and adapts to prevent miscategorizations.
Collapse
Affiliation(s)
- Marina Dubova
- Psychological and Brain Sciences, Indiana University
| | | |
Collapse
|
7
|
Bates CJ, Jacobs RA. Optimal attentional allocation in the presence of capacity constraints in uncued and cued visual search. J Vis 2021; 21:3. [PMID: 33944906 PMCID: PMC8107488 DOI: 10.1167/jov.21.5.3] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2020] [Accepted: 03/09/2021] [Indexed: 11/24/2022] Open
Abstract
The vision sciences literature contains a large diversity of experimental and theoretical approaches to the study of visual attention. We argue that this diversity arises, at least in part, from the field's inability to unify differing theoretical perspectives. In particular, the field has been hindered by a lack of a principled formal framework for simultaneously thinking about both optimal attentional processing and capacity-limited attentional processing, where capacity is limited in a general, task-independent manner. Here, we supply such a framework based on rate-distortion theory (RDT) and optimal lossy compression. Our approach defines Bayes-optimal performance when an upper limit on information processing rate is imposed. In this article, we compare Bayesian and RDT accounts in both uncued and cued visual search tasks. We start by highlighting a typical shortcoming of unlimited-capacity Bayesian models that is not shared by RDT models, namely, that they often overestimate task performance when information-processing demands are increased. Next, we reexamine data from two cued-search experiments that have previously been modeled as the result of unlimited-capacity Bayesian inference and demonstrate that they can just as easily be explained as the result of optimal lossy compression. To model cued visual search, we introduce the concept of a "conditional communication channel." This simple extension generalizes the lossy-compression framework such that it can, in principle, predict optimal attentional-shift behavior in any kind of perceptual task, even when inputs to the model are raw sensory data such as image pixels. To demonstrate this idea's viability, we compare our idealized model of cued search, which operates on a simplified abstraction of the stimulus, to a deep neural network version that performs approximately optimal lossy compression on the real (pixel-level) experimental stimuli.
Collapse
Affiliation(s)
| | - Robert A Jacobs
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, USA
| |
Collapse
|
8
|
Lai L, Gershman SJ. Policy compression: An information bottleneck in action selection. PSYCHOLOGY OF LEARNING AND MOTIVATION 2021. [DOI: 10.1016/bs.plm.2021.02.004] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
9
|
Bruning AL, Lewis-Peacock JA. Long-term memory guides resource allocation in working memory. Sci Rep 2020; 10:22161. [PMID: 33335170 PMCID: PMC7747625 DOI: 10.1038/s41598-020-79108-1] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2020] [Accepted: 12/01/2020] [Indexed: 01/02/2023] Open
Abstract
Working memory capacity is incredibly limited and thus it is important to use this resource wisely. Prior knowledge in long-term memory can aid in efficient encoding of information by allowing for the prioritization of novel stimuli over familiar ones. Here we used a full-report procedure in a visual working memory paradigm, where participants reported the location of six colored circles in any order, to examine the influence of prior information on resource allocation in working memory. Participants learned that one of the items appeared in a restricted range of locations, whereas the remaining items could appear in any location. We found that participants' memory performance benefited from learning this prior information. Specifically, response precision increased for all items when prior information was available for one of the items. Responses for both familiar and novel items were systematically ordered from highest to lowest precision. Participants tended to report the familiar item in the second half of the six responses and did so with greater precision than for novel items. Moreover, novel items that appeared near the center of the prior location were reported with worse precision than novel items that appeared elsewhere. This shows that people strategically allocated working memory resources by ignoring information that appeared in predictable locations and prioritizing the encoding of information that appeared in unpredictable locations. Together these findings demonstrate that people rely on long-term memory not only for remembering familiar items, but also for the strategic allocation of their limited capacity working memory resources.
Collapse
Affiliation(s)
- Allison L Bruning
- Department of Psychology, Center for Learning and Memory, University of Texas at Austin, 108 E Dean Keeton St, Stop A8000, Austin, TX, 78712, USA.
| | - Jarrod A Lewis-Peacock
- Department of Psychology, Center for Learning and Memory, University of Texas at Austin, 108 E Dean Keeton St, Stop A8000, Austin, TX, 78712, USA
| |
Collapse
|
10
|
Moreno-Bote R, Ramírez-Ruiz J, Drugowitsch J, Hayden BY. Heuristics and optimal solutions to the breadth-depth dilemma. Proc Natl Acad Sci U S A 2020; 117:19799-19808. [PMID: 32759219 PMCID: PMC7443877 DOI: 10.1073/pnas.2004929117] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023] Open
Abstract
In multialternative risky choice, we are often faced with the opportunity to allocate our limited information-gathering capacity between several options before receiving feedback. In such cases, we face a natural trade-off between breadth-spreading our capacity across many options-and depth-gaining more information about a smaller number of options. Despite its broad relevance to daily life, including in many naturalistic foraging situations, the optimal strategy in the breadth-depth trade-off has not been delineated. Here, we formalize the breadth-depth dilemma through a finite-sample capacity model. We find that, if capacity is small (∼10 samples), it is optimal to draw one sample per alternative, favoring breadth. However, for larger capacities, a sharp transition is observed, and it becomes best to deeply sample a very small fraction of alternatives, which roughly decreases with the square root of capacity. Thus, ignoring most options, even when capacity is large enough to shallowly sample all of them, is a signature of optimal behavior. Our results also provide a rich casuistic for metareasoning in multialternative decisions with bounded capacity using close-to-optimal heuristics.
Collapse
Affiliation(s)
- Rubén Moreno-Bote
- Center for Brain and Cognition, Universitat Pompeu Fabra, 08002 Barcelona, Spain;
- Department of Information and Communication Technologies, Universitat Pompeu Fabra, 08002 Barcelona, Spain
- Serra Húnter Fellow Programme, Universitat Pompeu Fabra, 08002 Barcelona, Spain
- Catalan Institution for Research and Advanced Studies-Academia, Universitat Pompeu Fabra, 08002 Barcelona, Spain
| | - Jorge Ramírez-Ruiz
- Center for Brain and Cognition, Universitat Pompeu Fabra, 08002 Barcelona, Spain
- Department of Information and Communication Technologies, Universitat Pompeu Fabra, 08002 Barcelona, Spain
| | - Jan Drugowitsch
- Department of Neurobiology, Harvard Medical School, Boston, MA 02115
| | - Benjamin Y Hayden
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455
- Center for Magnetic Resonance Research, University of Minnesota, Minneapolis, MN 55455
- Center for Neural Engineering, University of Minnesota, Minneapolis, MN 55455
| |
Collapse
|
11
|
Plancher G, Goldstone RL. How Do People Code Information in Working Memory When Items Share Features? Exp Psychol 2020; 67:169-177. [PMID: 32552545 DOI: 10.1027/1618-3169/a000480] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Abstract. A large literature suggests that the way we process information is influenced by the categories that we have learned. We examined whether, when we try to uniquely encode items in working memory, the information encoded depends on the other stimuli being simultaneously learned. Participants were required to memorize unknown aliens, presented one at the time, for immediate recognition of their features. Some aliens, called twins, were organized into pairs that shared every feature (nondiscriminative feature) except one (discriminative feature), while some other aliens, called hermits, did not share feature. We reasoned that if people develop unsupervised categories by creating a category for a pair of aliens, we should observe better feature identification performance for nondiscriminative features compared to hermit features, but not compared to discriminative features. On the contrary, if distinguishing features draw attention, we should observe better performance when a discriminative rather than nondiscriminative feature was probed. Overall, our results suggest that when items share features, people code items in working memory by focusing on similarities between items, establishing clusters of items in an unsupervised fashion not requiring feedback on cluster membership.
Collapse
Affiliation(s)
- Gaën Plancher
- Laboratoire d'Etude des Mécanismes Cognitifs, Université Lumière Lyon 2, Bron, France
| | | |
Collapse
|
12
|
The importance of constraints on constraints. Behav Brain Sci 2020; 43:e3. [PMID: 32159481 DOI: 10.1017/s0140525x19001572] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
The "resource-rational" approach is ambitious and worthwhile. A shortcoming of the proposed approach is that it fails to constrain what counts as a constraint. As a result, constraints used in different cognitive domains often have nothing in common. We describe an alternative framework that satisfies many of the desiderata of the resource-rational approach, but in a more disciplined manner.
Collapse
|
13
|
Jacobs RA, Xu C. Can multisensory training aid visual learning? A computational investigation. J Vis 2019; 19:1. [PMID: 31480074 DOI: 10.1167/19.11.1] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Although real-world environments are often multisensory, visual scientists typically study visual learning in unisensory environments containing visual signals only. Here, we use deep or artificial neural networks to address the question, Can multisensory training aid visual learning? We examine a network's internal representations of objects based on visual signals in two conditions: (a) when the network is initially trained with both visual and haptic signals, and (b) when it is initially trained with visual signals only. Our results demonstrate that a network trained in a visual-haptic environment (in which visual, but not haptic, signals are orientation-dependent) tends to learn visual representations containing useful abstractions, such as the categorical structure of objects, and also learns representations that are less sensitive to imaging parameters, such as viewpoint or orientation, that are irrelevant for object recognition or classification tasks. We conclude that researchers studying perceptual learning in vision-only contexts may be overestimating the difficulties associated with important perceptual learning problems. Although multisensory perception has its own challenges, perceptual learning can become easier when it is considered in a multisensory setting.
Collapse
Affiliation(s)
- Robert A Jacobs
- Department of Brain and Cognitive Sciences, University of Rochester, Rochester, NY, USA
| | - Chenliang Xu
- Department of Computer Science, University of Rochester, Rochester, NY, USA
| |
Collapse
|