1
|
Karakose-Akbiyik S, Sussman O, Wurm MF, Caramazza A. The Role of Agentive and Physical Forces in the Neural Representation of Motion Events. J Neurosci 2024; 44:e1363232023. [PMID: 38050107 PMCID: PMC10860628 DOI: 10.1523/jneurosci.1363-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 11/14/2023] [Accepted: 11/19/2023] [Indexed: 12/06/2023] Open
Abstract
How does the brain represent information about motion events in relation to agentive and physical forces? In this study, we investigated the neural activity patterns associated with observing animated actions of agents (e.g., an agent hitting a chair) in comparison to similar movements of inanimate objects that were either shaped solely by the physics of the scene (e.g., gravity causing an object to fall down a hill and hit a chair) or initiated by agents (e.g., a visible agent causing an object to hit a chair). Using an fMRI-based multivariate pattern analysis (MVPA), this design allowed testing where in the brain the neural activity patterns associated with motion events change as a function of, or are invariant to, agentive versus physical forces behind them. A total of 29 human participants (nine male) participated in the study. Cross-decoding revealed a shared neural representation of animate and inanimate motion events that is invariant to agentive or physical forces in regions spanning frontoparietal and posterior temporal cortices. In contrast, the right lateral occipitotemporal cortex showed a higher sensitivity to agentive events, while the left dorsal premotor cortex was more sensitive to information about inanimate object events that were solely shaped by the physics of the scene.
Collapse
Affiliation(s)
| | - Oliver Sussman
- Department of Psychology, Harvard University, Cambridge, Massachusetts 02138
| | - Moritz F Wurm
- Center for Mind/Brain Sciences - CIMeC, University of Trento, 38068 Rovereto, Italy
| | - Alfonso Caramazza
- Department of Psychology, Harvard University, Cambridge, Massachusetts 02138
- Center for Mind/Brain Sciences - CIMeC, University of Trento, 38068 Rovereto, Italy
| |
Collapse
|
2
|
Burk DC, Sheinberg DL. Neurons in inferior temporal cortex are sensitive to motion trajectory during degraded object recognition. Cereb Cortex Commun 2022; 3:tgac034. [PMID: 36168516 PMCID: PMC9499820 DOI: 10.1093/texcom/tgac034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 08/03/2022] [Accepted: 08/05/2022] [Indexed: 11/30/2022] Open
Abstract
Our brains continuously acquire sensory information and make judgments even when visual information is limited. In some circumstances, an ambiguous object can be recognized from how it moves, such as an animal hopping or a plane flying overhead. Yet it remains unclear how movement is processed by brain areas involved in visual object recognition. Here we investigate whether inferior temporal (IT) cortex, an area known for its relevance in visual form processing, has access to motion information during recognition. We developed a matching task that required monkeys to recognize moving shapes with variable levels of shape degradation. Neural recordings in area IT showed that, surprisingly, some IT neurons responded stronger to degraded shapes than clear ones. Furthermore, neurons exhibited motion sensitivity at different times during the presentation of the blurry target. Population decoding analyses showed that motion patterns could be decoded from IT neuron pseudo-populations. Contrary to previous findings, these results suggest that neurons in IT can integrate visual motion and shape information, particularly when shape information is degraded, in a way that has been previously overlooked. Our results highlight the importance of using challenging multifeature recognition tasks to understand the role of area IT in naturalistic visual object recognition.
Collapse
Affiliation(s)
- Diana C Burk
- Department of Neuroscience, Brown University , Providence, RI 02912 , United States
| | - David L Sheinberg
- Department of Neuroscience, Brown University , Providence, RI 02912 , United States
- Carney Institute for Brain Science, Brown University , Providence, RI 02912 , United States
| |
Collapse
|
3
|
Pramod RT, Cohen MA, Tenenbaum JB, Kanwisher N. Invariant representation of physical stability in the human brain. eLife 2022; 11:e71736. [PMID: 35635277 PMCID: PMC9150889 DOI: 10.7554/elife.71736] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Accepted: 05/10/2022] [Indexed: 11/13/2022] Open
Abstract
Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.
Collapse
Affiliation(s)
- RT Pramod
- Center for Brains, Minds and Machines, Massachusetts Institute of TechnologyCambridgeUnited States
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Michael A Cohen
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
- Amherst CollegeAmherstUnited States
| | - Joshua B Tenenbaum
- Center for Brains, Minds and Machines, Massachusetts Institute of TechnologyCambridgeUnited States
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
| | - Nancy Kanwisher
- Center for Brains, Minds and Machines, Massachusetts Institute of TechnologyCambridgeUnited States
- Department of Brain and Cognitive Sciences, Massachusetts Institute of TechnologyCambridgeUnited States
| |
Collapse
|
4
|
Li S, Zhang W, Zhang H, Zhang X, Leng Y. Proximal policy optimization with model-based methods. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2022. [DOI: 10.3233/jifs-211935] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Model-free reinforcement learning methods have successfully been applied to practical applications such as decision-making problems in Atari games. However, these methods have inherent shortcomings, such as a high variance and low sample efficiency. To improve the policy performance and sample efficiency of model-free reinforcement learning, we propose proximal policy optimization with model-based methods (PPOMM), a fusion method of both model-based and model-free reinforcement learning. PPOMM not only considers the information of past experience but also the prediction information of the future state. PPOMM adds the information of the next state to the objective function of the proximal policy optimization (PPO) algorithm through a model-based method. This method uses two components to optimize the policy: the error of PPO and the error of model-based reinforcement learning. We use the latter to optimize a latent transition model and predict the information of the next state. For most games, this method outperforms the state-of-the-art PPO algorithm when we evaluate across 49 Atari games in the Arcade Learning Environment (ALE). The experimental results show that PPOMM performs better or the same as the original algorithm in 33 games.
Collapse
Affiliation(s)
- Shuailong Li
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Wei Zhang
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China
| | | | - Xin Zhang
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China
- Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China
| | - Yuquan Leng
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China
- Guangdong Provincial Key Laboratory of Human-Augmentation and Rehabilitation Robotics in Universities, Southern University of Science and Technology, Shenzhen, China
| |
Collapse
|
5
|
Mangalam M, Fragaszy DM, Wagman JB, Day BM, Kelty-Stephen DG, Bongers RM, Stout DW, Osiurak F. On the psychological origins of tool use. Neurosci Biobehav Rev 2022; 134:104521. [PMID: 34998834 DOI: 10.1016/j.neubiorev.2022.104521] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2021] [Revised: 12/01/2021] [Accepted: 01/01/2022] [Indexed: 01/13/2023]
Abstract
The ubiquity of tool use in human life has generated multiple lines of scientific and philosophical investigation to understand the development and expression of humans' engagement with tools and its relation to other dimensions of human experience. However, existing literature on tool use faces several epistemological challenges in which the same set of questions generate many different answers. At least four critical questions can be identified, which are intimately intertwined-(1) What constitutes tool use? (2) What psychological processes underlie tool use in humans and nonhuman animals? (3) Which of these psychological processes are exclusive to tool use? (4) Which psychological processes involved in tool use are exclusive to Homo sapiens? To help advance a multidisciplinary scientific understanding of tool use, six author groups representing different academic disciplines (e.g., anthropology, psychology, neuroscience) and different theoretical perspectives respond to each of these questions, and then point to the direction of future work on tool use. We find that while there are marked differences among the responses of the respective author groups to each question, there is a surprising degree of agreement about many essential concepts and questions. We believe that this interdisciplinary and intertheoretical discussion will foster a more comprehensive understanding of tool use than any one of these perspectives (or any one of these author groups) would (or could) on their own.
Collapse
Affiliation(s)
- Madhur Mangalam
- Department of Physical Therapy, Movement and Rehabilitation Science, Northeastern University, Boston, Massachusetts 02115, USA.
| | | | - Jeffrey B Wagman
- Department of Psychology, Illinois State University, Normal, IL 61761, USA
| | - Brian M Day
- Department of Psychology, Butler University, Indianapolis, IN 46208, USA
| | | | - Raoul M Bongers
- Department of Human Movement Sciences, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, Netherlands
| | - Dietrich W Stout
- Department of Anthropology, Emory University, Atlanta, GA 30322, USA
| | - François Osiurak
- Laboratoire d'Etude des Mécanismes Cognitifs, Université de Lyon, Lyon 69361, France; Institut Universitaire de France, Paris 75231, France
| |
Collapse
|
6
|
Peters B, Kriegeskorte N. Capturing the objects of vision with neural networks. Nat Hum Behav 2021; 5:1127-1144. [PMID: 34545237 DOI: 10.1038/s41562-021-01194-6] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2019] [Accepted: 08/06/2021] [Indexed: 01/31/2023]
Abstract
Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioural studies have documented how object representations emerge through grouping, amodal completion, proto-objects and object files. By contrast, deep neural network models of visual object recognition remain largely tethered to sensory input, despite achieving human-level performance at labelling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving the development of deep neural network models that will put the object into object recognition.
Collapse
Affiliation(s)
- Benjamin Peters
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA.
| | - Nikolaus Kriegeskorte
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA. .,Department of Psychology, Columbia University, New York, NY, USA. .,Department of Neuroscience, Columbia University, New York, NY, USA. .,Department of Electrical Engineering, Columbia University, New York, NY, USA.
| |
Collapse
|
7
|
Stout D. The Cognitive Science of Technology. Trends Cogn Sci 2021; 25:964-977. [PMID: 34362661 DOI: 10.1016/j.tics.2021.07.005] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2021] [Revised: 07/09/2021] [Accepted: 07/13/2021] [Indexed: 01/23/2023]
Abstract
Technology is central to human life but hard to define and study. This review synthesizes advances in fields from anthropology to evolutionary biology and neuroscience to propose an interdisciplinary cognitive science of technology. The foundation of this effort is an evolutionarily motivated definition of technology that highlights three key features: material production, social collaboration, and cultural reproduction. This broad scope respects the complexity of the subject but poses a challenge for theoretical unification. Addressing this challenge requires a comparative approach to reduce the diversity of real-world technological cognition to a smaller number of recurring processes and relationships. To this end, a synthetic perceptual-motor hypothesis (PMH) for the evolutionary-developmental-cultural construction of technological cognition is advanced as an initial target for investigation.
Collapse
Affiliation(s)
- Dietrich Stout
- Department of Anthropology, Emory University, 1557 Dickey Drive, Atlanta, GA 30322, USA.
| |
Collapse
|
8
|
Burn Image Recognition of Medical Images Based on Deep Learning: From CNNs to Advanced Networks. Neural Process Lett 2021. [DOI: 10.1007/s11063-021-10459-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
9
|
Dasgupta I, Gershman SJ. Memory as a Computational Resource. Trends Cogn Sci 2021; 25:240-251. [PMID: 33454217 DOI: 10.1016/j.tics.2020.12.008] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2020] [Revised: 12/14/2020] [Accepted: 12/15/2020] [Indexed: 11/30/2022]
Abstract
Computer scientists have long recognized that naive implementations of algorithms often result in a paralyzing degree of redundant computation. More sophisticated implementations harness the power of memory by storing computational results and reusing them later. We review the application of these ideas to cognitive science, in four case studies (mental arithmetic, mental imagery, planning, and probabilistic inference). Despite their superficial differences, these cognitive processes share a common reliance on memory that enables efficient computation.
Collapse
Affiliation(s)
- Ishita Dasgupta
- Department of Computer Science, Princeton University, Princeton, NY, USA.
| | - Samuel J Gershman
- Department of Psychology, Center for Brain Science, Harvard University, Cambridge, MA, USA; Center for Brains, Minds, and Machines, MIT, Cambridge, MA, USA
| |
Collapse
|
10
|
Abstract
We show that the classical problem of three-dimensional (3D) size perception in obliquely viewed pictures can be understood by comparing human performance to the optimal geometric solution. A photograph seen from the camera position, can form the same retinal projection as the physical 3D scene, but retinal projections of sizes and shapes are distorted in oblique viewing. For real scenes, we previously showed that size and shape inconstancy result despite observers using the correct geometric back-transform, because some retinal images evoke misestimates of object slant or viewing elevation. Now, we examine how observers estimate 3D sizes in oblique views of pictures of objects lying on the ground in different poses. Compared to estimates for real scenes, in oblique views of pictures, sizes were seriously underestimated for objects at frontoparallel poses, but there was almost no change for objects perceived as pointing toward the viewer. The inverse of the function relating projected length to pose, camera elevation and viewing azimuth, gives the optimal correction factor for inferring correct 3D lengths if the elevation and azimuth are estimated accurately. Empirical correction functions had similar shapes to optimal, but lower amplitude. Measurements revealed that observers systematically underestimated viewing azimuth, similar to the frontoparallel bias for object pose perception. A model that adds underestimation of viewing azimuth to the geometrical back-transform, provided good fits to estimated 3D lengths from oblique views. These results add to accumulating evidence that observers use internalized projective geometry to perceive sizes, shapes, and poses in 3D scenes and their pictures.
Collapse
Affiliation(s)
- Akihito Maruya
- Graduate Center for Vision Research, State University of New York, New York, NY, USA.,
| | - Qasim Zaidi
- Graduate Center for Vision Research, State University of New York, New York, NY, USA.,
| |
Collapse
|
11
|
Schwettmann S, Tenenbaum JB, Kanwisher N. Invariant representations of mass in the human brain. eLife 2019; 8:46619. [PMID: 31845887 PMCID: PMC7007217 DOI: 10.7554/elife.46619] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2019] [Accepted: 12/10/2019] [Indexed: 01/14/2023] Open
Abstract
An intuitive understanding of physical objects and events is critical for successfully interacting with the world. Does the brain achieve this understanding by running simulations in a mental physics engine, which represents variables such as force and mass, or by analyzing patterns of motion without encoding underlying physical quantities? To investigate, we scanned participants with fMRI while they viewed videos of objects interacting in scenarios indicating their mass. Decoding analyses in brain regions previously implicated in intuitive physical inference revealed mass representations that generalized across variations in scenario, material, friction, and motion energy. These invariant representations were found during tasks without action planning, and tasks focusing on an orthogonal dimension (object color). Our results support an account of physical reasoning where abstract physical variables serve as inputs to a forward model of dynamics, akin to a physics engine, in parietal and frontal cortex.
Collapse
Affiliation(s)
- Sarah Schwettmann
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States.,Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, United States.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| | - Joshua B Tenenbaum
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States.,Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, United States.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States.,Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, United States
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, United States.,Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, United States.,McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, United States
| |
Collapse
|