What tool representation, intuitive physics, and action have in common: The brain's first-person physics engine

Monkeys engage in visual simulation to solve complex problems

Visual simulation-i.e., using internal reconstructions of the world to experience potential future versions of events that are not currently happening-is among the most sophisticated capacities of the human mind. But is this ability in fact uniquely human? To answer this question, we tested monkeys on a series of experiments involving the "Planko" game, which we have previously used to evoke visual simulation in human participants. We found that monkeys were able to successfully play the game using a simulation strategy, predicting the trajectory of a ball through a field of planks while demonstrating a level of accuracy and behavioral signatures comparable with those of humans. Computational analyses further revealed that the monkeys' strategy while playing Planko aligned with a recurrent neural network (RNN) that approached the task using a spontaneously learned simulation strategy. Finally, we carried out awake functional magnetic resonance imaging while monkeys played Planko. We found activity in motion-sensitive regions of the monkey brain during hypothesized simulation periods, even without any perceived visual motion cues. This neural result closely mirrors previous findings from human research, suggesting a shared mechanism of visual simulation across species. Taken together, these findings challenge traditional views of animal cognition, proposing that nonhuman primates possess a complex cognitive landscape, capable of invoking imaginative and predictive mental experiences to solve complex everyday problems.

Invariant representations in abstract concept grounding - the physical world in grounded cognition

Grounded cognition states that mental representations of concepts consist of experiential aspects. For example, the concept "cup" consists of the sensorimotor experiences from interactions with cups. Typical modalities in which concepts are grounded are: The sensorimotor system (including interoception), emotion, action, language, and social aspects. Here, we argue that this list should be expanded to include physical invariants (unchanging features of physical motion; e.g., gravity, momentum, friction). Research on physical reasoning consistently demonstrates that physical invariants are represented as fundamentally as other grounding substrates, and therefore should qualify. We assess several theories of concept representation (simulation, conceptual metaphor, conceptual spaces, predictive processing) and their positions on physical invariants. We find that the classic grounded cognition theories, simulation and conceptual metaphor theory, have not considered physical invariants, while conceptual spaces and predictive processing have. We conclude that physical invariants should be included into grounded cognition theories, and that the core mechanisms of simulation and conceptual metaphor theory are well suited to do this. Furthermore, conceptual spaces and predictive processing are very promising and should also be integrated with grounded cognition in the future.

Tangled Physics: Knots Strain Intuitive Physical Reasoning

Whereas decades of research have cataloged striking errors in physical reasoning, a resurgence of interest in intuitive physics has revealed humans' remarkable ability to successfully predict the unfolding of physical scenes. A leading interpretation intended to resolve these opposing results is that physical reasoning recruits a general-purpose mechanism that reliably models physical scenarios (explaining recent successes), but overly contrived tasks or impoverished and ecologically invalid stimuli can produce poor performance (accounting for earlier failures). But might there be tasks that persistently strain physical understanding, even in naturalistic contexts? Here, we explore this question by introducing a new intuitive physics task: evaluating the strength of knots and tangles. Knots are ubiquitous across cultures and time-periods, and evaluating them correctly often spells the difference between safety and peril. Despite this, 5 experiments show that observers fail to discern even very large differences in strength between knots. In a series of two-alternative forced-choice tasks, observers viewed a variety of simple "bends" (knots joining two pieces of thread) and decided which would require more force to undo. Though the strength of these knots is well-documented, observers' judgments completely failed to reflect these distinctions, across naturalistic photographs (E1), idealized renderings (E2), dynamic videos (E3), and even when accompanied by schematic diagrams of the knots' structures (E4). Moreover, these failures persisted despite accurate identification of the topological differences between the knots (E5); in other words, even when observers correctly perceived the underlying structure of the knot, they failed to correctly judge its strength. These results expose a blindspot in physical reasoning, placing new constraints on general-purpose theories of scene understanding.

Graspable foods and tools elicit similar responses in visual cortex

The extrastriatal visual cortex is known to exhibit distinct response profiles to complex stimuli of varying ecological importance (e.g. faces, scenes, and tools). Although food is primarily distinguished from other objects by its edibility, not its appearance, recent evidence suggests that there is also food selectivity in human visual cortex. Food is also associated with a common behavior, eating, and food consumption typically also involves the manipulation of food, often with hands. In this context, food items share many properties with tools: they are graspable objects that we manipulate in self-directed and stereotyped forms of action. Thus, food items may be preferentially represented in extrastriatal visual cortex in part because of these shared affordance properties, rather than because they reflect a wholly distinct kind of category. We conducted functional MRI and behavioral experiments to test this hypothesis. We found that graspable food items and tools were judged to be similar in their action-related properties and that the location, magnitude, and patterns of neural responses for images of graspable food items were similar in profile to the responses for tool stimuli. Our findings suggest that food selectivity may reflect the behavioral affordances of food items rather than a distinct form of category selectivity.

Fundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval

Motor learning is often viewed as a unitary process that operates outside of conscious awareness. This perspective has led to the development of sophisticated models designed to elucidate the mechanisms of implicit sensorimotor learning. In this review, we argue for a broader perspective, emphasizing the contribution of explicit strategies to sensorimotor learning tasks. Furthermore, we propose a theoretical framework for motor learning that consists of three fundamental processes: reasoning, the process of understanding action-outcome relationships; refinement, the process of optimizing sensorimotor and cognitive parameters to achieve motor goals; and retrieval, the process of inferring the context and recalling a control policy. We anticipate that this '3R' framework for understanding how complex movements are learned will open exciting avenues for future research at the intersection between cognition and action.

Grounding Cognition in Perceptual Experience

The aim of this Special Issue was to put forward a multifaceted reflection on the relevance of perceptual experience in affecting and modeling various aspects of cognitive performance [...].

Reciprocal interactions among parietal and occipito-temporal representations support everyday object-directed actions

Everyday interactions with common manipulable objects require the integration of conceptual knowledge about objects and actions with real-time sensory information about the position, orientation and volumetric structure of the grasp target. The ability to successfully interact with everyday objects involves analysis of visual form and shape, surface texture, material properties, conceptual attributes such as identity, function and typical context, and visuomotor processing supporting hand transport, grasp form, and object manipulation. Functionally separable brain regions across the dorsal and ventral visual pathways support the processing of these different object properties and, in cohort, are necessary for functional object use. Object-directed grasps display end-state-comfort: they anticipate in form and force the shape and material properties of the grasp target, and how the object will be manipulated after it is grasped. End-state-comfort is the default for everyday interactions with manipulable objects and implies integration of information across the ventral and dorsal visual pathways. We propose a model of how visuomotor and action representations in parietal cortex interact with object representations in ventral and lateral occipito-temporal cortex. One pathway, from the supramarginal gyrus to the middle and inferior temporal gyrus, supports the integration of action-related information, including hand and limb position (supramarginal gyrus) with conceptual attributes and an appreciation of the action goal (middle temporal gyrus). A second pathway, from posterior IPS to the fusiform gyrus and collateral sulcus supports the integration of grasp parameters (IPS) with the surface texture and material properties (e.g., weight distribution) of the grasp target. Reciprocal interactions among these regions are part of a broader network of regions that support everyday functional object interactions.

A dedicated mental resource for intuitive physics

Countless decisions and actions in daily life draw on a mental model of the physical structure and dynamics of the world - from stepping carefully around a patch of slippery pavement to stacking delicate produce in a shopping basket. People can make fast and accurate inferences about how physical interactions will unfold, but it remains unclear whether we do so by applying a general set of physical principles across scenarios, or instead by reasoning about the physics of individual scenarios in an ad-hoc fashion. Here, we hypothesized that humans possess a dedicated and flexible mental resource for physical inference, and we tested for such a resource using a battery of fine-tuned tasks to capture individual differences in performance. Despite varying scene contents across tasks, we found that performance was highly correlated among them and well-explained by a unitary intuitive physics resource, distinct from other facets of cognition such as spatial reasoning and working memory.

The Role of Agentive and Physical Forces in the Neural Representation of Motion Events

How does the brain represent information about motion events in relation to agentive and physical forces? In this study, we investigated the neural activity patterns associated with observing animated actions of agents (e.g., an agent hitting a chair) in comparison to similar movements of inanimate objects that were either shaped solely by the physics of the scene (e.g., gravity causing an object to fall down a hill and hit a chair) or initiated by agents (e.g., a visible agent causing an object to hit a chair). Using an fMRI-based multivariate pattern analysis (MVPA), this design allowed testing where in the brain the neural activity patterns associated with motion events change as a function of, or are invariant to, agentive versus physical forces behind them. A total of 29 human participants (nine male) participated in the study. Cross-decoding revealed a shared neural representation of animate and inanimate motion events that is invariant to agentive or physical forces in regions spanning frontoparietal and posterior temporal cortices. In contrast, the right lateral occipitotemporal cortex showed a higher sensitivity to agentive events, while the left dorsal premotor cortex was more sensitive to information about inanimate object events that were solely shaped by the physics of the scene.

On the functional brain networks involved in tool-related action understanding

Tool-use skills represent a significant cognitive leap in human evolution, playing a crucial role in the emergence of complex technologies. Yet, the neural mechanisms underlying such capabilities are still debated. Here we explore with fMRI the functional brain networks involved in tool-related action understanding. Participants viewed images depicting action-consistent (e.g., nail-hammer) and action-inconsistent (e.g., scarf-hammer) object-tool pairs, under three conditions: semantic (recognizing the tools previously seen in the pairs), mechanical (assessing the usability of the pairs), and control (looking at the pairs without explicit tasks). During the observation of the pairs, task-based left-brain functional connectivity differed within conditions. Compared to the control, both the semantic and mechanical conditions exhibited co-activations in dorsal (precuneus) and ventro-dorsal (inferior frontal gyrus) regions. However, the semantic condition recruited medial and posterior temporal areas, whereas the mechanical condition engaged inferior parietal and posterior temporal regions. Also, when distinguishing action-consistent from action-inconsistent pairs, an extensive frontotemporal neural circuit was activated. These findings support recent accounts that view tool-related action understanding as the combined product of semantic and mechanical knowledge. Furthermore, they emphasize how the left inferior parietal and anterior temporal lobes might be considered as hubs for the cross-modal integration of physical and conceptual knowledge, respectively.

Grounding Intuitive Physics in Perceptual Experience

This review article explores the foundation of laypeople's understanding of the physical world rooted in perceptual experience. Beginning with a concise historical overview of the study of intuitive physics, the article presents the hypothesis that laypeople possess accurate internalized representations of physical laws. A key aspect of this hypothesis is the contention that correct representations of physical laws emerge in ecological experimental conditions, where the scenario being examined resembles everyday life experiences. The article critically examines empirical evidence both supporting and challenging this claim, revealing that despite everyday-life-like conditions, fundamental misconceptions often persist. Many of these misconceptions can be attributed to a domain-general heuristic that arises from the overgeneralization of perceptual-motor experiences with physical objects. To conclude, the article delves into ongoing controversies and highlights promising future avenues in the field of intuitive physics, including action-judgment dissociations, insights from developmental psychology, and computational models integrating artificial intelligence.