Functional neuroanatomy of intuitive physical inference

When it all falls down: the relationship between intuitive physics and spatial cognition

Our intuitive understanding of physical dynamics is crucial in daily life. When we fill a coffee cup, stack items in a refrigerator, or navigate around a slippery patch of ice, we draw on our intuitions about how physical interactions will unfold. What mental machinery underlies our ability to form such inferences? Numerous aspects of cognition must contribute - for example, spatial thinking, temporal prediction, and working memory, to name a few. Is intuitive physics merely the sum of its parts - a collection of these and other related abilities that we apply to physical scenarios as we would to other tasks? Or does physical reasoning rest on something extra - a devoted set of mental resources that takes information from other cognitive systems as inputs? Here, we take a key step in addressing this question by relating individual differences on a physical prediction task to performance on spatial tasks, which may be most likely to account for intuitive physics abilities given the fundamentally spatial nature of physical interactions. To what degree can physical prediction performance be disentangled from spatial thinking? We tested 100 online participants in an “Unstable Towers” task and measures of spatial cognition and working memory. We found a positive relationship between intuitive physics and spatial skills, but there were substantial, reliable individual differences in physical prediction ability that could not be accounted for by spatial measures or working memory. Our findings point toward the separability of intuitive physics from spatial cognition.

How the brain represents mass

New fMRI experiments and machine learning are helping to identify how the mass of objects is processed in the brain.

Invariant representations of mass in the human brain

An intuitive understanding of physical objects and events is critical for successfully interacting with the world. Does the brain achieve this understanding by running simulations in a mental physics engine, which represents variables such as force and mass, or by analyzing patterns of motion without encoding underlying physical quantities? To investigate, we scanned participants with fMRI while they viewed videos of objects interacting in scenarios indicating their mass. Decoding analyses in brain regions previously implicated in intuitive physical inference revealed mass representations that generalized across variations in scenario, material, friction, and motion energy. These invariant representations were found during tasks without action planning, and tasks focusing on an orthogonal dimension (object color). Our results support an account of physical reasoning where abstract physical variables serve as inputs to a forward model of dynamics, akin to a physics engine, in parietal and frontal cortex.

Modeling human intuitions about liquid flow with particle-based simulation

Humans can easily describe, imagine, and, crucially, predict a wide variety of behaviors of liquids—splashing, squirting, gushing, sloshing, soaking, dripping, draining, trickling, pooling, and pouring—despite tremendous variability in their material and dynamical properties. Here we propose and test a computational model of how people perceive and predict these liquid dynamics, based on coarse approximate simulations of fluids as collections of interacting particles. Our model is analogous to a “game engine in the head”, drawing on techniques for interactive simulations (as in video games) that optimize for efficiency and natural appearance rather than physical accuracy. In two behavioral experiments, we found that the model accurately captured people’s predictions about how liquids flow among complex solid obstacles, and was significantly better than several alternatives based on simple heuristics and deep neural networks. Our model was also able to explain how people’s predictions varied as a function of the liquids’ properties (e.g., viscosity and stickiness). Together, the model and empirical results extend the recent proposal that human physical scene understanding for the dynamics of rigid, solid objects can be supported by approximate probabilistic simulation, to the more complex and unexplored domain of fluid dynamics.

Although most people struggle to learn physics in school, every human brain is a remarkable “intuitive physicist” when it comes to the quick, unconscious judgments we make in interacting with the world. Almost effortlessly, and with surprisingly high quantitative accuracy, we can judge when a plate placed near the edge of a table might be at risk of falling, or how far a partially filled glass of water can be tipped before the water is in danger of spilling. What kinds of computations in the brain support these abilities? We suggest an answer based on probabilistic inference operating over particle-based simulations, the same class of approximation methods used in video games to simulate convincing real-time interactions between objects in a virtual environment. This hypothesis can potentially account for people’s quantitative, graded judgments in diverse and novel situations including a wide array of materials and physical properties, without positing a large number of separate systems or heuristics. Here, we build on previous evidence that a system of approximate probabilistic simulation supports judgments about rigid objects (e.g. judging the stability of towers of blocks, as in the game Jenga), and ask whether people can also make systematic and accurate predictions about flowing and splashing liquids, such as water or honey. We show that it is possible to capture people’s quantitative predictions using a computational model that approximates the true underlying fluid dynamics to varying degrees of coarseness, and find that people’s responses are most consistent with a very coarse approximation; while typical engineering applications might use tens or hundreds of thousands of particles to simulate a fluid, the brain might get by with roughly a hundred particles. Furthermore, we find that people consistently underestimate the potential energy of a splashing liquid in our virtual scenes, and that our model captures this behavior.

Behavioral and oculomotor evidence for visual simulation of object movement

We regularly interact with moving objects in our environment. Yet, little is known about how we extrapolate the future movements of visually perceived objects. One possibility is that movements are experienced by a mental visual simulation, allowing one to internally picture an object's upcoming motion trajectory, even as the object itself remains stationary. Here we examined this possibility by asking human participants to make judgments about the future position of a falling ball on an obstacle-filled display. We found that properties of the ball's trajectory were highly predictive of subjects' reaction times and accuracy on the task. We also found that the eye movements subjects made while attempting to ascertain where the ball might fall had significant spatiotemporal overlap with those made while actually perceiving the ball fall. These findings suggest that subjects simulated the ball's trajectory to inform their responses. Finally, we trained a convolutional neural network to see whether this problem could be solved by simple image analysis as opposed to the more intricate simulation strategy we propose. We found that while the network was able to solve our task, the model's output did not effectively or consistently predict human behavior. This implies that subjects employed a different strategy for solving our task, and bolsters the conclusion that they were engaging in visual simulation. The current study thus provides support for visual simulation of motion as a means of understanding complex visual scenes and paves the way for future investigations of this phenomenon at a neural level.

The Neural Representations of Movement across Semantic Categories

Previous evidence from neuropsychological and neuroimaging studies suggests functional specialization for tools and related semantic knowledge in a left frontoparietal network. It is still debated whether these areas are involved in the representation of rudimentary movement-relevant knowledge regardless of semantic domains (animate vs. inanimate) or categories (tools vs. nontool objects). Here, we used fMRI to record brain activity while 13 volunteers performed two semantic judgment tasks on visually presented items from three different categories: animals, tools, and nontool objects. Participants had to judge two distinct semantic features: whether two items typically move in a similar way (e.g., a fan and a windmill move in circular motion) or whether they are usually found in the same environment (e.g., a seesaw and a swing are found in a playground). We investigated differences in overall activation (which areas are involved) as well as representational content (which information is encoded) across semantic features and categories. Results of voxel-wise mass univariate analysis showed that, regardless of semantic category, a dissociation emerges between processing information on prototypical location (involving the anterior temporal cortex and the angular gyrus) and movement (linked to left inferior parietal and frontal activation). Multivoxel pattern correlation analyses confirmed the representational segregation of networks encoding task- and category-related aspects of semantic processing. Taken together, these findings suggest that the left frontoparietal network is recruited to process movement properties of items (including both biological and nonbiological motion) regardless of their semantic category.

An integrative computational architecture for object-driven cortex

Computational architecture for object-driven cortex Objects in motion activate multiple cortical regions in every lobe of the human brain. Do these regions represent a collection of independent systems, or is there an overarching functional architecture spanning all of object-driven cortex? Inspired by recent work in artificial intelligence (AI), machine learning, and cognitive science, we consider the hypothesis that these regions can be understood as a coherent network implementing an integrative computational system that unifies the functions needed to perceive, predict, reason about, and plan with physical objects-as in the paradigmatic case of using or making tools. Our proposal draws on a modeling framework that combines multiple AI methods, including causal generative models, hybrid symbolic-continuous planning algorithms, and neural recognition networks, with object-centric, physics-based representations. We review evidence relating specific components of our proposal to the specific regions that comprise object-driven cortex, and lay out future research directions with the goal of building a complete functional and mechanistic account of this system.

Functional fixedness in tool use: Learning modality, limitations and individual differences

Functional fixedness is a cognitive bias that describes how previous knowledge of a tool's function can negatively impact the use of this tool in novel contexts. As such, functional fixedness disturbs the use of tools during mechanical problem solving. Little is known about whether this bias emerges from different experiences with tools, whether it occurs regardless of problem difficulty, or whether there are protective factors against it. To resolve the first issue, we created five experimental groups: Reading (R), Video (V), Manual (M), No Functional Fixedness (NFF), and No Training (NT). The R group learned to use tools by reading a description of their use, the V group by watching an instructional video, and the M group through direct instruction and active manipulation of the tools. To resolve the remaining two issues, we created mechanical puzzles of distinct difficulty and used tests of intuitive physics, fine motor skills, and creativity. Results showed that misleading functional knowledge is at the core of functional fixedness, and that this bias generates cognitive impasses in simple puzzles, but it does not play a role in higher difficulty problems. Additionally, intuitive physics and motor skills were protective factors against its emergence, but creativity did not influence it. Although functional fixedness leads to inaccurate problem solving, our results suggest that its effects are more limited than previously assumed.

Real World Problem-Solving

Real world problem-solving (RWPS) is what we do every day. It requires flexibility, resilience, resourcefulness, and a certain degree of creativity. A crucial feature of RWPS is that it involves continuous interaction with the environment during the problem-solving process. In this process, the environment can be seen as not only a source of inspiration for new ideas but also as a tool to facilitate creative thinking. The cognitive neuroscience literature in creativity and problem-solving is extensive, but it has largely focused on neural networks that are active when subjects are not focused on the outside world, i.e., not using their environment. In this paper, I attempt to combine the relevant literature on creativity and problem-solving with the scattered and nascent work in perceptually-driven learning from the environment. I present my synthesis as a potential new theory for real world problem-solving and map out its hypothesized neural basis. I outline some testable predictions made by the model and provide some considerations and ideas for experimental paradigms that could be used to evaluate the model more thoroughly.

Dissociating intuitive physics from intuitive psychology: Evidence from Williams syndrome

Prior work suggests that our understanding of how things work ("intuitive physics") and how people work ("intuitive psychology") are distinct domains of human cognition. Here we directly test the dissociability of these two domains by investigating knowledge of intuitive physics and intuitive psychology in adults with Williams syndrome (WS) - a genetic developmental disorder characterized by severely impaired spatial cognition, but relatively spared social cognition. WS adults and mental-age matched (MA) controls completed an intuitive physics task and an intuitive psychology task. If intuitive physics is a distinct domain (from intuitive psychology), then we should observe differential impairment on the physics task for individuals with WS compared to MA controls. Indeed, adults with WS performed significantly worse on the intuitive physics than the intuitive psychology task, relative to controls. These results support the hypothesis that knowledge of the physical world can be disrupted independently from knowledge of the social world.

Perceiving social interactions in the posterior superior temporal sulcus

Primates are highly attuned not just to social characteristics of individual agents, but also to social interactions between multiple agents. Here we report a neural correlate of the representation of social interactions in the human brain. Specifically, we observe a strong univariate response in the posterior superior temporal sulcus (pSTS) to stimuli depicting social interactions between two agents, compared with (i) pairs of agents not interacting with each other, (ii) physical interactions between inanimate objects, and (iii) individual animate agents pursuing goals and interacting with inanimate objects. We further show that this region contains information about the nature of the social interaction-specifically, whether one agent is helping or hindering the other. This sensitivity to social interactions is strongest in a specific subregion of the pSTS but extends to a lesser extent into nearby regions previously implicated in theory of mind and dynamic face perception. This sensitivity to the presence and nature of social interactions is not easily explainable in terms of low-level visual features, attention, or the animacy, actions, or goals of individual agents. This region may underlie our ability to understand the structure of our social world and navigate within it.

The Quest for the FFA and Where It Led

This article tells the story behind our first paper on the fusiform face area (FFA): how we chose the question, developed the methods, and followed the data to find the FFA and subsequently many other functionally specialized cortical regions. The paper's impact had less to do with the particular findings in the paper itself and more to do with the method that it promoted and the picture of the human mind and brain that it led to. The use of a functional localizer to define a candidate region in each subject individually enabled us not just to make pictures of brain activation, but also to ask principled, hypothesis-driven questions about a thing in nature. This method enabled stronger and more extensive tests of the function of each cortical region than had been possible before in humans and, as a result, has produced a large body of evidence that the human cortex contains numerous regions that are specifically engaged in particular mental processes. The growing inventory of cortical regions with distinctive and often very specific functions can be seen as an initial sketch of the basic components of the human mind. This sketch also serves as a roadmap into the vast and exciting new landscape of questions about the computations, structural connections, time course, development, plasticity, and evolution of each of these regions, as well as the hardest question of all: how do these regions work together to produce human intelligence?

Making Sense of Real-World Scenes

To interact with the world, we have to make sense of the continuous sensory input conveying information about our environment. A recent surge of studies has investigated the processes enabling scene understanding, using increasingly complex stimuli and sophisticated analyses to highlight the visual features and brain regions involved. However, there are two major challenges to producing a comprehensive framework for scene understanding. First, scene perception is highly dynamic, subserving multiple behavioral goals. Second, a multitude of different visual properties co-occur across scenes and may be correlated or independent. We synthesize the recent literature and argue that for a complete view of scene understanding, it is necessary to account for both differing observer goals and the contribution of diverse scene properties.