Neurocomputational theories of homeostatic control

Emergence of integrated behaviors through direct optimization for homeostasis

Homeostasis is a self-regulatory process, wherein an organism maintains a specific internal physiological state. Homeostatic reinforcement learning (RL) is a framework recently proposed in computational neuroscience to explain animal behavior. Homeostatic RL organizes the behaviors of autonomous embodied agents according to the demands of the internal dynamics of their bodies, coupled with the external environment. Thus, it provides a basis for real-world autonomous agents, such as robots, to continually acquire and learn integrated behaviors for survival. However, prior studies have generally explored problems pertaining to limited size, as the agent must handle observations of such coupled dynamics. To overcome this restriction, we developed an advanced method to realize scaled-up homeostatic RL using deep RL. Furthermore, several rewards for homeostasis have been proposed in the literature. We identified that the reward definition that uses the difference in drive function yields the best results. We created two benchmark environments for homeostasis and performed a behavioral analysis. The analysis showed that the trained agents in each environment changed their behavior based on their internal physiological states. Finally, we extended our method to address vision using deep convolutional neural networks. The analysis of a trained agent revealed that it has visual saliency rooted in the survival environment and internal representations resulting from multimodal input.

Homeostatic Reinforcement Theory Accounts for Sodium Appetitive State- and Taste-Dependent Dopamine Responding

Seeking and consuming nutrients is essential to survival and the maintenance of life. Dynamic and volatile environments require that animals learn complex behavioral strategies to obtain the necessary nutritive substances. While this has been classically viewed in terms of homeostatic regulation, recent theoretical work proposed that such strategies result from reinforcement learning processes. This theory proposed that phasic dopamine (DA) signals play a key role in signaling potentially need-fulfilling outcomes. To examine links between homeostatic and reinforcement learning processes, we focus on sodium appetite as sodium depletion triggers state- and taste-dependent changes in behavior and DA signaling evoked by sodium-related stimuli. We find that both the behavior and the dynamics of DA signaling underlying sodium appetite can be accounted for by a homeostatically regulated reinforcement learning framework (HRRL). We first optimized HRRL-based agents to sodium-seeking behavior measured in rodents. Agents successfully reproduced the state and the taste dependence of behavioral responding for sodium as well as for lithium and potassium salts. We then showed that these same agents account for the regulation of DA signals evoked by sodium tastants in a taste- and state-dependent manner. Our models quantitatively describe how DA signals evoked by sodium decrease with satiety and increase with deprivation. Lastly, our HRRL agents assigned equal preference for sodium versus the lithium containing salts, accounting for similar behavioral and neurophysiological observations in rodents. We propose that animals use orosensory signals as predictors of the internal impact of the consumed good and our results pose clear targets for future experiments. In sum, this work suggests that appetite-driven behavior may be driven by reinforcement learning mechanisms that are dynamically tuned by homeostatic need.

Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite

Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, reward learning, and decision-making. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multimodal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation via sodium appetitive behavior can be understood as a reinforcement learning process, and provide a mechanistic explanation for the underlying neural mechanisms of decision-making related to sodium appetite and homeostatic behavior.

Morality and the Brain: The Right Hemisphere and Doing Right

Morality, the set of shared attitudes and practices that regulate individual behavior to facilitate cohesion and well-being, is a function of the brain, yet its localization is uncertain. Neuroscientific study of morality has been conducted by examining departures from moral conduct after neurologic insult and by functional neuroimaging of moral decision-making in cognitively intact individuals. These investigations have yielded conflicting results: Acquired sociopathy, a syndromic surrogate for acquired immorality, has been reported predominantly after right frontotemporal lesions, whereas functional neuroimaging during moral decision-making has demonstrated bilateral activation. Although morality is bilaterally represented, the right hemisphere is clinically more critical in light of focal lesion data suggesting that moral behavior is subserved by a network of right frontotemporal structures and their subcortical connections. Evolution may have endowed the brain with bilaterally represented but unilaterally right-dominant morality. The unilateral dominance of morality permits concentration of an essential social cognitive function to support the perceptual and executive operations of moral behavior within a single hemisphere; the bilateral representation of morality allows activation of reserve tissue in the contralateral hemisphere in the event of an acquired hemispheric injury. The observed preponderance of right hemisphere lesions in individuals with acquired immorality offers a plausible hypothesis that can be tested in clinical settings. Advances in the neuroscience of morality promise to yield potentially transformative clinical and societal benefits. A deeper understanding of morality would help clinicians address disordered conduct after acquired neurologic insults and guide society in bolstering public health efforts to prevent brain disease.

Computational Models of Interoception and Body Regulation

To survive, organisms must effectively respond to the challenge of maintaining their physiological integrity in the face of an ever-changing environment. Preserving this homeostasis critically relies on adaptive behavior. In this review, we consider recent frameworks that extend classical homeostatic control via reflex arcs to include more flexible forms of adaptive behavior that take interoceptive context, experiences, and expectations into account. Specifically, we define a landscape for computational models of interoception, body regulation, and forecasting, address these models' unique challenges in relation to translational research efforts, and discuss what they can teach us about cognition as well as physical and mental health.

Introduction to the special issue on physics of mind

In recent years, both fields of physics and psychology have made important scientific advances. The emergence of new instruments gave rise to a data-driven neuroscience allowing us to learn about the state of the brain supporting known mental functions and conversely. In parallel, the appearance of new mathematics allowed the development of computational models describing fundamental brain functions and implementing them in technological applications. While emphasizing the methodology of physics, the special issue aims to bring together these trends in both the experimental and theoretical sciences in order to explain some of the most basic mental processes such as perception, cognition, emotion, consciousness, and learning. In this editorial, we define unsolved problems for brain and psychological sciences, discuss possible means toward their respective solutions, and outline some collaborative initiatives aiming toward these goals. The following problems are defined in gradual order of difficulty: what are the universal properties of human behavior across conditions and cultures? What have each culture learned over historical times and why should specific elements of knowledge be accumulated over cultural evolution? Can computational psychiatry help predict, understand, and cure mental disorders? What is the function of art and cultural artifacts such as music, fiction, or poetry for the cognitive system? How to explain the relation between first-person subjective experience and third-person objective physiological data? What neural mechanisms operate on which mental content at the highest levels of organization of the hierarchical brain? How do abstract ideas emerge from sensory-motor contingencies and what are the conditions for the birth of a new concept? Could symmetry play a role in psychogenesis and support the emergence of new hierarchical layers in cognition? How can we start addressing the question of meaning scientifically, and what does it entail for the physical sciences?