Bayesian action&perception: representing the world in the brain

High-dimensional cortical signals reveal rich bimodal and working memory-like representations among S1 neuron populations

Complexity is important for flexibility of natural behavior and for the remarkably efficient learning of the brain. Here we assessed the signal complexity among neuron populations in somatosensory cortex (S1). To maximize our chances of capturing population-level signal complexity, we used highly repeatable resolvable visual, tactile, and visuo-tactile inputs and neuronal unit activity recorded at high temporal resolution. We found the state space of the spontaneous activity to be extremely high-dimensional in S1 populations. Their processing of tactile inputs was profoundly modulated by visual inputs and even fine nuances of visual input patterns were separated. Moreover, the dynamic activity states of the S1 neuron population signaled the preceding specific input long after the stimulation had terminated, i.e., resident information that could be a substrate for a working memory. Hence, the recorded high-dimensional representations carried rich multimodal and internal working memory-like signals supporting high complexity in cortical circuitry operation.

Remembrance of things perceived: Adding thalamocortical function to artificial neural networks

Recent research has illuminated the complexity and importance of the thalamocortical system but it has been difficult to identify what computational functions it performs. Meanwhile, deep-learning artificial neural networks (ANNs) based on bio-inspired models of purely cortical circuits have achieved surprising success solving sophisticated cognitive problems associated historically with human intelligence. Nevertheless, the limitations and shortcomings of artificial intelligence (AI) based on such ANNs are becoming increasingly clear. This review considers how the addition of thalamocortical connectivity and its putative functions related to cortical attention might address some of those shortcomings. Such bio-inspired models are now providing both testable theories of biological cognition and improved AI technology, much of which is happening outside the usual academic venues.

Developing Intelligent Robots that Grasp Affordance

Humans and robots operating in unstructured environments both need to classify objects through haptic exploration and use them in various tasks, but currently they differ greatly in their strategies for acquiring such capabilities. This review explores nascent technologies that promise more convergence. A novel form of artificial intelligence classifies objects according to sensory percepts during active exploration and decides on efficient sequences of exploratory actions to identify objects. Representing objects according to the collective experience of manipulating them provides a substrate for discovering causality and affordances. Such concepts that generalize beyond explicit training experiences are an important aspect of human intelligence that has eluded robots. For robots to acquire such knowledge, they will need an extended period of active exploration and manipulation similar to that employed by infants. The efficacy, efficiency and safety of such behaviors depends on achieving smooth transitions between movements that change quickly from exploratory to executive to reflexive. Animals achieve such smoothness by using a hierarchical control scheme that is fundamentally different from those of conventional robotics. The lowest level of that hierarchy, the spinal cord, starts to self-organize during spontaneous movements in the fetus. This allows its connectivity to reflect the mechanics of the musculoskeletal plant, a bio-inspired process that could be used to adapt spinal-like middleware for robots. Implementation of these extended and essential stages of fetal and infant development is impractical, however, for mechatronic hardware that does not heal and replace itself like biological tissues. Instead such development can now be accomplished in silico and then cloned into physical robots, a strategy that could transcend human performance.

Remote cortical perturbation dynamically changes the network solutions to given tactile inputs in neocortical neurons

The neocortex has a globally encompassing network structure, which for each given input constrains the possible combinations of neuronal activations across it. Hence, its network contains solutions. But in addition, the cortex has an ever-changing multidimensional internal state, causing each given input to result in a wide range of specific neuronal activations. Here we use intracellular recordings in somatosensory cortex (SI) neurons of anesthetized rats to show that remote, subthreshold intracortical electrical perturbation can impact such constraints on the responses to a set of spatiotemporal tactile input patterns. Whereas each given input pattern normally induces a wide set of preferred response states, when combined with cortical perturbation response states that did not otherwise occur were induced and consequently made other response states less likely. The findings indicate that the physiological network structure can dynamically change as the state of any given cortical region changes, thereby enabling a rich, multifactorial, perceptual capability.

•

Tactile sensory input patterns evoke multi-structure cortical neuron responses

•

Multi-structure responses are shown to be impacted by remote cortical regions

•

Highly dynamic neuron responses reflects global cortical information integration

•

Perception hence depends on globally distributed activity at the time of input

Multi-structure Cortical States Deduced From Intracellular Representations of Fixed Tactile Input Patterns

The brain has a never-ending internal activity, whose spatiotemporal evolution interacts with external inputs to constrain their impact on brain activity and thereby how we perceive them. We used reproducible touch-related spatiotemporal sensory inputs and recorded intracellularly from rat (Sprague-Dawley, male) neocortical neurons to characterize this interaction. The synaptic responses, or the summed input of the networks connected to the neuron, varied greatly to repeated presentations of the same tactile input pattern delivered to the tip of digit 2. Surprisingly, however, these responses tended to sort into a set of specific time-evolving response types, unique for each neuron. Further, using a set of eight such tactile input patterns, we found each neuron to exhibit a set of specific response types for each input provided. Response types were not determined by the global cortical state, but instead likely depended on the time-varying state of the specific subnetworks connected to each neuron. The fact that some types of responses recurred indicates that the cortical network had a non-continuous landscape of solutions for these tactile inputs. Therefore, our data suggest that sensory inputs combine with the internal dynamics of the brain networks, thereby causing them to fall into one of the multiple possible perceptual attractor states. The neuron-specific instantiations of response types we observed suggest that the subnetworks connected to each neuron represent different components of those attractor states. Our results indicate that the impact of cortical internal states on external inputs is substantially more richly resolvable than previously shown.

It's in your hands: How variable perception affects grasping estimates in virtual reality

Successful interaction within one’s environment is contingent upon one’s ability to accurately perceive the extent over which actions can be performed, referred to as action boundaries. As our possibilities for action are subject to variability, it is necessary for individuals to be able to update their perceived action boundaries to accommodate for variance. While research has shown that individuals can update their action boundaries to accommodate for variability, it is unclear how the perceptual system calibrates to this variance to inform our action boundaries. This study investigated the influence of perceptual motor variability by analysing the effect of random and systematic variability on perceived grasp ability in virtual reality. Participants estimated grasp ability following perceptual-motor experience with a constricted, normal, extended, or variable grasp. In Experiment 1, participants experienced all three grasping abilities (constricted, normal, extended) 33% of the time. In Experiment 2 participants experienced the constricted and normal grasps 25% of the time, and the extended grasp 50% of the time. The results indicated that when perceptual-motor feedback is inconsistent, the perceptual system disregards the frequency of perceptual-motor experience with the different action capabilities and considers each action capability experienced as a type, and subsequently calibrates to the average action boundary experienced by type.

A new approach to medical diagnostic decision support

Data mining is a powerful tool to reduce costs and mitigate errors in the diagnostic analysis and repair of complex engineered system, but it has yet to be applied systematically to the most complex and socially expensive system - the human body. The currently available approaches of knowledge-based and pattern-based artificial intelligence are unsuited to the iterative and often subjective nature of clinician-patient interactions. Furthermore, current electronic health records generally have poor design and low quality for such data mining. Bayesian methods have been developed to suggest multiple possible diagnoses given a set of clinical findings, but the larger problem is advising the physician on useful next steps. A new approach based on inverting Bayesian inference allows identification of the diagnostic actions that are most likely to disambiguate a differential diagnosis at each point in a patient's work-up. This can be combined with personalized cost information to suggest a cost-effective path to the clinician. Because the software is tracking the clinician's decision-making process, it can provide salient suggestions for both diagnoses and diagnostic tests in standard, coded formats that need only to be selected. This would reduce the need to type in free text, which is prone to ambiguities, omissions and errors. As the database of high-quality records grows, the scope, utility and acceptance of the system should also grow automatically, without requiring expert updating or correction.

Learning to use Muscles

The human musculoskeletal system is highly complex mechanically. Its neural control must deal successfully with this complexity to perform the diverse, efficient, robust and usually graceful behaviors of which humans are capable. Most of those behaviors might be performed by many different subsets of its myriad possible states, so how does the nervous system decide which subset to use? One solution that has received much attention over the past 50 years would be for the nervous system to be fundamentally limited in the patterns of muscle activation that it can access, a concept known as muscle synergies or movement primitives. Another solution, based on engineering control methodology, is for the nervous system to compute the single optimal pattern of muscle activation for each task according to a cost function. This review points out why neither appears to be the solution used by humans. There is a third solution that is based on trial-and-error learning, recall and interpolation of sensorimotor programs that are good-enough rather than limited or optimal. The solution set acquired by an individual during the protracted development of motor skills starting in infancy forms the basis of motor habits, which are inherently low-dimensional. Such habits give rise to muscle usage patterns that are consistent with synergies but do not reflect fundamental limitations of the nervous system and can be shaped by training or disability. This habit-based strategy provides a robust substrate for the control of new musculoskeletal structures during evolution as well as for efficient learning, athletic training and rehabilitation therapy.

Motor modules account for active perception of force

Despite longstanding evidence suggesting a relation between action and perception, the mechanisms underlying their integration are still unclear. It has been proposed that to simplify the sensorimotor integration processes underlying active perception, the central nervous system (CNS) selects patterns of movements aimed at maximizing sampling of task-related sensory input. While previous studies investigated the action-perception loop focusing on the role of higher-level features of motor behavior (e.g., kinematic invariants, effort), the present study explored and quantified the contribution of lower-level organization of motor control. We tested the hypothesis that the coordinated recruitment of group of muscles (i.e., motor modules) engaged to counteract an external force contributes to participants’ perception of the same force. We found that: 1) a model describing the modulation of a subset of motor modules involved in the motor task accounted for about 70% of participants’ perceptual variance; 2) an alternative model, incompatible with the motor modules hypothesis, accounted for significantly lower variance of participants’ detection performance. Our results provide empirical evidence of the potential role played by muscle activation patterns in active perception of force. They also suggest that a modular organization of motor control may mediate not only coordination of multiple muscles, but also perceptual inference.

Proprioceptive Recognition with Artificial Neural Networks Based on Organizations of Spinocerebellar Tract and Cerebellum

Muscle kinematics and kinetics are nonlinearly encoded by proprioceptors, and the changes in muscle length and velocity are integrated into Ia afferent. Besides, proprioceptive signals from multiple muscles are probably mixed in afferent pathways, which all lead to difficulties in proprioceptive recognition for the cerebellum. In this study, the artificial neural networks, whose organizations are biologically based on the spinocerebellar tract and cerebellum, are utilized to decode the proprioceptive signals. Consistent with the controversy of the proprioceptive division in the dorsal spinocerebellar tract, the spinocerebellar tract networks incorporated two distinct inferences, (1) the centralized networks, which mixed Ia, II, and Ib and processed them together; (2) the decentralized networks, which processed Ia, II, and Ib afferents separately. The cerebellar networks were based on the Marr-Albus model to recognize the kinematic states. The networks were trained by a specific movement, and the trained networks were subsequently required to predict kinematic states of six movements. The results demonstrated that the centralized networks, which were more consistent with the physiological findings in recent years, had better recognition accuracy than the decentralized networks, and the networks were still effective even when proprioceptive afferents from multiple muscles were integrated.

Auditory-induced response in the primary sensory cortex of rodents

The details of auditory response at the subthreshold level in the rodent primary somatosensory cortex, the barrel cortex, have not been studied extensively, although several phenomenological reports have been published. Multisensory features may act as neuronal representations of links between inputs from one sensory modality to other sensory modalities. Here, we examined the basic multisensory postsynaptic responses in the rodent barrel cortex using in vivo whole-cell recordings of neurons. We observed robust responses to acoustic stimuli in most barrel cortex neurons. Acoustically evoked responses were mediated by hearing and reached approximately 60% of the postsynaptic response amplitude elicited by strong somatosensory stimuli. Compared to tactile stimuli, auditory stimuli evoked postsynaptic potentials with a longer latency and longer duration. Specifically, auditory stimuli in barrel cortex neurons appeared to trigger "up states", episodes associated with membrane depolarization and increased synaptic activity. Taken together, our data suggest that barrel cortex neurons have multisensory properties, with distinct synaptic mechanisms underlying tactile and non-tactile responses.

Intracellular Dynamics in Cuneate Nucleus Neurons Support Self-Stabilizing Learning of Generalizable Tactile Representations

How the brain represents the external world is an unresolved issue for neuroscience, which could provide fundamental insights into brain circuitry operation and solutions for artificial intelligence and robotics. The neurons of the cuneate nucleus form the first interface for the sense of touch in the brain. They were previously shown to have a highly skewed synaptic weight distribution for tactile primary afferent inputs, suggesting that their connectivity is strongly shaped by learning. Here we first characterized the intracellular dynamics and inhibitory synaptic inputs of cuneate neurons in vivo and modeled their integration of tactile sensory inputs. We then replaced the tactile inputs with input from a sensorized bionic fingertip and modeled the learning-induced representations that emerged from varied sensory experiences. The model reproduced both the intrinsic membrane dynamics and the synaptic weight distributions observed in cuneate neurons in vivo. In terms of higher level model properties, individual cuneate neurons learnt to identify specific sets of correlated sensors, which at the population level resulted in a decomposition of the sensor space into its recurring high-dimensional components. Such vector components could be applied to identify both past and novel sensory experiences and likely correspond to the fundamental haptic input features these neurons encode in vivo. In addition, we show that the cuneate learning architecture is robust to a wide range of intrinsic parameter settings due to the neuronal intrinsic dynamics. Therefore, the architecture is a potentially generic solution for forming versatile representations of the external world in different sensor systems.

Sensory Flow as a Basis for a Novel Distance Cue in Freely Behaving Electric Fish

The sensory input that an animal receives is directly linked to its motor activity. Behavior thus enables animals to influence their sensory input, a concept referred to as active sensing. How such behavior can serve as a scaffold for generating sensory information is of general scientific interest. In this article, we investigate how behavior can shape sensory information by using some unique features of the sensorimotor system of the weakly electric fish. Based on quantitative behavioral characterizations and computational reconstruction of sensory input, we show how electrosensory flow is actively created during highly patterned, spontaneous behavior in Gnathonemus petersii. The spatiotemporal structure of the sensory input provides information for the computation of a novel distance cue, which allows for a continuous estimation of distance. This has significant advantages over previously known nondynamic distance estimators as determined from electric image blur. Our investigation of the sensorimotor interactions in pulsatile electrolocation shows, for the first time, that the electrosensory flow contains behaviorally relevant information accessible only through active behavior. As patterned sensory behaviors are a shared feature of (active) sensory systems, our results have general implications for the understanding of (active) sensing, with the proposed sensory flow-based measure being potentially pertinent to a broad range of sensory modalities.

SIGNIFICANCE STATEMENT

Acquisition of sensory information depends on motion, as either an animal or its sensors move. Behavior can thus actively influence the sensory flow; and in this way, behavior can be seen as a manifestation of the brain's integrative functions. The properties of the active pulsatile electrolocation system in Gnathonemus petersii allow for the sensory input to be computationally reconstructed, enabling us to link the informational content of spatiotemporal sensory dynamics to behavior. Our study reveals a novel sensory cue for estimating depth that is actively generated by the fishes' behavior. The physical and behavioral similarities between electrolocation and other active sensory systems suggest that this may be a mechanism shared by (active) sensory systems.

Biomimetic Active Touch with Fingertips and Whiskers

This study provides a synthetic viewpoint that compares, contrasts, and draws commonalities for biomimetic perception over a range of tactile sensors and tactile stimuli. Biomimetic active perception is formulated from three principles: (i) evidence accumulation based on leading models of perceptual decision making; (ii) action selection with an evidence-based policy, here based on overt focal attention; and (iii) sensory encoding of evidence based on neural coding. Two experiments with each of three biomimetic tactile sensors are considered: the iCub (capacitive) fingertip, the TacTip (optical) tactile sensor, and BIOTACT whiskers. For each sensor, one experiment considers a similar task (perception of shape and location) and the other a different tactile perception task. In all experiments, active perception with a biomimetic action selection policy based on focal attention outperforms passive perception with static or random action selection. The active perception also consistently reaches superresolved accuracy (hyperacuity) finer than the spacing between tactile elements. Biomimetic active touch thus offers a common approach for biomimetic tactile sensors to accurately and robustly characterize and explore non-trivial, uncertain environments analogous to how animals perceive the natural world.

The Representation of Objects in Apraxia: From Action Execution to Error Awareness

Apraxia is a well-known syndrome characterized by the sufferer's inability to perform routine gestures. In an attempt to understand the syndrome better, various different theories have been developed and a number of classifications of different subtypes have been proposed. In this article review, we will address these theories with a specific focus on how the use of objects helps us to better understand upper limb apraxia. With this aim, we will consider transitive vs. intransitive action dissociation as well as less frequent types of apraxia involving objects, i.e., constructive apraxia and magnetic apraxia. Pantomime and the imitation of objects in use are also considered with a view to dissociating the various different components involved in upper limb apraxia. Finally, we discuss the evidence relating to action recognition and awareness of errors in the execution of actions. Various different components concerning the use of objects emerge from our analysis and the results show that knowledge of an object and sensory-motor representations are supported by other functions such as spatial and body representations, executive functions and monitoring systems.

Lower neuronal variability in the monkey dorsolateral prefrontal than posterior parietal cortex

The dorsolateral prefrontal and posterior parietal cortex are two brain areas involved in cognitive functions such as spatial attention and working memory. When tested with identical tasks, only subtle differences in firing rate are present between neurons recorded in the two areas. In this article we report that major differences in neuronal variability characterize the two areas during working memory. The Fano factors of spike counts in dorsolateral prefrontal neurons were consistently lower than those of the posterior parietal cortex across a range of tasks, epochs, and conditions in the same monkeys. Variability differences were observed despite minor differences in firing rates between the two areas in the tasks tested and higher overall firing rate in the prefrontal than in the posterior parietal sample. Other measures of neuronal discharge variability, such as the coefficient of variation of the interspike interval, displayed the same pattern of lower prefrontal variability. Fano factor values were negatively correlated with performance in the working memory task, suggesting that higher neuronal variability was associated with diminished task performance. The results indicate that information involving remembered stimuli is more reliably represented in the prefrontal than the posterior parietal cortex based on the variability of neuronal responses, and suggest functional differentiation between the two areas beyond differences in firing rate.

Emergence of Cognition from Action

Theories of brain function have evolved through multiple stages. The first proposition was that brain networks support a set of reflex responses, with current sensory inputs producing immediate motor outputs. The behaviorist paradigm suggested that actions can always be explained as a response to immediate external cues. In response to these views, the cognitive paradigm argued that behavior cannot be understood simply as input-output functions because the hidden layers of brain generate unpredictability. The central processing was termed "cognition." Here we propose a neuroscience-based model of cognition. Our core hypothesis is that cognition depends on internal models of the animal and its world, where internally generated sequences can serve to perform "what if" scenarios and anticipate the possible consequences of alternative actions without actually testing them, and aid in the decisions of overt actions. We support our hypotheses by several examples of recent experimental findings and show how externally guided cell assembly sequences become internalized to support cognitive functions.