Illusions of force perception: the role of sensori-motor predictions, visual information, and motor errors

Attribution of sensory prediction error to perception of muscle fatigue

Sensory prediction-error is vital to discriminating whether sensory inputs are caused externally or are the consequence of self-action, thereby contributing to a stable perception of the external world and building sense of agency. However, it remains unexplored whether prediction error of self-action is also used to estimate the internal body condition. To address this point, we examined whether prediction error affects the perceived intensity of muscle fatigue. Participants evaluated fatigue while maintaining repetitive finger movements. To provide prediction error, we inserted a temporal delay into online visual feedback of self-movements. The results show that the subjective rating of muscle fatigue significantly increased under the delayed visual feedback, suggesting that prediction error enhances the perception of muscle fatigue. Furthermore, we introduced visual feedback that preceded actual finger movements to test whether the temporal direction of the mismatch is crucial in estimating muscle fatigue. We found that perceived fatigue was significantly weaker with preceding visual feedback compared to normal feedback, showing that the perception of muscle fatigue is affected by the signed prediction-error. Our findings support the idea that the brain flexibly attributes prediction errors to a self-origin with keeping sense of agency, or external origin by considering contexts and error characteristics.

The Impact of Nonminimum-Phase Zeros on Human-in-the-Loop Control Systems

We present results from an experiment in which 33 human subjects interact with a dynamic system 40 times over a one-week period. The subjects are divided into three groups. For each interaction, a subject performs a command-following task, where the reference command is the same for all trials and all subjects. However, each group interacts with a different dynamic system, which is represented by a transfer function. The transfer functions have the same poles but different zeros. One has a minimum-phase zero , another has a nonminimum-phase zero , and the last has a slower (i.e., closer to the imaginary axis) nonminimum-phase zero z_sn ∈ (0,z_n) . The experimental results show that nonminimum-phase zeros tend to make dynamic systems more difficult for humans to learn to control. We use a subsystem identification algorithm to identify the control strategy that each subject uses on each trial. The identification results show that the identified feedforward controllers approximate the inverse dynamics of the system with which the subjects interact better on the last trial than on the first trial. However, the subjects interacting with the minimum-phase system are able to approximate the inverse dynamics in feedforward more accurately than the subjects interacting with the nonminimum-phase system. This observation suggests that nonminimum-phase zeros are an impediment to approximating inverse dynamics in feedforward. Finally, we provide evidence that humans rely on feedforward-step-like-control strategies with systems (e.g., nonminimum-phase systems) for which it is difficult to approximate the inverse dynamics in feedforward.

Size, weight, and expectations

The size-weight illusion is well-known: if two equally heavy objects differ in size, the large one feels lighter than the small one. Most explanations for this illusion assume that because the information about the relevant attribute (weight itself) is unreliable, information about an irrelevant but correlated attribute (size) is used as well. If such reasoning is correct, one would expect that the illusion can be inverted: if size information is unreliable, weight information will be used to judge size. We explored whether such a weight-size illusion exists by asking participants to lift Styrofoam balls that were coated with glow in the dark paint. The balls (2 sizes, 3 weights) were lifted using a pulley system in complete darkness at 2 distances. Participants reported the size using free magnitude estimation. The visual size information was indeed unreliable: balls that were presented at a 20% larger distance were judged 15% smaller. Nevertheless, the judgments of size were not systematically affected by the 20% weight change (differences < 0.5%). We conclude that because the weight-size illusion does not exist, the mechanism behind the size-weight illusion is specific for judging heaviness.

Tandem internal models execute motor learning in the cerebellum

In performing skillful movement, humans use predictions from internal models formed by repetition learning. However, the computational organization of internal models in the brain remains unknown. Here, we demonstrate that a computational architecture employing a tandem configuration of forward and inverse internal models enables efficient motor learning in the cerebellum. The model predicted learning adaptations observed in hand-reaching experiments in humans wearing a prism lens and explained the kinetic components of these behavioral adaptations. The tandem system also predicted a form of subliminal motor learning that was experimentally validated after training intentional misses of hand targets. Patients with cerebellar degeneration disease showed behavioral impairments consistent with tandemly arranged internal models. These findings validate computational tandemization of internal models in motor control and its potential uses in more complex forms of learning and cognition.

The Roles of Feedback and Feedforward as Humans Learn to Control Unknown Dynamic Systems

We present results from an experiment in which human subjects interact with an unknown dynamic system 40 times during a two-week period. During each interaction, subjects are asked to perform a command-following (i.e., pursuit tracking) task. Each subject's performance at that task improves from the first trial to the last trial. For each trial, we use subsystem identification to estimate each subject's feedforward (or anticipatory) control, feedback (or reactive) control, and feedback time delay. Over the 40 trials, the magnitudes of the identified feedback controllers and the identified feedback time delays do not change significantly. In contrast, the identified feedforward controllers do change significantly. By the last trial, the average identified feedforward controller approximates the inverse of the dynamic system. This observation provides evidence that a fundamental component of human learning is updating the anticipatory control until it models the inverse dynamics.

Active inference and robot control: a case study

Active inference is a general framework for perception and action that is gaining prominence in computational and systems neuroscience but is less known outside these fields. Here, we discuss a proof-of-principle implementation of the active inference scheme for the control or the 7-DoF arm of a (simulated) PR2 robot. By manipulating visual and proprioceptive noise levels, we show under which conditions robot control under the active inference scheme is accurate. Besides accurate control, our analysis of the internal system dynamics (e.g. the dynamics of the hidden states that are inferred during the inference) sheds light on key aspects of the framework such as the quintessentially multimodal nature of control and the differential roles of proprioception and vision. In the discussion, we consider the potential importance of being able to implement active inference in robots. In particular, we briefly review the opportunities for modelling psychophysiological phenomena such as sensory attenuation and related failures of gain control, of the sort seen in Parkinson's disease. We also consider the fundamental difference between active inference and optimal control formulations, showing that in the former the heavy lifting shifts from solving a dynamical inverse problem to creating deep forward or generative models with dynamics, whose attracting sets prescribe desired behaviours.

Forward modelling the rubber hand: illusion of ownership modifies motor-sensory predictions by the brain

The question of how we attribute observed body parts as our own, and the consequences of this attribution on our sensory-motor processes, is fundamental to understand how our brain distinguishes between self and other. Previous studies have identified interactions between the illusion of ownership, and multi-sensory integration and cross-sensory predictions by the brain. Here we show that illusory ownership additionally modifies the motor-sensory predictions by the brain. In our preliminary experiments, we observed a new numbness illusion following the classical rubber-hand illusion (RHI); brushing only the rubber hand after induction of the RHI results in illusory numbness in one's real hand. Previous studies have shown that self-generated actions (like tickling) are attenuated by motor-sensory predictions by the so-called forward model. Motivated by this finding, here we examined whether the numbness illusion after the RHI is different when the rubber hand is brushed oneself, compared with when the brushing is performed by another. We observed that, all other conditions remaining the same, haptic perception in the real hand was lower (numbness higher) during self-generated brushing. Our result suggests that RHI reorganizes the forward model, such that we predict haptic consequences of self-generated motor actions on the rubber hand.

Human creativity, evolutionary algorithms, and predictive representations: The mechanics of thought trials

Creative thinking is arguably the pinnacle of cerebral functionality. Like no other mental faculty, it has been omnipotent in transforming human civilizations. Probing the neural basis of this most extraordinary capacity, however, has been doggedly frustrated. Despite a flurry of activity in cognitive neuroscience, recent reviews have shown that there is no coherent picture emerging from the neuroimaging work. Based on this, we take a different route and apply two well established paradigms to the problem. First is the evolutionary framework that, despite being part and parcel of creativity research, has no informed experimental work in cognitive neuroscience. Second is the emerging prediction framework that recognizes predictive representations as an integrating principle of all cognition. We show here how the prediction imperative revealingly synthesizes a host of new insights into the way brains process variation-selection thought trials and present a new neural mechanism for the partial sightedness in human creativity. Our ability to run offline simulations of expected future environments and action outcomes can account for some of the characteristic properties of cultural evolutionary algorithms running in brains, such as degrees of sightedness, the formation of scaffolds to jump over unviable intermediate forms, or how fitness criteria are set for a selection process that is necessarily hypothetical. Prospective processing in the brain also sheds light on how human creating and designing - as opposed to biological creativity - can be accompanied by intentions and foresight. This paper raises questions about the nature of creative thought that, as far as we know, have never been asked before.

Cerebral Hemodynamic Responses During Dynamic Posturography: Analysis with a Multichannel Near-Infrared Spectroscopy System

To investigate cortical roles in standing balance, cortical hemodynamic activity was recorded from the right hemisphere using near-infrared spectroscopy (NIRS) while subjects underwent the sensory organization test (SOT) protocol that systematically disrupts sensory integration processes (i.e., somatosensory or visual inputs or both). Eleven healthy men underwent the SOT during NIRS recording. Group statistical analyses were performed based on changes in oxygenated hemoglobin concentration in 10 different cortical regions of interest and on a general linear analysis with NIRS statistical parametric mapping. The statistical analyses indicated significant activation in the right frontal operculum (f-Op), right parietal operculum (p-Op), and right superior temporal gyrus (STG), right posterior parietal cortex (PPC), right dorsal and ventral premotor cortex (PMC), and the supplementary motor area (SMA) under various conditions. The activation patterns in response to specific combinations of SOT conditions suggested that (1) f-Op, p-Op, and STG are essential for sensory integration when standing balance is perturbed; (2) the SMA is involved in the execution of volitional action and establishment of new motor programs to maintain postural balance; and (3) the PPC and PMC are involved in the updating and computation of spatial reference frames during instances of sensory conflict between vestibular and visual information.

The basal ganglia select the expected sensory input used for predictive coding

While considerable evidence supports the notion that lower-level interpretation of incoming sensory information is guided by top-down sensory expectations, less is known about the source of the sensory expectations or the mechanisms by which they are spread. Predictive coding theory proposes that sensory expectations flow down from higher-level association areas to lower-level sensory cortex. A separate theory of the role of prediction in cognition describes "emulations" as linked representations of potential actions and their associated expected sensation that are hypothesized to play an important role in many aspects of cognition. The expected sensations in active emulations are proposed to be the top-down expectation used in predictive coding. Representations of the potential action and expected sensation in emulations are claimed to be instantiated in distributed cortical networks. Combining predictive coding with emulations thus provides a theoretical link between the top-down expectations that guide sensory expectations and the cortical networks representing potential actions. Now moving to theories of action selection, the basal ganglia has long been proposed to select between potential actions by reducing inhibition to the cortical network instantiating the desired action plan. Integration of these isolated theories leads to the novel hypothesis that reduction in inhibition from the basal ganglia selects not just action plans, but entire emulations, including the sensory input expected to result from the action. Basal ganglia disinhibition is hypothesized to both initiate an action and also allow propagation of the action's associated sensory expectation down towards primary sensory cortex. This is a novel proposal for the role of the basal ganglia in biasing perception by selecting the expected sensation, and initiating the top-down transmission of those expectations in predictive coding.

Sensory synergy as environmental input integration

The development of a method to feed proper environmental inputs back to the central nervous system (CNS) remains one of the challenges in achieving natural movement when part of the body is replaced with an artificial device. Muscle synergies are widely accepted as a biologically plausible interpretation of the neural dynamics between the CNS and the muscular system. Yet the sensorineural dynamics of environmental feedback to the CNS has not been investigated in detail. In this study, we address this issue by exploring the concept of sensory synergy. In contrast to muscle synergy, we hypothesize that sensory synergy plays an essential role in integrating the overall environmental inputs to provide low-dimensional information to the CNS. We assume that sensor synergy and muscle synergy communicate using these low-dimensional signals. To examine our hypothesis, we conducted posture control experiments involving lateral disturbance with nine healthy participants. Proprioceptive information represented by the changes on muscle lengths were estimated by using the musculoskeletal model analysis software SIMM. Changes on muscles lengths were then used to compute sensory synergies. The experimental results indicate that the environmental inputs were translated into the two dimensional signals and used to move the upper limb to the desired position immediately after the lateral disturbance. Participants who showed high skill in posture control were found to be likely to have a strong correlation between sensory and muscle signaling as well as high coordination between the utilized sensory synergies. These results suggest the importance of integrating environmental inputs into suitable low-dimensional signals before providing them to the CNS. This mechanism should be essential when designing the prosthesis' sensory system to make the controller simpler.

The effect of temporal perception on weight perception

A successful catch of a falling ball requires an accurate estimation of the timing for when the ball hits the hand. In a previous experiment in which participants performed ball-catching task in virtual reality environment, we accidentally found that the weight of a falling ball was perceived differently when the timing of ball load force to the hand was shifted from the timing expected from visual information. Although it is well known that spatial information of an object, such as size, can easily deceive our perception of its heaviness, the relationship between temporal information and perceived heaviness is still not clear. In this study, we investigated the effect of temporal factors on weight perception. We conducted ball-catching experiments in a virtual environment where the timing of load force exertion was shifted away from the visual contact timing (i.e., time when the ball hit the hand in the display). We found that the ball was perceived heavier when force was applied earlier than visual contact and lighter when force was applied after visual contact. We also conducted additional experiments in which participants were conditioned to one of two constant time offsets prior to testing weight perception. After performing ball-catching trials with 60 ms advanced or delayed load force exertion, participants’ subjective judgment on the simultaneity of visual contact and force exertion changed, reflecting a shift in perception of time offset. In addition, timing of catching motion initiation relative to visual contact changed, reflecting a shift in estimation of force timing. We also found that participants began to perceive the ball as lighter after conditioning to 60 ms advanced offset and heavier after the 60 ms delayed offset. These results suggest that perceived heaviness depends not on the actual time offset between force exertion and visual contact but on the subjectively perceived time offset between them and/or estimation error in force timing.

Emulation as an integrating principle for cognition

Emulations, defined as ongoing internal representations of potential actions and the futures those actions are expected to produce, play a critical role in directing human bodily activities. Studies of gross motor behavior, perception, allocation of attention, response to errors, interoception, and homeostatic activities, and higher cognitive reasoning suggest that the proper execution of all these functions relies on emulations. Further evidence supports the notion that reinforcement learning in humans is aimed at updating emulations, and that action selection occurs via the advancement of preferred emulations toward realization of their action and environmental prediction. Emulations are hypothesized to exist as distributed active networks of neurons in cortical and sub-cortical structures. This manuscript ties together previously unrelated theories of the role of prediction in different aspects of human information processing to create an integrated framework for cognition.

Illusory force perception following a voluntary limb movement

We present a novel illusion in which participants report constant forces on their hand as steadily increasing. Participants made discrete reaching movements perturbed by a lateral force that increased with the distance moved; when stationary at the end of the movement, a true constant force was perceived to increase. We tested perceived subjective equality by increasing or decreasing the force. The illusion was significantly stronger when the perturbation was applied during active movement. We conclude that the unusual context of moving against lateral spring forces results in participants failing to predict steady lateral forces at the end of their movement, and causes an illusion of increasing forces even after movement termination. This result further emphasizes the role of action prediction in sensory perception.

Bayesian and "anti-Bayesian" biases in sensory integration for action and perception in the size-weight illusion

Which is heavier: a pound of lead or a pound of feathers? This classic trick question belies a simple but surprising truth: when lifted, the pound of lead feels heavier--a phenomenon known as the size-weight illusion. To estimate the weight of an object, our CNS combines two imperfect sources of information: a prior expectation, based on the object's appearance, and direct sensory information from lifting it. Bayes' theorem (or Bayes' law) defines the statistically optimal way to combine multiple information sources for maximally accurate estimation. Here we asked whether the mechanisms for combining these information sources produce statistically optimal weight estimates for both perceptions and actions. We first studied the ability of subjects to hold one hand steady when the other removed an object from it, under conditions in which sensory information about the object's weight sometimes conflicted with prior expectations based on its size. Since the ability to steady the supporting hand depends on the generation of a motor command that accounts for lift timing and object weight, hand motion can be used to gauge biases in weight estimation by the motor system. We found that these motor system weight estimates reflected the integration of prior expectations with real-time proprioceptive information in a Bayesian, statistically optimal fashion that discounted unexpected sensory information. This produces a motor size-weight illusion that consistently biases weight estimates toward prior expectations. In contrast, when subjects compared the weights of two objects, their perceptions defied Bayes' law, exaggerating the value of unexpected sensory information. This produces a perceptual size-weight illusion that biases weight perceptions away from prior expectations. We term this effect "anti-Bayesian" because the bias is opposite that seen in Bayesian integration. Our findings suggest that two fundamentally different strategies for the integration of prior expectations with sensory information coexist in the nervous system for weight estimation.

Action and behavior: a free-energy formulation

We have previously tried to explain perceptual inference and learning under a free-energy principle that pursues Helmholtz's agenda to understand the brain in terms of energy minimization. It is fairly easy to show that making inferences about the causes of sensory data can be cast as the minimization of a free-energy bound on the likelihood of sensory inputs, given an internal model of how they were caused. In this article, we consider what would happen if the data themselves were sampled to minimize this bound. It transpires that the ensuing active sampling or inference is mandated by ergodic arguments based on the very existence of adaptive agents. Furthermore, it accounts for many aspects of motor behavior; from retinal stabilization to goal-seeking. In particular, it suggests that motor control can be understood as fulfilling prior expectations about proprioceptive sensations. This formulation can explain why adaptive behavior emerges in biological agents and suggests a simple alternative to optimal control theory. We illustrate these points using simulations of oculomotor control and then apply to same principles to cued and goal-directed movements. In short, the free-energy formulation may provide an alternative perspective on the motor control that places it in an intimate relationship with perception.

What's next? New evidence for prediction in human vision

Everyday visual experience involves making implicit predictions, as revealed by our surprise when something disturbs our expectations. Many theories of vision have been premised on the central role played by prediction. Yet, implicit prediction in human vision has been difficult to assess in the laboratory, and many results have not distinguished between the indisputably important role of memory and the future-oriented aspect of prediction. Now, a new and unexpected finding - that humans can resume an interrupted visual search much faster than they can start a new search - offers new hope, because the rapid resumption of a search seems to depend on participants forming an implicit prediction of what they will see after the interruption. These findings combined with results of recent neurophysiology studies provide a framework for studying implicit prediction in perception.