Gravity in the Brain as a Reference for Space and Time Perception

Invariant representations in abstract concept grounding - the physical world in grounded cognition

Grounded cognition states that mental representations of concepts consist of experiential aspects. For example, the concept "cup" consists of the sensorimotor experiences from interactions with cups. Typical modalities in which concepts are grounded are: The sensorimotor system (including interoception), emotion, action, language, and social aspects. Here, we argue that this list should be expanded to include physical invariants (unchanging features of physical motion; e.g., gravity, momentum, friction). Research on physical reasoning consistently demonstrates that physical invariants are represented as fundamentally as other grounding substrates, and therefore should qualify. We assess several theories of concept representation (simulation, conceptual metaphor, conceptual spaces, predictive processing) and their positions on physical invariants. We find that the classic grounded cognition theories, simulation and conceptual metaphor theory, have not considered physical invariants, while conceptual spaces and predictive processing have. We conclude that physical invariants should be included into grounded cognition theories, and that the core mechanisms of simulation and conceptual metaphor theory are well suited to do this. Furthermore, conceptual spaces and predictive processing are very promising and should also be integrated with grounded cognition in the future.

Human estimates of descending objects' motion are more accurate than those of ascending objects regardless of gravity information

Humans can accurately estimate and track object motion, even if it accelerates. Research shows that humans exhibit superior estimation and tracking performance for descending (falling) than ascending (rising) objects. Previous studies presented ascending and descending targets along the gravitational and body axes in an upright posture. Thus, it is unclear whether humans rely on congruent information between the direction of the target motion and gravity or the direction of the target motion and longitudinal body axes. Two experiments were conducted to explore these possibilities. In Experiment 1, participants estimated the arrival time at a goal for both upward and downward motion of targets along the longitudinal body axis in the upright (both axes of target motion and gravity congruent) and supine (both axes incongruent) postures. In Experiment 2, smooth pursuit eye movements were assessed while tracking both targets in the same postures. Arrival time estimation and smooth pursuit eye movement performance were consistently more accurate for downward target motion than for upward motion, irrespective of posture. These findings suggest that the visual experience of seeing an object moving along an observer's leg side in everyday life may influence the ability to accurately estimate and track the descending object's motion.

Interception of vertically approaching objects: temporal recruitment of the internal model of gravity and contribution of optical information

Introduction: Recent views posit that precise control of the interceptive timing can be achieved by combining on-line processing of visual information with predictions based on prior experience. Indeed, for interception of free-falling objects under gravity's effects, experimental evidence shows that time-to-contact predictions can be derived from an internal gravity representation in the vestibular cortex. However, whether the internal gravity model is fully engaged at the target motion outset or reinforced by visual motion processing at later stages of motion is not yet clear. Moreover, there is no conclusive evidence about the relative contribution of internalized gravity and optical information in determining the time-to-contact estimates. Methods: We sought to gain insight on this issue by asking 32 participants to intercept free falling objects approaching directly from above in virtual reality. Object motion had durations comprised between 800 and 1100 ms and it could be either congruent with gravity (1 g accelerated motion) or not (constant velocity or -1 g decelerated motion). We analyzed accuracy and precision of the interceptive responses, and fitted them to Bayesian regression models, which included predictors related to the recruitment of a priori gravity information at different times during the target motion, as well as based on available optical information. Results: Consistent with the use of internalized gravity information, interception accuracy and precision were significantly higher with 1 g motion. Moreover, Bayesian regression indicated that interceptive responses were predicted very closely by assuming engagement of the gravity prior 450 ms after the motion onset, and that adding a predictor related to on-line processing of optical information improved only slightly the model predictive power. Discussion: Thus, engagement of a priori gravity information depended critically on the processing of the first 450 ms of visual motion information, exerting a predominant influence on the interceptive timing, compared to continuously available optical information. Finally, these results may support a parallel processing scheme for the control of interceptive timing.

Rhythmic tapping to a moving beat motion kinematics overrules natural gravity

Beat induction is the cognitive ability that allows humans to listen to a regular pulse in music and move in synchrony with it. Although auditory rhythmic cues induce more consistent synchronization than flashing visual metronomes, this auditory-visual asymmetry can be canceled by visual moving stimuli. Here, we investigated whether the naturalness of visual motion or its kinematics could provide a synchronization advantage over flashing metronomes. Subjects were asked to tap in sync with visual metronomes defined by vertically accelerating/decelerating motion, either congruent or not with natural gravity; horizontally accelerating/decelerating motion; or flashing stimuli. We found that motion kinematics was the predominant factor determining rhythm synchronization, as accelerating moving metronomes in any cardinal direction produced more precise and predictive tapping than decelerating or flashing conditions. Our results support the notion that accelerating visual metronomes convey a strong sense of beat, as seen in the cueing movements of an orchestra director.

Delay and Acceleration Threshold of Movement Perception in Patients Suffering from Vertigo or Dizziness

Background: The objective was to evaluate the delay and the acceleration threshold (AT) of movement perception in a population of patients suffering from dizziness and analyze the factors influencing these parameters. Methods: This prospective study included 256 adult subjects: 16 control and 240 patients (5 acute unilateral vestibular loss, 13 compensated unilateral loss, 32 Meniere diseases, 48 persistent postural-perceptual dizziness (PPPD), 95 benign paroxysmal positional vertigo (BPPV), 10 central cases, 19 bilateral vestibulopathy, 14 vestibular migraine, and 4 age-related dizziness). Patients were evaluated for the sound–movement synchronicity perception (maximum delay between the bed oscillation peak and a beep perceived as synchronous, PST) and AT during a pendular movement on a swinging bed. Results: We observed higher PST in women and in senior patients regardless of etiology. AT was higher in senior patients. AT was not influenced by etiology except in patients with bilateral vestibulopathy who had higher thresholds. AT was related to unipodal stance performance, past history of fall, and stop-walking-when-talking test. Conclusions: Delay and acceleration thresholds appear to be coherent with clinical findings and open insights on the exploration of symptoms that cannot be explained by routine otoneurological tests.

Spaceflight alters reaction time and duration judgment of astronauts

We report a study on astronauts aimed at characterizing duration judgment before, during, and after long-duration stays on board the International Space Station. Ten astronauts and a control group of 15 healthy (non-astronaut) participants performed a duration reproduction task and a duration production task using a visual target duration ranging from 2 to 38 s. Participants also performed a reaction time test for assessing attention. Compared to control participants and preflight responses, the astronauts’ reaction time increased during spaceflight. Also, during spaceflight, time intervals were under-produced while counting aloud and under-reproduced when there was a concurrent reading task. We hypothesize that time perception during spaceflight is altered by two mechanisms: (a) an acceleration of the internal clock through the changes in vestibular inputs in microgravity, and (b) difficulties in attention and working memory when a concurrent reading task is present. Prolonged isolation in confined areas, weightlessness, stress related to workload, and high-performance expectations could account for these cognitive impairments.

Humans sense by touch the location of objects that roll in handheld containers

Humans use active touch to gain behaviourally relevant information from their environment, including information about contained objects. Although most common, the perceptual basis of interacting with containers remains largely unexplored. The first aim of this study was to determine how accurately people can sense, by touch only, the location of a contained rolling object. Experiment 1 used tubes containing physical balls and demonstrated a considerable degree of accuracy in estimating the rolled distance. The second aim was to identify the relative effectiveness of the various available physical cues. Experiment 2 employed virtual reality technology to present, in isolation and in various combinations, the constituent haptic cues produced by a rolling ball, which are, the mechanical noise during rolling, the jolts from an impact with an internal wall, and the intensity and timing of the jolts resulting from elastic bounces. The rolling noise was of primary importance to the perceptual estimation task suggesting that the implementation of the laws of motion is based on an analysis of the ball's movement velocity. Although estimates became more accurate when the rolling and impact cues were combined, they were not necessarily more precise. The presence of elastic bounces did not affect performance.

The brain-body disconnect: A somatic sensory basis for trauma-related disorders

Although the manifestation of trauma in the body is a phenomenon well-endorsed by clinicians and traumatized individuals, the neurobiological underpinnings of this manifestation remain unclear. The notion of somatic sensory processing, which encompasses vestibular and somatosensory processing and relates to the sensory systems concerned with how the physical body exists in and relates to physical space, is introduced as a major contributor to overall regulatory, social-emotional, and self-referential functioning. From a phylogenetically and ontogenetically informed perspective, trauma-related symptomology is conceptualized to be grounded in brainstem-level somatic sensory processing dysfunction and its cascading influences on physiological arousal modulation, affect regulation, and higher-order capacities. Lastly, we introduce a novel hierarchical model bridging somatic sensory processes with limbic and neocortical mechanisms regulating an individual's emotional experience and sense of a relational, agentive self. This model provides a working framework for the neurobiologically informed assessment and treatment of trauma-related conditions from a somatic sensory processing perspective.

Express: Cognition in Zero Gravity: Effects of Non-Terrestrial Gravity on Human Behaviour

As humanity prepares for deep space exploration, understanding the impact of spaceflight on bodily physiology is critical. While the effects of non-terrestrial gravity on the body are well established, little is known about its impact on human behaviour and cognition. Astronauts often describe dramatic alterations in sensorimotor functioning, including orientation, postural control and balance. Changes in cognitive functioning as well as in socio-affective processing have also been observed. Here we have reviewed the key literature and explored the impact of non-terrestrial gravity across three key functional domains: sensorimotor, cognition, and socio-affective processing. We have proposed a neuroanatomical model to account for the effects of non-terrestrial gravity in these domains. Understanding the impact of non-terrestrial gravity on human behaviour has never been more timely and it will help mitigate against risks in both commercial and non-commercial spaceflight.

Neural Control of Stopping and Stabilizing the Arm

Stopping is a crucial yet under-studied action for planning and producing meaningful and efficient movements. In this review, we discuss classical human psychophysics studies as well as those using engineered systems that aim to develop models of motor control of the upper limb. We present evidence for a hybrid model of motor control, which has an evolutionary advantage due to division of labor between cerebral hemispheres. Stopping is a fundamental aspect of movement that deserves more attention in research than it currently receives. Such research may provide a basis for understanding arm stabilization deficits that can occur following central nervous system (CNS) damage.

Contributions of Body-Orientation to Mental Ball Dropping Task During Out-of-Body Experiences

Out-of-body experiences (OBEs) provide fascinating insights into our understanding of bodily self-consciousness and the workings of the brain. Studies that examined individuals with brain lesions reported that OBEs are generally characterized by participants experiencing themselves outside their physical body (i.e., disembodied feeling) (Blanke and Arzy, 2005). Based on such a characterization, it has been shown that it is possible to create virtual OBEs in immersive virtual environments (Ehrsson, 2007; Ionta et al., 2011b; Bourdin et al., 2017). However, the extent to which body-orientation influences virtual OBEs is not well-understood. Thus, in the present study, 30 participants (within group design) experienced a full-body ownership illusion (synchronous visuo-tactile stimulation only) induced with a gender-matched full-body virtual avatar seen from the first-person perspective (1PP). At the beginning of the experiment, participants performed a mental ball dropping (MBD) task, seen from the location of their virtual avatar, to provide a baseline measurement. After this, a full-body ownership illusion (embodiment phase) was induced in all participants. This was followed by the virtual OBE illusion phase of the experiment (disembodiment phase) in which the first-person viewpoint was switched to a third-person perspective (3PP), and participants' disembodied viewpoint was gradually raised to 14 m above the virtual avatar, from which altitude they repeated the MBD task. During the experiment, this procedure was conducted twice, and the participants were allocated first to the supine or the standing body position at random. Results of the MBD task showed that the participants experienced increased MBD durations during the supine condition compared to the standing condition. Furthermore, although the findings from the subjective reports confirmed the previous findings of virtual OBEs, no significant difference between the two postures was found for body ownership. Taken together, the findings of the current study make further contributions to our understanding of both the vestibular system and time perception during OBEs.

Human movements do not look the same in a tilted world: Gravitational constraints influence the perception of biological motion

We investigated whether gravitational constraints influence the interaction of visual, proprioceptive and vestibular cues for Biological Motion Perception (BMP). Participants were asked to distinguish between plausible and random point-light movements, while passively placed in either an upright or a tilted body orientation. Manipulating the body orientation with respect to gravity leads to different gravitational signals transmitted by the visual, proprioceptive, and vestibular systems. Participants were overall faster in distinguishing plausible point-light movements than random movements. Critically, response times for biologically plausible point-light movements - but not for random movements - were significantly prolonged in the tilted body orientation. Our results suggest that BMP depends not only on the spatial-temporal cues embedded in point-light movements but also rely on the congruency between current gravitational signals detected by the sensory systems and our previous knowledge of terrestrial gravity. STATEMENT OF RELEVANCE: As humankind is preparing for a new space age, understanding how gravity influences behaviour and cognition has never been more pressing. All living organisms have evolved to survive in a terrestrial gravitational field. Although we cannot consciously feel gravity, it has an impact in our life: it affects how we move and interact with the external environment. The sensory signals from the vestibular system are continuously combined with visual and proprioceptive cues to help us in maintaining a stable representation of the world. Here we placed participants in a tilted body orientation and were able to determine that a conflict between prior gravitational knowledge and what was actively sensed about gravity affected human Biological Movement Perception. Humans suffer changes in perception under non-terrestrial gravity conditions that may potentially compromise performance during space exploration.

Watching the Effects of Gravity. Vestibular Cortex and the Neural Representation of "Visual" Gravity

Gravity is a physical constraint all terrestrial species have adapted to through evolution. Indeed, gravity effects are taken into account in many forms of interaction with the environment, from the seemingly simple task of maintaining balance to the complex motor skills performed by athletes and dancers. Graviceptors, primarily located in the vestibular otolith organs, feed the Central Nervous System with information related to the gravity acceleration vector. This information is integrated with signals from semicircular canals, vision, and proprioception in an ensemble of interconnected brain areas, including the vestibular nuclei, cerebellum, thalamus, insula, retroinsula, parietal operculum, and temporo-parietal junction, in the so-called vestibular network. Classical views consider this stage of multisensory integration as instrumental to sort out conflicting and/or ambiguous information from the incoming sensory signals. However, there is compelling evidence that it also contributes to an internal representation of gravity effects based on prior experience with the environment. This a priori knowledge could be engaged by various types of information, including sensory signals like the visual ones, which lack a direct correspondence with physical gravity. Indeed, the retinal accelerations elicited by gravitational motion in a visual scene are not invariant, but scale with viewing distance. Moreover, the "visual" gravity vector may not be aligned with physical gravity, as when we watch a scene on a tilted monitor or in weightlessness. This review will discuss experimental evidence from behavioral, neuroimaging (connectomics, fMRI, TMS), and patients' studies, supporting the idea that the internal model estimating the effects of gravity on visual objects is constructed by transforming the vestibular estimates of physical gravity, which are computed in the brainstem and cerebellum, into internalized estimates of virtual gravity, stored in the vestibular cortex. The integration of the internal model of gravity with visual and non-visual signals would take place at multiple levels in the cortex and might involve recurrent connections between early visual areas engaged in the analysis of spatio-temporal features of the visual stimuli and higher visual areas in temporo-parietal-insular regions.

Where is my hand in space? The internal model of gravity influences proprioception

Knowing where our limbs are in space is crucial for a successful interaction with the external world. Joint position sense (JPS) relies on both cues from muscle spindles and joint mechanoreceptors, as well as the effort required to move. However, JPS may also rely on the perceived external force on the limb, such as the gravitational field. It is well known that the internal model of gravity plays a large role in perception and behaviour. Thus, we have explored whether direct vestibular-gravitational cues could influence JPS. Participants passively estimated the position of their hand while they were upright and therefore aligned with terrestrial gravity, or pitch-tilted 45° backwards from gravity. Overall participants overestimated the position of their hand in both upright and tilted postures; however, the proprioceptive bias was significantly reduced when participants were tilted. Our findings therefore suggest that the internal model of gravity may influence and update JPS in order to allow the organism to interact with the environment.

An Implanted Vestibular Prosthesis Improves Spatial Orientation in Animals with Severe Vestibular Damage

Gravity is a pervasive environmental stimulus, and accurate graviception is required for optimal spatial orientation and postural stability. The primary graviceptors are the vestibular organs, which include angular velocity (semicircular canals) and linear acceleration (otolith organs) sensors. Graviception is degraded in patients with vestibular damage, resulting in spatial misperception and imbalance. Since minimal therapy is available for these patients, substantial effort has focused on developing a vestibular prosthesis or vestibular implant (VI) that reproduces information normally provided by the canals (since reproducing otolith function is very challenging technically). Prior studies demonstrated that angular eye velocity responses could be driven by canal VI-mediated angular head velocity information, but it remains unknown whether a canal VI could improve spatial perception and posture since these behaviors require accurate estimates of angular head position in space relative to gravity. Here, we tested the hypothesis that a canal VI that transduces angular head velocity and provides this information to the brain via motion-modulated electrical stimulation of canal afferent nerves could improve the perception of angular head position relative to gravity in monkeys with severe vestibular damage. Using a subjective visual vertical task, we found that normal female monkeys accurately sensed the orientation of the head relative to gravity during dynamic tilts, that this ability was degraded following bilateral vestibular damage, and improved when the canal VI was used. These results demonstrate that a canal VI can improve graviception in vestibulopathic animals, suggesting that it could reduce the disabling perceptual and postural deficits experienced by patients with severe vestibular damage.SIGNIFICANCE STATEMENT Patients with vestibular damage experience impaired vision, spatial perception, and balance, symptoms that could potentially respond to a vestibular implant (VI). Anatomic features facilitate semicircular canal (angular velocity) prosthetics but inhibit approaches with the otolith (linear acceleration) organs, and canal VIs that sense angular head velocity can generate compensatory eye velocity responses in vestibulopathic subjects. Can the brain use canal VI head velocity information to improve estimates of head orientation (e.g., head position relative to gravity), which is a prerequisite for accurate spatial perception and posture? Here we show that a canal VI can improve the perception of head orientation in vestibulopathic monkeys, results that are highly significant because they suggest that VIs mimicking canal function can improve spatial orientation and balance in vestibulopathic patients.

The Effects of Visual Parabolic Motion on the Subjective Vertical and on Interception

Observers typically present a strong bias in estimating the orientation of a visual bar when their body is tilted >60° in the roll plane and in the absence of visual background information. Known as the A-effect, this phenomenon likely results from the under-compensation of body tilt. Static visual cues can reduce such bias in the perceived vertical. Yet, it is unknown whether dynamic visual cues would be also effective. Here we presented projectile motions of a visual target along parabolic trajectories with different orientations relative to physical gravity. The aim of the experiment was twofold: First, we assessed whether the projectile motions could bias the estimation of the perceived orientation of a visual bar, measured with a classical subjective visual vertical (SVV) task. Second, we evaluated whether the ability to estimate time-to-contact of the visual target in an interception task was influenced by the orientation of these parabolic trajectories. Two groups of participants performed the experiment, either with their head and body tilted 90° along the roll plane or in an upright position. We found that the perceived orientation of the visual bar in the SVV task was affected by the orientation of the parabolic trajectories. This result was present in the tilted but not in the upright participants. In the interception task, the timing error increased linearly as a function of the orientation of the parabola. These results support the hypothesis that a gravity vector estimated from dynamic visual stimuli contributes to the subjective visual vertical.

Gravity prior in human behaviour: a perceptual or semantic phenomenon?

Humans show a gravitational advantage in perception: we are more precise at judging the speed of downwards-moving than upwards-moving objects, indicating that gravitational acceleration is an internalised prior. However, it is unclear whether this gravity prior is based on purely perceptual cues or whether it can incorporate semantic knowledge. Previous research has used only objects which are known to comply with gravity, possibly confounding semantic and perceptual cues. Here we have addressed this question by asking participants to judge the speed of objects that typically move coherently with gravity (ball) or against it (rocket). Our results showed a perceptual advantage for falling stimuli, irrespective of object identity, suggesting the gravity prior is based on perceptual cues.

The gravitational imprint on sensorimotor planning and control

Humans excel at learning complex tasks, and elite performers such as musicians or athletes develop motor skills that defy biomechanical constraints. All actions require the movement of massive bodies. Of particular interest in the process of sensorimotor learning and control is the impact of gravitational forces on the body. Indeed, efficient control and accurate internal representations of the body configuration in space depend on our ability to feel and anticipate the action of gravity. Here we review studies on perception and sensorimotor control in both normal and altered gravity. Behavioral and modeling studies together suggested that the nervous system develops efficient strategies to take advantage of gravitational forces across a wide variety of tasks. However, when the body was exposed to altered gravity, the rate and amount of adaptation exhibited substantial variation from one experiment to another and sometimes led to partial adjustment only. Overall, these results support the hypothesis that the brain uses a multimodal and flexible representation of the effect of gravity on our body and movements. Future work is necessary to better characterize the nature of this internal representation and the extent to which it can adapt to novel contexts.

EMT Transcription Factors Are Involved in the Altered Cell Adhesion under Simulated Microgravity Effect or Overloading by Regulation of E-cadherin

In order to study the effect of stress changes on cell adhesion, HUVEC, and MCF-7 cells were treated with simulated microgravity effect (SMG) and overloading (OL). Methods: Rotating Wall Vessel (2D-RWVS) bioreactor was used to create different culture conditions. In addition, the alteration of cell adhesion states, adhesion proteins, and relating factors of adhesion molecules under these two conditions were detected using cell adhesion assay, immunofluorescence, western blot, and qRT-PCR technology. Results: The results showed that the adhesion of cells decreased under SMG, while increased under OL. The expressions of integrin β1, paxillin, and E-cadherin under SMG condition were down-regulated as compared to that of the control group showing a time-dependent pattern of the decreasing. However, under OL condition, the expressions of adhesion proteins were up-regulated as compared to that of the control group, with a time-dependent pattern of increasing. EMT transcription factors Snail, twist, and ZEB1 were up-regulated under SMG while down-regulated under OL. Conclusion: Collectively our results indicated that cells could respond to stress changes to regulate the expressions of adhesion proteins and adapt their adhesion state to the altered mechanical environment. The altered cell adhesion in response to the mechanical stress may involve the changed expression of EMT-inducing factors, Snail, Twist, and ZEB1under the SMG/OL conditions.

Seeing Gravity: Gait Adaptations to Visual and Physical Inclines - A Virtual Reality Study

Using advanced virtual reality technology, we demonstrate that exposure to virtual inclinations visually simulating inclined walking induces gait modulations in a manner consistent with expected gravitational forces (i.e., acting upon a free body), suggesting vision-based perception of gravity. The force of gravity critically impacts the regulation of our movements. However, how humans perceive and incorporate gravity into locomotion is not well understood. In this study, we introduce a novel paradigm for exposing humans to incongruent sensory information under conditions constrained by distinct gravitational effects, facilitating analysis of the consistency of human locomotion with expected gravitational forces. Young healthy adults walked under conditions of actual physical inclinations as well as virtual inclinations. We identify and describe ‘braking’ and ‘exertion’ effects – locomotor adaptations accommodating gravito-inertial forces associated with physical inclines. We show that purely visual cues (from virtual inclinations) induce consistent locomotor adaptations to counter expected gravity-based changes, consistent with indirect prediction mechanisms. Specifically, downhill visual cues activate the braking effect in anticipation of a gravitational boost, whereas uphill visual cues promote an exertion effect in anticipation of gravitational deceleration. Although participants initially rely upon vision to accommodate environmental changes, a sensory reweighting mechanism gradually reprioritizes body-based cues over visual ones. A high-level neural model outlines a putative pathway subserving the observed effects. Our findings may be pivotal in designing virtual reality-based paradigms for understanding perception and action in complex environments with potential translational benefits.

The role of cortical areas hMT/V5+ and TPJ on the magnitude of representational momentum and representational gravity: a transcranial magnetic stimulation study

The perceived vanishing location of a moving target is systematically displaced forward, in the direction of motion-representational momentum-, and downward, in the direction of gravity-representational gravity. Despite a wealth of research on the factors that modulate these phenomena, little is known regarding their neurophysiological substrates. The present experiment aims to explore which role is played by cortical areas hMT/V5+, linked to the processing of visual motion, and TPJ, thought to support the functioning of an internal model of gravity, in modulating both effects. Participants were required to perform a standard spatial localization task while the activity of the right hMT/V5+ or TPJ sites was selectively disrupted with an offline continuous theta-burst stimulation (cTBS) protocol, interspersed with control blocks with no stimulation. Eye movements were recorded during all spatial localizations. Results revealed an increase in representational gravity contingent on the disruption of the activity of hMT/V5+ and, conversely, some evidence suggested a bigger representational momentum when TPJ was stimulated. Furthermore, stimulation of hMT/V5+ led to a decreased ocular overshoot and to a time-dependent downward drift of gaze location. These outcomes suggest that a reciprocal balance between perceived kinematics and anticipated dynamics might modulate these spatial localization responses, compatible with a push-pull mechanism.

Perception of Looming Motion in Virtual Reality Egocentric Interception Tasks

Motion in depth is commonly misperceived in Virtual Reality (VR), making it difficult to intercept moving objects, for example, in games. We investigate whether motion cues could be modified to improve these interactions in VR. We developed a time-to-contact estimation task, in which observers ($n=18$n=18) had to indicate by button press when a looming virtual object would collide with their head. We show that users consistently underestimate speed. We construct a user-specific model of motion-in-depth perception, and use this model to propose a novel method to modify monocular depth cues tailored to the specific user, correcting individual response errors in speed estimation. A user study was conducted in a simulated baseball environment and observers were asked to hit a looming baseball back in the direction of the pitcher. The study was conducted with and without intervention and demonstrates the effectiveness of the method in reducing interception errors following cue modifications. The intervention was particularly effective at fast ball speeds where performance is most limited by the user's sensorimotor constraints. The proposed approach is easy to implement and could improve the user experience of interacting with dynamic virtual environments.

Ocular tracking of occluded ballistic trajectories: Effects of visual context and of target law of motion

In tracking a moving target, the visual context may provide cues for an observer to interpret the causal nature of the target motion and extract features to which the visual system is weakly sensitive, such as target acceleration. This information could be critical when vision of the target is temporarily impeded, requiring visual motion extrapolation processes. Here we investigated how visual context influences ocular tracking of motion either congruent or not with natural gravity. To this end, 28 subjects tracked computer-simulated ballistic trajectories either perturbed in the descending segment with altered gravity effects (0g/2g) or retaining natural-like motion (1g). Shortly after the perturbation (550 ms), targets disappeared for either 450 or 650 ms and became visible again until landing. Target motion occurred with either quasi-realistic pictorial cues or a uniform background, presented in counterbalanced order. We analyzed saccadic and pursuit movements after 0g and 2g target-motion perturbations and for corresponding intervals of unperturbed 1g trajectories, as well as after corresponding occlusions. Moreover, we considered the eye-to-target distance at target reappearance. Tracking parameters differed significantly between scenarios: With a neutral background, eye movements did not depend consistently on target motion, whereas with pictorial background they showed significant dependence, denoting better tracking of accelerated targets. These results suggest that oculomotor control is tuned to realistic properties of the visual scene.

Body orientation contributes to modelling the effects of gravity for target interception in humans

KEY POINTS

It is known that interception of targets accelerated by gravity involves internal models coupled with visual signals. Non-visual signals related to head and body orientation relative to gravity may also contribute, although their role is poorly understood. In a novel experiment, we asked pitched observers to hit a virtual target approaching with an acceleration that was either coherent or incoherent with their pitch-tilt. Initially, the timing errors were large and independent of the coherence between target acceleration and observer's pitch. With practice, however, the timing errors became substantially smaller in the coherent conditions. The results show that information about head and body orientation can contribute to modelling the effects of gravity on a moving target. Orientation cues from vestibular and somatosensory signals might be integrated with visual signals in the vestibular cortex, where the internal model of gravity is assumed to be encoded.

ABSTRACT

Interception of moving targets relies on visual signals and internal models. Less is known about the additional contribution of non-visual cues about head and body orientation relative to gravity. We took advantage of Galileo's law of motion along an incline to demonstrate the effects of vestibular and somatosensory cues about head and body orientation on interception timing. Participants were asked to hit a ball rolling in a gutter towards the eyes, resulting in image expansion. The scene was presented in a head-mounted display, without any visual information about gravity direction. In separate blocks of trials participants were pitched backwards by 20° or 60°, whereas ball acceleration was randomized across trials so as to be compatible with rolling down a slope of 20° or 60°. Initially, the timing errors were large, independently of the coherence between ball acceleration and pitch angle, consistent with responses based exclusively on visual information because visual stimuli were identical at both tilts. At the end of the experiment, however, the timing errors were systematically smaller in the coherent conditions than the incoherent ones. Moreover, the responses were significantly (P = 0.007) earlier when participants were pitched by 60° than when they were pitched by 20°. Therefore, practice with the task led to incorporation of information about head and body orientation relative to gravity for response timing. Instead, posture did not affect response timing in a control experiment in which participants hit a static target in synchrony with the last of a predictable series of stationary audiovisual stimuli.

Gravity modulates behaviour control strategy

Human behaviour is a trade-off between exploitation of familiar resources and exploration of new ones. In a challenging environment-such as outer space-making the correct decision is vital. On Earth, gravity is always there, and is an important reference for behaviour. Thus, altered gravitational signals may affect behaviour control strategies. Here, we investigated whether changing the body's orientation to the gravitational vector would modulate the balance between routine and novel behaviour. Participants completed a random number generation task while upright or supine. We found decreased randomness when participants were supine. In particular, the degree of equiprobability of pairs of consecutive responses was reduced in the supine orientation. Online gravitational signals may shape the balance between exploitation and exploration, in favour of more stereotyped and routine responses.

Altered time judgements highlight common mechanisms of time and space perception

Space, numbers and time share similar processing mechanisms mediated by parietal cortex. In parallel to the spatial representation of numbers along a horizontal line, temporal information is mapped on a horizontal axis with short intervals (and the past) represented to the left of long intervals (and the future). Little is known about the representation of time in the presence of visuo-spatial deficits. We here report two experiments on the comparative judgment of time. Experiment 1 required patients with left-sided neglect to indicate which of two consecutively presented silent intervals was longer. Their judgments were better if the first interval was longer and they judged the first interval longer on trials in which the two intervals were equally long. These results were not present in right-hemispheric damaged patients without neglect and healthy controls. They are in line with a previously reported finding in a single patient with neglect, but not readily compatible with findings of neglect patients' comparative length judgments. In Experiment 2, healthy participants' performance on an identical task improved for trials with a first-longer interval after caloric vestibular stimulation (CVS) of the right ear with warm water.

Balance in Blind Subjects: Cane and Fingertip Touch Induce Similar Extent and Promptness of Stance Stabilization

Subjects with low vision often use a cane when standing and walking autonomously in everyday life. One aim of this study was to assess differences in the body stabilizing effect produced by the contact of the cane with the ground or by the fingertip touch of a firm surface. Another aim was to estimate the promptness of balance stabilization (or destabilization) on adding (or withdrawing) the haptic input from cane or fingertip. Twelve blind subjects and two subjects with severe visual impairment participated in two experimental protocols while maintaining the tandem Romberg posture on a force platform. In one protocol, subjects lowered the cane to a second platform on the ground and lifted it in sequence at their own pace. In the other protocol, they touched an instrumented pad with the index finger and withdrew the finger from the pad in sequence. In both protocols, subjects were asked to exert a force not granting mechanical stabilization. Under steady-state condition, the finger touch or the contact of the cane with the ground significantly reduced (to ∼78% and ∼86%, respectively) the amplitude of medio-lateral oscillation of the centre of foot pressure (CoP). Oscillation then increased when haptic information was removed. The delay to the change in body oscillation after the haptic shift was longer for addition than withdrawal of the haptic information (∼1.4 s and ∼0.7 s, respectively; p < 0.001), but was not different between the two haptic conditions (finger and cane). Similar stabilizing effects of input from cane on the ground and from fingertip touch, and similar latencies to integrate haptic cue from both sources, suggest that the process of integration of the input for balance control is initiated by the haptic stimulus at the interface cane-hand. Use of a tool is as helpful as the fingertip input, and does not produce different stabilization. Further, the latencies to haptic cue integration (from fingertip or cane) are similar to those previously found in a group of sighted subjects, suggesting that integration delays for automatic balance stabilization are not modified by visual impairment. Haptic input from a tool is easily exploited by the neural circuits subserving automatic balance stabilization in blind people, and its use should be enforced by sensory-enhancing devices and appropriate training.

Rolling Motion Along an Incline: Visual Sensitivity to the Relation Between Acceleration and Slope

People easily intercept a ball rolling down an incline, despite its acceleration varies with the slope in a complex manner. Apparently, however, they are poor at detecting anomalies when asked to judge artificial animations of descending motion. Since the perceptual deficiencies have been reported in studies involving a limited visual context, here we tested the hypothesis that judgments of naturalness of rolling motion are consistent with physics when the visual scene incorporates sufficient cues about environmental reference and metric scale, roughly comparable to those present when intercepting a ball. Participants viewed a sphere rolling down an incline located in the median sagittal plane, presented in 3D wide-field virtual reality. In different experiments, either the slope of the plane or the sphere acceleration were changed in arbitrary combinations, resulting in a kinematics that was either consistent or inconsistent with physics. In Experiment 1 (slope adjustment), participants were asked to modify the slope angle until the resulting motion looked natural for a given ball acceleration. In Experiment 2 (acceleration adjustment), instead, they were asked to modify the acceleration until the motion on a given slope looked natural. No feedback about performance was provided. For both experiments, we found that participants were rather accurate at finding the match between slope angle and ball acceleration congruent with physics, but there was a systematic effect of the initial conditions: accuracy was higher when the participants started the exploration from the combination of slope and acceleration corresponding to the congruent conditions than when they started far away from the congruent conditions. In Experiment 3, participants modified the slope angle based on an adaptive staircase, but the target never coincided with the starting condition. Here we found a generally accurate performance, irrespective of the target slope. We suggest that, provided the visual scene includes sufficient cues about environmental reference and metric scale, joint processing of slope and acceleration may facilitate the detection of natural motion. Perception of rolling motion may rely on the kind of approximate, probabilistic simulations of Newtonian mechanics that have previously been called into play to explain complex inferences in rich visual scenes.

Supramodal agnosia for oblique mirror orientation in patients with periventricular leukomalacia

Periventricular leukomalacia (PVL) is characterized by focal necrosis at the level of the periventricular white matter, often observed in preterm infants. PVL is frequently associated with motor impairment and with visual deficits affecting primary stages of visual processes as well as higher visual cognitive abilities. Here we describe six PVL subjects, with normal verbal IQ, showing orientation perception deficits in both the haptic and visual domains. Subjects were asked to compare the orientation of two stimuli presented simultaneously or sequentially, using both a two alternative forced choice (2AFC) orientation-discrimination and a matching procedure. Visual stimuli were oriented gratings or bars or collinear short lines embedded within a random pattern. Haptic stimuli comprised two rotatable wooden sticks. PVL patients performed at chance in discriminating the oblique orientation, both for visual and haptic stimuli. Moreover when asked to reproduce the oblique orientation, they often oriented the stimulus along the symmetric mirror orientation. The deficit generalized to stimuli varying in many low level features, was invariant for spatiotopic object orientation, and also occurred for sequential presentations. The deficit was specific to oblique orientations, and not for horizontal or vertical stimuli. These findings show that PVL can affect a specific network involved with the supramodal perception of mirror symmetry orientation.

Mental imagery of gravitational motion

There is considerable evidence that gravitational acceleration is taken into account in the interaction with falling targets through an internal model of Earth gravity. Here we asked whether this internal model is accessed also when target motion is imagined rather than real. In the main experiments, naïve participants grasped an imaginary ball, threw it against the ceiling, and caught it on rebound. In different blocks of trials, they had to imagine that the ball moved under terrestrial gravity (1g condition) or under microgravity (0g) as during a space flight. We measured the speed and timing of the throwing and catching actions, and plotted ball flight duration versus throwing speed. Best-fitting duration-speed curves estimate the laws of ball motion implicit in the participant's performance. Surprisingly, we found duration-speed curves compatible with 0g for both the imaginary 0g condition and the imaginary 1g condition, despite the familiarity with Earth gravity effects and the added realism of performing the throwing and catching actions. In a control experiment, naïve participants were asked to throw the imaginary ball vertically upwards at different heights, without hitting the ceiling, and to catch it on its way down. All participants overestimated ball flight durations relative to the durations predicted by the effects of Earth gravity. Overall, the results indicate that mental imagery of motion does not have access to the internal model of Earth gravity, but resorts to a simulation of visual motion. Because visual processing of accelerating/decelerating motion is poor, visual imagery of motion at constant speed or slowly varying speed appears to be the preferred mode to perform the tasks.

Relative Time Compression for Slow-Motion Stimuli through Rapid Recalibration

A number of psychophysical studies have shown that moving stimuli appear to last longer than static stimuli. Here, we report that the perceived duration for slow moving stimuli can be shorter than for static stimuli under specific circumstances. Observers were tested using natural movies presented at various speeds (0.0× = static, 0.25× = slow, or 1.9× = fast, relative to original speed) and indicated whether test duration was perceived as longer or shorter than comparison movies presented at their original speed. While fast movies were perceived as longer than slow and static movies (in accordance with previous studies), we found that slow movies were perceived as shorter (i.e., time compressed) compared to static images. Similar results were obtained for artificial stimuli consisting of drifting gratings. However, time compression for slow stimuli disappeared if comparison stimuli were replaced by a white static disk that removed repetitive exposures to moving stimuli. Results suggest that duration estimation is modulated by contextual effects induced by the specific diet – or distribution – of prior visual stimuli to which observers are exposed. A simple model, which includes a rapid recalibration of human time estimation via adaptation to preceding stimuli, succeeds in reproducing our experimental data.

Differential contributions to the interception of occluded ballistic trajectories by the temporoparietal junction, area hMT/V5+, and the intraparietal cortex

The ability to catch objects when transiently occluded from view suggests their motion can be extrapolated. Intraparietal cortex (IPS) plays a major role in this process along with other brain structures, depending on the task. For example, interception of objects under Earth's gravity effects may depend on time-to-contact predictions derived from integration of visual signals processed by hMT/V5+ with a priori knowledge of gravity residing in the temporoparietal junction (TPJ). To investigate this issue further, we disrupted TPJ, hMT/V5+, and IPS activities with transcranial magnetic stimulation (TMS) while subjects intercepted computer-simulated projectile trajectories perturbed randomly with either hypo- or hypergravity effects. In experiment 1, trajectories were occluded either 750 or 1,250 ms before landing. Three subject groups underwent triple-pulse TMS (tpTMS, 3 pulses at 10 Hz) on one target area (TPJ | hMT/V5+ | IPS) and on the vertex (control site), timed at either trajectory perturbation or occlusion. In experiment 2, trajectories were entirely visible and participants received tpTMS on TPJ and hMT/V5+ with same timing as experiment 1 tpTMS of TPJ, hMT/V5+, and IPS affected differently the interceptive timing. TPJ stimulation affected preferentially responses to 1-g motion, hMT/V5+ all response types, and IPS stimulation induced opposite effects on 0-g and 2-g responses, being ineffective on 1-g responses. Only IPS stimulation was effective when applied after target disappearance, implying this area might elaborate memory representations of occluded target motion. Results are compatible with the idea that IPS, TPJ, and hMT/V5+ contribute to distinct aspects of visual motion extrapolation, perhaps through parallel processing.NEW & NOTEWORTHY Visual extrapolation represents a potential neural solution to afford motor interactions with the environment in the face of missing information. We investigated relative contributions by temporoparietal junction (TPJ), hMT/V5+, and intraparietal cortex (IPS), cortical areas potentially involved in these processes. Parallel organization of visual extrapolation processes emerged with respect to the target's motion causal nature: TPJ was primarily involved for visual motion congruent with gravity effects, IPS for arbitrary visual motion, whereas hMT/V5+ contributed at earlier processing stages.

Gravity as a Strong Prior: Implications for Perception and Action

In the future, humans are likely to be exposed to environments with altered gravity conditions, be it only visually (Virtual and Augmented Reality), or visually and bodily (space travel). As visually and bodily perceived gravity as well as an interiorized representation of earth gravity are involved in a series of tasks, such as catching, grasping, body orientation estimation and spatial inferences, humans will need to adapt to these new gravity conditions. Performance under earth gravity discrepant conditions has been shown to be relatively poor, and few studies conducted in gravity adaptation are rather discouraging. Especially in VR on earth, conflicts between bodily and visual gravity cues seem to make a full adaptation to visually perceived earth-discrepant gravities nearly impossible, and even in space, when visual and bodily cues are congruent, adaptation is extremely slow. We invoke a Bayesian framework for gravity related perceptual processes, in which earth gravity holds the status of a so called “strong prior”. As other strong priors, the gravity prior has developed through years and years of experience in an earth gravity environment. For this reason, the reliability of this representation is extremely high and overrules any sensory information to its contrary. While also other factors such as the multisensory nature of gravity perception need to be taken into account, we present the strong prior account as a unifying explanation for empirical results in gravity perception and adaptation to earth-discrepant gravities.

Effects of visual motion consistent or inconsistent with gravity on postural sway

Vision plays an important role in postural control, and visual perception of the gravity-defined vertical helps maintaining upright stance. In addition, the influence of the gravity field on objects' motion is known to provide a reference for motor and non-motor behavior. However, the role of dynamic visual cues related to gravity in the control of postural balance has been little investigated. In order to understand whether visual cues about gravitational acceleration are relevant for postural control, we assessed the relation between postural sway and visual motion congruent or incongruent with gravity acceleration. Postural sway of 44 healthy volunteers was recorded by means of force platforms while they watched virtual targets moving in different directions and with different accelerations. Small but significant differences emerged in sway parameters with respect to the characteristics of target motion. Namely, for vertically accelerated targets, gravitational motion (GM) was associated with smaller oscillations of the center of pressure than anti-GM. The present findings support the hypothesis that not only static, but also dynamic visual cues about direction and magnitude of the gravitational field are relevant for balance control during upright stance.

Vestibular stimulation interferes with the dynamics of an internal representation of gravity

The remembered vanishing location of a moving target has been found to be displaced downward in the direction of gravity (representational gravity) and more so with increasing retention intervals, suggesting that the visual spatial updating recruits an internal model of gravity. Despite being consistently linked with gravity, few inquiries have been made about the role of vestibular information in these trends. Previous experiments with static tilting of observers' bodies suggest that under conflicting cues between the idiotropic vector and vestibular signals, the dynamic drift in memory is reduced to a constant displacement along the body's main axis. The present experiment aims to replicate and extend these outcomes while keeping the observers' bodies unchanged in relation to physical gravity by varying the gravito-inertial acceleration using a short-radius centrifuge. Observers were shown, while accelerated to varying degrees, targets moving along several directions and were required to indicate the perceived vanishing location after a variable interval. Increases of the gravito-inertial force (up to 1.4G), orthogonal to the idiotropic vector, did not affect the direction of representational gravity, but significantly disrupted its time course. The role and functioning of an internal model of gravity for spatial perception and orientation are discussed in light of the results.

The vestibular body: Vestibular contributions to bodily representations

Vestibular signals are integrated with signals from other sensory modalities. This convergence could reflect an important mechanism for maintaining the perception of the body. Here we review the current literature in order to develop a framework for understanding how the vestibular system contributes to body representation. According to recent models, we distinguish between three processes for body representation, and we look at whether vestibular signals might influence each process. These are (i) somatosensation, the primary sensory processing of somatic stimuli, (ii) somatoperception, the processes of constructing percepts and experiences of somatic objects and events and (iii) somatorepresentation, the knowledge about the body as a physical object in the world. Vestibular signals appear to contribute to all three levels in this model of body processing. Thus, the traditional view of the vestibular system as a low-level, dedicated orienting module tends to underestimate the pervasive role of vestibular input in bodily self-awareness.