Visual delay affects force scaling and weight perception during object lifting in virtual reality

Multisensory information about changing object properties can be used to quickly correct predictive force scaling for object lifting

Sensory information about object properties, such as size or material, can be used to make an estimate of object weight and to generate an accurate motor plan to lift the object. When object properties change, the motor plan needs to be corrected based on the new information. The current study investigated whether such corrections could be made quickly, after the movement was initiated. Participants had to grasp and lift objects of different weights that could be indicated with different cues. During the reaching phase, the cue could change to indicate a different weight and participants had to quickly adjust their planned forces in order to lift the object skilfully. The object weight was cued with different object sizes (Experiment 1) or materials (Experiment 2) and the cue was presented in different sensory modality conditions: visually, haptically or both (visuohaptic). Results showed that participants could adjust their planned forces based on both size and material. Furthermore, corrections could be made in the visual, haptic and visuohaptic conditions, although the multisensory condition did not outperform the conditions with one sensory modality. These results suggest that motor plans can be quickly corrected based on sensory information about object properties from different sensory modalities. These findings provide insights into the information that can be shared between brain areas for the online control of hand-object interactions.

Force Control During the Precision Grip Translates to Virtual Reality

When grasping and manipulating objects we implicitly adapt grip forces according to the physical parameters of the object. We integrate visual, cutaneous, and force feedback to estimate these parameters and adapt our control accordingly. Using virtual reality, both feedback integration and control can be investigated in ways that are not possible using real-life objects. Here, we present our custom-built virtual reality setup and show its validity for use in human studies of fine motor control. Participants grasped and lifted virtual objects with different weights. We show that, consistent with lifting real objects, all participants adapt their grip forces to the object mass, and do so on a trial-by-trial basis. Compared to similar studies with real objects and full feedback, grip forces were increased, and adaptation required more trials. This study successfully demonstrated that grip force control in the precision grip translates to virtual reality. While our setup can be used for similar work in the future, subsequent virtual reality experiments should include a longer adaptation phase compared to classic setups.

Brain functional networks analysis of five fingers grasping in virtual reality environment

This study investigated the effects of different center of mass (COM) of the grasping device and visual time-delay on the information interaction between brain regions during five-finger grasping process. Nine healthy right-handed subjects used five fingers to grasp a special device in a virtual reality (VR) environment. Two independent variables were set in the experiment: the COM of the grasping device and the visual delay time. Place a 50 g mass randomly at five different directions of the grasping device base. The three levels of visual delay time appear randomly. The kinematics and dynamics and electroencephalogram (EEG) signals were recorded during the experiment. The brain network was constructed based on multiplex horizontal visibility graph (MHVG). Interlayer mutual information (MI) and phase locking value (PLV) were calculated to quantify the network, while clustering coefficient (C), shortest path length (L) and overall network efficiency (E) are selected to quantify the network characteristic. Statistical results show that when the mass is located in the radial side, during the load phase of grasping, the C and E is significantly higher than that in the proximal, ulnar and medial side, and L was significantly lower than that in the proximal and radial side. This shows that when grasping an object with a COM bias on the radial side, the process of brain feedforward control has higher level of information interaction and ability and it can build stronger sensorimotor memory. It is also found that the brain network features of theta, beta and gamma bands of EEG are positively correlated, especially between beta and gamma bands, which suggests there is a coupling relationship between different bands in information processing and transmission.Clinical Relevance- This study explains the neural mechanism of grasping control from the topological structure of the whole brain network level and the informatics.

Familiarity with a Tool Influences Peripersonal Space and Primary Motor Cortex Excitability of Muscles Involved in Haptic Contact

Long-term experience with a tool stably enlarges peripersonal space (PPS). Also, gained experience with a tool modulates internal models of action. The aim of this work was to understand whether the familiarity with a tool influences both PPS and motor representation. Toward this goal, we tested in 13 expert fencers through a multisensory integration paradigm the embodiment in their PPS of a personal (pE) or a common (cE) épée. Then, we evaluated the primary motor cortex excitability of proximal (ECR) and distal (APB) muscles during a motor imagery (MI) task of an athletic gesture when athletes handled these tools. Results showed that pE enlarges subjects' PPS, while cE does not. Moreover, during MI, handling tools increased cortical excitability of ECR muscle. Notably, APB's cortical excitability during MI only increased with pE as a function of its embodiment in PPS. These findings indicate that the familiarity with a tool specifically enlarges PPS and modulates the cortical motor representation of those muscles involved in the haptic contact with it.

Grip Force Control During Virtual Interaction With Deformable and Rigid Objects Via a Haptic Gripper

Most studies of grip force control focus on the manipulation of rigid objects. However, in virtual and teleoperation applications, objects are often elastic in the pinch degree-of-freedom, and are manipulated using a handle that presents haptic feedback to the user. When designing controllers for haptic grippers, it is crucial to understand how grip force is controlled when manipulating rigid and deformable objects. Here, we used a virtual teleoperation setup with a haptic gripper interface to investigate grip force control in virtual environments. Ten participants lifted virtual objects and performed vertical, cyclic motions using a haptic gripper. We manipulated the control signal to the virtual interface, the haptic properties of the gripper, and the visual properties of the virtual objects to test their effect on the grip force control. We found that participants modulated their grip force as a function of the anticipated load force in all of the experimental conditions. The control signal and properties of the haptic gripper, but not the visual properties of the objects, affected the baseline and the extent of the grip force modulation. These results can provide design guidelines for haptic grippers that elicit natural grasping in virtual and bilateral teleoperation applications.

Reach-to-Grasp: A Multisensory Experience

The reach-to-grasp movement is ordinarily performed in everyday living activities and it represents a key behavior that allows humans to interact with their environment. Remarkably, it serves as an experimental test case for probing the multisensory architecture of goal-oriented actions. This review focuses on experimental evidence that enhances or modifies how we might conceptualize the "multisensory" substrates of prehension. We will review evidence suggesting that how reach-to-grasp movements are planned and executed is influenced by information coming from different sensory modalities such as vision, proprioception, audition, taste, and olfaction. The review closes with some considerations about the predominant role of the multisensory constituents in shaping prehensile behavior and how this might be important for future research developments, especially in the rehabilitative domain.

Visual Feedback of Object Motion Direction Influences the Timing of Grip Force Modulation During Object Manipulation

During manipulation, object slipping is prevented by modulating the grip force (GF) in synchrony with motion-related inertial forces, i.e., load force (LF). However, due to conduction delays of the sensory system, GF must be modulated in advance based on predictions of LF changes. It has been proposed that such predictive force control relies on internal representations, i.e., internal models, of the relation between the dynamic of the environment and movement kinematics. Somatosensory and visual feedback plays a primary role in building these internal representations. For instance, it has been shown that manipulation-dependent somatosensory signals contribute to building internal representations of gravity in normal and altered gravitational contexts. Furthermore, delaying the timing of visual feedback of object displacement has been shown to affect GF. Here, we explored whether and the extent to which spatial features of visual feedback movement, such as motion direction, may contribute to GF control. If this were the case, a spatial mismatch between actual (somatosensory) and visual feedback of object motion would elicit changes in GF modulation. We tested this hypothesis by asking participants to generate vertical object movements while visual feedback of object position was congruent (0° rotation) or incongruent (180° or 90°) with the actual object displacement. The role of vision on GF control was quantified by the temporal shift of GF modulation as a function of visual feedback orientation and actual object motion direction. GF control was affected by visual feedback when this was incongruent in the vertical (180°), but not horizontal dimension. Importantly, 180° visual feedback rotation delayed and anticipated GF modulation during upward and downward actual movements, respectively. Our findings suggest that during manipulation, spatial features of visual feedback motion are used to predict upcoming LF changes. Furthermore, the present study provides evidence that an internal model of gravity contributes to GF control by influencing sensory reweighting processes during object manipulation.

Flexible organization of grip force control during movement frequency scaling

The grip force applied to maintain grasp of a handheld object has been typically reported as tightly coupled to the load force exerted by the object as it is actively manipulated, occurring proportionally and consistently in phase with changes in load force. However, continuous grip force-load force coupling breaks down when overall load force levels and oscillation amplitudes are lower (Grover F, Lamb M, Bonnette S, Silva PL, Lorenz T, Riley MA. Exp Brain Res 236: 2531–2544, 2018) or more predictable (Grover FM, Nalepka P, Silva PL, Lorenz T, Riley MA. Exp Brain Res 237: 687–703, 2019). Under these circumstances, grip force is instead only intermittently coupled to load force; continuous coupling is prompted only when load force levels or variations become sufficiently high or unpredictable. The current study investigated the nature of the transition between continuous and intermittent modes of grip force control by scaling the load force level and the oscillation amplitude continuously in time by means of scaling the required frequency of movement oscillations. Participants grasped a cylindrical object between the thumb and forefinger and oscillated their arm about the shoulder in the sagittal plane. Oscillation frequencies were paced with a metronome that scaled through an ascending or descending frequency progression. Due to greater accelerations, faster frequencies produced greater overall load force levels and more pronounced load oscillations. We observed smooth but nonlinear transitions between clear regimes of intermittent and continuous grip force-load force coordination, for both scaling directions, indicating that grip force control can flexibly reorganize as parameters affecting grasp (e.g., variations in load force) change over time.

NEW & NOTEWORTHY Grip force (GF) is synchronously coupled to changing load forces (LF) during object manipulation when LF levels are high or unpredictable, but only intermittently coupled to LF during less challenging grasp conditions. This study characterized the nature of transitions between synchronous and intermittent GF-LF coupling, revealing a smooth but nonlinear change in intermittent GF modulation in response to continuous scaling of LF amplitude. Intermittent, “drift-and-act” control may provide an alternative framework for understanding GF-LF coupling.

Dynamic size-weight changes after object lifting reduce the size-weight illusion

In the size-weight illusion, the smaller object from two equally weighted objects is typically judged as being heavier. One explanation is that the mismatch between the weight expectation based on object size and actual sensory feedback influences heaviness perception. In most studies, the size of an object is perceived before its weight. We investigated whether size changes would influence weight judgement if both would be perceived simultaneously. We used virtual reality to change the size and weight of an object after lifting and asked participants to judge whether the object became lighter or heavier. We found that simultaneous size-weight changes greatly reduced the size-weight illusion to perceptual biases below discrimination thresholds. In a control experiment in which we used a standard size-weight illusion protocol with sequential lifts of small and large objects in the same virtual reality setup, we found a larger, typical perceptual bias. These results show that the size-weight illusion is smaller when size and weight information is perceived simultaneously. This provides support for the prediction mismatch theory explaining the size-weight illusion. The comparison between perceived and expected weight during the lifting phase could be a critical brain mechanism for mediating the size-weight illusion.

Force perceptual bias caused by muscle activity in unimanual steering

This study sought to investigate whether force perceptual bias was affected by differences in posture while steering an automobile using a psychophysical experiment to examine the relationship with muscle activity. The human perceptual characteristics of weight and force are known to be nonlinear, and a perceptual bias can occur, that is, bias that causes a perception of something that is larger or smaller than the actual scale. This is considered to be caused by physical and/or psychological conditions. Sense of effort is believed to be one influential factor. It is known to correlate with muscle activity intensity, and bias may be caused by muscle activity changes. In the current study, we hypothesized that force perceptual bias would depend on posture due to the intensity of muscle activity changes caused by changing postures during steering operation. By investigating this hypothesis, we can clarify the relationship between sense of effort and muscle activity. To investigate this issue, we conducted a psychophysical experiment to confirm postural dependence, and estimated muscle activity using a three-dimensional musculoskeletal model simulation with postural and arm force data during the experiment. In addition, prediction of bias was conducted based on a simulation in the psychophysical experiment using these data. The results revealed that bias existed, as measured by differences in postures. Additionally, a significant moderate correlation was found between the predicted bias and the actual bias, indicating the existence of a relationship between muscle activity and bias.

Visual delay affects force scaling and weight perception during object lifting in virtual reality

Lifting an object requires precise scaling of fingertip forces based on a prediction of object weight. At object contact, a series of tactile and visual events arise that need to be rapidly processed online to fine-tune the planned motor commands for lifting the object. The brain mechanisms underlying multisensory integration serially at transient sensorimotor events, a general feature of actions requiring hand-object interactions, are not yet understood. In this study we tested the relative weighting between haptic and visual signals when they are integrated online into the motor command. We used a new virtual reality setup to desynchronize visual feedback from haptics, which allowed us to probe the relative contribution of haptics and vision in driving participants’ movements when they grasped virtual objects simulated by two force-feedback robots. We found that visual delay changed the profile of fingertip force generation and led participants to perceive objects as heavier than when lifts were performed without visual delay. We further modeled the effect of vision on motor output by manipulating the extent to which delayed visual events could bias the force profile, which allowed us to determine the specific weighting the brain assigns to haptics and vision. Our results show for the first time how visuo-haptic integration is processed at discrete sensorimotor events for controlling object-lifting dynamics and further highlight the organization of multisensory signals online for controlling action and perception.

NEW & NOTEWORTHY Dexterous hand movements require rapid integration of information from different senses, in particular touch and vision, at different key time points as movement unfolds. The relative weighting between vision and haptics for object manipulation is unknown. We used object lifting in virtual reality to desynchronize visual and haptic feedback and find out their relative weightings. Our findings shed light on how rapid multisensory integration is processed over a series of discrete sensorimotor control points.