Perception and Action in Teleoperated Needle Insertion

Human Stabilization of Delay-Induced Instability of Haptic Rendering in a Stiffness Discrimination Task

Towards developing a coupled stability theory for haptic systems, we study the interaction of operators with time-delayed force feedback. In this work, we analyzed and validated experimentally the stability boundaries of an uncoupled system - without considering the human. We then designed an experiment in which the participants used a haptic device to interact with virtual elastic force fields in a stiffness discrimination task. We compared the performance and kinematics of users in uncoupled-unstable and uncoupled-stable conditions and characterized the stabilizing contribution of the users. We found that the users were able to perform the task regardless of the uncoupled-stability conditions. In addition, in uncoupled-unstable conditions, users maintained movement characteristics that were important for exploratory mediation, such as depth and duration of the movement, whereas other characteristics were not preserved. The results were reproduced in a simulation of the human controller that combined an inverse model and an optimal feedback controller. Adequate performance under the uncoupled-unstable yet coupled-stable conditions supports the potential benefit of designing for coupled stability of haptic systems. This could lead to the use of less conservative controllers than state-of-the-art solutions in haptic and teleoperation systems, and advance the fidelity of haptic feedback.

Effects of visuomotor delays on the control of movement and on perceptual localization in the presence and absence of visual targets

The sensory system constantly deals with delayed feedback. Recent studies showed that playing a virtual game of pong with delayed feedback caused hypermetric reaching movements. We investigated whether this effect is associated with a perceptual bias. In addition, we examined the importance of the target in causing hypermetric movements. In a first experiment, participants played a delayed pong game and blindly reached to presented targets. Following each reaching movement, they assessed the position of the invisible cursor. We found that participants performed hypermetric movements but reported that the invisible cursor reached the target, suggesting that they were unaware of the hypermetria and that their perception was biased toward the target rather than toward their hand position. In a second experiment, we removed the visual target, and strikingly, the hypermetria vanished. Moreover, participants reported that the invisible cursor was located with their hand. Taking these results together, we conclude that the adaptation to the visuomotor delay during the pong game selectively affected the execution of goal directed movements, resulting in hypermetria and perceptual bias when movements are directed toward visual targets but not when such targets are absent.NEW & NOTEWORTHY Recent studies showed that adaptation to visuomotor delays causes hypermetric movements in the absence of visual feedback, suggesting that visuomotor delay is represented using current state information. We report that this adaptation also affects perception. Importantly, both the motor and perceptual effects are selective to the representations that are used in the execution of goal-directed movements toward visual targets.

Visuo-Haptic Discrimination of Viscoelastic Materials

In our daily lives, we interact with different types of deformable materials. Regarding their mechanical behavior, some of those materials lie in a range that is between purely elastic and purely viscous. This range of mechanical behavior is described as viscoelasticity. In certain types of haptic interactions, such as assessment of ripeness of fruit, firmness of cheese, and consistency of organ tissue, we rely heavily on our haptic perception of viscoelastic materials. The relationship between the mechanical behavior of viscoelastic materials and our perception of them has been investigated in the field of psychorheology. However, our knowledge on how we perceive viscoelastic materials is still quite limited though some research work has already been done on purely elastic and purely viscous materials. History- and frequency-dependent behavior of viscoelastic materials result in a complex time-dependent response, which requires relatively more sophisticated models to investigate their behavior than those of purely elastic and viscous materials. In this study, we model viscoelasticity using a "springpot" (i.e., fractional-order derivative element) and express its behavior in the frequency domain using two physical parameters-"magnitude" and "phase" of complex stiffness. In the frequency domain, we are able to devise signal detection experiments where we can investigate the perception of viscoelastic materials using the perceptual terms of "firmness" and "bounciness," corresponding to the physical parameters of "magnitude" and "phase." The results of our experiments show that the just-noticeable difference (JND) for bounciness increases linearly with increasing "phase," following Weber's law, while the JND for firmness is surprisingly independent of the level of "phase."

Adaptation to Laterally Asymmetrical Visuomotor Delay Has an Effect on Action But Not on Perception

When interacting with the environment, the sensorimotor system faces temporal and spatial discrepancies between sensory inputs, such as delay in sensory information transmission, and asymmetrical visual inputs across space. These discrepancies can affect motor control and the representation of space. We recently showed that adaptation to a laterally asymmetric delay in the visual feedback induces neglect-like effects in blind drawing movements, expressed by asymmetrical elongation of circles that are drawn in different workspaces and directions; this establishes a possible connection between delayed feedback and asymmetrical spatial processing in the control of action. In the current study, we investigate whether such adaptation also influences visual perception. In addition, we examined transfer to another motor task – a line bisection task that is commonly used to detect spatial disorders, and extend these results to examine the mapping of these neglect-like effects. We performed two sets of experiments in which participants executed lateral reaching movements, and were exposed to visual feedback delay only in the left workspace. We examined transfer of adaptation to a perceptual line bisection task – answers about the perceived midline of lines that were presented in different directions and workspaces, and to a blind motor line bisection task – reaching movements toward the centers of similar lines. We found that the adaptation to the asymmetrical delay transferred to the control of lateral movements, but did not affect the perceived location of the midlines. Our results clarify the effect of asymmetrical delayed visual feedback on perception and action, and provide potential insights on the link between visuomotor delay and neurological disorders such as the hemispatial neglect syndrome.

Neglect-Like Effects on Drawing Symmetry Induced by Adaptation to a Laterally Asymmetric Visuomotor Delay

In daily interactions, our sensorimotor system accounts for spatial and temporal discrepancies between the senses. Functional lateralization between hemispheres causes differences in attention and in the control of action across the left and right workspaces. In addition, differences in transmission delays between modalities affect movement control and internal representations. Studies on motor impairments such as hemispatial neglect syndrome suggested a link between lateral spatial biases and temporal processing. To understand this link, we computationally modeled and experimentally validated the effect of laterally asymmetric delay in visual feedback on motor learning and its transfer to the control of drawing movements without visual feedback. In the behavioral experiments, we asked healthy participants to perform lateral reaching movements while adapting to delayed visual feedback in either left, right, or both workspaces. We found that the adaptation transferred to blind drawing and caused movement elongation, which is consistent with a state representation of the delay. However, the pattern of the spatial effect varied between conditions: whereas adaptation to delay in only the left workspace or in the whole workspace caused selective leftward elongation, adaptation to delay in only the right workspace caused drawing elongation in both directions. We simulated arm movements according to different models of perceptual and motor spatial asymmetry in the representation of delay and found that the best model that accounts for our results combines both perceptual and motor asymmetry between the hemispheres. These results provide direct evidence for an asymmetrical processing of delayed visual feedback that is associated with both perceptual and motor biases that are similar to those observed in hemispatial neglect syndrome.

Switching in Feedforward Control of Grip Force During Tool-Mediated Interaction With Elastic Force Fields

Switched systems are common in artificial control systems. Here, we suggest that the brain adopts a switched feedforward control of grip forces during manipulation of objects. We measured how participants modulated grip force when interacting with soft and rigid virtual objects when stiffness varied continuously between trials. We identified a sudden phase transition between two forms of feedforward control that differed in the timing of the synchronization between the anticipated load force and the applied grip force. The switch occurred several trials after a threshold stiffness level in the range 100–200 N/m. These results suggest that in the control of grip force, the brain acts as a switching control system. This opens new research questions as to the nature of the discrete state variables that drive the switching.

Force feedback delay affects perception of stiffness but not action, and the effect depends on the hand used but not on the handedness

Interaction with an object often requires the estimation of its mechanical properties. We examined whether the hand that is used to interact with the object and their handedness affected people's estimation of these properties using stiffness estimation as a test case. We recorded participants' responses on a stiffness discrimination of a virtual elastic force field and the grip force applied on the robotic device during the interaction. In half of the trials, the robotic device delayed the participants' force feedback. Consistent with previous studies, delayed force feedback biased the perceived stiffness of the force field. Interestingly, in both left-handed and right-handed participants, for the delayed force field, there was even less perceived stiffness when participants used their left hand than their right hand. This result supports the idea that haptic processing is affected by laterality in the brain, not by handedness. Consistent with previous studies, participants adjusted their applied grip force according to the correct size and timing of the load force regardless of the hand that was used, the handedness, or the delay. This suggests that in all of these conditions, participants were able to form an accurate internal representation of the anticipated trajectory of the load force (size and timing) and that this representation was used for accurate control of grip force independently of the perceptual bias. Thus these results provide additional evidence for the dissociation between action and perception in the processing of delayed information. NEW & NOTEWORTHY Introducing delay to force feedback during interaction with an elastic force field biases the perceived stiffness of the force field. We show that this bias depends on the hand that was used for probing but not on handedness. At the same time, both left-handed and right-handed participants adjusted their applied grip force while using either their left or right hands in anticipation of the correct magnitude and timing despite the delay in load force.

Representing delayed force feedback as a combination of current and delayed states

To adapt to deterministic force perturbations that depend on the current state of the hand, internal representations are formed to capture the relationships between forces experienced and motion. However, information from multiple modalities travels at different rates, resulting in intermodal delays that require compensation for these internal representations to develop. To understand how these delays are represented by the brain, we presented participants with delayed velocity-dependent force fields, i.e., forces that depend on hand velocity either 70 or 100 ms beforehand. We probed the internal representation of these delayed forces by examining the forces the participants applied to cope with the perturbations. The findings showed that for both delayed forces, the best model of internal representation consisted of a delayed velocity and current position and velocity. We show that participants relied initially on the current state, but with adaptation, the contribution of the delayed representation to adaptation increased. After adaptation, when the participants were asked to make movements with a higher velocity for which they had not previously experienced with the delayed force field, they applied forces that were consistent with current position and velocity as well as delayed velocity representations. This suggests that the sensorimotor system represents delayed force feedback using current and delayed state information and that it uses this representation when generalizing to faster movements.NEW & NOTEWORTHY The brain compensates for forces in the body and the environment to control movements, but it is unclear how it does so given the inherent delays in information transmission and processing. We examined how participants cope with delayed forces that depend on their arm velocity 70 or 100 ms beforehand. After adaptation, participants applied opposing forces that revealed a partially correct representation of the perturbation using the current and the delayed information.

Haptic Feedback in Needle Insertion Modeling and Simulation

Needle insertion is the most basic skill in medical care, and training has to be imparted not only for physicians but also for nurses and paramedics. In most needle insertion procedures, haptic feedback from the needle is the main stimulus in which novices need training. For better patient safety, the classical methods of training the haptic skills have to be replaced with simulators based on new robotic and graphics technologies. This paper reviews the current advances in needle insertion modeling, classified into three sections: needle insertion models, tissue deformation models, and needle-tissue interaction models. Although understated in the literature, the classical and dynamic friction models, which are critical for needle insertion modeling, are also discussed. The experimental setup or the needle simulators that have been developed to validate the models are described. The need of psychophysics for needle simulators and psychophysical parameter analysis of human perception in needle insertion are discussed, which are completely ignored in the literature.

The effect of force feedback delay on stiffness perception and grip force modulation during tool-mediated interaction with elastic force fields

During interaction with objects, we form an internal representation of their mechanical properties. This representation is used for perception and for guiding actions, such as in precision grip, where grip force is modulated with the predicted load forces. In this study, we explored the relationship between grip force adjustment and perception of stiffness during interaction with linear elastic force fields. In a forced-choice paradigm, participants probed pairs of virtual force fields while grasping a force sensor that was attached to a haptic device. For each pair, they were asked which field had higher level of stiffness. In half of the pairs, the force feedback of one of the fields was delayed. Participants underestimated the stiffness of the delayed field relatively to the nondelayed, but their grip force characteristics were similar in both conditions. We analyzed the magnitude of the grip force and the lag between the grip force and the load force in the exploratory probing movements within each trial. Right before answering which force field had higher level of stiffness, both magnitude and lag were similar between delayed and nondelayed force fields. These results suggest that an accurate internal representation of environment stiffness and time delay was used for adjusting the grip force. However, this representation did not help in eliminating the bias in stiffness perception. We argue that during performance of a perceptual task that is based on proprioceptive feedback, separate neural mechanisms are responsible for perception and action-related computations in the brain.

Identifying haptic exploratory procedures by analyzing hand dynamics and contact force

Haptic exploratory procedures (EPs) are prototypical hand movements that are linked to the acquisition of specific object properties. In studies of haptic perception, hand movements are often classified into these EPs. Here, we aim to investigate several EPs in a quantitative manner to understand how hand dynamics and contact forces differ between them. These dissimilarities are then used to construct an EP identification model capable of discriminating between EPs based on the index finger position and contact force. The extent to which the instructed EPs were distinct, repeatable, and similar across subjects was confirmed by showing that more than 95 percent of the analyzed trials were classified correctly. Finally, the method is employed to investigate haptic exploratory behavior during similarity judgments based on several object properties. It seems that discrimination based on material properties (hardness, roughness, and temperature) yields more consistent classification results compared to discrimination based on the acquisition of shape information.

Adaptive Human Control Gains during Precision Grip

Eight human test subjects attempted to track a desired position trajectory with an instrumented manipulandum (MN). The test subjects used the MN with three different levels of stiffness. A transfer function was developed to represent the human application of a precision grip from the data when the test subjects initially displaced the MN so as to learn the position mapping from the MN onto the display. Another transfer function was formed from the data of the remainder of the experiments, after significant displacement of the MN occurred. Both of these transfer functions accurately modelled the system dynamics for a portion of the experiments, but neither was accurate for the duration of the experiments because the human grip dynamics changed while learning the position mapping. Thus, an adaptive system model was developed to describe the learning process of the human test subjects as they displaced the MN in order to gain knowledge of the position mapping. The adaptive system model was subsequently validated following comparison with the human test subject data. An examination of the average absolute error between the position predicted by the adaptive model and the actual experimental data yielded an overall average error of 0.34mm for all three levels of stiffness.

Cutaneous Force Feedback as a Sensory Subtraction Technique in Haptics

A novel sensory substitution technique is presented. Kinesthetic and cutaneous force feedback are substituted by cutaneous feedback (CF) only, provided by two wearable devices able to apply forces to the index finger and the thumb, while holding a handle during a teleoperation task. The force pattern, fed back to the user while using the cutaneous devices, is similar, in terms of intensity and area of application, to the cutaneous force pattern applied to the finger pad while interacting with a haptic device providing both cutaneous and kinesthetic force feedback. The pattern generated using the cutaneous devices can be thought as a subtraction between the complete haptic feedback (HF) and the kinesthetic part of it. For this reason, we refer to this approach as sensory subtraction instead of sensory substitution. A needle insertion scenario is considered to validate the approach. The haptic device is connected to a virtual environment simulating a needle insertion task. Experiments show that the perception of inserting a needle using the cutaneous-only force feedback is nearly indistinguishable from the one felt by the user while using both cutaneous and kinesthetic feedback. As most of the sensory substitution approaches, the proposed sensory subtraction technique also has the advantage of not suffering from stability issues of teleoperation systems due, for instance, to communication delays. Moreover, experiments show that the sensory subtraction technique outperforms sensory substitution with more conventional visual feedback (VF).

Toward Perceiving Robots as Humans: Three Handshake Models Face the Turing-Like Handshake Test

In the Turing test a computer model is deemed to "think intelligently" if it can generate answers that are indistinguishable from those of a human. We developed an analogous Turing-like handshake test to determine if a machine can produce similarly indistinguishable movements. The test is administered through a telerobotic system in which an interrogator holds a robotic stylus and interacts with another party - artificial or human with varying levels of noise. The interrogator is asked which party seems to be more human. Here, we compare the human-likeness levels of three different models for handshake: (1) Tit-for-Tat model, (2) λ model, and (3) Machine Learning model. The Tit-for-Tat and the Machine Learning models generated handshakes that were perceived as the most human-like among the three models that were tested. Combining the best aspects of each of the three models into a single robotic handshake algorithm might allow us to advance our understanding of the way the nervous system controls sensorimotor interactions and further improve the human-likeness of robotic handshakes.