Haptic dominance in form perception: vision versus proprioception

Modified Egocentric Viewpoint for Softer Seated Experience in Virtual Reality

Users in a prolonged experience of virtual reality adopt a sitting position according to their task, as they do in the real world. However, inconsistencies in the haptic feedback from a chair they sit on in the real world and that which is expected in the virtual world decrease the feeling of presence. We aimed to change the perceived haptic features of a chair by shifting the position and angle of the users' viewpoints in the virtual reality environment. The targeted features in this study were seat softness and backrest flexibility. To enhance the seat softness, we shifted the virtual viewpoint using an exponential formula soon after a user's bottom contacted the seat surface. The flexibility of the backrest was manipulated by moving the viewpoint, which followed the tilt of the virtual backrest. These shifts make users feel as if their body moves along with the viewpoint; as a result, they would perceive pseudo-softness or flexibility consistently with the body movement. Based on subjective evaluations, we confirmed that the participants perceived the seat as being softer and the backrest as being more flexible than the actual ones. These results demonstrated that only shifting the viewpoint could change the participants' perceptions of the haptic features of their seats, although significant changes created strong discomfort.

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

From an early age, humans learn to develop an intuition for the physical nature of the objects around them by using exploratory behaviors. Such exploration provides observations of how objects feel, sound, look, and move as a result of actions applied on them. Previous works in robotics have shown that robots can also use such behaviors (e.g., lifting, pressing, shaking) to infer object properties that camera input alone cannot detect. Such learned representations are specific to each individual robot and cannot currently be transferred directly to another robot with different sensors and actions. Moreover, sensor failure can cause a robot to lose a specific sensory modality which may prevent it from using perceptual models that require it as input. To address these limitations, we propose a framework for knowledge transfer across behaviors and sensory modalities such that: (1) knowledge can be transferred from one or more robots to another, and, (2) knowledge can be transferred from one or more sensory modalities to another. We propose two different models for transfer based on variational auto-encoders and encoder-decoder networks. The main hypothesis behind our approach is that if two or more robots share multi-sensory object observations of a shared set of objects, then those observations can be used to establish mappings between multiple features spaces, each corresponding to a combination of an exploratory behavior and a sensory modality. We evaluate our approach on a category recognition task using a dataset in which a robot used 9 behaviors, coupled with 4 sensory modalities, performed multiple times on 100 objects. The results indicate that sensorimotor knowledge about objects can be transferred both across behaviors and across sensory modalities, such that a new robot (or the same robot, but with a different set of sensors) can bootstrap its category recognition models without having to exhaustively explore the full set of objects.

The motor vertical in the absence of gravicentric cues

When participants are asked to flip an omnidirectional switch "down", the direction of their responses depends mainly on gravicentric, less so on egocentric and least on visual cues about the vertical (Lackner and DiZio, Exp. Brain Res. 130:2-26, 2000). Here we evaluate response direction when gravicentric cues are not available. Participants flipped an omnidirectional switch "down" when gravito-inertial force acted orthogonally to the response plane on earth (session E), and when it was near zero during parabolic flights (session P). We found that the relative weight of visual cues was similar in both sessions, and it was similar to that in an earlier study where participants stood upright. Across all three data sets, the weight of visual cues averaged 0.09. The relative weight of egocentric cues was also similar in both sessions, averaging 0.87; however, it was significantly lower in the earlier study with upright participants, where it averaged 0.43. We further found that informative and noninformative tactile stimulation had no substantial effects on response direction, which suggests that the earlier reported anchoring effect of tactile signals for the perceived vertical may not extend to the motor vertical. We conclude that the absence of gravicentric cues is compensated by a higher weight of egocentric cues, but not by a higher weight of visual cues. As a consequence, astronauts, divers and persons who work on ground in a horizontal body posture may mishandle equipment because of their strong reliance on egocentric cues.

Intrasensory Attention: Kinaesthetic versus Cutaneous Inputs

Blindfolded participants felt pairs of raised-line drawings simultaneously, one with each index finger. The stimuli presented at each fingertip were 180° rotations of each other (eg 6 and 9). One finger moved (either actively or passively), and this in turn caused movement of a matched raised line underneath the stationary finger on the other hand, in a yoked manner. Thus, a 6 at the moving finger would be felt as a 9 on the stationary finger. On all trials there was a raised line moving underneath the stationary fully passive finger. For the moving finger, a raised line was present on only half of the trials. When a raised line could be felt at the moving fingertip, the shape followed by this finger was more often reported than was the shape present at the other (stationary) fingertip. However, when no line was present under the moving finger (ie when movement became the major cue for shape), subjects reported experiencing the shape moved under the stationary fingertip. Results are interpreted as an indication that cutaneous information can be more ‘attention-getting’ than kinaesthetic information, and are considered to support the modality-appropriateness theory.

Visual influence on haptic torque perception

The brain receives input from multiple sensory modalities simultaneously, yet we experience the outside world as a single integrated percept. This integration process must overcome instances where perceptual information conflicts across sensory modalities. Under such conflicts, the relative weighting of information from each modality typically depends on the given task. For conflicts between visual and haptic modalities, visual information has been shown to influence haptic judgments of object identity, spatial features (e.g., location, size), texture, and heaviness. Here we test a novel instance of haptic-visual conflict in the perception of torque. We asked participants to hold a left-right unbalanced object while viewing a potentially left-right mirror-reversed image of the object. Despite the intuition that the more proximal haptic information should dominate the perception of torque, we find that visual information exerts substantial influences on torque perception even when participants know that visual information is unreliable.

Interactive object recognition using proprioceptive and auditory feedback

In this paper we propose a method for interactive recognition of household objects using proprioceptive and auditory feedback. In our experiments, the robot observed the changes in its proprioceptive and auditory sensory streams while performing five exploratory behaviors (lift, shake, drop, crush, and push) on 50 common household objects (e.g. bottles, cups, balls, toys, etc.). The robot was tasked with recognizing the objects it was manipulating by feeling them and listening to the sounds that they make without using any visual information. The results show that both proprioception and audio, coupled with exploratory behaviors, can be used successfully for object recognition. Furthermore, the robot was able to integrate feedback from the two modalities, to achieve even better recognition accuracy. Finally, the results show that the robot can boost its recognition rate even further by applying multiple different exploratory behaviors on the object.

Visual effects of haptic feedback are large but local

Vision generally provides reliable predictions for touch and motor-control, but some classes of stimuli evoke visual illusions. Using haptic feedback on virtual 3-D surfaces, we tested the function of touch in such cases. Our experiments show that in the perception of 3-D shapes from texture cues, haptic information can dominate vision in some cases, changing percepts qualitatively from convex to concave and concave to slant. The effects take time to develop, do not outlive the cessation of the feedback, are attenuated by distance, and drastically reduced by gaps in the surface. These dynamic shifts in qualitative perceived shapes could be invaluable in neural investigations that test whether haptic feedback modifies selective activation of neurons or changes the shape-tuning of neurons responsible for percepts of 3-D shapes.

Sensory dominance in combinations of audio, visual and haptic stimuli

Participants presented with auditory, visual, or bi-sensory audio-visual stimuli in a speeded discrimination task, fail to respond to the auditory component of the bi-sensory trials significantly more often than they fail to respond to the visual component--a 'visual dominance' effect. The current study investigated further the sensory dominance phenomenon in all combinations of auditory, visual and haptic stimuli. We found a similar visual dominance effect also in bi-sensory trials of combined haptic-visual stimuli, but no bias towards either sensory modality in bi-sensory trials of haptic-auditory stimuli. When presented with tri-sensory trials of combined auditory-visual-haptic stimuli, participants made more errors of responding only to two corresponding sensory signals than errors of responding only to a single sensory modality, however, there were no biases towards either sensory modality (or sensory pairs) in the distribution of both types of errors (i.e. responding only to a single stimulus or to pairs of stimuli). These results suggest that while vision can dominate both the auditory and the haptic sensory modalities, it is limited to bi-sensory combinations in which the visual signal is combined with another single stimulus. However, in a tri-sensory combination when a visual signal is presented simultaneously with both the auditory and the haptic signals, the probability of missing two signals is much smaller than of missing only one signal and therefore the visual dominance disappears.

Position coding in a video-controlled pointing task with a rotated visual display: evidence for individual differences in visuo-proprioceptive interaction

In video-controlled tasks, visuomotor performance is generally initially poor with rotated visual display, but improves through trial-by-trial learning. We hypothesise that the inaccurate processing of the visual hand-to-target vector mainly results from the persistent influence of non-visual information relating to arm posture. To test this hypothesis, arm-related proprioceptive and visual information were independently manipulated in a video-controlled pointing task. Analysis of movement vectors revealed that the target was located according to the visual hand but its proprioceptive orientation (Allelocentric(1) system of reference, N = 10), or according to the proprioceptive hand location and orientation (Egocentric system of reference, N = 8). The prevalence of one system of reference correlated with the accuracy of proprioceptive signals informing about arm posture. One obstacle in mastering video-controlled task results thus from the persistent influence of proprioceptive information in the spatial coding of visual goals for action, which however differs across individuals.

What role does multisensory integration play in the visuotactile perception of texture?

Previous research has shown that vision and touch are both effective at many roughness discrimination tasks; however, there is no evidence that using both senses simultaneously improves discrimination performance. We investigated the nature of this failure to integrate multisensory inputs, using three varieties of forced-choice discrimination tasks. In Experiment 1, visual, tactile and bimodal roughness discriminations were made between pairs of fabric stimuli. Bimodal discriminations were typically performed with a sensitivity somewhere between that observed for the unimodal presentations. In Experiment 2, a similar design was used except that during the stimulus presentation, one interval contained a unimodal (vision or touch) stimulus, the other interval a bimodal stimulus presentation. Bias toward the bimodal interval would indicate an increase in the magnitude of perceived roughness for such presentations. No such bias was found. In Experiment 3, participants made single-interval, bimodal discriminations, determining whether a rough stimulus was presented to touch, to vision, to both modalities, or to neither modality. The improved performance seen for the dual-target vs. single-target presentations was best modelled as arising from a trialwise division of attention between vision and touch. Overall, these results suggest that vision and touch act as independent sources of roughness information, where the necessity to divide attention across both modalities reduces the discriminative ability in (or information available from) each of these individual modalities.

Haptic perception of the horizontal by blind and low-vision individuals

We examined haptic perception of the horizontal in visually impaired people. Blind people (late blind and congenitally blind), persons with very low vision, and blindfolded sighted individuals felt raised-line drawings of jars at four angles. They had to demonstrate their understanding that water remains horizontal, despite jar tilt, by selecting the correct raised-line drawing given four choices. Low-vision subjects, with near perfect scores, performed significantly better than the other groups of subjects. While the late-blind and blindfolded sighted subjects performed slightly better than the congenitally blind participants, the difference between the late-blind and congenitally blind groups was nonsignificant. The performance of the congenitally blind subjects indicates that visual experience is not necessary for the development of an understanding that water level stays horizontal, given container tilt.

Experience-dependent visual cue integration based on consistencies between visual and haptic percepts

We study the hypothesis that observers can use haptic percepts as a standard against which the relative reliabilities of visual cues can be judged, and that these reliabilities determine how observers combine depth information provided by these cues. Using a novel visuo-haptic virtual reality environment, subjects viewed and grasped virtual objects. In Experiment 1, subjects were trained under motion relevant conditions, during which haptic and visual motion cues were consistent whereas haptic and visual texture cues were uncorrelated, and texture relevant conditions, during which haptic and texture cues were consistent whereas haptic and motion cues were uncorrelated. Subjects relied more on the motion cue after motion relevant training than after texture relevant training, and more on the texture cue after texture relevant training than after motion relevant training. Experiment 2 studied whether or not subjects could adapt their visual cue combination strategies in a context-dependent manner based on context-dependent consistencies between haptic and visual cues. Subjects successfully learned two cue combination strategies in parallel, and correctly applied each strategy in its appropriate context. Experiment 3, which was similar to Experiment 1 except that it used a more naturalistic experimental task, yielded the same pattern of results as Experiment 1 indicating that the findings do not depend on the precise nature of the experimental task. Overall, the results suggest that observers can involuntarily compare visual and haptic percepts in order to evaluate the relative reliabilities of visual cues, and that these reliabilities determine how cues are combined during three-dimensional visual perception.

Intersensory conflict between vision and touch: the response modality dominates when precise, attention-riveting judgments are required

In four experiments, reducing lenses were used to minify vision and generate intersensory size conflicts between vision and touch. Subjects made size judgments, using either visual matching or haptic matching. In visual matching, the subjects chose from a set of visible squares that progressively increased in size. In haptic matching, the subjects selected matches from an array of tangible wooden squares. In Experiment 1, it was found that neither sense dominated when subjects exposed to an intersensory discrepancy made their size estimates by using either visual matching or haptic matching. Size judgments were nearly indentical for conflict subjects making visual or haptic matches. Thus, matching modality did not matter in Experiment 1. In Experiment 2, it was found that subjects were influenced by the sight of their hands, which led to increases in the magnitude of their size judgments. Sight of the hands produced more accurate judgments, with subjects being better able to compensate for the illusory effects of the reducing lens. In two additional experiments, it was found that when more precise judgments were required and subjects had to generate their own size estimates, the response modality dominated. Thus, vision dominated in Experiment 3, where size judgments derived from viewing a metric ruler, whereas touch dominated in Experiment 4, where subjects made size estimates with a pincers posture of their hands. It is suggested that matching procedures are inadequate for assessing intersensory dominance relations. These results qualify the position (Hershberger & Misceo, 1996) that the modality of size estimates influences the resolution of intersensory conflicts. Only when required to self-generate more precise judgments did subjects rely on one sense, either vision or touch. Thus, task and attentional requirements influence dominance relations, and vision does not invariably prevail over touch.

Haptic estimates of discordant visual-haptic size vary developmentally

In two size-conflict experiments, children viewed various squares through a reducing (1/2) lens while manually grasping them through a hand-concealing cloth. Then, using either vision or touch, they selected a match from a set of comparison squares. Forty 6-, 9-, and 12-year-olds participated in Experiment 1. Vision dominated the visual estimates of all three age groups; however, for the haptic estimates, the dominant modality varied developmentally: Vision dominated the 6-year-olds' haptic estimates, whereas neither modality dominated the 9-year-olds' haptic estimates, and touch dominated the 12-year-olds' haptic estimates. In Experiment 2, 24 six-year-olds were tested, as in Experiment 1; however, half of them were shown the size-distorting effects of the lens just prior to testing. Although this reduced the visual dominance of their haptic estimates, the effect was weak and short-lived. The haptic dominance seen in the data of the 12-year-olds was conspicuously absent.

Large scale differences between haptic and visual judgments of curvature

Large systematic scale differences show up when subjects are asked to match the curvature of a haptically presented surface with a visually presented cross section. Scaling factors ranging from 0.3 to 3.0 were found. Most often (for five out of six subjects) haptically estimated curvature was overestimated relative to visual curvature. We conclude that mutually inconsistent representations of surface curvature coexist in a single observer in the haptic and visual modalities. Another conclusion can be drawn with certainty, namely that at least one of the two modalities does not provide the observer with veridical information. The results for the left and the right hand of a subject were very similar, though for two subjects there existed a small significant difference in scaling. It seems likely that the curvature representations of the two hands are not independent of one another, but our data do not permit us to be conclusive.

Touch dominates haptic estimates of discordant visual-haptic size

Observers (72 college students) estimated the size of plastic squares that they held in their fingers and simultaneously viewed through a reducing lens that halved the squares' visual size. The squares were grasped from below through a cloth that prevented direct sight of the hand. Each estimate was a match selected later, either haptically or visually, from a set of comparison squares. Vision dominated the visual estimates and touch dominated the haptic estimates, whether or not the observers knew in advance which type of estimate they would be asked to make. Neither modality inherently dominates perceived size.

Influence of visual guidance on braille recognition: low lighting also helps touch

This study was an attempt to clarify the mechanisms responsible for the benefits of visual guidance in tactual braille recognition. Subjects touched +90 degrees tilted braille under normal room lighting, or with low lighting, with or without visual guidance. Both visual information about finger angle and spatial reference information were manipulated with stained glass and light-emitting diodes. The provision of visual information about finger angle alone was no help to braille recognition, and performance was low. Adding visual spatial reference information to vision of finger angle raised performance. However, recognition accuracy was also substantially improved by low lighting. The benefits of darkness for haptics did not generalize to the reading of upright, two-letter braille words. It was proposed that extraneous visual information may distract sighted subjects in haptic tasks that require mental rotation of visual images.