Predicting precision grip grasp locations on three-dimensional objects

Ouvrai opens access to remote virtual reality studies of human behavioural neuroscience

Modern virtual reality (VR) devices record six-degree-of-freedom kinematic data with high spatial and temporal resolution and display high-resolution stereoscopic three-dimensional graphics. These capabilities make VR a powerful tool for many types of behavioural research, including studies of sensorimotor, perceptual and cognitive functions. Here we introduce Ouvrai, an open-source solution that facilitates the design and execution of remote VR studies, capitalizing on the surge in VR headset ownership. This tool allows researchers to develop sophisticated experiments using cutting-edge web technologies such as WebXR to enable browser-based VR, without compromising on experimental design. Ouvrai's features include easy installation, intuitive JavaScript templates, a component library managing front- and backend processes and a streamlined workflow. It integrates with Firebase, Prolific and Amazon Mechanical Turk and provides data processing utilities for analysis. Unlike other tools, Ouvrai remains free, with researchers managing their web hosting and cloud database via personal Firebase accounts. Ouvrai is not limited to VR studies; researchers can also develop and run desktop or touchscreen studies using the same streamlined workflow. Through three distinct motor learning experiments, we confirm Ouvrai's efficiency and viability for conducting remote VR studies.

High-level aftereffects reveal the role of statistical features in visual shape encoding

Visual shape perception is central to many everyday tasks, from object recognition to grasping and handling tools.1,2,3,4,5,6,7,8,9,10 Yet how shape is encoded in the visual system remains poorly understood. Here, we probed shape representations using visual aftereffects-perceptual distortions that occur following extended exposure to a stimulus.11,12,13,14,15,16,17 Such effects are thought to be caused by adaptation in neural populations that encode both simple, low-level stimulus characteristics17,18,19,20 and more abstract, high-level object features.21,22,23 To tease these two contributions apart, we used machine-learning methods to synthesize novel shapes in a multidimensional shape space, derived from a large database of natural shapes.24 Stimuli were carefully selected such that low-level and high-level adaptation models made distinct predictions about the shapes that observers would perceive following adaptation. We found that adaptation along vector trajectories in the high-level shape space predicted shape aftereffects better than simple low-level processes. Our findings reveal the central role of high-level statistical features in the visual representation of shape. The findings also hint that human vision is attuned to the distribution of shapes experienced in the natural environment.

Distinct Neural Components of Visually Guided Grasping during Planning and Execution

Selecting suitable grasps on three-dimensional objects is a challenging visuomotor computation, which involves combining information about an object (e.g., its shape, size, and mass) with information about the actor's body (e.g., the optimal grasp aperture and hand posture for comfortable manipulation). Here, we used functional magnetic resonance imaging to investigate brain networks associated with these distinct aspects during grasp planning and execution. Human participants of either sex viewed and then executed preselected grasps on L-shaped objects made of wood and/or brass. By leveraging a computational approach that accurately predicts human grasp locations, we selected grasp points that disentangled the role of multiple grasp-relevant factors, that is, grasp axis, grasp size, and object mass. Representational Similarity Analysis revealed that grasp axis was encoded along dorsal-stream regions during grasp planning. Grasp size was first encoded in ventral stream areas during grasp planning then in premotor regions during grasp execution. Object mass was encoded in ventral stream and (pre)motor regions only during grasp execution. Premotor regions further encoded visual predictions of grasp comfort, whereas the ventral stream encoded grasp comfort during execution, suggesting its involvement in haptic evaluation. These shifts in neural representations thus capture the sensorimotor transformations that allow humans to grasp objects.SIGNIFICANCE STATEMENT Grasping requires integrating object properties with constraints on hand and arm postures. Using a computational approach that accurately predicts human grasp locations by combining such constraints, we selected grasps on objects that disentangled the relative contributions of object mass, grasp size, and grasp axis during grasp planning and execution in a neuroimaging study. Our findings reveal a greater role of dorsal-stream visuomotor areas during grasp planning, and, surprisingly, increasing ventral stream engagement during execution. We propose that during planning, visuomotor representations initially encode grasp axis and size. Perceptual representations of object material properties become more relevant instead as the hand approaches the object and motor programs are refined with estimates of the grip forces required to successfully lift the object.

HANDdata - first-person dataset including proximity and kinematics measurements from reach-to-grasp actions

HANDdata is a dataset designed to provide hand kinematics and proximity vision data during reach to grasp actions of non-virtual objects, specifically tailored for autonomous grasping of a robotic hand, and with particular attention to the reaching phase. Thus, we sought to capture target object characteristics from radar and time-of-flight proximity sensors, as well as details of the reach-to-grasp action by looking at wrist and fingers kinematics, and at hand-object interaction main events. We structured the data collection as a sequence of static and grasping tasks, organized by increasing levels of complexity. HANDdata is a first-person, reach-to-grasp dataset that includes almost 6000 human-object interactions from 29 healthy adults, with 10 standardized objects of 5 different shapes and 2 kinds of materials. We believe that such data collection can be of value for researchers interested in autonomous grasping robots for healthcare and industrial applications, as well as for those interested in radar-based computer vision and in basic aspects of sensorimotor control and manipulation.

Not only perception but also grasping actions can obey Weber's law

Weber's law, the principle that the uncertainty of perceptual estimates increases proportionally with object size, is regularly violated when considering the uncertainty of the grip aperture during grasping movements. The origins of this perception-action dissociation are debated and are attributed to various reasons, including different coding of visual size information for perception and action, biomechanical factors, the use of positional information to guide grasping, or, sensorimotor calibration. Here, we contrasted these accounts and compared perceptual and grasping uncertainties by asking people to indicate the visually perceived center of differently sized objects (Perception condition) or to grasp and lift the same objects with the requirement to achieve a balanced lift (Action condition). We found that the variability (uncertainty) of contact positions increased as a function of object size in both perception and action. The adherence of the Action condition to Weber's law and the consequent absence of a perception-action dissociation contradict the predictions based on different coding of visual size information and sensorimotor calibration. These findings provide clear evidence that human perceptual and visuomotor systems rely on the same visual information and suggest that the previously reported violations of Weber's law in grasping movements should be attributed to other factors.

Having several options does not increase the time it takes to make a movement to an adequate end point

Throughout the day, people constantly make choices such as where to direct their gaze or place their foot. When making such movement choices, there are usually multiple acceptable options, although some are more advantageous than others. How much time does it take to make such choices and to what extent is the most advantageous option chosen from the available alternatives? To find out, we asked participants to collect points by tapping on any of several targets with their index finger. It did not take participants more time to direct their movements to an advantageous target when there were more options. Participants chose targets that were advantageous because they were easier to reach. Targets could be easier to reach because the finger was already moving in their direction when they appeared, or because they were larger or oriented along the movement direction so that the finger could move faster towards them without missing them. When the target’s colour indicated that it was worth more points they chose it slightly less fast, presumably because it generally takes longer to respond to colour than to respond to attributes such as size. They also chose it less often than they probably should have, presumably because the advantage of choosing it was established arbitrarily. We conclude that having many options does not increase the time it takes to move to an adequate target.

Effects of visual and visual-haptic perception of material rigidity on reaching and grasping in the course of development

The development of material property perception for grasping objects is not well explored during early childhood. Therefore, we investigated infants', 3-year-old children's, and adults' unimanual grasping behavior and reaching kinematics for objects of different rigidity using a 3D motion capture system. In Experiment 1, 11-month-old infants and for purposes of comparison adults, and in Experiment 2, 3-year old children were encouraged to lift relatively heavy objects with one of two handles differing in rigidity after visual (Condition 1) and visual-haptic exploration (Condition 2). Experiment 1 revealed that 11-months-olds, after visual object exploration, showed no significant material preference, and thus did not consider the material to facilitate grasping. After visual-haptic object exploration and when grasping the contralateral handles, infants showed an unexpected preference for the soft handles, which were harder to use to lift the object. In contrast, adults generally grasped the rigid handle exploiting their knowledge about efficient and functional grasping in both conditions. Reaching kinematics were barely affected by rigidity, but rather by condition and age. Experiment 2 revealed that 3-year-olds no longer exhibit a preference for grasping soft handles, but still no adult-like preference for rigid handles in both conditions. This suggests that material rigidity plays a minor role in infants' grasping behavior when only visual material information is available. Also, 3-year-olds seem to be on an intermediate level in the development from (1) preferring the pleasant sensation of a soft fabric, to (2) preferring the efficient rigid handle.

An image-computable model of human visual shape similarity

Shape is a defining feature of objects, and human observers can effortlessly compare shapes to determine how similar they are. Yet, to date, no image-computable model can predict how visually similar or different shapes appear. Such a model would be an invaluable tool for neuroscientists and could provide insights into computations underlying human shape perception. To address this need, we developed a model (‘ShapeComp’), based on over 100 shape features (e.g., area, compactness, Fourier descriptors). When trained to capture the variance in a database of >25,000 animal silhouettes, ShapeComp accurately predicts human shape similarity judgments between pairs of shapes without fitting any parameters to human data. To test the model, we created carefully selected arrays of complex novel shapes using a Generative Adversarial Network trained on the animal silhouettes, which we presented to observers in a wide range of tasks. Our findings show that incorporating multiple ShapeComp dimensions facilitates the prediction of human shape similarity across a small number of shapes, and also captures much of the variance in the multiple arrangements of many shapes. ShapeComp outperforms both conventional pixel-based metrics and state-of-the-art convolutional neural networks, and can also be used to generate perceptually uniform stimulus sets, making it a powerful tool for investigating shape and object representations in the human brain.

The ability to describe and compare shapes is crucial in many scientific domains from visual object recognition to computational morphology and computer graphics. Across disciplines, considerable effort has been devoted to the study of shape and its influence on object recognition, yet an important stumbling block is the quantitative characterization of shape similarity. Here we develop a psychophysically validated model that takes as input an object’s shape boundary and provides a high-dimensional output that can be used for predicting visual shape similarity. With this precise control of shape similarity, the model’s description of shape is a powerful tool that can be used across the neurosciences and artificial intelligence to test role of shape in perception and the brain.

Stereopsis contributes to the predictive control of grip forces during prehension

Binocular viewing is associated with a superior prehensile performance, which is particularly evident in the latter part of the reach as the hand approaches and makes contact with the target object. However, the visuomotor mechanisms through which binocular vision serves prehension are not fully understood. This study assessed the role of stereopsis in the predictive control of grasping by measuring grip force. Twenty participants performed a precision reach-to-grasp task in four viewing conditions: binocular, monocular, and with reduced stereoacuity (200 arc sec, > 400 arc sec). Monocular, compared to binocular viewing, was associated with a fourfold increase in grasp errors, a 56% increase in grasp duration, 22% decrease in grip force at 50 ms following grasp initiation, and the time of peak force occurred 40% later after grasp initiation (all p < 0.05). Grasp performance was also disrupted when viewing with reduced stereoacuity. Notably, grip force at the time of object lift-off was comparable between all viewing conditions. These results demonstrate that binocular stereopsis contributes to the efficient programming of grip forces. Specifically, stereopsis may provide important sensory information that enables the central nervous system to engage in predictive control of grasping.

An elasticity-curvature illusion decouples cutaneous and proprioceptive cues in active exploration of soft objects

Our sense of touch helps us encounter the richness of our natural world. Across a myriad of contexts and repetitions, we have learned to deploy certain exploratory movements in order to elicit perceptual cues that are salient and efficient. The task of identifying optimal exploration strategies and somatosensory cues that underlie our softness perception remains relevant and incomplete. Leveraging psychophysical evaluations combined with computational finite element modeling of skin contact mechanics, we investigate an illusion phenomenon in exploring softness; where small-compliant and large-stiff spheres are indiscriminable. By modulating contact interactions at the finger pad, we find this elasticity-curvature illusion is observable in passive touch, when the finger is constrained to be stationary and only cutaneous responses from mechanosensitive afferents are perceptible. However, these spheres become readily discriminable when explored volitionally with musculoskeletal proprioception available. We subsequently exploit this phenomenon to dissociate relative contributions from cutaneous and proprioceptive signals in encoding our percept of material softness. Our findings shed light on how we volitionally explore soft objects, i.e., by controlling surface contact force to optimally elicit and integrate proprioceptive inputs amidst indiscriminable cutaneous contact cues. Moreover, in passive touch, e.g., for touch-enabled displays grounded to the finger, we find those spheres are discriminable when rates of change in cutaneous contact are varied between the stimuli, to supplant proprioceptive feedback.

Friction is preferred over grasp configuration in precision grip grasping

How humans visually select where to grasp an object depends on many factors, including grasp stability and preferred grasp configuration. We examined how endpoints are selected when these two factors are brought into conflict: Do people favor stable grasps or do they prefer their natural grasp configurations? Participants reached to grasp one of three cuboids oriented so that its two corners were either aligned with, or rotated away from, each individual's natural grasp axis (NGA). All objects were made of brass (mass: 420 g), but the surfaces of their sides were manipulated to alter friction: 1) all-brass, 2) two opposing sides covered with wood, and the other two remained of brass, or 3) two opposing sides covered with sandpaper, and the two remaining brass sides smeared with Vaseline. Grasps were evaluated as either clockwise (thumb to the left of finger in frontal plane) or counterclockwise of the NGA. Grasp endpoints depended on both object orientation and surface material. For the all-brass object, grasps were bimodally distributed in the NGA-aligned condition but predominantly clockwise in the NGA-unaligned condition. These data reflected participants' natural grasp configuration independently of surface material. When grasping objects with different surface materials, endpoint selection changed: Participants sacrificed their usual grasp configuration to choose the more stable object sides. A model in which surface material shifts participants' preferred grip angle proportionally to the perceived friction of the surfaces accounts for our results. Our findings demonstrate that a stable grasp is more important than a biomechanically comfortable grasp configuration.NEW & NOTEWORTHY When grasping an object, humans can place their fingers at several positions on its surface. The selection of these endpoints depends on many factors, with two of the most important being grasp stability and grasp configuration. We put these two factors in conflict and examine which is considered more important. Our results highlight that humans are not reluctant to adopt unusual grasp configurations to satisfy grasp stability.

Humans Can Visually Judge Grasp Quality and Refine Their Judgments Through Visual and Haptic Feedback

How humans visually select where to grasp objects is determined by the physical object properties (e.g., size, shape, weight), the degrees of freedom of the arm and hand, as well as the task to be performed. We recently demonstrated that human grasps are near-optimal with respect to a weighted combination of different cost functions that make grasps uncomfortable, unstable, or impossible, e.g., due to unnatural grasp apertures or large torques. Here, we ask whether humans can consciously access these rules. We test if humans can explicitly judge grasp quality derived from rules regarding grasp size, orientation, torque, and visibility. More specifically, we test if grasp quality can be inferred (i) by using visual cues and motor imagery alone, (ii) from watching grasps executed by others, and (iii) through performing grasps, i.e., receiving visual, proprioceptive and haptic feedback. Stimuli were novel objects made of 10 cubes of brass and wood (side length 2.5 cm) in various configurations. On each object, one near-optimal and one sub-optimal grasp were selected based on one cost function (e.g., torque), while the other constraints (grasp size, orientation, and visibility) were kept approximately constant or counterbalanced. Participants were visually cued to the location of the selected grasps on each object and verbally reported which of the two grasps was best. Across three experiments, participants were required to either (i) passively view the static objects and imagine executing the two competing grasps, (ii) passively view videos of other participants grasping the objects, or (iii) actively grasp the objects themselves. Our results show that, for a majority of tested objects, participants could already judge grasp optimality from simply viewing the objects and imagining to grasp them, but were significantly better in the video and grasping session. These findings suggest that humans can determine grasp quality even without performing the grasp-perhaps through motor imagery-and can further refine their understanding of how to correctly grasp an object through sensorimotor feedback but also by passively viewing others grasp objects.