Memory influences haptic perception of softness

SORI: A softness-rendering interface to unravel the nature of softness perception

Tactile perception of softness serves a critical role in the survival, well-being, and social interaction among various species, including humans. This perception informs activities from food selection in animals to medical palpation for disease detection in humans. Despite its fundamental importance, a comprehensive understanding of how softness is neurologically and cognitively processed remains elusive. Previous research has demonstrated that the somatosensory system leverages both cutaneous and kinesthetic cues for the sensation of softness. Factors such as contact area, depth, and force play a particularly critical role in sensations experienced at the fingertips. Yet, existing haptic technologies designed to explore this phenomenon are limited, as they often couple force and contact area, failing to provide a real-world experience of softness perception. Our research introduces the softness-rendering interface (SORI), a haptic softness display designed to bridge this knowledge gap. Unlike its predecessors, SORI has the unique ability to decouple contact area and force, thereby allowing for a quantitative representation of softness sensations at the fingertips. Furthermore, SORI incorporates individual physical fingertip properties and model-based softness cue estimation and mapping to provide a highly personalized experience. Utilizing this method, SORI quantitatively replicates the sensation of softness on stationary, dynamic, homogeneous, and heterogeneous surfaces. We demonstrate that SORI accurately renders the surfaces of both virtual and daily objects, thereby presenting opportunities across a range of fields, from teleoperation to medical technology. Finally, our proposed method and SORI will expedite psychological and neuroscience research to unlock the nature of softness perception.

Surgery in the Next Space Missions

In the coming years, missions to the Moon and Mars shall be the new goals of space flight. The complexity of these missions due to the great distance from Earth and the unforeseen obstacles to settle on another planet have given rise to great concerns for crew health and survival. The need for advanced crew autonomy and a different approach to surgical emergency require new protocols and devices to help future crew medical officers and other crew members in a task of unprecedented difficulty. Hence, the increasing variety of schedules, devices, and protocols being developed. A serious health problem, such as an emerging surgical disease or severe trauma, can jeopardize the mission and survival of the entire crew. Many other difficulties are present in deep-space missions or settlements on other planets, such as communication and supply, also medical, delays, and shortage, and the presence of radiation. Progress in advanced technologies as well as the evolution of robotic surgery and the use of artificial intelligence are other topics of this review. In this particular area of research, even if we are still very far from an "intelligent robot", this evolution must be evaluated in the light of legislative and ethical considerations. This topic was presented at the annual meeting of the American College of Surgeons-Italy Chapter in 2021.

Assessing the representational structure of softness activated by words

Softness is an important material property that can be judged directly, by interacting with an object, but also indirectly, by simply looking at an image of a material. The latter is likely possible by filling in relevant multisensory information from prior experiences with soft materials. Such experiences are thought to lead to associations that make up our representations about perceptual softness. Here, we investigate the structure of this representational space when activated by words, and compare it to haptic and visual perceptual spaces that we obtained in earlier work. To this end, we performed an online study where people rated different sensory aspects of soft materials, presented as written names. We compared the results with the previous studies where identical ratings were made on the basis of visual and haptic information. Correlation and Procrustes analyses show that, overall, the representational spaces of verbally presented materials were similar to those obtained from haptic and visual experiments. However, a classifier analysis showed that verbal representations could better be predicted from those obtained from visual than from haptic experiments. In a second study we rule out that these larger discrepancies in representations between verbal and haptic conditions could be due to difficulties in material identification in haptic experiments. We discuss the results with respect to the recent idea that at perceived softness is a multidimensional construct.

Hand explorations are determined by the characteristics of the perceptual space of real-world materials from silk to sand

Perceiving mechanical properties of objects, i.e., how they react to physical forces, is a crucial ability in many aspects of life, from choosing an avocado to picking your clothes. There is, a wide variety of materials that differ substantially in their mechanical properties. For example, both, silk and sand deform and change shape in response to exploration forces, but each does so in very different ways. Studies show that the haptic perceptual space has multiple dimensions corresponding to the physical properties of textures, however in these experiments the range of materials or exploratory movements were restricted. Here we investigate the perceptual dimensionality in a large set of real materials in a free haptic exploration task. Thirty-two participants actively explored deformable and non-deformable materials with their hands and rated them on several attributes. Using the semantic differential technique, video analysis and linear classification, we found four haptic dimensions, each associated with a distinct set of hand and finger movements during active exploration. Taken together our findings suggest that the physical, particularly the mechanical, properties of a material systematically affect how it is explored on a much more fine-grained level than originally thought.

Getting Insights From Twitter: What People Want to Touch in Daily Life

Understanding what people want to touch in daily life has been one of the central topics in the fields of haptic science, engineering, and marketing. Several studies have addressed the topic, however, their findings were highly dependent on the experimental stimuli in the laboratory environment. In this study, we tried to gain insights into targets that people want to touch in daily life by conducting a Twitter survey. We collected a considerable amount of Japanese Twitter posts containing references to the desire for touch. To capture the motivation underlying these desires in relation to haptic properties, we used several queries that comprehensively covered exploratory procedures. The results showed that targets that people wanted to touch depended on the exploratory procedures in the queries used (e.g., "want to stroke" tended to target people and animals). We also found that these targets were desirable to touch not only for their haptic properties but also as a means of communicating with other people or living animals. Our findings would be important for understanding human haptic function in the real world and for developing consumer haptic displays and applications.

Visual and Tactile Sensory Systems Share Common Features in Object Recognition

Although we use our visual and tactile sensory systems interchangeably for object recognition on a daily basis, little is known about the mechanism underlying this ability. This study examined how 3D shape features of objects form two congruent and interchangeable visual and tactile perceptual spaces in healthy male and female participants. Since active exploration plays an important role in shape processing, a virtual reality environment was used to visually explore 3D objects called digital embryos without using the tactile sense. In addition, during the tactile procedure, blindfolded participants actively palpated a 3D-printed version of the same objects with both hands. We first demonstrated that the visual and tactile perceptual spaces were highly similar. We then extracted a series of 3D shape features to investigate how visual and tactile exploration can lead to the correct identification of the relationships between objects. The results indicate that both modalities share the same shape features to form highly similar veridical spaces. This finding suggests that visual and tactile systems might apply similar cognitive processes to sensory inputs that enable humans to rely merely on one modality in the absence of another to recognize surrounding objects.

The look and feel of soft are similar across different softness dimensions

The softness of objects can be perceived through several senses. For instance, to judge the softness of a cat's fur, we do not only look at it, we often also run our fingers through its coat. Recently, we have shown that haptically perceived softness covaries with the compliance, viscosity, granularity, and furriness of materials (Dovencioglu, Üstün, Doerschner, & Drewing, 2020). However, it is unknown whether vision can provide similar information about the various aspects of perceived softness. Here, we investigated this question in an experiment with three conditions: in the haptic condition, blindfolded participants explored materials with their hands, in the static visual condition participants were presented with close-up photographs of the same materials, and in the dynamic visual condition participants watched videos of the hand-material interactions that were recorded in the haptic condition. After haptically or visually exploring the materials, participants rated them on various attributes. Our results show a high overall perceptual correspondence among the three experimental conditions. With a few exceptions, this correspondence tended to be strongest between haptic and dynamic visual conditions. These results are discussed with respect to information potentially available through the senses, or through prior experience, when judging the softness of materials.

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

Time-dependent Cues Encode the Minimum Exploration Time in Discriminating Naturalistic Compliances

Our perception of compliance is informed by multi-dimensional tactile cues. Compared with stationary cues at terminal contact, time-dependent cues may afford optimal efficiency, speed, and fidelity. In this work, we investigate strategies by which temporal cues may encode compliances by modulating our exploration time. Two potential perceptual strategies are considered, inspired by memory representations within and between explorations. For either strategy, we introduce a unique computational approach. First, a curve similarity analysis, of accumulating touch force between sequentially explored compliances, generates a minimum time for discrimination. Second, a Kalman filtering approach derives a recognition time from progressive integration of stiffness estimates over time within a single exploration. Human-subjects experiments are conducted for both single finger touch and pinch grasp. The results indicate that for either strategy, by employing a more natural pinch grasp, time-dependent cues afford greater efficiency by reducing the exploration time, especially for harder objects. Moreover, compared to single finger touch, pinch grasp improves discrimination rates in judging plum ripeness. The time-dependent strategies as defined here appear promising, and may tie with the time-scales over which we make perceptual judgments.

Neonatal Assessment Manual Score: Is There a Role of a Novel, Structured Touch-Based Evaluation in Neonatal Intensive Care Unit?

Despite the technological improvements in monitoring preterm infants in the neonatal intensive care unit, routine care in the neonatal ward is primarily based on manual procedures. Although manual clinical procedures play a critical role in neonatology, little attention has been paid to palpation as a clinical assessment tool. Palpation is a clinical evaluation tool that relies mostly on the senses of touch and proprioception. Based on recent studies investigating the role and clinical effectiveness of touch in full-term and preterm babies, this paper proposes an evaluative touch-based procedure-the Neonatal Assessment Manual Score (NAME) model-that could be useful in the neonatal ward and describes its rationale. The operator applies gentle light pressures to the infant's body. In essence, the touch stimulates low-threshold afferent fibers that could influence the interoceptive cerebral network and the autonomic nervous system, thus altering the blood flow and breathing rhythm. These events could change how bodily fluids distribute among body segments and hence the body volume. The volume modification could be felt manually through haptic perception owing to the high sensitivity of the fingers. On the basis of their clinical conditions and stage of development, infants will respond differently to the applied pressures. Evaluating the infant's response, the operator produces a score of "bad," "marginal," or "good" for communicating quickly and clearly the infant's conditions to other professionals. Because the NAME model is intended for every professional who is used to touch-based procedures, if future studies confirmed its validity and reliability in clinical practice, the NAME model could become a part of the neonatal ward routine care for better assessing and managing the infant's conditions, even during emergencies.