Visuo-Haptic Exploration for Multimodal Memory

Affect Recognition in Hand-Object Interaction Using Object-Sensed Tactile and Kinematic Data

We investigate the recognition of the affective states of a person performing an action with an object, by processing the object-sensed data. We focus on sequences of basic actions such as grasping and rotating, which are constituents of daily-life interactions. iCube, a 5 cm cube, was used to collect tactile and kinematics data that consist of tactile maps (without information on the pressure applied to the surface), and rotations. We conduct two studies: classification of i) emotions and ii) the vitality forms. In both, the participants perform a semi-structured task composed of basic actions. For emotion recognition, 237 trials by 11 participants associated with anger, sadness, excitement, and gratitude were used to train models using 10 hand-crafted features. The classifier accuracy reaches up to 82.7%. Interestingly, the same classifier when learned exclusively with the tactile data performs on par with its counterpart modeled with all 10 features. For the second study, 1135 trials by 10 participants were used to classify two vitality forms. The best-performing model differentiated gentle actions from rude ones with an accuracy of 84.85%. The results also confirm that people touch objects differently when performing these basic actions with different affective states and attitudes.

Early blindness modulates haptic object recognition

Haptic object recognition is usually an efficient process although slower and less accurate than its visual counterpart. The early loss of vision imposes a greater reliance on haptic perception for recognition compared to the sighted. Therefore, we may expect that congenitally blind persons could recognize objects through touch more quickly and accurately than late blind or sighted people. However, the literature provided mixed results. Furthermore, most of the studies on haptic object recognition focused on performance, devoting little attention to the exploration procedures that conducted to that performance. In this study, we used iCube, an instrumented cube recording its orientation in space as well as the location of the points of contact on its faces. Three groups of congenitally blind, late blind and age and gender-matched blindfolded sighted participants were asked to explore the cube faces where little pins were positioned in varying number. Participants were required to explore the cube twice, reporting whether the cube was the same or it differed in pins disposition. Results showed that recognition accuracy was not modulated by the level of visual ability. However, congenitally blind touched more cells simultaneously while exploring the faces and changed more the pattern of touched cells from one recording sample to the next than late blind and sighted. Furthermore, the number of simultaneously touched cells negatively correlated with exploration duration. These findings indicate that early blindness shapes haptic exploration of objects that can be held in hands.

Mental Rotation Skill Shapes Haptic Exploration Strategies

Haptic exploration strategies have been traditionally studied focusing on hand movements and neglecting how objects are moved in space. However, in daily life situations touch and movement cannot be disentangled. Furthermore, the relation between object manipulation as well as performance in haptic tasks and spatial skill is still little understood. In this study, we used iCube, a sensorized cube recording its orientation in space as well as the location of the points of contact on its faces. Participants had to explore the cube faces where little pins were positioned in varying number and count the number of pins on the faces with either even or odd number of pins. At the end of this task, they also completed a standard visual mental rotation test (MRT). Results showed that higher MRT scores were associated with better performance in the task with iCube both in term of accuracy and exploration speed and exploration strategies associated with better performance were identified. High performers tended to rotate the cube so that the explored face had the same spatial orientation (i.e., they preferentially explored the upward face and rotated iCube to explore the next face in the same orientation). They also explored less often twice the same face and were faster and more systematic in moving from one face to the next. These findings indicate that iCube could be used to infer subjects' spatial skill in a more natural and unobtrusive fashion than with standard MRTs.

Three-Dimensional Virtual and Printed Prototypes in Complex Congenital and Pediatric Cardiac Surgery-A Multidisciplinary Team-Learning Experience

Three-dimensional (3D) virtual modeling and printing advances individualized medicine and surgery. In congenital cardiac surgery, 3D virtual models and printed prototypes offer advantages of better understanding of complex anatomy, hands-on preoperative surgical planning and emulation, and improved communication within the multidisciplinary team and to patients. We report our single center team-learning experience about the realization and validation of possible clinical benefits of 3D-printed models in surgical planning of complex congenital cardiac surgery. CT-angiography raw data were segmented into 3D-virtual models of the heart-great vessels. Prototypes were 3D-printed as rigid “blood-volume” and flexible “hollow”. The accuracy of the models was evaluated intraoperatively. Production steps were realized in the framework of a clinical/research partnership. We produced 3D prototypes of the heart-great vessels for 15 case scenarios (nine males, median age: 11 months) undergoing complex intracardiac repairs. Parity between 3D models and intraoperative structures was within 1 mm range. Models refined diagnostics in 13/15, provided new anatomic information in 9/15. As a team-learning experience, all complex staged redo-operations (13/15; Aristotle-score mean: 10.64 ± 1.95) were rehearsed on the 3D models preoperatively. 3D-printed prototypes significantly contributed to an improved/alternative operative plan on the surgical approach, modification of intracardiac repair in 13/15. No operative morbidity/mortality occurred. Our clinical/research partnership provided coverage for the extra time/labor and material/machinery not financed by insurance. 3D-printed models provided a team-learning experience and contributed to the safety of complex congenital cardiac surgeries. A clinical/research partnership may open avenues for bioprinting of patient-specific implants.

Enhancing Organizational Memory Through Virtual Memoryscapes: Does It Work?

Enhancing cognitive memory through virtual reality represents an issue, that has never been investigated in organizational settings. Here, we compared a virtual memoryscape (treatment) – an immersive virtual environment used by subjects as a shared memory tool based on spatial navigation – with respect to the traditional individual-specific mnemonic tool based on the “method of loci” (control). A memory task characterized by high ecological validity was administered to 82 subjects employed by large banking group. Memory recall was measured, for both groups, immediately after the task (Phase 1) and one week later (Phase 2). Results show that (i) in Phase 1, the method of loci was more efficient in terms of recalling information than the to the virtual memoryscape; (ii) in Phase 2, there was no difference. Compared to the method of loci, the virtual memoryscape presents the advantages – relevant for organizations – of being collective, controllable, dynamic, and non-manipulable.

Functional heterogeneity in the left lateral posterior parietal cortex during visual and haptic crossmodal dot-surface matching

BACKGROUND

Vision and touch are thought to contribute information to object perception in an independent but complementary manner. The left lateral posterior parietal cortex (LPPC) has long been associated with multisensory information processing, and it plays an important role in visual and haptic crossmodal information retrieval. However, it remains unclear how LPPC subregions are involved in visuo-haptic crossmodal retrieval processing.

METHODS

In the present study, we used an fMRI experiment with a crossmodal delayed match-to-sample paradigm to reveal the functional role of LPPC subregions related to unimodal and crossmodal dot-surface retrieval.

RESULTS

The visual-to-haptic condition enhanced the activity of the left inferior parietal lobule relative to the haptic unimodal condition, whereas the inverse condition enhanced the activity of the left superior parietal lobule. By contrast, activation of the left intraparietal sulcus did not differ significantly between the crossmodal and unimodal conditions. Seed-based resting connectivity analysis revealed that these three left LPPC subregions engaged distinct networks, confirming their different functions in crossmodal retrieval processing.

CONCLUSION

Taken together, the findings suggest that functional heterogeneity of the left LPPC during visuo-haptic crossmodal dot-surface retrieval processing reflects that the left LPPC does not simply contribute to retrieval of past information; rather, each subregion has a specific functional role in resolving different task requirements.