1
|
Zhao Y, Lu E, Zeng Y. Brain-inspired bodily self-perception model for robot rubber hand illusion. PATTERNS (NEW YORK, N.Y.) 2023; 4:100888. [PMID: 38106608 PMCID: PMC10724368 DOI: 10.1016/j.patter.2023.100888] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Revised: 08/21/2023] [Accepted: 11/07/2023] [Indexed: 12/19/2023]
Abstract
The core of bodily self-consciousness involves perceiving ownership of one's body. A central question is how body illusions like the rubber hand illusion (RHI) occur. Existing theoretical models still lack satisfying computational explanations from connectionist perspectives, especially for how the brain encodes body perception and generates illusions from neuronal interactions. Moreover, the integration of disability experiments is also neglected. Here, we integrate biological findings of bodily self-consciousness to propose a brain-inspired bodily self-perception model by which perceptions of bodily self are autonomously constructed without any supervision signals. We successfully validated the model with six RHI experiments and a disability experiment on an iCub humanoid robot and simulated environments. The results show that our model can not only well-replicate the behavioral and neural data of monkeys in biological experiments but also reasonably explain the causes and results of RHI at the neuronal level, thus contributing to the revelation of mechanisms underlying RHI.
Collapse
Affiliation(s)
- Yuxuan Zhao
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Enmeng Lu
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Yi Zeng
- Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
- Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, China
- School of Future Technology, University of Chinese Academy of Sciences, Beijing 100049, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
- Center for Long-term Artificial Intelligence, Beijing, China
| |
Collapse
|
2
|
Bernasconi F, Noel JP, Park HD, Faivre N, Seeck M, Spinelli L, Schaller K, Blanke O, Serino A. Audio-Tactile and Peripersonal Space Processing Around the Trunk in Human Parietal and Temporal Cortex: An Intracranial EEG Study. Cereb Cortex 2019; 28:3385-3397. [PMID: 30010843 PMCID: PMC6095214 DOI: 10.1093/cercor/bhy156] [Citation(s) in RCA: 38] [Impact Index Per Article: 7.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2018] [Accepted: 06/14/2018] [Indexed: 12/04/2022] Open
Abstract
Interactions with the environment happen within one’s peripersonal space (PPS)—the space surrounding the body. Studies in monkeys and humans have highlighted a multisensory distributed cortical network representing the PPS. However, knowledge about the temporal dynamics of PPS processing around the trunk is lacking. Here, we recorded intracranial electroencephalography (iEEG) in humans while administering tactile stimulation (T), approaching auditory stimuli (A), and the 2 combined (AT). To map PPS, tactile stimulation was delivered when the sound was far, intermediate, or close to the body. The 19% of the electrodes showed AT multisensory integration. Among those, 30% showed a PPS effect, a modulation of the response as a function of the distance between the sound and body. AT multisensory integration and PPS effects had similar spatiotemporal characteristics, with an early response (~50 ms) in the insular cortex, and later responses (~200 ms) in precentral and postcentral gyri. Superior temporal cortex showed a different response pattern with AT multisensory integration at ~100 ms without a PPS effect. These results, represent the first iEEG delineation of PPS processing in humans and show that PPS and multisensory integration happen at similar neural sites and time periods, suggesting that PPS representation is based on a spatial modulation of multisensory integration.
Collapse
Affiliation(s)
- Fosco Bernasconi
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Swiss Federal Institute of Technology (Ecole Polytechnique Fédérale de Lausanne), Geneva, Switzerland.,Center for Neuroprosthetics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Jean-Paul Noel
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Swiss Federal Institute of Technology (Ecole Polytechnique Fédérale de Lausanne), Geneva, Switzerland.,Neuroscience Graduate Program, Vanderbilt University, Nashville, USA.,Vanderbilt Brain Institute, Vanderbilt University, Nashville, USA
| | - Hyeong Dong Park
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Swiss Federal Institute of Technology (Ecole Polytechnique Fédérale de Lausanne), Geneva, Switzerland.,Center for Neuroprosthetics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Nathan Faivre
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Swiss Federal Institute of Technology (Ecole Polytechnique Fédérale de Lausanne), Geneva, Switzerland.,Center for Neuroprosthetics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne, Geneva, Switzerland.,Centre d'Economie de la Sorbonne, CNRS UMR 8174, Paris, France
| | - Margitta Seeck
- Presurgical Epilepsy Evaluation Unit, Neurology Department, University Hospital of Geneva, Geneva, Switzerland
| | - Laurent Spinelli
- Presurgical Epilepsy Evaluation Unit, Neurology Department, University Hospital of Geneva, Geneva, Switzerland
| | - Karl Schaller
- Department of Neurosurgery, Geneva University Hospital (HUG), 4 Rue Gabrielle-Perret-Gentil, Geneva, Switzerland
| | - Olaf Blanke
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Swiss Federal Institute of Technology (Ecole Polytechnique Fédérale de Lausanne), Geneva, Switzerland.,Center for Neuroprosthetics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne, Geneva, Switzerland.,Centre d'Economie de la Sorbonne, CNRS UMR 8174, Paris, France
| | - Andrea Serino
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Swiss Federal Institute of Technology (Ecole Polytechnique Fédérale de Lausanne), Geneva, Switzerland.,Center for Neuroprosthetics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne, Geneva, Switzerland.,MySpace Lab, Department of Clinical Neuroscience, Centre Hospitalier Universitaire Vaudois (CHUV), University of Lausanne, Lausanne, Switzerland
| |
Collapse
|
4
|
An Application of Deep Learning to Tactile Data for Object Recognition under Visual Guidance. SENSORS 2019; 19:s19071534. [PMID: 30934907 PMCID: PMC6480322 DOI: 10.3390/s19071534] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/27/2019] [Revised: 03/18/2019] [Accepted: 03/25/2019] [Indexed: 11/16/2022]
Abstract
Drawing inspiration from haptic exploration of objects by humans, the current work proposes a novel framework for robotic tactile object recognition, where visual information in the form of a set of visually interesting points is employed to guide the process of tactile data acquisition. Neuroscience research confirms the integration of cutaneous data as a response to surface changes sensed by humans with data from joints, muscles, and bones (kinesthetic cues) for object recognition. On the other hand, psychological studies demonstrate that humans tend to follow object contours to perceive their global shape, which leads to object recognition. In compliance with these findings, a series of contours are determined around a set of 24 virtual objects from which bimodal tactile data (kinesthetic and cutaneous) are obtained sequentially and by adaptively changing the size of the sensor surface according to the object geometry for each object. A virtual Force Sensing Resistor array (FSR) is employed to capture cutaneous cues. Two different methods for sequential data classification are then implemented using Convolutional Neural Networks (CNN) and conventional classifiers, including support vector machines and k-nearest neighbors. In the case of conventional classifiers, we exploit contourlet transformation to extract features from tactile images. In the case of CNN, two networks are trained for cutaneous and kinesthetic data and a novel hybrid decision-making strategy is proposed for object recognition. The proposed framework is tested both for contours determined blindly (randomly determined contours of objects) and contours determined using a model of visual attention. Trained classifiers are tested on 4560 new sequential tactile data and the CNN trained over tactile data from object contours selected by the model of visual attention yields an accuracy of 98.97% which is the highest accuracy among other implemented approaches.
Collapse
|
5
|
Xue Z, Zeng X, Koehl L, Shen L. Interpretation of Fabric Tactile Perceptions through Visual Features for Textile Products. J SENS STUD 2016. [DOI: 10.1111/joss.12201] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
Affiliation(s)
- Z. Xue
- Department of clothing design and engineering, School of Textiles and Clothing; Jiangnan university; Wuxi Jiangsu province 214122 P.R China
- Research group of human centered design (HCD), Laboratoire de Génie et Matériaux Textiles (GEMTEX), Ecole Nationale Supérieure des Arts et Industries Textiles (ENSAIT); 2 allée Louise et Victor Champier, BP30329, F-59056 Roubaix Cedex 1 France
| | - X. Zeng
- Research group of human centered design (HCD), Laboratoire de Génie et Matériaux Textiles (GEMTEX), Ecole Nationale Supérieure des Arts et Industries Textiles (ENSAIT); 2 allée Louise et Victor Champier, BP30329, F-59056 Roubaix Cedex 1 France
| | - L. Koehl
- Research group of human centered design (HCD), Laboratoire de Génie et Matériaux Textiles (GEMTEX), Ecole Nationale Supérieure des Arts et Industries Textiles (ENSAIT); 2 allée Louise et Victor Champier, BP30329, F-59056 Roubaix Cedex 1 France
| | - L. Shen
- Department of clothing design and engineering, School of Textiles and Clothing; Jiangnan university; Wuxi Jiangsu province 214122 P.R China
| |
Collapse
|
6
|
Kilteni K, Maselli A, Kording KP, Slater M. Over my fake body: body ownership illusions for studying the multisensory basis of own-body perception. Front Hum Neurosci 2015; 9:141. [PMID: 25852524 PMCID: PMC4371812 DOI: 10.3389/fnhum.2015.00141] [Citation(s) in RCA: 205] [Impact Index Per Article: 22.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2014] [Accepted: 02/28/2015] [Indexed: 11/13/2022] Open
Abstract
Which is my body and how do I distinguish it from the bodies of others, or from objects in the surrounding environment? The perception of our own body and more particularly our sense of body ownership is taken for granted. Nevertheless, experimental findings from body ownership illusions (BOIs), show that under specific multisensory conditions, we can experience artificial body parts or fake bodies as our own body parts or body, respectively. The aim of the present paper is to discuss how and why BOIs are induced. We review several experimental findings concerning the spatial, temporal, and semantic principles of crossmodal stimuli that have been applied to induce BOIs. On the basis of these principles, we discuss theoretical approaches concerning the underlying mechanism of BOIs. We propose a conceptualization based on Bayesian causal inference for addressing how our nervous system could infer whether an object belongs to our own body, using multisensory, sensorimotor, and semantic information, and we discuss how this can account for several experimental findings. Finally, we point to neural network models as an implementational framework within which the computational problem behind BOIs could be addressed in the future.
Collapse
Affiliation(s)
- Konstantina Kilteni
- Event Lab, Department of Personality, Evaluation and Psychological Treatment, University of Barcelona Barcelona, Spain ; IR3C Institute for Brain, Cognition, and Behaviour, University of Barcelona Barcelona, Spain
| | - Antonella Maselli
- Event Lab, Department of Personality, Evaluation and Psychological Treatment, University of Barcelona Barcelona, Spain
| | - Konrad P Kording
- Sensory Motor Performance Program, Rehabilitation Institute of Chicago Chicago, IL, USA ; Department of Physical Medicine and Rehabilitation, Northwestern University Chicago, IL, USA ; Department of Physiology, Northwestern University Chicago, IL, USA
| | - Mel Slater
- Event Lab, Department of Personality, Evaluation and Psychological Treatment, University of Barcelona Barcelona, Spain ; IR3C Institute for Brain, Cognition, and Behaviour, University of Barcelona Barcelona, Spain ; Institució Catalana de Recerca i Estudis Avançats, Passeig Lluís Companys 23 Barcelona, Spain
| |
Collapse
|
8
|
A neural network model can explain ventriloquism aftereffect and its generalization across sound frequencies. BIOMED RESEARCH INTERNATIONAL 2013; 2013:475427. [PMID: 24228250 PMCID: PMC3818813 DOI: 10.1155/2013/475427] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/29/2013] [Revised: 08/28/2013] [Accepted: 08/28/2013] [Indexed: 11/17/2022]
Abstract
Exposure to synchronous but spatially disparate auditory and visual stimuli produces a perceptual shift of sound location towards the visual stimulus (ventriloquism effect). After adaptation to a ventriloquism situation, enduring sound shift is observed in the absence of the visual stimulus (ventriloquism aftereffect). Experimental studies report opposing results as to aftereffect generalization across sound frequencies varying from aftereffect being confined to the frequency used during adaptation to aftereffect generalizing across some octaves. Here, we present an extension of a model of visual-auditory interaction we previously developed. The new model is able to simulate the ventriloquism effect and, via Hebbian learning rules, the ventriloquism aftereffect and can be used to investigate aftereffect generalization across frequencies. The model includes auditory neurons coding both for the spatial and spectral features of the auditory stimuli and mimicking properties of biological auditory neurons. The model suggests that different extent of aftereffect generalization across frequencies can be obtained by changing the intensity of the auditory stimulus that induces different amounts of activation in the auditory layer. The model provides a coherent theoretical framework to explain the apparently contradictory results found in the literature. Model mechanisms and hypotheses are discussed in relation to neurophysiological and psychophysical data.
Collapse
|
9
|
Wang WY, Hu L, Valentini E, Xie XB, Cui HY, Hu Y. Dynamic characteristics of multisensory facilitation and inhibition. Cogn Neurodyn 2012; 6:409-19. [PMID: 24082962 DOI: 10.1007/s11571-012-9197-x] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2011] [Revised: 02/14/2012] [Accepted: 03/01/2012] [Indexed: 11/29/2022] Open
Abstract
Multimodal integration, which mainly refers to multisensory facilitation and multisensory inhibition, is the process of merging multisensory information in the human brain. However, the neural mechanisms underlying the dynamic characteristics of multimodal integration are not fully understood. The objective of this study is to investigate the basic mechanisms of multimodal integration by assessing the intermodal influences of vision, audition, and somatosensory sensations (the influence of multisensory background events to the target event). We used a timed target detection task, and measured both behavioral and electroencephalographic responses to visual target events (green solid circle), auditory target events (2 kHz pure tone) and somatosensory target events (1.5 ± 0.1 mA square wave pulse) from 20 normal participants. There were significant differences in both behavior performance and ERP components when comparing the unimodal target stimuli with multimodal (bimodal and trimodal) target stimuli for all target groups. Significant correlation among reaction time and P3 latency was observed across all target conditions. The perceptual processing of auditory target events (A) was inhibited by the background events, while the perceptual processing of somatosensory target events (S) was facilitated by the background events. In contrast, the perceptual processing of visual target events (V) remained impervious to multisensory background events.
Collapse
Affiliation(s)
- W Y Wang
- Chinese Academy of Medical Science and Peking Union Medical College, Institute of Biomedical Engineering, Tianjin, China ; Department of Orthopaedics and Traumatology, Duchess of Kent Children's Hospital, The University of Hong Kong, 12 Sandy Bay Road, Hong Kong, China
| | | | | | | | | | | |
Collapse
|