1
|
Cary E, Lahdesmaki I, Badde S. Audiovisual simultaneity windows reflect temporal sensory uncertainty. Psychon Bull Rev 2024:10.3758/s13423-024-02478-4. [PMID: 38388825 DOI: 10.3758/s13423-024-02478-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/04/2024] [Indexed: 02/24/2024]
Abstract
The ability to judge the temporal alignment of visual and auditory information is a prerequisite for multisensory integration and segregation. However, each temporal measurement is subject to error. Thus, when judging whether a visual and auditory stimulus were presented simultaneously, observers must rely on a subjective decision boundary to distinguish between measurement error and truly misaligned audiovisual signals. Here, we tested whether these decision boundaries are relaxed with increasing temporal sensory uncertainty, i.e., whether participants make the same type of adjustment an ideal observer would make. Participants judged the simultaneity of audiovisual stimulus pairs with varying temporal offset, while being immersed in different virtual environments. To obtain estimates of participants' temporal sensory uncertainty and simultaneity criteria in each environment, an independent-channels model was fitted to their simultaneity judgments. In two experiments, participants' simultaneity decision boundaries were predicted by their temporal uncertainty, which varied unsystematically with the environment. Hence, observers used a flexibly updated estimate of their own audiovisual temporal uncertainty to establish subjective criteria of simultaneity. This finding implies that, under typical circumstances, audiovisual simultaneity windows reflect an observer's cross-modal temporal uncertainty.
Collapse
Affiliation(s)
- Emma Cary
- Department of Psychology, Tufts University, Medford, MA, 02155, USA
| | - Ilona Lahdesmaki
- Department of Psychology, Tufts University, Medford, MA, 02155, USA
| | - Stephanie Badde
- Department of Psychology, Tufts University, Medford, MA, 02155, USA.
| |
Collapse
|
2
|
Chiyohara S, Furukawa JI, Noda T, Morimoto J, Imamizu H. Proprioceptive short-term memory in passive motor learning. Sci Rep 2023; 13:20826. [PMID: 38012253 PMCID: PMC10682388 DOI: 10.1038/s41598-023-48101-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 11/22/2023] [Indexed: 11/29/2023] Open
Abstract
A physical trainer often physically guides a learner's limbs to teach an ideal movement, giving the learner proprioceptive information about the movement to be reproduced later. This instruction requires the learner to perceive kinesthetic information and store the instructed information temporarily. Therefore, (1) proprioceptive acuity to accurately perceive the taught kinesthetics and (2) short-term memory to store the perceived information are two critical functions for reproducing the taught movement. While the importance of proprioceptive acuity and short-term memory has been suggested for active motor learning, little is known about passive motor learning. Twenty-one healthy adults (mean age 25.6 years, range 19-38 years) participated in this study to investigate whether individual learning efficiency in passively guided learning is related to these two functions. Consequently, learning efficiency was significantly associated with short-term memory capacity. In particular, individuals who could recall older sensory stimuli showed better learning efficiency. However, no significant relationship was observed between learning efficiency and proprioceptive acuity. A causal graph model found a direct influence of memory on learning and an indirect effect of proprioceptive acuity on learning via memory. Our findings suggest the importance of a learner's short-term memory for effective passive motor learning.
Collapse
Affiliation(s)
- Shinya Chiyohara
- Brain Information Communication Research Laboratory Group, Advanced Telecommunications Research Institute International (ATR), Keihanna Science City, Kyoto, 619-0288, Japan
| | - Jun-Ichiro Furukawa
- Brain Information Communication Research Laboratory Group, Advanced Telecommunications Research Institute International (ATR), Keihanna Science City, Kyoto, 619-0288, Japan
- Man-Machine Collaboration Research Team, Guardian Robot Project, RIKEN, Kyoto, Japan
| | - Tomoyuki Noda
- Brain Information Communication Research Laboratory Group, Advanced Telecommunications Research Institute International (ATR), Keihanna Science City, Kyoto, 619-0288, Japan
| | - Jun Morimoto
- Brain Information Communication Research Laboratory Group, Advanced Telecommunications Research Institute International (ATR), Keihanna Science City, Kyoto, 619-0288, Japan.
- Man-Machine Collaboration Research Team, Guardian Robot Project, RIKEN, Kyoto, Japan.
- Graduate School of Informatics, Kyoto University, Kyoto, Japan.
| | - Hiroshi Imamizu
- Brain Information Communication Research Laboratory Group, Advanced Telecommunications Research Institute International (ATR), Keihanna Science City, Kyoto, 619-0288, Japan
- Department of Psychology, Graduate School of Humanities and Sociology, The University of Tokyo, Hongo 7-3-1, Bunkyo-Ku, Tokyo, 113-0033, Japan
- Research Into Artifacts, Center for Engineering, School of Engineering, The University of Tokyo, Hongo 7-3-1, Bunkyo-Ku, Tokyo, 113-8656, Japan
| |
Collapse
|
3
|
Bruns P, Röder B. Development and experience-dependence of multisensory spatial processing. Trends Cogn Sci 2023; 27:961-973. [PMID: 37208286 DOI: 10.1016/j.tics.2023.04.012] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2022] [Revised: 04/24/2023] [Accepted: 04/25/2023] [Indexed: 05/21/2023]
Abstract
Multisensory spatial processes are fundamental for efficient interaction with the world. They include not only the integration of spatial cues across sensory modalities, but also the adjustment or recalibration of spatial representations to changing cue reliabilities, crossmodal correspondences, and causal structures. Yet how multisensory spatial functions emerge during ontogeny is poorly understood. New results suggest that temporal synchrony and enhanced multisensory associative learning capabilities first guide causal inference and initiate early coarse multisensory integration capabilities. These multisensory percepts are crucial for the alignment of spatial maps across sensory systems, and are used to derive more stable biases for adult crossmodal recalibration. The refinement of multisensory spatial integration with increasing age is further promoted by the inclusion of higher-order knowledge.
Collapse
Affiliation(s)
- Patrick Bruns
- Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany.
| | - Brigitte Röder
- Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
| |
Collapse
|
4
|
Badde S, Landy MS, Adams WJ. Multisensory causal inference is feature-specific, not object-based. Philos Trans R Soc Lond B Biol Sci 2023; 378:20220345. [PMID: 37545302 PMCID: PMC10404918 DOI: 10.1098/rstb.2022.0345] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2023] [Accepted: 06/18/2023] [Indexed: 08/08/2023] Open
Abstract
Multisensory integration depends on causal inference about the sensory signals. We tested whether implicit causal-inference judgements pertain to entire objects or focus on task-relevant object features. Participants in our study judged virtual visual, haptic and visual-haptic surfaces with respect to two features-slant and roughness-against an internal standard in a two-alternative forced-choice task. Modelling of participants' responses revealed that the degree to which their perceptual judgements were based on integrated visual-haptic information varied unsystematically across features. For example, a perceived mismatch between visual and haptic roughness would not deter the observer from integrating visual and haptic slant. These results indicate that participants based their perceptual judgements on a feature-specific selection of information, suggesting that multisensory causal inference proceeds not at the object level but at the level of single object features. This article is part of the theme issue 'Decision and control processes in multisensory perception'.
Collapse
Affiliation(s)
- Stephanie Badde
- Department of Psychology, Tufts University, 490 Boston Avenue, Medford, MA 02155, USA
| | - Michael S. Landy
- Department of Psychology and Center of Neural Science, New York University, 6 Washington Place, New York, NY 10003, USA
| | - Wendy J. Adams
- Department of Psychology, University of Southampton, 44 Highfield Campus, Southampton SO17 1BJ, UK
| |
Collapse
|
5
|
Fassold ME, Locke SM, Landy MS. Feeling lucky? prospective and retrospective cues for sensorimotor confidence. PLoS Comput Biol 2023; 19:e1010740. [PMID: 37363929 DOI: 10.1371/journal.pcbi.1010740] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Accepted: 06/08/2023] [Indexed: 06/28/2023] Open
Abstract
On a daily basis, humans interact with the outside world using judgments of sensorimotor confidence, constantly evaluating our actions for success. We ask, what sensory and motor-execution cues are used in making these judgements and when are they available? Two sources of temporally distinct information are prospective cues, available prior to the action (e.g., knowledge of motor noise and past performance), and retrospective cues specific to the action itself (e.g., proprioceptive measurements). We investigated the use of these two cues in two tasks, a secondary motor-awareness task and a main task in which participants reached toward a visual target with an unseen hand and then made a continuous judgment of confidence about the success of the reach. Confidence was reported by setting the size of a circle centered on the reach-target location, where a larger circle reflects lower confidence. Points were awarded if the confidence circle enclosed the true endpoint, with fewer points returned for larger circles. This incentivized accurate reaches and attentive reporting to maximize the score. We compared three Bayesian-inference models of sensorimotor confidence based on either prospective cues, retrospective cues, or both sources of information to maximize expected gain (i.e., an ideal-performance model). Our findings showed two distinct strategies: participants either performed as ideal observers, using both prospective and retrospective cues to make the confidence judgment, or relied solely on prospective information, ignoring retrospective cues. Thus, participants can make use of retrospective cues, evidenced by the behavior observed in our motor-awareness task, but these cues are not always included in the computation of sensorimotor confidence.
Collapse
Affiliation(s)
- Marissa E Fassold
- Dept. of Psychology, New York University, New York, New York, United States of America
| | - Shannon M Locke
- Laboratoire des Systèmes Perceptifs, Département d'Études Cognitives, École Normale Supérieure, PSL University, CNRS, Paris, France
| | - Michael S Landy
- Dept. of Psychology, New York University, New York, New York, United States of America
- Center for Neural Science, New York University, New York, New York, United States of America
| |
Collapse
|
6
|
Murray CA, Shams L. Crossmodal interactions in human learning and memory. Front Hum Neurosci 2023; 17:1181760. [PMID: 37266327 PMCID: PMC10229776 DOI: 10.3389/fnhum.2023.1181760] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2023] [Accepted: 05/02/2023] [Indexed: 06/03/2023] Open
Abstract
Most studies of memory and perceptual learning in humans have employed unisensory settings to simplify the study paradigm. However, in daily life we are often surrounded by complex and cluttered scenes made up of many objects and sources of sensory stimulation. Our experiences are, therefore, highly multisensory both when passively observing the world and when acting and navigating. We argue that human learning and memory systems are evolved to operate under these multisensory and dynamic conditions. The nervous system exploits the rich array of sensory inputs in this process, is sensitive to the relationship between the sensory inputs, and continuously updates sensory representations, and encodes memory traces based on the relationship between the senses. We review some recent findings that demonstrate a range of human learning and memory phenomena in which the interactions between visual and auditory modalities play an important role, and suggest possible neural mechanisms that can underlie some surprising recent findings. We outline open questions as well as directions of future research to unravel human perceptual learning and memory.
Collapse
Affiliation(s)
- Carolyn A. Murray
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, United States
| | - Ladan Shams
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, United States
- Department of Bioengineering, Neuroscience Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA, United States
| |
Collapse
|
7
|
Federici A, Bernardi G, Senna I, Fantoni M, Ernst MO, Ricciardi E, Bottari D. Crossmodal plasticity following short-term monocular deprivation. Neuroimage 2023; 274:120141. [PMID: 37120043 DOI: 10.1016/j.neuroimage.2023.120141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 04/21/2023] [Accepted: 04/27/2023] [Indexed: 05/01/2023] Open
Abstract
A brief period of monocular deprivation (MD) induces short-term plasticity of the adult visual system. Whether MD elicits neural changes beyond visual processing is yet unclear. Here, we assessed the specific impact of MD on neural correlates of multisensory processes. Neural oscillations associated with visual and audio-visual processing were measured for both the deprived and the non-deprived eye. Results revealed that MD changed neural activities associated with visual and multisensory processes in an eye-specific manner. Selectively for the deprived eye, alpha synchronization was reduced within the first 150 ms of visual processing. Conversely, gamma activity was enhanced in response to audio-visual events only for the non-deprived eye within 100-300 ms after stimulus onset. The analysis of gamma responses to unisensory auditory events revealed that MD elicited a crossmodal upweight for the non-deprived eye. Distributed source modeling suggested that the right parietal cortex played a major role in neural effects induced by MD. Finally, visual and audio-visual processing alterations emerged for the induced component of the neural oscillations, indicating a prominent role of feedback connectivity. Results reveal the causal impact of MD on both unisensory (visual and auditory) and multisensory (audio-visual) processes and, their frequency-specific profiles. These findings support a model in which MD increases excitability to visual events for the deprived eye and audio-visual and auditory input for the non-deprived eye.
Collapse
Affiliation(s)
- A Federici
- MoMiLab, IMT School for Advanced Studies Lucca, 55100 Lucca, Italy.
| | - G Bernardi
- MoMiLab, IMT School for Advanced Studies Lucca, 55100 Lucca, Italy
| | - I Senna
- Applied Cognitive Psychology, Ulm University, 89081 Ulm, Germany
| | - M Fantoni
- MoMiLab, IMT School for Advanced Studies Lucca, 55100 Lucca, Italy
| | - M O Ernst
- Applied Cognitive Psychology, Ulm University, 89081 Ulm, Germany
| | - E Ricciardi
- MoMiLab, IMT School for Advanced Studies Lucca, 55100 Lucca, Italy
| | - D Bottari
- MoMiLab, IMT School for Advanced Studies Lucca, 55100 Lucca, Italy
| |
Collapse
|
8
|
Tsay JS, Tan S, Chu MA, Ivry RB, Cooper EA. Low Vision Impairs Implicit Sensorimotor Adaptation in Response to Small Errors, But Not Large Errors. J Cogn Neurosci 2023; 35:736-748. [PMID: 36724396 PMCID: PMC10512469 DOI: 10.1162/jocn_a_01969] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/03/2023]
Abstract
Successful goal-directed actions require constant fine-tuning of the motor system. This fine-tuning is thought to rely on an implicit adaptation process that is driven by sensory prediction errors (e.g., where you see your hand after reaching vs. where you expected it to be). Individuals with low vision experience challenges with visuomotor control, but whether low vision disrupts motor adaptation is unknown. To explore this question, we assessed individuals with low vision and matched controls with normal vision on a visuomotor task designed to isolate implicit adaptation. We found that low vision was associated with attenuated implicit adaptation only for small visual errors, but not for large visual errors. This result highlights important constraints underlying how low-fidelity visual information is processed by the sensorimotor system to enable successful implicit adaptation.
Collapse
|
9
|
Hong F, Badde S, Landy MS. Repeated exposure to either consistently spatiotemporally congruent or consistently incongruent audiovisual stimuli modulates the audiovisual common-cause prior. Sci Rep 2022; 12:15532. [PMID: 36109544 PMCID: PMC9478143 DOI: 10.1038/s41598-022-19041-7] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Accepted: 08/23/2022] [Indexed: 11/09/2022] Open
Abstract
AbstractTo estimate an environmental property such as object location from multiple sensory signals, the brain must infer their causal relationship. Only information originating from the same source should be integrated. This inference relies on the characteristics of the measurements, the information the sensory modalities provide on a given trial, as well as on a cross-modal common-cause prior: accumulated knowledge about the probability that cross-modal measurements originate from the same source. We examined the plasticity of this cross-modal common-cause prior. In a learning phase, participants were exposed to a series of audiovisual stimuli that were either consistently spatiotemporally congruent or consistently incongruent; participants’ audiovisual spatial integration was measured before and after this exposure. We fitted several Bayesian causal-inference models to the data; the models differed in the plasticity of the common-source prior. Model comparison revealed that, for the majority of the participants, the common-cause prior changed during the learning phase. Our findings reveal that short periods of exposure to audiovisual stimuli with a consistent causal relationship can modify the common-cause prior. In accordance with previous studies, both exposure conditions could either strengthen or weaken the common-cause prior at the participant level. Simulations imply that the direction of the prior-update might be mediated by the degree of sensory noise, the variability of the measurements of the same signal across trials, during the learning phase.
Collapse
|
10
|
Bruns P, Li L, Guerreiro MJ, Shareef I, Rajendran SS, Pitchaimuthu K, Kekunnaya R, Röder B. Audiovisual spatial recalibration but not integration is shaped by early sensory experience. iScience 2022; 25:104439. [PMID: 35874923 PMCID: PMC9301879 DOI: 10.1016/j.isci.2022.104439] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2021] [Revised: 02/14/2022] [Accepted: 05/06/2022] [Indexed: 11/15/2022] Open
Affiliation(s)
- Patrick Bruns
- Biological Psychology and Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
- Corresponding author
| | - Lux Li
- Biological Psychology and Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
- Department of Epidemiology and Biostatistics, Schulich School of Medicine & Dentistry, Western University, London, ON N6G 2M1, Canada
| | - Maria J.S. Guerreiro
- Biological Psychology and Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
- Biological Psychology, Department of Psychology, School of Medicine and Health Sciences, University of Oldenburg, 26111 Oldenburg, Germany
| | - Idris Shareef
- Jasti V Ramanamma Children’s Eye Care Centre, LV Prasad Eye Institute, Hyderabad, Telangana 500034, India
| | - Siddhart S. Rajendran
- Biological Psychology and Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
- Jasti V Ramanamma Children’s Eye Care Centre, LV Prasad Eye Institute, Hyderabad, Telangana 500034, India
| | - Kabilan Pitchaimuthu
- Biological Psychology and Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
- Jasti V Ramanamma Children’s Eye Care Centre, LV Prasad Eye Institute, Hyderabad, Telangana 500034, India
| | - Ramesh Kekunnaya
- Jasti V Ramanamma Children’s Eye Care Centre, LV Prasad Eye Institute, Hyderabad, Telangana 500034, India
| | - Brigitte Röder
- Biological Psychology and Neuropsychology, University of Hamburg, 20146 Hamburg, Germany
| |
Collapse
|