Cross-modal associations between vision, touch, and audition influence visual search through top-down attention, not bottom-up capture

Distinguishing between straight and curved sounds: Auditory shape in pitch, loudness, and tempo gestures

Sound-based trajectories or sound gestures draw links to spatiokinetic processes. For instance, a gliding, decreasing pitch conveys an analogous downward motion or fall. Whereas the gesture's pitch orientation and range convey its meaning and magnitude, respectively, the way in which pitch changes over time can be conceived of as gesture shape, which to date has rarely been studied in isolation. This article reports on an experiment that studied the perception of shape in uni-directional pitch, loudness, and tempo gestures, each assessed for four physical scalings. Gestures could increase or decrease over time and comprised different frequency and sound level ranges, durations, and different scaling contexts. Using a crossmodal-matching task, participants could reliably distinguish between pitch and loudness gestures and relate them to analogous visual line segments. Scalings based on equivalent-rectangular bandwidth (ERB) rate for pitch and raw signal amplitude for loudness were matched closest to a straight line, whereas other scalings led to perceptions of exponential or logarithmic curvatures. The investigated tempo gestures, by contrast, did not yield reliable differences. The reliable, robust perception of gesture shape for pitch and loudness has implications on various sound-design applications, especially those cases that rely on crossmodal mappings, e.g., visual analysis or control interfaces like audio waveforms or spectrograms. Given its perceptual relevance, auditory shape appears to be an integral part of sound gestures, while illustrating how crossmodal correspondences can underpin auditory perception.

Exploring crossmodal correspondences for future research in human movement augmentation

"Crossmodal correspondences" are the consistent mappings between perceptual dimensions or stimuli from different sensory domains, which have been widely observed in the general population and investigated by experimental psychologists in recent years. At the same time, the emerging field of human movement augmentation (i.e., the enhancement of an individual's motor abilities by means of artificial devices) has been struggling with the question of how to relay supplementary information concerning the state of the artificial device and its interaction with the environment to the user, which may help the latter to control the device more effectively. To date, this challenge has not been explicitly addressed by capitalizing on our emerging knowledge concerning crossmodal correspondences, despite these being tightly related to multisensory integration. In this perspective paper, we introduce some of the latest research findings on the crossmodal correspondences and their potential role in human augmentation. We then consider three ways in which the former might impact the latter, and the feasibility of this process. First, crossmodal correspondences, given the documented effect on attentional processing, might facilitate the integration of device status information (e.g., concerning position) coming from different sensory modalities (e.g., haptic and visual), thus increasing their usefulness for motor control and embodiment. Second, by capitalizing on their widespread and seemingly spontaneous nature, crossmodal correspondences might be exploited to reduce the cognitive burden caused by additional sensory inputs and the time required for the human brain to adapt the representation of the body to the presence of the artificial device. Third, to accomplish the first two points, the benefits of crossmodal correspondences should be maintained even after sensory substitution, a strategy commonly used when implementing supplementary feedback.

Social touch alters newborn monkey behavior

In humans, infants respond positively to slow, gentle stroking-processed by C-tactile (CT) nerve fibers-by showing reductions in stress and increases in eye contact, smiling, and positive vocalizations. More frequent maternal touch is linked to greater activity and connectivity strength in social brain regions, and increases children's attention to and learning of faces. It has been theorized that touch may prime children for social interactions and set them on a path towards healthy social cognitive development. However, less is known about the effects of touch on young infants' psychological development, especially in the newborn period, a highly sensitive period of transition with rapid growth in sensory and social processing. It remains untested whether newborns can distinguish CT-targeted touch from other types of touch, or whether there are benefits of touch for newborns' social, emotional, or cognitive development. In the present study, we experimentally investigated the acute effects of touch in newborn monkeys, a common model for human social development. Rhesus macaques (Macaca mulatta), like humans, are highly social, have complex mother-infant interactions with frequent body contact for the first weeks of life, making them an excellent model of infant sociality. Infant monkeys in the present study were reared in a neonatal nursery, enabling control over their early environment, including all caregiver interactions. One-week-old macaque infants (N = 27) participated in three 5-minute counter-balanced caregiver interactions, all with mutual gaze: stroking head and shoulders (CT-targeted touch), stroking palms of hands and soles of feet (Non-CT touch), or no stroking (No-touch). Immediately following the interaction, infants watched social and nonsocial videos and picture arrays including faces and objects, while we tracked their visual attention with remote eye tracking. We found that, during the caregiver interactions, infants behaved differently while being touched compared to the no-touch condition, irrespective of the body part touched. Most notably, in both touch conditions, infants exhibited fewer stress-related behaviors-self-scratching, locomotion, and contact time with a comfort object-compared to when they were not touched. Following CT-targeted touch, infants were faster to orient to the picture arrays compared to the other interaction conditions, suggesting CT-targeted touch may activate or prime infants' attentional orienting system. In the No-touch condition infants attended longer to the nonsocial compared to the social video, possibly reflecting a baseline preference for nonsocial stimuli. In contrast, in both touch conditions, infants' looked equally to the social and nonsocial videos, suggesting that touch may influence the types of visual stimuli that hold infants' attention. Collectively, our results reveal that newborn macaques responded positively to touch, and touch appeared to influence some aspects of their subsequent attention, although we found limited evidence that these effects are mediated by CT fibers. These findings suggest that newborn touch may broadly support infants' psychological development, and may have early evolutionary roots, shared across primates. This study illustrates the unique insight offered by nonhuman primates for exploring early infant social touch, revealing that touch may positively affect emotional and attentional development as early as the newborn period.

Are crossmodal correspondences relative or absolute? Sequential effects on speeded classification

Crossmodal correspondences have often been demonstrated using congruency effects between pairs of stimuli in different sensory modalities that vary along separate dimensions. To date, however, it is still unclear the extent to which these correspondences are relative versus absolute in nature: that is, whether they result from pre-defined values that rigidly link the two dimensions or rather result from flexible values related to the previous occurrence of the crossmodal stimuli. Here, we investigated this issue in a speeded classification task featuring the correspondence between auditory pitch and visual size (e.g., congruent correspondence between high pitch/small disc and low pitch/large disc). Participants classified the size of the visual stimuli (large vs. small) while hearing concurrent high- or low-pitched task-irrelevant sounds. On some trials, visual stimuli were paired instead with "intermediate" pitch, that could be interpreted differently according to the auditory stimulus on the preceding trial (i.e., as "lower" following the presentation of a high pitch tone, but as "higher" following the presentation of a low pitch tone). Performance on sequence-congruent trials (e.g., when a small disc paired with the intermediate-pitched tone was preceded by a low pitch tone) was compared to sequence-incongruent trials (e.g., when a small disc paired with the intermediate-pitch tone was by a high-pitched tone). The results revealed faster classification responses on sequence-congruent than on sequence-incongruent trials. This demonstrates that the effect of the pitch/size correspondence is relative in nature, and subjected to trial-by-trial interpretation of the stimulus pair.

Crossmodal Correspondences: Four Challenges

The renewed interest that has emerged around the topic of crossmodal correspondences in recent years has demonstrated that crossmodal matchings and mappings exist between the majority of sensory dimensions, and across all combinations of sensory modalities. This renewed interest also offers a rapidly-growing list of ways in which correspondences affect--or interact with--metaphorical understanding, feelings of 'knowing', behavioral tasks, learning, mental imagery, and perceptual experiences. Here we highlight why, more generally, crossmodal correspondences matter to theories of multisensory interactions.

The COGs (context, object, and goals) in multisensory processing

Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and "top-down" control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer's goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications.