Audiotactile interactions in temporal perception

Touch Helps Hearing: Evidence From Continuous Audio-Tactile Stimulation

OBJECTIVES

Identifying target sounds in challenging environments is crucial for daily experiences. It is important to note that it can be enhanced by nonauditory stimuli, for example, through lip-reading in an ongoing conversation. However, how tactile stimuli affect auditory processing is still relatively unclear. Recent studies have shown that brief tactile stimuli can reliably facilitate auditory perception, while studies using longer-lasting audio-tactile stimulation yielded conflicting results. This study aimed to investigate the impact of ongoing pulsating tactile stimulation on basic auditory processing.

DESIGN

In experiment 1, the electroencephalogram (EEG) was recorded while 24 participants performed a loudness-discrimination task on a 4-Hz modulated tone-in-noise and received either in-phase, anti-phase, or no 4-Hz electrotactile stimulation above the median nerve. In experiment 2, another 24 participants were presented with the same tactile stimulation as before, but performed a tone-in-noise detection task while their selective auditory attention was manipulated.

RESULTS

We found that in-phase tactile stimulation enhanced EEG responses to the tone, whereas anti-phase tactile stimulation suppressed these responses. No corresponding tactile effects on loudness-discrimination performance were observed in experiment 1. Using a yes/no paradigm in experiment 2, we found that in-phase tactile stimulation, but not anti-phase tactile stimulation, improved detection thresholds. Selective attention also improved thresholds but did not modulate the observed benefit from in-phase tactile stimulation.

CONCLUSIONS

Our study highlights that ongoing in-phase tactile input can enhance basic auditory processing as reflected in scalp EEG and detection thresholds. This might have implications for the development of hearing enhancement technologies and interventions.

Developmental changes in audiotactile event perception

The fission and fusion illusions provide measures of multisensory integration. The sound-induced tap fission illusion occurs when a tap is paired with two distractor sounds, resulting in the perception of two taps; the sound-induced tap fusion illusion occurs when two taps are paired with a single sound, resulting in the perception of a single tap. Using these illusions, we measured integration in three groups of children (9-, 11-, and 13-year-olds) and compared them with a group of adults. Based on accuracy, we derived a measure of magnitude of illusion and used a signal detection analysis to estimate perceptual discriminability and decisional criterion. All age groups showed a significant fission illusion, whereas only the three groups of children showed a significant fusion illusion. When compared with adults, the 9-year-olds showed larger fission and fusion illusions (i.e., reduced discriminability and greater bias), whereas the 11-year-olds were adult-like for fission but showed some differences for fusion: significantly worse discriminability and marginally greater magnitude and criterion. The 13-year-olds were adult-like on all measures. Based on the pattern of data, we speculate that the developmental trajectories for fission and fusion differ. We discuss these developmental results in the context of three non-mutually exclusive theoretical frameworks: sensory dominance, maximum likelihood estimation, and causal inference.

Social Context and Tool Use Can Modulate Interpersonal Comfort Space

Recent research has investigated whether the representation of space around the body, in terms of reach-action (imagining of reaching another person) and comfort-social (tolerance of the other's proximity) spaces, may reflect a shared sensorimotor basis. Some studies exploiting motor plasticity induced by tool use have not observed sensorimotor identity (i.e., the same mechanisms that underlie, based on sensory information, the representation of proximal space in terms of action possibilities, goal-directed motor actions, and anticipation of the sensorimotor consequences), whereas evidence to the contrary has also emerged. Since the data are not fully convergent, here we wondered whether or not the combination of motor plasticity induced by tool use and the processing of the role of social context might reflect a similar modulation in both spaces. To this end, we conducted a randomized control trial with three groups of participants (N = 62) in which reaching and comfort distances were measured in Pre- and Post-tool-use sessions. The tool-use sessions were conducted under different conditions: (i) in the presence of a social stimulus (determining the social context) (Tool plus Mannequin group); (ii) without any stimulus (Only Tool group); (iii) in the presence of a box (Tool plus Object group) as a control condition. Results showed an extension of comfort distance in the Post-tool session of the Tool plus Mannequin group compared with the other conditions. Conversely, the reaching distance was larger after tool use than at the Pre-tool-use session, independently of the experimental conditions. Our findings suggest that motor plasticity impacts reaching and comfort spaces to different degrees; while reaching space is markedly sensitive to motor plasticity, comfort space needs qualification of social context information.

Integration of somatosensory and motor-related information in the auditory system

An ability to integrate information provided by different sensory modalities is a fundamental feature of neurons in many brain areas. Because visual and auditory inputs often originate from the same external object, which may be located some distance away from the observer, the synthesis of these cues can improve localization accuracy and speed up behavioral responses. By contrast, multisensory interactions occurring close to the body typically involve a combination of tactile stimuli with other sensory modalities. Moreover, most activities involving active touch generate sound, indicating that stimuli in these modalities are frequently experienced together. In this review, we examine the basis for determining sound-source distance and the contribution of auditory inputs to the neural encoding of space around the body. We then consider the perceptual consequences of combining auditory and tactile inputs in humans and discuss recent evidence from animal studies demonstrating how cortical and subcortical areas work together to mediate communication between these senses. This research has shown that somatosensory inputs interface with and modulate sound processing at multiple levels of the auditory pathway, from the cochlear nucleus in the brainstem to the cortex. Circuits involving inputs from the primary somatosensory cortex to the auditory midbrain have been identified that mediate suppressive effects of whisker stimulation on auditory thalamocortical processing, providing a possible basis for prioritizing the processing of tactile cues from nearby objects. Close links also exist between audition and movement, and auditory responses are typically suppressed by locomotion and other actions. These movement-related signals are thought to cancel out self-generated sounds, but they may also affect auditory responses via the associated somatosensory stimulation or as a result of changes in brain state. Together, these studies highlight the importance of considering both multisensory context and movement-related activity in order to understand how the auditory cortex operates during natural behaviors, paving the way for future work to investigate auditory-somatosensory interactions in more ecological situations.

Auditory clicks elicit equivalent temporal frequency perception to tactile pulses: A cross-modal psychophysical study

Both hearing and touch are sensitive to the frequency of mechanical oscillations—sound waves and tactile vibrations, respectively. The mounting evidence of parallels in temporal frequency processing between the two sensory systems led us to directly address the question of perceptual frequency equivalence between touch and hearing using stimuli of simple and more complex temporal features. In a cross-modal psychophysical paradigm, subjects compared the perceived frequency of pulsatile mechanical vibrations to that elicited by pulsatile acoustic (click) trains, and vice versa. Non-invasive pulsatile stimulation designed to excite a fixed population of afferents was used to induce desired temporal spike trains at frequencies spanning flutter up to vibratory hum (>50 Hz). The cross-modal perceived frequency for regular test pulse trains of either modality was a close match to the presented stimulus physical frequency up to 100 Hz. We then tested whether the recently discovered “burst gap” temporal code for frequency, that is shared by the two senses, renders an equivalent cross-modal frequency perception. When subjects compared trains comprising pairs of pulses (bursts) in one modality against regular trains in the other, the cross-sensory equivalent perceptual frequency best corresponded to the silent interval between the successive bursts in both auditory and tactile test stimuli. These findings suggest that identical acoustic and vibrotactile pulse trains, regardless of pattern, elicit equivalent frequencies, and imply analogous temporal frequency computation strategies in both modalities. This perceptual correspondence raises the possibility of employing a cross-modal comparison as a robust standard to overcome the prevailing methodological limitations in psychophysical investigations and strongly encourages cross-modal approaches for transmitting sensory information such as translating pitch into a similar pattern of vibration on the skin.

FeelMusic: Enriching Our Emotive Experience of Music through Audio-Tactile Mappings

We present and evaluate the concept of FeelMusic and evaluate an implementation of it. It is an augmentation of music through the haptic translation of core musical elements. Music and touch are intrinsic modes of affective communication that are physically sensed. By projecting musical features such as rhythm and melody into the haptic domain, we can explore and enrich this embodied sensation; hence, we investigated audio-tactile mappings that successfully render emotive qualities. We began by investigating the affective qualities of vibrotactile stimuli through a psychophysical study with 20 participants using the circumplex model of affect. We found positive correlations between vibration frequency and arousal across participants, but correlations with valence were specific to the individual. We then developed novel FeelMusic mappings by translating key features of music samples and implementing them with “Pump-and-Vibe”, a wearable interface utilising fluidic actuation and vibration to generate dynamic haptic sensations. We conducted a preliminary investigation to evaluate the FeelMusic mappings by gathering 20 participants’ responses to the musical, tactile and combined stimuli, using valence ratings and descriptive words from Hevner’s adjective circle to measure affect. These mappings, and new tactile compositions, validated that FeelMusic interfaces have the potential to enrich musical experiences and be a means of affective communication in their own right. FeelMusic is a tangible realisation of the expression “feel the music”, enriching our musical experiences.

Audiotactile interactions in the mouse cochlear nucleus

Multisensory integration of auditory and tactile information occurs already at the level of the cochlear nucleus. Rodents use their whiskers for tactile perception to guide them in their exploration of the world. As nocturnal animals with relatively poor vision, audiotactile interactions are of great importance for this species. Here, the influence of whisker deflections on sound-evoked spiking in the cochlear nucleus was investigated in vivo in anesthetized mice. Multichannel, silicon-probe electrophysiological recordings were obtained from both the dorsal and ventral cochlear nucleus. Whisker deflections evoked an increased spiking activity in fusiform cells of the dorsal cochlear nucleus and t-stellate cells in ventral cochlear nucleus, whereas bushy cells in the ventral cochlear nucleus showed a more variable response. The response to broadband noise stimulation increased in fusiform cells and primary-like bushy cells when the sound stimulation was preceded (~ 20 ms) by whisker stimulation. Multi-sensory integration of auditory and whisker input can thus occur already in this early brainstem nucleus, emphasizing the importance of early integration of auditory and somatosensory information.

Dark, loud, and compact sounds induce frisson

Frisson is characterised by tingling and tickling sensations with positive or negative feelings. However, it is still unknown what factors affect the intensity of frisson. We conducted experiments on the stimulus characteristics and individual’s mood states and personality traits. Participants filled out self-reported questionnaires, including the Profile of Mood States, Beck Depression Inventory, and Big Five Inventory. They continuously indicated the subjective intensity of frisson throughout a 17-min experiment while listening to binaural brushing and tapping sounds through headphones. In the interviews after the experiments, participants reported that tingling and tickling sensations mainly originated on their ears, neck, shoulders, and back. Cross-correlation results showed that the intensity of frisson was closely linked to the acoustic features of auditory stimuli, including their amplitude, spectral centroid, and spectral bandwidth. This suggests that proximal sounds with dark and compact timbre trigger frisson. The peak of correlation between frisson and the acoustic feature was observed 2 s after the acoustic feature changed, suggesting that bottom-up auditory inputs modulate skin-related modalities. We also found that participants with anxiety were sensitive to frisson. Our results provide important clues to understanding the mechanisms of auditory–somatosensory interactions.

The Influence of Auditory Cues on Bodily and Movement Perception

The sounds that result from our movement and that mark the outcome of our actions typically convey useful information concerning the state of our body and its movement, as well as providing pertinent information about the stimuli with which we are interacting. Here we review the rapidly growing literature investigating the influence of non-veridical auditory cues (i.e., inaccurate in terms of their context, timing, and/or spectral distribution) on multisensory body and action perception, and on motor behavior. Inaccurate auditory cues provide a unique opportunity to study cross-modal processes: the ability to detect the impact of each sense when they provide a slightly different message is greater. Additionally, given that similar cross-modal processes likely occur regardless of the accuracy or inaccuracy of sensory input, studying incongruent interactions are likely to also help us predict interactions between congruent inputs. The available research convincingly demonstrates that perceptions of the body, of movement, and of surface contact features (e.g., roughness) are influenced by the addition of non-veridical auditory cues. Moreover, auditory cues impact both motor behavior and emotional valence, the latter showing that sounds that are highly incongruent with the performed movement induce feelings of unpleasantness (perhaps associated with lower processing fluency). Such findings are relevant to the design of auditory cues associated with product interaction, and the use of auditory cues in sport performance and therapeutic situations given the impact on motor behavior.

Auditory Frequency Representations in Human Somatosensory Cortex

Recent studies have challenged the traditional notion of modality-dedicated cortical systems by showing that audition and touch evoke responses in the same sensory brain regions. While much of this work has focused on somatosensory responses in auditory regions, fewer studies have investigated sound responses and representations in somatosensory regions. In this functional magnetic resonance imaging (fMRI) study, we measured BOLD signal changes in participants performing an auditory frequency discrimination task and characterized activation patterns related to stimulus frequency using both univariate and multivariate analysis approaches. Outside of bilateral temporal lobe regions, we observed robust and frequency-specific responses to auditory stimulation in classically defined somatosensory areas. Moreover, using representational similarity analysis to define the relationships between multi-voxel activation patterns for all sound pairs, we found clear similarity patterns for auditory responses in the parietal lobe that correlated significantly with perceptual similarity judgments. Our results demonstrate that auditory frequency representations can be distributed over brain regions traditionally considered to be dedicated to somatosensation. The broad distribution of auditory and tactile responses over parietal and temporal regions reveals a number of candidate brain areas that could support general temporal frequency processing and mediate the extensive and robust perceptual interactions between audition and touch.

Crossmodal Correspondences in the Sounds of Chinese Instruments

People tend to associate stimuli from different sensory modalities, a phenomenon known as crossmodal correspondences. We conducted two experiments to investigate how Chinese participants associated musical notes produced by four types of Chinese instruments (bowed strings, plucked strings, winds, and percussion instruments) with different colors, taste terms, and fabric textures. Specifically, the participants were asked to select a sound to match each color patch or taste term in Experiment 1 and to match the experience of touching each fabric in Experiment 2. The results demonstrated some associations between pitch and color, taste term, or the smoothness of fabrics. Moreover, certain types of Chinese instruments were preferentially chosen to match some of the colors, taste terms, and the texture of certain fabrics. These findings therefore provided insights about the perception of Chinese music and shed light on how to apply the multisensory features of sounds to enhance the composition, performance, and appreciation of music.

Movement Improves the Quality of Temporal Perception and Decision-Making

A critical aspect of behavior is that mobile organisms must be able to precisely determine where and when to move. A better understanding of the mechanisms underlying precise movement timing and action planning is therefore crucial to understanding how we interact with the world around us. Recent evidence suggests that our experience of time is directly and intrinsically computed within the motor system, consistent with the theory of embodied cognition. To investigate the role of the motor system, we tested human subjects (n = 40) on a novel task combining reaching and time estimation. In this task, subjects were required to move a robotic manipulandum to one of two physical locations to categorize a concurrently timed suprasecond. Critically, subjects were divided into two groups: one in which movement during the interval was unrestricted and one in which they were restricted from moving until the stimulus interval had elapsed. Our results revealed a higher degree of precision for subjects in the free-moving group. A further experiment (n = 14) verified that these findings were not due to proximity to the target, counting strategies, bias, or movement length. A final experiment (n = 10) replicated these findings using a within-subjects design, performing a time reproduction task, in which movement during encoding of the interval led to more precise performance. Our findings suggest that time estimation may be instantiated within the motor system as an ongoing readout of timing judgment and confidence.

Multisensory perceptual interactions between higher-order temporal frequency signals

Naturally occurring signals in audition and touch can be complex and marked by temporal variations in frequency and amplitude. Auditory frequency sweep processing has been studied extensively; however, much less is known about sweep processing in touch because studies have primarily focused on the perception of simple sinusoidal vibrations. Given the extensive interactions between audition and touch in the frequency processing of pure tone signals, we reasoned that these senses might also interact in the processing of higher-order frequency representations like sweeps. In a series of psychophysical experiments, we characterized the influence of auditory distractors on the ability of participants to discriminate tactile frequency sweeps. Auditory frequency sweeps systematically biased the tactile perception of sweep direction. Importantly, auditory cues exerted little influence on tactile sweep direction perception when the sounds and vibrations occupied different absolute frequency ranges or when the sounds consisted of intensity sweeps. Thus, audition and touch interact in frequency sweep perception in a frequency- and feature-specific manner. Our results demonstrate that audio-tactile interactions are not constrained to the processing of simple sinusoids. Because higher-order frequency representations may be synthesized from simpler representations, our findings imply that multisensory interactions in the temporal frequency domain span multiple hierarchical levels in sensory processing. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Synchronising to a frequency while estimating time of vibro-tactile stimuli

It is well known that subjective time perception can be modified by the emotional experience related to a specific event, by pharmaceutical compounds or by sensory stimuli. As for the latter, visual and auditory stimuli have been widely studied compared to tactile ones. Two experiments were conducted using different vibratory frequencies to stimulate participants who were asked to reproduce stimulus duration. Experiment 1 compared differences in reproduced times for 8-s stimuli ranging between 0.5 and 6 Hz in 100 participants who performed a time reproduction task with the stimulus present or absent during the reproduction. The task was done under prospective and retrospective paradigms. Experiment 2 assessed differences in reproduced times by 80 participants under vibrotactile stimulation of two frequencies simultaneously delivered to each hand, frequencies with specific proportions of 0.5 and 0.75 times the standard frequency for two groups of standard frequency (2 or 12 Hz). Reproduced times in Experiment 1 did not show significant differences among frequencies. Significant differences were found for the absence/presence condition, solely, in prospective tasks, where estimations were longer in the absence of the vibrotactile stimulus. Significant differences were found in Experiment 2 for reproduced time by participants between groups of standard frequency. Data analysis suggests the need to improve the understanding of the subjective time perception processes for higher frequencies considering the intensity modulation based on the amplitude and frequency relation. Results open the possibility of designing new protocols in the study of time perception and other cognitive functions.

Reciprocal Interactions Between Audition and Touch in Flutter Frequency Perception

In both audition and touch, sensory cues comprising repeating events are perceived either as a continuous signal or as a stream of temporally discrete events (flutter), depending on the events' repetition rate. At high repetition rates (>100 Hz), auditory and tactile cues interact reciprocally in pitch processing. The frequency of a cue experienced in one modality systematically biases the perceived frequency of a cue experienced in the other modality. Here, we tested whether audition and touch also interact in the processing of low-frequency stimulation. We also tested whether multisensory interactions occurred if the stimulation in one modality comprised click trains and the stimulation in the other modality comprised amplitude-modulated signals. We found that auditory cues bias touch and tactile cues bias audition on a flutter discrimination task. Even though participants were instructed to attend to a single sensory modality and ignore the other cue, the flutter rate in the attended modality is perceived to be similar to that of the distractor modality. Moreover, we observed similar interaction patterns regardless of stimulus type and whether the same stimulus types were experienced by both senses. Combined with earlier studies, our results suggest that the nervous system extracts and combines temporal rate information from multisensory environmental signals, regardless of stimulus type, in both the low- and high temporal frequency domains. This function likely reflects the importance of temporal frequency as a fundamental feature of our multisensory experience.

Memory for Random Time Patterns in Audition, Touch, and Vision

Perception deals with temporal sequences of events, like series of phonemes for audition, dynamic changes in pressure for touch textures, or moving objects for vision. Memory processes are thus needed to make sense of the temporal patterning of sensory information. Recently, we have shown that auditory temporal patterns could be learned rapidly and incidentally with repeated exposure [Kang et al., 2017]. Here, we tested whether rapid incidental learning of temporal patterns was specific to audition, or if it was a more general property of sensory systems. We used a same behavioral task in three modalities: audition, touch, and vision, for stimuli having identical temporal statistics. Participants were presented with sequences of acoustic pulses for audition, motion pulses to the fingertips for touch, or light pulses for vision. Pulses were randomly and irregularly spaced, with all inter-pulse intervals in the sub-second range and all constrained to be longer than the temporal acuity in any modality. This led to pulse sequences with an average inter-pulse interval of 166 ms, a minimum inter-pulse interval of 60 ms, and a total duration of 1.2 s. Results showed that, if a random temporal pattern re-occurred at random times during an experimental block, it was rapidly learned, whatever the sensory modality. Moreover, patterns first learned in the auditory modality displayed transfer of learning to either touch or vision. This suggests that sensory systems may be exquisitely tuned to incidentally learn re-occurring temporal patterns, with possible cross-talk between the senses.

Selective Attention Gates the Interactive Crossmodal Coupling between Perceptual Systems

Sensory cortical systems often activate in parallel, even when stimulation is experienced through a single sensory modality [1-3]. Co-activations may reflect the interactive coupling between information-linked cortical systems or merely parallel but independent sensory processing. We report causal evidence consistent with the hypothesis that human somatosensory cortex (S1), which co-activates with auditory cortex during the processing of vibrations and textures [4-9], interactively couples to cortical systems that support auditory perception. In a series of behavioral experiments, we used transcranial magnetic stimulation (TMS) to probe interactions between the somatosensory and auditory perceptual systems as we manipulated attention state. Acute TMS over S1 impairs auditory frequency perception when subjects simultaneously attend to auditory and tactile frequency, but not when attention is directed to audition alone. Auditory frequency perception is unaffected by TMS over visual cortex, thus confirming the privileged interactions between the somatosensory and auditory systems in temporal frequency processing [10-13]. Our results provide a key demonstration that selective attention can modulate the functional properties of cortical systems thought to support specific sensory modalities. The gating of crossmodal coupling by selective attention may critically support multisensory interactions and feature-specific perception.

Robust Subthreshold Cross-modal Modulation of Auditory Response by Cutaneous Electrical Stimulation in First- and Higher-order Auditory Thalamic Nuclei

Conventional extracellular recording has revealed cross-modal alterations of auditory cell activities by cutaneous electrical stimulation of the hindpaw in first- and higher-order auditory thalamic nuclei (Donishi et al., 2011). Juxta-cellular recording and labeling techniques were used in the present study to examine the cross-modal alterations in detail, focusing on possible nucleus and/or cell type-related distinctions in modulation. Recordings were obtained from 80 cells of anesthetized rats. Cutaneous electrical stimulation, which did not elicit unit discharges, i.e., subthreshold effects, modulated early (onset) and/or late auditory responses of first- (64%) and higher-order nucleus cells (77%) with regard to response magnitude, latency and/or burst spiking. Attenuation predominated in the modulation of response magnitude and burst spiking, and delay predominated in the modulation of response time. Striking alterations of burst spiking took place in higher-order nucleus cells, which had the potential to exhibit higher propensities for burst spiking as compared to first-order nucleus cells. A subpopulation of first-order nucleus cells showing modulation in early response magnitude in the caudal domain of the nucleus had larger cell bodies and higher propensities for burst spiking as compared to cells showing no modulation. These findings suggest that somatosensory influence is incorporated into parallel channels in auditory thalamic nuclei to impose distinct impacts on cortical and subcortical sensory processing. Further, cutaneous electrical stimulation given after early auditory responses modulated late responses. Somatosensory influence is likely to affect ongoing auditory processing at any time without being coincident with sound onset in a narrow temporal window.

Reciprocal Interference Between Audition and Touch in the Perception of Duration

Audition and touch interact with one another and share a number of similarities; however, little is known about their interplay in the perception of temporal duration. The present study intended to investigate whether the temporal duration of an irrelevant auditory or tactile stimulus could modulate the perceived duration of a target stimulus presented in the other modality (i.e., tactile or auditory) adopting both a between-participants (Experiment 1) and a within-participants (Experiment 2) experimental design. In a two-alternative forced-choice task, participants decided which of two events in a target modality was longer. The simultaneously distractor stimuli were presented with a duration that was either congruent or incongruent to the target’s. Results showed that both the auditory and tactile modalities affected duration judgments in the incongruent condition, decreasing performance in both experiments. Moreover, in Experiment 1, the tactile modality enhanced the perception of auditory stimuli in the congruent condition, but audition did not facilitate performance for the congruent condition in the tactile modality; this tactile enhancement of audition was not found in Experiment 2. To the best of our knowledge, this is the first study documenting audiotactile interactions in the perception of duration, and suggests that audition and touch might modulate one another in a more balanced manner, in contrast to audiovisual pairings. The findings support previous evidence as to the shared links and reciprocal influences when audition and touch interact with one another.

Synchronizing to auditory and tactile metronomes: a test of the auditory-motor enhancement hypothesis

Humans show a striking advantage for synchronizing movements with discretely timed auditory metronomes (e.g., clicking sounds) over temporally matched visual metronomes (e.g., flashing lights), suggesting enhanced auditory-motor coupling for rhythmic processing. Does the auditory advantage persist for other modalities (not just vision)? Here, nonmusicians finger tapped to the beat of auditory, tactile, and bimodal metronomes. Stimulus magnitude and rhythmic complexity were also manipulated. In conditions involving a large area of stimulation and simple rhythmic sequences, tactile synchronization closely matched auditory. Although this finding shows a limitation to the hypothesis of enhanced auditory-motor coupling for rhythmic processing, other findings clearly support it. First, there was a robust advantage with auditory information for synchronization with complex rhythm sequences; moreover, in complex sequences a measure of error correction was found only when auditory information was present. Second, higher order grouping was evident only when auditory information was present.

Music and speech distractors disrupt sensorimotor synchronization: effects of musical training

Humans display a natural tendency to move to the beat of music, more than to the rhythm of any other auditory stimulus. We typically move with music, but rarely with speech. This proclivity is apparent early during development and can be further developed over the years via joint dancing, singing, or instrument playing. Synchronization of movement to the beat can thus improve with age, but also with musical experience. In a previous study, we found that music perturbed synchronization with a metronome more than speech fragments; music superiority disappeared when distractors shared isochrony and the same meter (Dalla Bella et al., PLoS One 8(8):e71945, 2013). Here, we examined if the interfering effect of music and speech distractors in a synchronization task is influenced by musical training. Musicians and non-musicians synchronized by producing finger force pulses to the sounds of a metronome while music and speech distractors were presented at one of various phase relationships with respect to the target. Distractors were familiar musical excerpts and fragments of children poetry comparable in terms of beat/stress isochrony. Music perturbed synchronization with the metronome more than speech did in both groups. However, the difference in synchronization error between music and speech distractors was smaller for musicians than for non-musicians, especially when the peak force of movement is reached. These findings point to a link between musical training and timing of sensorimotor synchronization when reacting to music and speech distractors.

Maternal alterations induced by exposure to an unfamiliar home cage in early underfed dams

The expression of different behavioral components in the adult rat depends on a number of early influences, including age, hormones, manipulations of sensory cues, and perinatal undernutrition, all of which impact the development of brain areas underlying adaptive processes, maternal behavior, and the response to novelty. The current study investigates the effects of pre- and neonatal undernutrition on various components of maternal behavior of dams exposed to the challenge of an unfamiliar home cage on days 4, 8, and 12 of lactation. Food restriction was initiated from gestational day (G) G6 to G19 when dams received 50-70% of the normal balanced diet, followed by 100% from G20 to G21. After birth, pups were underfed by alternating every 12h between two lactating dams, one of which, had ligated nipples. Weaning was at 25days of age followed by an ad libitum diet until postpartum day 90, when females were mated, and subsequently tested for maternal behavior in an unfamiliar cage. The results indicated that in early underfed mothers the frequency of handling wood shavings and of, approaching, licking, crouching, and grasping pups for retrieval was significantly reduced. Moreover, self-grooming increased substantially in the underfed dams, but the frequency of rearing was reduced. Additionally, the body weight of pups nursed by early underfed dams was significantly lower than that of control pups. These findings suggest a relation between early food restriction and the deficient maternal care observed when these dams were challenged by exposure to an unfamiliar home cage.

Walking to a multisensory beat

Living in a complex and multisensory environment demands constant interaction between perception and action. In everyday life it is common to combine efficiently simultaneous signals coming from different modalities. There is evidence of a multisensory benefit in a variety of laboratory tasks (temporal judgement, reaction time tasks). It is less clear if this effect extends to ecological tasks, such as walking. Furthermore, benefits of multimodal stimulation are linked to temporal properties such as the temporal window of integration and temporal recalibration. These properties have been examined in tasks involving single, non-repeating stimulus presentations. Here we investigate the same temporal properties in the context of a rhythmic task, namely audio-tactile stimulation during walking. The effect of audio-tactile rhythmic cues on gait variability and the ability to synchronize to the cues was studied in young adults. Participants walked with rhythmic cues presented at different stimulus-onset asynchronies. We observed a multisensory benefit by comparing audio-tactile to unimodal stimulation. Moreover, both the temporal window of integration and temporal recalibration mediated the response to multimodal stimulation. In sum, rhythmic behaviours obey the same principles as temporal discrimination and detection behaviours and thus can also benefit from multimodal stimulation.

Haptic modality takes its time: Dynamic of activations of sensory modalities in perceptual and memory processes

Based on claims resulting from grounded cognition theory that perceptual and memory processes are using the same distributed systems, the present study investigated the temporal aspect of access to memory traces through haptic and auditory modalities. Unlike in the case of visual or auditory components, the perception of a vibrotactile component is more sequential in nature and therefore cannot be fully processed before the end of the signal. The present study explores the dynamic of components activation in a situation of audio-vibrotactile asynchrony. We used a short-term priming paradigm consisting of an association phase (between a vibration and sound) and a test phase testing priming effect of a vibrotactile stimulation on the processing of a target sound. Results showed an interference with a simultaneous processing and a facilitation with a sequential processing. The temporality process of perceptual components is also important at a memory level.

Precision and Bias in Approximate Numerical Judgment in Auditory, Tactile, and Cross-modal Presentation

Many studies have claimed that the numerosity of any set of discrete elements can be depicted by a genuinely abstract number representation, irrespective of whether they are presented in a visual, auditory, or tactile modality. However, in behavioral studies, some inconsistencies have been observed in the performance of number comparisons among different modalities. In this study, we have tested whether numerical comparisons of auditory, tactile, and cross-modal presentations would differ under adequate control of stimulus presentation, and, if so, how they would differ. The unimodal and cross-modal stimuli pairs were presented in sequential manner. We measured the Weber fractions (i.e., precision) and points of subjective equality (i.e., accuracy) of numerical discriminations in auditory, tactile, and crossmodal conditions. The results showed that the Weber fractions are constant over standard stimuli, indicating that the Weber's law holds for the range of numerical values that was tested. Furthermore, the Weber fractions are consistent over unimodal and cross-modal comparisons, and this indicates that there is no additional noise involved in the cross-modal comparisons. Interestingly, the bias measure showed that the number of auditory stimuli is systematically overestimated compared with that of tactile stimuli.

How are overlapping time intervals perceived? Evidence for a weighted sum of segments model

This study investigated the way in which people time two overlapping intervals. Timing models already proposed in the literature predict different effects of the degree of overlap on each estimate, and empirical findings were compared to these predictions. Two unimodal experiments (in which each to-be-timed interval was a visual stimulus) and one bimodal experiment (in which one to-be-timed interval was auditory and the other visual) were conducted. The estimate of the first interval was either unaffected or decreased, and the estimate of the second interval consistently increased as the intervals were more temporally separated. The only model in the literature that could account for such result patterns is a single pacemaker single accumulator structure with an additional recency weighting (see the weighted sum of segments model). That is, participants appear to segment the two overlapping intervals into three non-overlapping and overlapping segments, time these segments separately, and then combine them to estimate each interval. Importantly, a recency weighting, determined by the time that has passed since the end of that segment, is also applied to each segment in the summation process. Further, in the bimodal experiment the order in which the stimuli of different modalities were presented affected the way in which they were timed, a finding that none of the current models can explain. This highlights that a comprehensive model of interval timing must consider not only the modalities of to-be-timed intervals but also the order in which different modalities must be timed.

Temporal factors affecting somatosensory-auditory interactions in speech processing

Speech perception is known to rely on both auditory and visual information. However, sound-specific somatosensory input has been shown also to influence speech perceptual processing (Ito et al., 2009). In the present study, we addressed further the relationship between somatosensory information and speech perceptual processing by addressing the hypothesis that the temporal relationship between orofacial movement and sound processing contributes to somatosensory–auditory interaction in speech perception. We examined the changes in event-related potentials (ERPs) in response to multisensory synchronous (simultaneous) and asynchronous (90 ms lag and lead) somatosensory and auditory stimulation compared to individual unisensory auditory and somatosensory stimulation alone. We used a robotic device to apply facial skin somatosensory deformations that were similar in timing and duration to those experienced in speech production. Following synchronous multisensory stimulation the amplitude of the ERP was reliably different from the two unisensory potentials. More importantly, the magnitude of the ERP difference varied as a function of the relative timing of the somatosensory–auditory stimulation. Event-related activity change due to stimulus timing was seen between 160 and 220 ms following somatosensory onset, mostly around the parietal area. The results demonstrate a dynamic modulation of somatosensory–auditory convergence and suggest the contribution of somatosensory information for speech processing process is dependent on the specific temporal order of sensory inputs in speech production.

Temporal integration windows for naturalistic visual sequences

There is increasing evidence that the brain possesses mechanisms to integrate incoming sensory information as it unfolds over time-periods of 2-3 seconds. The ubiquity of this mechanism across modalities, tasks, perception and production has led to the proposal that it may underlie our experience of the subjective present. A critical test of this claim is that this phenomenon should be apparent in naturalistic visual experiences. We tested this using movie-clips as a surrogate for our day-to-day experience, temporally scrambling them to require (re-) integration within and beyond the hypothesized 2-3 second interval. Two independent experiments demonstrate a step-wise increase in the difficulty to follow stimuli at the hypothesized 2-3 second scrambling condition. Moreover, only this difference could not be accounted for by low-level visual properties. This provides the first evidence that this 2-3 second integration window extends to complex, naturalistic visual sequences more consistent with our experience of the subjective present.

Audio-tactile integration in congenitally and late deaf cochlear implant users

Several studies conducted in mammals and humans have shown that multisensory processing may be impaired following congenital sensory loss and in particular if no experience is achieved within specific early developmental time windows known as sensitive periods. In this study we investigated whether basic multisensory abilities are impaired in hearing-restored individuals with deafness acquired at different stages of development. To this aim, we tested congenitally and late deaf cochlear implant (CI) recipients, age-matched with two groups of hearing controls, on an audio-tactile redundancy paradigm, in which reaction times to unimodal and crossmodal redundant signals were measured. Our results showed that both congenitally and late deaf CI recipients were able to integrate audio-tactile stimuli, suggesting that congenital and acquired deafness does not prevent the development and recovery of basic multisensory processing. However, we found that congenitally deaf CI recipients had a lower multisensory gain compared to their matched controls, which may be explained by their faster responses to tactile stimuli. We discuss this finding in the context of reorganisation of the sensory systems following sensory loss and the possibility that these changes cannot be “rewired” through auditory reafferentation.

Concurrent emotional pictures modulate temporal order judgments of spatially separated audio-tactile stimuli

Although attention can be captured toward high-arousal stimuli, little is known about how perceiving emotion in one modality influences the temporal processing of non-emotional stimuli in other modalities. We addressed this issue by presenting observers spatially uninformative emotional pictures while they performed an audio-tactile temporal-order judgment (TOJ) task. In Experiment 1, audio-tactile stimuli were presented at the same location straight ahead of the participants, who had to judge "which modality came first?". In Experiments 2 and 3, the audio-tactile stimuli were delivered one to the left and the other to the right side, and participants had to judge "which side came first?". We found both negative and positive high-arousal pictures to significantly bias TOJs towards the tactile and away from the auditory event when the audio-tactile stimuli were spatially separated; by contrast, there was no such bias when the audio-tactile stimuli originated from the same location. To further examine whether this bias is attributable to the emotional meanings conveyed by the pictures or to their high arousal effect, we compared and contrasted the influences of near-body threat vs. remote threat (emotional) pictures on audio-tactile TOJs in Experiment 3. The bias manifested only in the near-body threat condition. Taken together, the findings indicate that visual stimuli conveying meanings of near-body interaction activate a sensorimotor functional link prioritizing the processing of tactile over auditory signals when these signals are spatially separated. In contrast, audio-tactile signals from the same location engender strong crossmodal integration, thus counteracting modality-based attentional shifts induced by the emotional pictures.

Dissociation of psychophysical and EEG steady-state response measures of cross-modal temporal correspondence for amplitude modulated acoustic and vibrotactile stimulation

Research examining multisensory integration suggests that the correspondence of stimulus characteristics across modalities (cross-modal correspondence) can have a dramatic influence on both neurophysiological and perceptual responses to multimodal stimulation. The current study extends prior research by examining the cross-modal correspondence of amplitude modulation rate for simultaneous acoustic and vibrotactile stimulation using EEG and perceptual measures of sensitivity to amplitude modulation. To achieve this, psychophysical thresholds and steady-state responses (SSRs) were measured for acoustic and vibrotactile amplitude modulated (AM) stimulation for 21 and 40 Hz AM rates as a function of the cross-modal correspondence. The study design included three primary conditions to determine whether the changes in the SSR and psychophysical thresholds were due to the cross-modal temporal correspondence of amplitude modulated stimuli: NONE (AM in one modality only), SAME (the same AM rate for each modality) and DIFF (different AM rates for each modality). The results of the psychophysical analysis showed that AM detection thresholds for the simultaneous AM conditions (i.e., SAME and DIFF) were significantly higher (i.e., lower sensitivity) than AM detection thresholds for the stimulation of a single modality (i.e., NONE). SSR results showed significant effects of SAME and DIFF conditions on SSR activity. The different pattern of results for perceptual and SSR measures of cross-modal correspondence of AM rate indicates a dissociation between entrained cortical activity (i.e., SSR) and perception.

Auditory time-interval perception as causal inference on sound sources

Perception of a temporal pattern in a sub-second time scale is fundamental to conversation, music perception, and other kinds of sound communication. However, its mechanism is not fully understood. A simple example is hearing three successive sounds with short time intervals. The following misperception of the latter interval is known: underestimation of the latter interval when the former is a little shorter or much longer than the latter, and overestimation of the latter when the former is a little longer or much shorter than the latter. Although this misperception of auditory time intervals for simple stimuli might be a cue to understanding the mechanism of time-interval perception, there exists no model that comprehensively explains it. Considering a previous experiment demonstrating that illusory perception does not occur for stimulus sounds with different frequencies, it might be plausible to think that the underlying mechanism of time-interval perception involves a causal inference on sound sources: herein, different frequencies provide cues for different causes. We construct a Bayesian observer model of this time-interval perception. We introduce a probabilistic variable representing the causality of sounds in the model. As prior knowledge, the observer assumes that a single sound source produces periodic and short time intervals, which is consistent with several previous works. We conducted numerical simulations and confirmed that our model can reproduce the misperception of auditory time intervals. A similar phenomenon has also been reported in visual and tactile modalities, though the time ranges for these are wider. This suggests the existence of a common mechanism for temporal pattern perception over modalities. This is because these different properties can be interpreted as a difference in time resolutions, given that the time resolutions for vision and touch are lower than those for audition.

Effects of delayed auditory and visual feedback on sequence production

This study represents the first systematic comparison of the relative contributions of auditory and visual feedback to sequence production. Participants learned an isochronous melody that they performed on a keyboard and attempted to perform this sequence at a prescribed rate while auditory and visual feedback were manipulated. Delayed auditory feedback (DAF) and delayed visual feedback (DVF) both tended to slow production of the sequence. These effects were additive. There was no modulation of this effect of delay in either modality by the absence of feedback in the other. In contrast with past research, DAF did not increase timing variability, though DVF did. Motion analyses ruled out differences in salience of visual feedback between delayed and non-delayed conditions as an explanation of the effects of DVF. The results suggest that the effects of delayed feedback may be attributable to both sensorimotor interference and to conflicting information across feedback channels.

The effects of voluntary movements on auditory-haptic and haptic-haptic temporal order judgments

In two experiments we investigated the effects of voluntary movements on temporal haptic perception. Measures of sensitivity (JND) and temporal alignment (PSS) were obtained from temporal order judgments made on intermodal auditory-haptic (Experiment 1) or intramodal haptic (Experiment 2) stimulus pairs under three movement conditions. In the baseline, static condition, the arm of the participants remained stationary. In the passive condition, the arm was displaced by a servo-controlled motorized device. In the active condition, the participants moved voluntarily. The auditory stimulus was a short, 500Hz tone presented over headphones and the haptic stimulus was a brief suprathreshold force pulse applied to the tip of the index finger orthogonally to the finger movement. Active movement did not significantly affect discrimination sensitivity on the auditory-haptic stimulus pairs, whereas it significantly improved sensitivity in the case of the haptic stimulus pair, demonstrating a key role for motor command information in temporal sensitivity in the haptic system. Points of subjective simultaneity were by-and-large coincident with physical simultaneity, with one striking exception in the passive condition with the auditory-haptic stimulus pair. In the latter case, the haptic stimulus had to be presented 45ms before the auditory stimulus in order to obtain subjective simultaneity. A model is proposed to explain the discrimination performance.

When time is space: evidence for a mental time line

Time and space are tightly linked in the physical word. Recently, several lines of evidence have suggested that the mental representation of time might be spatial in nature. For instance, time-space interactions have been described as a strong preference to associate the past with the left space and the future with the right space. Here we review the growing evidence of interactions between time and space processing, systematized according to the type of interaction being investigated. We present the empirical findings supporting the possibility that humans represent the subjective time flow on a spatially oriented "mental time line" that is accessed through spatial attention mechanisms. The heterogeneous time-space interactions are then compared with the number-space interactions described in the numerical cognition literature. An alternative hypothesis, which maintains a common system for magnitude processing, including time, space, and number, is also discussed. Finally, we extend the discussion to the more general issue of how the representation of these concepts might be grounded into the cortical circuits that support spatial attention and sensorimotor transformations.

Response errors explain the failure of independent-channels models of perception of temporal order

Independent-channels models of perception of temporal order (also referred to as threshold models or perceptual latency models) have been ruled out because two formal properties of these models (monotonicity and parallelism) are not borne out by data from ternary tasks in which observers must judge whether stimulus A was presented before, after, or simultaneously with stimulus B. These models generally assume that observed responses are authentic indicators of unobservable judgments, but blinks, lapses of attention, or errors in pressing the response keys (maybe, but not only, motivated by time pressure when reaction times are being recorded) may make observers misreport their judgments or simply guess a response. We present an extension of independent-channels models that considers response errors and we show that the model produces psychometric functions that do not satisfy monotonicity and parallelism. The model is illustrated by fitting it to data from a published study in which the ternary task was used. The fitted functions describe very accurately the absence of monotonicity and parallelism shown by the data. These characteristics of empirical data are thus consistent with independent-channels models when response errors are taken into consideration. The implications of these results for the analysis and interpretation of temporal order judgment data are discussed.

Evaluating User Response to In-Car Haptic Feedback Touchscreens Using the Lane Change Test

Touchscreen interfaces are widely used in modern technology, from mobile devices to in-car infotainment systems. However, touchscreens impose significant visual workload demands on the user which have safety implications for use in cars. Previous studies indicate that the application of haptic feedback can improve both performance of and affective response to user interfaces. This paper reports on and extends the findings of a 2009 study conducted to evaluate the effects of different combinations of touchscreen visual, audible, and haptic feedback on driving and task performance, affective response, and subjective workload; the initial findings of which were originally published in (M. J. Pitts et al., 2009). A total of 48 non-expert users completed the study. A dual-task approach was applied, using the Lane Change Test as the driving task and realistic automotive use case touchscreen tasks. Results indicated that, while feedback type had no effect on driving or task performance, preference was expressed for multimodal feedback over visual alone. Issues relating to workload and cross-modal interaction were also identified.