Changes in auditory frequency guide visual-spatial attention

Interference from semantically distracting sounds in action scene search

Research on visual searching has highlighted the role of crossmodal interactions between semantically congruent visual and auditory stimuli. Typically, such sounds facilitate performance. Conversely, semantically incongruent sounds may impair visual search efficiency for action scenes, though consensus has yet to be reached. This study investigated whether interference effects occur within the action-scene search paradigm. Participants performed a search task involving four simultaneously presented video stimuli, accompanied by one of three sound conditions: sound congruent with the target, congruent with a distractor, or a control sound. Auditory interference was observed, though it was relatively weak and varied across conditions rather than being simply present or absent. The observed variability in interference effects may align with the established view that observers typically ignore semantic distractor information in goal-directed searches, except in cases where the strength of target designation is compromised. These findings offer insights into the complex interplay between auditory and visual stimuli in action scene searches, suggesting that these underlying mechanisms may also apply to other paradigms, such as those involving conventional real object searches.

Involvement of the superior colliculi in crossmodal correspondences

There is an increasing body of evidence suggesting that there are low-level perceptual processes involved in crossmodal correspondences. In this study, we investigate the involvement of the superior colliculi in three basic crossmodal correspondences: elevation/pitch, lightness/pitch, and size/pitch. Using a psychophysical design, we modulate visual input to the superior colliculus to test whether the superior colliculus is required for behavioural crossmodal congruency effects to manifest in an unspeeded multisensory discrimination task. In the elevation/pitch task, superior colliculus involvement is required for a behavioural elevation/pitch congruency effect to manifest in the task. In the lightness/pitch and size/pitch task, we observed a behavioural elevation/pitch congruency effect regardless of superior colliculus involvement. These results suggest that the elevation/pitch correspondence may be processed differently to other low-level crossmodal correspondences. The implications of a distributed model of crossmodal correspondence processing in the brain are discussed.

How to prevent cognitive overload in the walking-arithmetic dual task among patients with Parkinson's disease

BACKGROUND

Participants with Parkinson's disease (PD) may experience difficulty during certain dual-task (DT) tests. Thus, it is necessary to keep the cognitive load within the limits of their ability.

OBJECTIVE

To identify cognitive overload and its influence on the walking and auditory addition and subtraction (AAS, all values within the range of 0-20) DT performance of patients with PD.

STUDY DESIGN

A cross-sectional observational study with convenience sampling.

SETTING

Outpatient clinic of the Department of Neurology.

SUBJECTS

Sixteen patients with PD and 15 sex- and age- matched people elderly healthy controls (HCs).

METHODS

Verbal calculation responses and gait parameters were collected from the two groups in the 2-min single arithmetic task (2-min SAT), 2-min single walking task (2-min SWT), and 2-min walking-arithmetic dual task (2-min WADT).

RESULTS

The group differences in the lower-limb gait parameters increased in the 2-min WADT (P < 0.01), and those in the arm, trunk, and waist parameters did not change (P > 0.05). In the 2-min SAT, the calculation speed of the PD group was significantly lower than that of the HC group (P < 0.01). In the 2-min WADT, both groups made more errors (P < 0.05), especially the PD group (P = 0.00). PD group miscalculations occurred in the first half of the 2-min SAT but were uniformly distributed in the 2-min WADT. The HC group and PD group had subtraction self-correction rates of 31.25% and 10.25%, respectively. The PD group tended to make subtraction errors when the value of the first operand was 20 or 13.46 ± 2.60 and when the value of the second and third operands were 7.75 ± 2.51 (P = 0.3657) and 8.50 ± 4.04 (P = 0.170), respectively.

CONCLUSIONS

Cognitive overload was observed in patients with PD. This was mainly reflected in the failure of gait control and accurate calculation, indicated by gait parameters of the lower limbs and accuracy of calculation. To impose a constant cognitive load, the amount added or subtracted, especially in subtraction with borrowing, should not be mixed during a sequential arithmetic problem in the DT, and equations with the value of the first operand equal to 20 or approximately 13, the value of the second operand approximately 7, or the value of the third operand of approximately 9 should be excluded in the AAS DT.

TRIAL REGISTRATION

Clinical trial registration number: ChiCTR1800020158.

Effect of Target Semantic Consistency in Different Sequence Positions and Processing Modes on T2 Recognition: Integration and Suppression Based on Cross-Modal Processing

In the rapid serial visual presentation (RSVP) paradigm, sound affects participants' recognition of targets. Although many studies have shown that sound improves cross-modal processing, researchers have not yet explored the effects of sound semantic information with respect to different locations and processing modalities after removing sound saliency. In this study, the RSVP paradigm was used to investigate the difference between attention under conditions of consistent and inconsistent semantics with the target (Experiment 1), as well as the difference between top-down (Experiment 2) and bottom-up processing (Experiment 3) for sounds with consistent semantics with target 2 (T2) at different sequence locations after removing sound saliency. The results showed that cross-modal processing significantly improved attentional blink (AB). The early or lagged appearance of sounds consistent with T2 did not affect participants' judgments in the exogenous attentional modality. However, visual target judgments were improved with endogenous attention. The sequential location of sounds consistent with T2 influenced the judgment of auditory and visual congruency. The results illustrate the effects of sound semantic information in different locations and processing modalities.

The effect of different combinations of physical activity and natural environment videos on children's attention levels between class breaks

BACKGROUND

Physical activity intervention and watching natural environment videos have been proven to improve young children's attention levels. However, evidence comparing the improvement effects of different combinations of the two activities has rarely been reported. By comparing the differences in the improvement effects of four combinations of physical activities and watching natural environment videos on young children's attention levels, this study can enrich the evidence in this research field and also provide a reference for arranging effective intervention methods for children's attention recovery between classes.

METHOD

A total of 152 children aged 4 to 6 years were recruited and randomly divided into four intervention groups: (1) physical activity intervention first and thereafter watching a natural environment video group (activity + video group), (2) watching a natural environment video first and thereafter the physical activity intervention group (video + activity group), (3) physical activity-based group, and (4) natural environment video-based group. Physical activity involved 4 min of moderate-intensity basic physical fitness combination training. The subjects wore the Pico Neo pioneer version of the VR glasses all-in-one machine to watch a natural environment video. Thereafter, population sociological variables and daily physical activity levels were investigated. Auditory and visual sustained attention tests were performed before and after intervention in each group.

RESULT

The auditory attention post-test scores of the four groups showed an improvement trend compared with the pretest scores. In particular, the activity + video group (F = 10.828; ɳp2 = 0.226; p = 0.002) and natural environment video-based group (F = 9.452; ɳp2 = 0.203; p = 0.004) have the best improvement effect. For visual attention, only the activity + video group showed a significant improvement trend (F = 4.287; ɳp2 = 0.104; p = 0.045), while the other three groups showed a downward trend in scores.

CONCLUSIONS

Among the different intervention combinations, the physical activity intervention first and watching natural environment videos thereafter group has the best effect on improving children's attention during recess. Physical activity interventions at the end of recess adversely affect young children's visual attention levels at the beginning of the class. Therefore, this study recommends that children should not engage in physical activity interventions in the second half of the class break. Lastly, the current research recommends presenting the content of physical activity interventions first and further improving their attention thereafter by watching natural environment videos.

Pitch-elevation and pitch-size cross-modal correspondences do not affect temporal ventriloquism

Temporal ventriloquism refers to the shift in the perceived timing of a visual stimulus towards a transient auditory stimulus presented close in time. This effect is demonstrated by greater sensitivity of temporal order judgments of two visual stimuli when a sound is presented before the first visual stimulus and after the second visual stimulus. Recent studies suggest that temporal ventriloquism is affected by cross-modal correspondence between auditory pitch and visual elevation but not by correspondence between pitch and visual size. Here we examined the possibility that these results do not reflect a difference in the effects of different types of cross-modal correspondences on temporal ventriloquism but are rather mediated by shifts in visual-spatial attention triggered by preceding auditory stimuli. In Experiment 1, we replicated the results of previous studies that varied with the type of correspondence. Experiment 2 investigated the effects of the second audiovisual stimuli's asynchrony while the first audiovisual stimuli were synchronized. The results, unlike in Experiment 1, revealed that the magnitude of the temporal ventriloquism effect did not change with the congruency of pitch-elevation correspondence. Experiment 3 also indicated that the asynchrony of the first audiovisual stimuli modulated visual discrimination sensitivity irrespective of temporal ventriloquism. These results suggest that cross-modal correspondences do not affect temporal ventriloquism. Greater visual sensitivity when audiovisual stimuli are congruent with pitch-elevation correspondence may be attributable to shifts in visual attention caused by pitches of the preceding auditory stimulus, which speeds up detection of the first visual stimulus.

Effects of pitch and musical sounds on body-representations when moving with sound

The effects of music on bodily movement and feelings, such as when people are dancing or engaged in physical activity, are well-documented—people may move in response to the sound cues, feel powerful, less tired. How sounds and bodily movements relate to create such effects? Here we deconstruct the problem and investigate how different auditory features affect people’s body-representation and feelings even when paired with the same movement. In three experiments, participants executed a simple arm raise synchronised with changing pitch in simple tones (Experiment 1), rich musical sounds (Experiment 2) and within different frequency ranges (Experiment 3), while we recorded indirect and direct measures on their movement, body-representations and feelings. Changes in pitch influenced people’s general emotional state as well as the various bodily dimensions investigated—movement, proprioceptive awareness and feelings about one’s body and movement. Adding harmonic content amplified the differences between ascending and descending sounds, while shifting the absolute frequency range had a general effect on movement amplitude, bodily feelings and emotional state. These results provide new insights in the role of auditory and musical features in dance and exercise, and have implications for the design of sound-based applications supporting movement expression, physical activity, or rehabilitation.

Multisensory Proximity and Transition Cues for Improving Target Awareness in Narrow Field of View Augmented Reality Displays

Augmented reality applications allow users to enrich their real surroundings with additional digital content. However, due to the limited field of view of augmented reality devices, it can sometimes be difficult to become aware of newly emerging information inside or outside the field of view. Typical visual conflicts like clutter and occlusion of augmentations occur and can be further aggravated especially in the context of dense information spaces. In this article, we evaluate how multisensory cue combinations can improve the awareness for moving out-of-view objects in narrow field of view augmented reality displays. We distinguish between proximity and transition cues in either visual, auditory or tactile manner. Proximity cues are intended to enhance spatial awareness of approaching out-of-view objects while transition cues inform the user that the object just entered the field of view. In study 1, user preference was determined for 6 different cue combinations via forced-choice decisions. In study 2, the 3 most preferred modes were then evaluated with respect to performance and awareness measures in a divided attention reaction task. Both studies were conducted under varying noise levels. We show that on average the Visual-Tactile combination leads to 63% and Audio-Tactile to 65% faster reactions to incoming out-of-view augmentations than their Visual-Audio counterpart, indicating a high usefulness of tactile transition cues. We further show a detrimental effect of visual and audio noise on performance when feedback included visual proximity cues. Based on these results, we make recommendations to determine which cue combination is appropriate for which application.

Haptic-payment: Exploring vibration feedback as a means of reducing overspending in mobile payment

The proliferation of mobile payment applications in recent years has decoupled the physical act of paying from the consumption experience. Prior research suggests that this decreases the psychological sense of loss or 'pain' that consumers feel when making a purchase with more direct payment types (such as cash) and leads them to spend more money. To help address this issue, the present research explores, designs, and tests haptic vibration feedback configurations aimed at restoring the 'pain' of paying with cashless payment options (i.e., online and mobile payment). Counter-intuitively, the present research finds that lower- (vs. higher-) intensity vibration feedback reduces participants' reported willingness-to-spend when compared to a control group that does not receive any vibration feedback. This work is one of the first to explore the role of haptic vibration feedback in nudging consumers to reduce their spending when using cashless payment methods.

Response orientation modulates pitch-space relationships: the ROMPR effect

Congruency between auditory and visuospatial stimuli has been previously shown to affect responses to multisensory stimulus pairs, and congruency between stimuli and response devices may play a role in response speed and accuracy. Across two experiments, we tested whether the accuracy and speed of pitch judgments were affected by a congruent or incongruent paired visual stimulus, and whether the relationship was modulated by response orientation. In Experiment 1, participants using a vertically (transversely) oriented keyboard demonstrated a large crossmodal vertical effect, but a minimal crossmodal horizontal effect. In contrast, Experiment 2 used a horizontally oriented keyboard, while also examining whether musical training impacts pitch judgments. As in the first experiment, we found an effect of response mapping on pitch judgments; these results suggest that vertical visual stimuli are processed automatically, while the effects of horizontal visual stimuli are decisional and require a compatible response orientation. Based on these findings, we propose an effect we call the ROMPR effect: response orientation modulates pitch-space relationships. Unexpectedly, non-musicians demonstrated significant ROMPR effects while trained musicians did not. We suggest that non-musicians are more likely to use visual information when making spatial location judgments of pitch: unlike musicians, they have not been trained to rely exclusively on auditory information during pitch processing. We also discuss alternative explanations of the data: namely, that there is a need to disambiguate audiovisual congruency from visual-response congruency with modulations of experimental design.

The Role of Selective Attention in Cross-modal Interactions between Auditory and Visual Features

Evans and Treisman (2010) showed systematic interactions between audition and vision when participants made speeded classifications in one modality while supposedly ignoring another. We found perceptual facilitation between high pitch and high visual position, high spatial frequency and small size, and interference between high pitch and low position, low spatial frequency and large size, while the converse was the case between low pitch and the same visual features. The present study examined the role of selective attention in these cross-modal interactions. Participants performed speeded classification or search tasks of low or high load while attempting to ignore irrelevant stimuli in a different modality. In both paradigms, congruency between the visual and the irrelevant auditory stimulus had an equal effect in the low and in the high perceptual load conditions. A third experiment tested divided attention, requiring participants to compare stimuli across modalities and respond to the visual-auditory compound. The congruency effect was as large with attention focused on one modality as when it was divided across both. These findings offer converging evidence that cross-modal interactions between corresponding basic features are independent of selective attention.

Coarse to Fine Audio-Visual Size Correspondences Develop During Primary School Age

Developmental studies have shown that children can associate visual size with non-visual and apparently unrelated stimuli, such as pure tone frequencies. Most research to date has focused on audio-visual size associations by showing that children can associate low pure tone frequencies with large objects, and high pure tone frequencies with small objects. Researchers relate these findings to coarser association, i.e., less precise associations for which binary categories of stimuli are used such as in the case of low versus high frequencies and large versus small visual stimuli. This study investigates how finer, more precise, crossmodal audio-visual associations develop during primary school age (from 6 to 11 years old). To unveil such patterns, we took advantage of a range of auditory pure tones and tested how primary school children match sounds with visually presented shapes. We tested 66 children (6-11 years old) in an audio-visual matching task involving a range of pure tone frequencies. Visual stimuli were circles or angles of different sizes. We asked participants to indicate the shape matching the sound. All children associated large objects/angles with low pitch, and small objects/angles with high pitch sounds. Interestingly, older children made greater use of intermediate visual sizes to provide their responses. Indeed, audio-visual associations for finer differences between stimulus features such as size and pure tone frequencies, may develop later depending on the maturation of supramodal size perception processes. Considering our results, we suggest that audio-visual size correspondences can be used for educational purposes by aiding the discrimination of sizes, including angles of different aperture. Moreover, their use should be shaped according to children's specific developmental stage.

Reaching for the high note: judgments of auditory pitch are affected by kinesthetic position

Auditory pitch is represented on a vertical continuum (e.g., high vs. low). In three experiments, we examined whether the kinesthetic experience of moving in a particular direction (e.g., walking up vs. down stairs; reaching up vs. down) would affect judgments of auditory pitch. Participants listened to three tones twice each, once while moving upward and once while moving downward, and estimated the pitch of each tone. In all experiments, participants' judgments of the tones' pitch were biased in the direction of their movement. Moreover, this effect is not due to visibility of the movement or to using a numerical response method. Our results suggest that kinesthetic information from one's own bodily movements biases pitch estimation, and several possible mechanisms for the effect are discussed.

Auditory pitch glides influence time-to-contact judgements of visual stimuli

A common experimental task used to study the accuracy of estimating when a moving object arrives at a designated location is the time-to-contact (TTC) task. The previous studies have shown evidence that sound motion cues influence TTC estimates of a visual moving object. However, the extent to which sound can influence TTC of visual targets still remains unclear. Some studies on the crossmodal correspondence between pitch and speed suggest that descending pitch sounds are associated with faster speeds compared to ascending pitch sounds due to an internal model of gravity. Other studies have shown an opposite pitch-speed mapping (i.e., ascending pitch associated with faster speeds) and no influence of gravity heuristics. Here, we explored whether auditory pitch glides, a continuous pure tone sound either ascending or descending in pitch, influence TTC estimates of a vertically moving visual target and if any observed effects are consistent with a gravity-centered or gravity-unrelated pitch-speed mapping. Subjects estimated when a disc moving either upward or downward at a constant speed reached a visual landmark after the disc disappeared behind an occluder under three conditions: with an accompanying ascending pitch glide, with a descending pitch glide, or with no sound. Overall, subjects underestimated TTC with ascending pitch glides and overestimated TTC with descending pitch glides, compared to the no-sound condition. These biases in TTC were consistent in both disc motion directions. These results suggest that subjects adopted a gravity-unrelated pitch-speed mapping where ascending pitch is associated with faster speeds and descending pitch associated with slower speeds.

Lightness/pitch and elevation/pitch crossmodal correspondences are low-level sensory effects

We tested the sensory versus decisional origins of two established audiovisual crossmodal correspondences (CMCs; lightness/pitch and elevation/pitch), applying a signal discrimination paradigm to low-level stimulus features and controlling for attentional cueing. An audiovisual stimulus randomly varied along two visual dimensions (lightness: black/white; elevation: high/low) and one auditory dimension (pitch: high/low), and participants discriminated either only lightness, only elevation, or both lightness and elevation. The discrimination task and the stimulus duration varied between subjects. To investigate the influence of crossmodal congruency, we considered the effect of each CMC (lightness/pitch and elevation/pitch) on the sensitivity and criterion of each discrimination as a function of stimulus duration. There were three main findings. First, discrimination sensitivity was significantly higher for visual targets paired congruently (compared with incongruently) with tones while criterion was unaffected. Second, the sensitivity increase occurred for all stimulus durations, ruling out attentional cueing effects. Third, the sensitivity increase was feature specific such that only the CMC that related to the feature being discriminated influenced sensitivity (i.e. lightness congruency only influenced lightness discrimination and elevation congruency only influenced elevation discrimination in the single and dual task conditions). We suggest that these congruency effects reflect low-level sensory processes.

Analysing the Impact of Music on the Perception of Red Wine via Temporal Dominance of Sensations

Several studies have examined how music may affect the evaluation of food and drink, but the vast majority have not observed how this interaction unfolds in time. This seems to be quite relevant, since both music and the consumer experience of food/drink are time-varying in nature. In the present study we sought to fix this gap, using Temporal Dominance of Sensations (TDS), a method developed to record the dominant sensory attribute at any given moment in time, to examine the impact of music on the wine taster's perception. More specifically, we assessed how the same red wine might be experienced differently when tasters were exposed to various sonic environments (two pieces of music plus a silent control condition). The results revealed diverse patterns of dominant flavours for each sound condition, with significant differences in flavour dominance in each music condition as compared to the silent control condition. Moreover, musical correspondence analysis revealed that differences in perceived dominance of acidity and bitterness in the wine were correlated in the temporality of the experience, with changes in basic auditory attributes. Potential implications for the role of attention in auditory flavour modification and opportunities for future studies are discussed.

On the Relative Nature of (Pitch-Based) Crossmodal Correspondences

This review deals with the question of the relative vs absolute nature of crossmodal correspondences, with a specific focus on those correspondences involving the auditory dimension of pitch. Crossmodal correspondences have been defined as the often-surprising crossmodal associations that people experience between features, attributes, or dimensions of experience in different sensory modalities, when either physically present, or else merely imagined. In the literature, crossmodal correspondences have often been contrasted with synaesthesia in that the former are frequently said to be relative phenomena (e.g., it is the higher-pitched of two sounds that is matched with the smaller of two visual stimuli, say, rather than there being a specific one-to-one crossmodal mapping between a particular pitch of sound and size of object). By contrast, in the case of synaesthesia, the idiosyncratic mapping between inducer and concurrent tends to be absolute (e.g., it is a particular sonic inducer that elicits a specific colour concurrent). However, a closer analysis of the literature soon reveals that the distinction between relative and absolute in the case of crossmodal correspondences may not be as clear-cut as some commentators would have us believe. Furthermore, it is important to note that the relative vs absolute question may receive different answers depending on the particular (class of) correspondence under empirical investigation.

Upper nasal hemifield location and nonspatial auditory tones accelerate visual detection during dichoptic viewing

Visual performance is asymmetric across the visual field, but locational biases that occur during dichoptic viewing are not well understood. In this study, we characterized horizontal, vertical and naso-temporal biases in visual target detection during dichoptic stimulation and explored whether the detection was facilitated by non-spatial auditory tones associated with the target’s location.

The detection time for single monocular targets that were suppressed from view with a 10 Hz dynamic noise mask presented to the other eye was measured at the 4° intercardinal location of each eye with the breaking Continuous Flash Suppression (b-CFS) technique. Each target was either combined with a sound (i.e., high or low pitch tone) that was congruent or incongruent with its vertical location (i.e., upper or lower visual field) or presented without a sound. The results indicated faster detection of targets in the upper rather than lower visual field and faster detection of targets in the nasal than temporal hemifield of each eye. Sounds generally accelerated target detection, but the tone pitch-elevation congruency did not further enhance performance. These findings suggest that visual detection during dichoptic viewing differs from standard viewing conditions with respect to location-related perceptual biases and crossmodal modulation of visual perception. These differences should be carefully considered in experimental designs employing dichoptic stimulation techniques and in display applications that utilize dichoptic viewing.

Seeing music: The perception of melodic 'ups and downs' modulates the spatial processing of visual stimuli

Musical melodies have "peaks" and "valleys". Although the vertical component of pitch and music is well-known, the mechanisms underlying its mental representation still remain elusive. We show evidence regarding the importance of previous experience with melodies for crossmodal interactions to emerge. The impact of these crossmodal interactions on other perceptual and attentional processes was also studied. Melodies including two tones with different frequency (e.g., E4 and D3) were repeatedly presented during the study. These melodies could either generate strong predictions (e.g., E4-D3-E4-D3-E4-[D3]) or not (e.g., E4-D3-E4-E4-D3-[?]). After the presentation of each melody, the participants had to judge the colour of a visual stimulus that appeared in a position that was, according to the traditional vertical connotations of pitch, either congruent (e.g., high-low-high-low-[up]), incongruent (high-low-high-low-[down]) or unpredicted with respect to the melody. Behavioural and electroencephalographic responses to the visual stimuli were obtained. Congruent visual stimuli elicited faster responses at the end of the experiment than at the beginning. Additionally, incongruent visual stimuli that broke the spatial prediction generated by the melody elicited larger P3b amplitudes (reflecting 'surprise' responses). Our results suggest that the passive (but repeated) exposure to melodies elicits spatial predictions that modulate the processing of other sensory events.

Covert Exogenous Cross-Modality Orienting between Audition and Vision

Control of visual attention by auditory stimuli is explored in seven previously unpublished experiments that were presented at conferences in the late 1980s. Reaction time (RT) to luminance targets was found to be affected by the spatial congruence between the target and a preceding or simultaneous, and non-informative, auditory event, suggesting that localizable auditory stimuli exogenously (rapidly and automatically) capture visual attention. These cuing effects were obtained in the absence of eye movements and do not appear to be mediated merely by criterion adjustments. When the information value of the auditory event was placed in conflict with its location (i.e., a tone on the right indicated that the visual target was likely to appear on the left), it was found that at short stimulus onset asynchronies (SOAs) reaction time (RT) was faster for targets at the stimulated location, an effect that disappeared within 500 ms and was reversed by 1000 ms. This demonstrates that it requires over 500 ms for endogenous orienting in response to probabilistic information about target location to overcome the powerful exogenous control of visual attention by localizable auditory stimulation. Simple RT to auditory stimuli was unaffected by the spatial congruence of a preceding or simultaneous visual stimulus. When uninformative, neither pitch contours (rising/falling tones) nor pitch (high/low tones) produced significant visual orienting along the vertical midline. When the direction of a pitch contour indicated the likely location of a visual target, participants were able to shift their attention if the relation between the natural meaning and the probabilistic information was compatible (e.g., rising contour signaled that a upper target was likely) but not when it was incompatible. The relation of these 30-year-old experiments to contemporary findings and ideas is discussed.

Audio-visual interaction in visual motion detection: Synchrony versus Asynchrony

OBJECTIVE

Detection and identification of moving targets is of paramount importance in everyday life, even if it is not widely tested in optometric practice, mostly for technical reasons. There are clear indications in the literature that in perception of moving targets, vision and hearing interact, for example in noisy surrounds and in understanding speech. The main aim of visual perception, the ability that optometry aims to optimize, is the identification of objects, from everyday objects to letters, but also the spatial orientation of subjects in natural surrounds. To subserve this aim, corresponding visual and acoustic features from the rich spectrum of signals supplied by natural environments have to be combined.

METHODS

Here, we investigated the influence of an auditory motion stimulus on visual motion detection, both with a concrete (left/right movement) and an abstract auditory motion (increase/decrease of pitch).

RESULTS

We found that incongruent audiovisual stimuli led to significantly inferior detection compared to the visual only condition. Additionally, detection was significantly better in abstract congruent than incongruent trials. For the concrete stimuli the detection threshold was significantly better in asynchronous audiovisual conditions than in the unimodal visual condition.

CONCLUSION

We find a clear but complex pattern of partly synergistic and partly inhibitory audio-visual interactions. It seems that asynchrony plays only a positive role in audiovisual motion while incongruence mostly disturbs in simultaneous abstract configurations but not in concrete configurations. As in speech perception in hearing-impaired patients, patients suffering from visual deficits should be able to benefit from acoustic information.

Contingent sounds change the mental representation of one's finger length

Mental body-representations are highly plastic and can be modified after brief exposure to unexpected sensory feedback. While the role of vision, touch and proprioception in shaping body-representations has been highlighted by many studies, the auditory influences on mental body-representations remain poorly understood. Changes in body-representations by the manipulation of natural sounds produced when one’s body impacts on surfaces have recently been evidenced. But will these changes also occur with non-naturalistic sounds, which provide no information about the impact produced by or on the body? Drawing on the well-documented capacity of dynamic changes in pitch to elicit impressions of motion along the vertical plane and of changes in object size, we asked participants to pull on their right index fingertip with their left hand while they were presented with brief sounds of rising, falling or constant pitches, and in the absence of visual information of their hands. Results show an “auditory Pinocchio” effect, with participants feeling and estimating their finger to be longer after the rising pitch condition. These results provide the first evidence that sounds that are not indicative of veridical movement, such as non-naturalistic sounds, can induce a Pinocchio-like change in body-representation when arbitrarily paired with a bodily action.

Volitional Mechanisms Mediate the Cuing Effect of Pitch on Attention Orienting: The Influences of Perceptual Difficulty and Response Pressure

Our cognitive system tends to link auditory pitch with spatial location in a specific manner (ie high-pitched sounds are usually associated with an upper location, and low sounds are associated with a lower location). Recent studies have demonstrated that this cross-modality association biases the allocation of visual attention and affects performance despite the auditory stimuli being irrelevant to the behavioural task. There is, however, a discrepancy between studies in their interpretation of the underlying mechanisms. Whereas we have previously claimed that the pitch-location mapping is mediated by volitional shifts of attention (Chiou & Rich, 2012, Perception, 41: , 339-353), other researchers suggest that this cross-modal effect reflects automatic shifts of attention (Mossbridge, Grabowecky, & Suzuki, 2011, Cognition, 121: , 133-139). Here we report a series of three experiments examining the effects of perceptual and response-related pressure on the ability of nonpredictive pitch to bias visual attention. We compare it with two control cues: a predictive pitch that triggers voluntary attention shifts and a salient peripheral flash that evokes involuntary shifts. The results show that the effect of nonpredictive pitch is abolished by pressure at either perceptual or response levels. By contrast, the effects of the two control cues remain significant, demonstrating the robustness of informative and perceptually salient stimuli in directing attention. This distinction suggests that, in contexts of high perceptual demand and response pressure, cognitive resources are primarily engaged by the task-relevant stimuli, which effectively prevents uninformative pitch from orienting attention to its cross-modally associated location. These findings are consistent with the hypothesis that the link between pitch and location affects attentional deployment via volitional rather than automatic mechanisms.

Which Direction Is up for a High Pitch?

Low- and high-pitched sounds are perceptually associated with low and high visuospatial elevations, respectively. The spatial properties of this association are not well understood. Here we report two experiments that investigated whether low and high tones can be used as spatial cues to upright for self-orientation and identified the spatial frame(s) of reference used in perceptually binding auditory pitch to visuospatial 'up' and 'down'. In experiment 1, participants' perceptual upright (PU) was measured while lying on their right side with and without high- and low-pitched sounds played through speakers above their left ear and below their right ear. The sounds were ineffective in moving the perceived upright from a direction intermediate between the body and gravity towards the direction indicated by the sounds. In experiment 2, we measured the biasing effects of ascending and descending tones played through headphones on ambiguous vertical or horizontal visual motion created by combining gratings drifting in opposite directions while participants either sat upright or laid on their right side. Ascending and descending tones biased the interpretation of ambiguous motion along both the gravitational vertical and the long-axis of the body with the strongest effect along the body axis. The combination of these two effects showed that axis of maximum effect of sound corresponded approximately to the direction of the perceptual upright, compatible with the idea that 'high' and 'low' sounds are defined along this axis.

Cognitive Styles Differentiate Crossmodal Correspondences Between Pitch Glide and Visual Apparent Motion

Crossmodal correspondences are the automatic associations that most people have between different basic sensory stimulus attributes, dimensions, or features. For instance, people often show a systematic tendency to associate moving objects with changing pitches. Cognitive styles are defined as an individual's consistent approach to think, perceive, and remember information, and they reflect qualitative rather than quantitative differences between individuals in their thinking processes. Here we asked whether cognitive styles played a role in modulating the crossmodal interaction. We used the visual Ternus display in our study, since it elicits two distinct apparent motion percepts: element motion (with a shorter interval between the two Ternus frames) and group motion (with a longer interval between the two frames). We examined the audiovisual correspondences between the visual Ternus movement directions (upward or downward) and the changes of pitches of concurrent glides (ascending frequency or descending frequency). Moreover, we measured the cognitive styles (with the Embedded Figure Test) for each participant. The results showed that congruent correspondence between pitch-ascending (decreasing) glides and moving upward (downward) visual directions led to a more dominant percept of 'element motion', and such an effect was typically observed in the field-independent group. Importantly, field-independent participants demonstrated a high efficiency for identifying the properties of audiovisual events and applying the crossmodal correspondence in crossmodal interaction. The results suggest cognitive styles could differentiate crossmodal correspondences in crossmodal interaction.

Pitch perception deficits in nonverbal learning disability

The nonverbal learning disability (NLD) is a neurological dysfunction that affects cognitive functions predominantly related to the right hemisphere such as spatial and abstract reasoning. Previous evidence in healthy adults suggests that acoustic pitch (i.e., the relative difference in frequency between sounds) is, under certain conditions, encoded in specific areas of the right hemisphere that also encode the spatial elevation of external objects (e.g., high vs. low position). Taking this evidence into account, we explored the perception of pitch in preadolescents and adolescents with NLD and in a group of healthy participants matched by age, gender, musical knowledge and handedness. Participants performed four speeded tests: a stimulus detection test and three perceptual categorization tests based on colour, spatial position and pitch. Results revealed that both groups were equally fast at detecting visual targets and categorizing visual stimuli according to their colour. In contrast, the NLD group showed slower responses than the control group when categorizing space (direction of a visual object) and pitch (direction of a change in sound frequency). This pattern of results suggests the presence of a subtle deficit at judging pitch in NLD along with the traditionally-described difficulties in spatial processing.

Direction of Auditory Pitch-Change Influences Visual Search for Slope From Graphs

Linear trend (slope) is important information conveyed by graphs. We investigated how sounds influenced slope detection in a visual search paradigm. Four bar graphs or scatter plots were presented on each trial. Participants looked for a positive-slope or a negative-slope target (in blocked trials), and responded to targets in a go or no-go fashion. For example, in a positive-slope-target block, the target graph displayed a positive slope while other graphs displayed negative slopes (a go trial), or all graphs displayed negative slopes (a no-go trial). When an ascending or descending sound was presented concurrently, ascending sounds slowed detection of negative-slope targets whereas descending sounds slowed detection of positive-slope targets. The sounds had no effect when they immediately preceded the visual search displays, suggesting that the results were due to crossmodal interaction rather than priming. The sounds also had no effect when targets were words describing slopes, such as "positive," "negative," "increasing," or "decreasing," suggesting that the results were unlikely due to semantic-level interactions. Manipulations of spatiotemporal similarity between sounds and graphs had little effect. These results suggest that ascending and descending sounds influence visual search for slope based on a general association between the direction of auditory pitch-change and visual linear trend.

How big is this sound? Crossmodal association between pitch and size in infants

We examined 4- and 6-month-old infants' sensitivity to the perceptual association between pitch and object size. Crossmodal correspondence effects were observed in 6-month-old infants but not in younger infants, suggesting that experience and/or further maturation is needed to fully develop this crossmodal association.

Musicians are more consistent: Gestural cross-modal mappings of pitch, loudness and tempo in real-time

Cross-modal mappings of auditory stimuli reveal valuable insights into how humans make sense of sound and music. Whereas researchers have investigated cross-modal mappings of sound features varied in isolation within paradigms such as speeded classification and forced-choice matching tasks, investigations of representations of concurrently varied sound features (e.g., pitch, loudness and tempo) with overt gestures-accounting for the intrinsic link between movement and sound-are scant. To explore the role of bodily gestures in cross-modal mappings of auditory stimuli we asked 64 musically trained and untrained participants to represent pure tones-continually sounding and concurrently varied in pitch, loudness and tempo-with gestures while the sound stimuli were played. We hypothesized musical training to lead to more consistent mappings between pitch and height, loudness and distance/height, and tempo and speed of hand movement and muscular energy. Our results corroborate previously reported pitch vs. height (higher pitch leading to higher elevation in space) and tempo vs. speed (increasing tempo leading to increasing speed of hand movement) associations, but also reveal novel findings pertaining to musical training which influenced consistency of pitch mappings, annulling a commonly observed bias for convex (i.e., rising-falling) pitch contours. Moreover, we reveal effects of interactions between musical parameters on cross-modal mappings (e.g., pitch and loudness on speed of hand movement), highlighting the importance of studying auditory stimuli concurrently varied in different musical parameters. Results are discussed in light of cross-modal cognition, with particular emphasis on studies within (embodied) music cognition. Implications for theoretical refinements and potential clinical applications are provided.

Natural auditory scene statistics shapes human spatial hearing

Human perception, cognition, and action are laced with seemingly arbitrary mappings. In particular, sound has a strong spatial connotation: Sounds are high and low, melodies rise and fall, and pitch systematically biases perceived sound elevation. The origins of such mappings are unknown. Are they the result of physiological constraints, do they reflect natural environmental statistics, or are they truly arbitrary? We recorded natural sounds from the environment, analyzed the elevation-dependent filtering of the outer ear, and measured frequency-dependent biases in human sound localization. We find that auditory scene statistics reveals a clear mapping between frequency and elevation. Perhaps more interestingly, this natural statistical mapping is tightly mirrored in both ear-filtering properties and in perceived sound location. This suggests that both sound localization behavior and ear anatomy are fine-tuned to the statistics of natural auditory scenes, likely providing the basis for the spatial connotation of human hearing.

Why we are not all synesthetes (not even weakly so)

A little over a decade ago, Martino and Marks (Current Directions in Psychological Science 10:61-65, 2001) put forward the influential claim that cases of intuitive matchings between stimuli in different sensory modalities should be considered as a weak form of synesthesia. Over the intervening years, many other researchers have agreed-at the very least, implicitly-with this position (e.g., Bien, ten Oever, Goebel, & Sack NeuroImage 59:663-672, 2012; Eagleman Cortex 45:1266-1277, 2009; Esterman, Verstynen, Ivry, & Robertson Journal of Cognitive Neuroscience 18:1570-1576, 2006; Ludwig, Adachi, & Matzuzawa Proceedings of the National Academy of Sciences of the United States of America 108:20661-20665, 2011; Mulvenna & Walsh Trends in Cognitive Sciences 10:350-352, 2006; Sagiv & Ward 2006; Zellner, McGarry, Mattern-McClory, & Abreu Chemical Senses 33:211-222:2008). Here, though, we defend the separatist view, arguing that these cases are likely to form distinct kinds of phenomena despite their superficial similarities. We believe that crossmodal correspondences should be studied in their own right and not assimilated, either in terms of the name used or in terms of the explanation given, to synesthesia. To conflate these two phenomena is both inappropriate and potentially misleading. Below, we critically evaluate the evidence concerning the descriptive and constitutive features of crossmodal correspondences and synesthesia and highlight how they differ. Ultimately, we wish to provide a general definition of crossmodal correspondences as acquired, malleable, relative, and transitive pairings between sensory dimensions and to provide a framework in which to integrate the nonsystematic cataloguing of new cases of crossmodal correspondences, a tendency that has increased in recent years.

Auditory rhythms are systemically associated with spatial-frequency and density information in visual scenes

A variety of perceptual correspondences between auditory and visual features have been reported, but few studies have investigated how rhythm, an auditory feature defined purely by dynamics relevant to speech and music, interacts with visual features. Here, we demonstrate a novel crossmodal association between auditory rhythm and visual clutter. Participants were shown a variety of visual scenes from diverse categories and asked to report the auditory rhythm that perceptually matched each scene by adjusting the rate of amplitude modulation (AM) of a sound. Participants matched each scene to a specific AM rate with surprising consistency. A spatial-frequency analysis showed that scenes with greater contrast energy in midrange spatial frequencies were matched to faster AM rates. Bandpass-filtering the scenes indicated that greater contrast energy in this spatial-frequency range was associated with an abundance of object boundaries and contours, suggesting that participants matched more cluttered scenes to faster AM rates. Consistent with this hypothesis, AM-rate matches were strongly correlated with perceived clutter. Additional results indicated that both AM-rate matches and perceived clutter depend on object-based (cycles per object) rather than retinal (cycles per degree of visual angle) spatial frequency. Taken together, these results suggest a systematic crossmodal association between auditory rhythm, representing density in the temporal domain, and visual clutter, representing object-based density in the spatial domain. This association may allow for the use of auditory rhythm to influence how visual clutter is perceived and attended.

Rapid volitional control of apparent motion during percept generation

How rapidly can one voluntarily influence percept generation? The time course of voluntary visual-spatial attention is well studied, but the time course of intentional control over percept generation is relatively unknown. We investigated the latter question using "one-shot" apparent motion. When a vertical or horizontal pair of squares is replaced by its 90º-rotated version, the bottom-up signal is ambiguous. From this ambiguous signal, it is known that people can intentionally generate a percept of rotation in a desired direction (clockwise or counterclockwise). To determine the time course of this intentional control, we instructed participants to voluntarily induce rotation in a precued direction (clockwise rotation when a high-pitched tone was heard, and counterclockwise rotation when a low-pitched tone was heard), and then to report the direction of rotation that was actually perceived. We varied the delay between the instructional cue and the rotated frame (cue-lead time) from 0 to 1,067 ms. Intentional control became more effective with longer cue-lead times (asymptotically effective at 533 ms). Notably, intentional control was reliable even with a zero cue-lead time; control experiments ruled out response bias and the development of an auditory-visual association as explanations. This demonstrates that people can interpret an auditory cue and intentionally generate a desired motion percept surprisingly rapidly, entirely within the subjectively instantaneous moment in which the visual system constructs a percept of apparent motion.

How automatic are crossmodal correspondences?

The last couple of years have seen a rapid growth of interest (especially amongst cognitive psychologists, cognitive neuroscientists, and developmental researchers) in the study of crossmodal correspondences - the tendency for our brains (not to mention the brains of other species) to preferentially associate certain features or dimensions of stimuli across the senses. By now, robust empirical evidence supports the existence of numerous crossmodal correspondences, affecting people's performance across a wide range of psychological tasks - in everything from the redundant target effect paradigm through to studies of the Implicit Association Test, and from speeded discrimination/classification tasks through to unspeeded spatial localisation and temporal order judgment tasks. However, one question that has yet to receive a satisfactory answer is whether crossmodal correspondences automatically affect people's performance (in all, or at least in a subset of tasks), as opposed to reflecting more of a strategic, or top-down, phenomenon. Here, we review the latest research on the topic of crossmodal correspondences to have addressed this issue. We argue that answering the question will require researchers to be more precise in terms of defining what exactly automaticity entails. Furthermore, one's answer to the automaticity question may also hinge on the answer to a second question: Namely, whether crossmodal correspondences are all 'of a kind', or whether instead there may be several different kinds of crossmodal mapping (e.g., statistical, structural, and semantic). Different answers to the automaticity question may then be revealed depending on the type of correspondence under consideration. We make a number of suggestions for future research that might help to determine just how automatic crossmodal correspondences really are.