Crossmodal interference on counting performance: Evidence for shared attentional resources

During the act of counting, our perceptual system may rely on information coming from different sensory channels. However, when the information coming from different sources is discordant, such as in the case of a de-synchronization between visual stimuli to be counted and irrelevant auditory stimuli, the performance in a sequential counting task might deteriorate. Such deterioration may originate from two different mechanisms, both linked to exogenous attention attracted by auditory stimuli. Indeed, exogenous auditory triggers may infiltrate our internal "counter", interfering with the counting process, resulting in an overcount; alternatively, the exogenous auditory triggers may disrupt the internal "counter" by deviating participants' attention from the visual stimuli, resulting in an undercount. We tested these hypotheses by asking participants to count visual discs sequentially appearing on the screen while listening to task-irrelevant sounds, in systematically varied conditions: visual stimuli could be synchronized or de-synchronized with sounds; they could feature regular or irregular pacing; and their speed presentation could be fast (approx. 3/sec), moderate (approx. 2/sec), or slow (approx. 1.5/sec). Our results support the second hypothesis since participants tend to undercount visual stimuli in all harder conditions (de-synchronized, irregular, fast sequences). We discuss these results in detail, adding novel elements to the study of crossmodal interference.

On the Relative Nature of (Pitch-Based) Crossmodal Correspondences

This review deals with the question of the relative vs absolute nature of crossmodal correspondences, with a specific focus on those correspondences involving the auditory dimension of pitch. Crossmodal correspondences have been defined as the often-surprising crossmodal associations that people experience between features, attributes, or dimensions of experience in different sensory modalities, when either physically present, or else merely imagined. In the literature, crossmodal correspondences have often been contrasted with synaesthesia in that the former are frequently said to be relative phenomena (e.g., it is the higher-pitched of two sounds that is matched with the smaller of two visual stimuli, say, rather than there being a specific one-to-one crossmodal mapping between a particular pitch of sound and size of object). By contrast, in the case of synaesthesia, the idiosyncratic mapping between inducer and concurrent tends to be absolute (e.g., it is a particular sonic inducer that elicits a specific colour concurrent). However, a closer analysis of the literature soon reveals that the distinction between relative and absolute in the case of crossmodal correspondences may not be as clear-cut as some commentators would have us believe. Furthermore, it is important to note that the relative vs absolute question may receive different answers depending on the particular (class of) correspondence under empirical investigation.

Which Direction Is up for a High Pitch?

Low- and high-pitched sounds are perceptually associated with low and high visuospatial elevations, respectively. The spatial properties of this association are not well understood. Here we report two experiments that investigated whether low and high tones can be used as spatial cues to upright for self-orientation and identified the spatial frame(s) of reference used in perceptually binding auditory pitch to visuospatial 'up' and 'down'. In experiment 1, participants' perceptual upright (PU) was measured while lying on their right side with and without high- and low-pitched sounds played through speakers above their left ear and below their right ear. The sounds were ineffective in moving the perceived upright from a direction intermediate between the body and gravity towards the direction indicated by the sounds. In experiment 2, we measured the biasing effects of ascending and descending tones played through headphones on ambiguous vertical or horizontal visual motion created by combining gratings drifting in opposite directions while participants either sat upright or laid on their right side. Ascending and descending tones biased the interpretation of ambiguous motion along both the gravitational vertical and the long-axis of the body with the strongest effect along the body axis. The combination of these two effects showed that axis of maximum effect of sound corresponded approximately to the direction of the perceptual upright, compatible with the idea that 'high' and 'low' sounds are defined along this axis.

Cross-Sensory Correspondences: Heaviness is Dark and Low-Pitched

Everyday language reveals how stimuli encoded in one sensory feature domain can possess qualities normally associated with a different domain (e.g., higher pitch sounds are bright, light in weight, sharp, and thin). Such cross-sensory associations appear to reflect crosstalk among aligned (corresponding) feature dimensions, including brightness, heaviness, and sharpness. Evidence for heaviness being one such dimension is very limited, with heaviness appearing primarily as a verbal associate of other feature contrasts (e.g., darker objects and lower pitch sounds are heavier than their opposites). Given the presumed bidirectionality of the crosstalk between corresponding dimensions, heaviness should itself induce the cross-sensory associations observed elsewhere, including with brightness and pitch. Taking care to dissociate effects arising from the size and mass of an object, this is confirmed. When hidden objects varying independently in size and mass are lifted, objects that feel heavier are judged to be darker and to make lower pitch sounds than objects feeling less heavy. These judgements track the changes in perceived heaviness induced by the size-weight illusion. The potential involvement of language, natural scene statistics, and Bayesian processes in correspondences, and the effects they induce, is considered.

Synesthesia strengthens sound-symbolic cross-modal correspondences

Synesthesia is a phenomenon in which an experience in one domain is accompanied by an involuntary secondary experience in another, unrelated domain; in classical synesthesia, these associations are arbitrary and idiosyncratic. Cross-modal correspondences refer to universal associations between seemingly unrelated sensory features, e.g., auditory pitch and visual size. Some argue that these phenomena form a continuum, with classical synesthesia being an exaggeration of universal cross-modal correspondences, whereas others contend that the two are quite different, since cross-modal correspondences are non-arbitrary, non-idiosyncratic, and do not involve secondary experiences. Here, we used the implicit association test to compare synesthetes' and non-synesthetes' sensitivity to cross-modal correspondences. We tested the associations between auditory pitch and visual elevation, auditory pitch and visual size, and sound-symbolic correspondences between auditory pseudowords and visual shapes. Synesthetes were more sensitive than non-synesthetes to cross-modal correspondences involving sound-symbolic, but not low-level sensory, associations. We conclude that synesthesia heightens universally experienced cross-modal correspondences, but only when these involve sound symbolism. This is only partly consistent with the idea of a continuum between synesthesia and cross-modal correspondences, but accords with the idea that synesthesia is a high-level, post-perceptual phenomenon, with spillover of the abilities of synesthetes into domains outside their synesthesias. To our knowledge, this is the first demonstration that synesthetes, relative to non-synesthetes, experience stronger cross-modal correspondences outside their synesthetic domains.

Synaesthetic interactions across vision and audition

In everyday life our senses are exposed to a constant influx of sensory signals. The brain binds signals into a coherent percept based on temporal, spatial or semantic correspondences. In addition, synaesthetic correspondences may form important cues for multisensory binding. This study focussed on the synaesthetic correspondences between auditory pitch and visual size. While high pitch has been associated with small objects in static contexts, recent research has surprisingly found that increasing size is linked with rising pitch. The current study presented participants with small/large visual circles/discs together with high/low pitched pure tones in an intersensory selective attention paradigm. Whilst fixating a central cross participants discriminated between small and large visual size in the visual modality or between high and low pitch in the auditory modality. Across a series of five experiments, we observed convergent evidence that participants associated small visual size with low pitch and large visual size with high pitch. In other words, we observed the pitch-size mapping that has previously been observed only for dynamic contexts. We suggest that these contradictory findings may emerge because participants can interpret visual size as an index of permanent object size or distance (e.g. in motion) from the observer. Moreover, the pitch-size mapping may depend not only on relative but also on the absolute levels of pitch and size of the presented stimuli.

Crossmodal Correspondence Between Auditory Pitch and Visual Elevation Affects Temporal Ventriloquism

Temporal ventriloquism is the shift in perceived timing of a visual stimulus that occurs when an auditory stimulus is presented close in time. This study investigated whether crossmodal correspondence between auditory pitch and visual elevation modulates temporal ventriloquism. Participants were presented two visual stimuli (above and below fixation) across a range of stimulus onset asynchronies and were asked to judge the order of the events. A task-irrelevant auditory click was presented shortly before the first and another shortly after the second visual stimulus. There were two pitches used (low and high) and the congruency between the auditory and visual stimuli was manipulated. The results show that incongruent pairings between pitch and elevation abolish temporal ventriloquism. In contrast, the crossmodal correspondence effect was absent when the direction of the pitch change was fixed within sessions, reducing the saliency of the pitch change. The results support previous studies suggesting that in addition to spatial and temporal factors, crossmodal correspondences can influence binding of information across the senses, although these effects are likely to be dependent on the saliency of the crossmodal mapping.

Cross-Modal Correspondence Among Vision, Audition, and Touch in Natural Objects: An Investigation of the Perceptual Properties of Wood

Certain systematic relationships are often assumed between information conveyed from multiple sensory modalities; for instance, a small figure and a high pitch may be perceived as more harmonious. This phenomenon, termed cross-modal correspondence, may result from correlations between multi-sensory signals learned in daily experience of the natural environment. If so, we would observe cross-modal correspondences not only in the perception of artificial stimuli but also in perception of natural objects. To test this hypothesis, we reanalyzed data collected previously in our laboratory examining perceptions of the material properties of wood using vision, audition, and touch. We compared participant evaluations of three perceptual properties (surface brightness, sharpness of sound, and smoothness) of the wood blocks obtained separately via vision, audition, and touch. Significant positive correlations were identified for all properties in the audition–touch comparison, and for two of the three properties regarding in the vision–touch comparison. By contrast, no properties exhibited significant positive correlations in the vision–audition comparison. These results suggest that we learn correlations between multi-sensory signals through experience; however, the strength of this statistical learning is apparently dependent on the particular combination of sensory modalities involved.

Reliability and validity of the computerized Revised Token Test: comparison of reading and listening versions in persons with and without aphasia

PURPOSE

This study assessed the reliability and validity of intermodality associations and differences in persons with aphasia (PWA) and healthy controls (HC) on a computerized listening and 3 reading versions of the Revised Token Test (RTT; McNeil & Prescott, 1978).

METHOD

Thirty PWA and 30 HC completed the test versions, including a complete replication. Reading versions varied according to stimulus presentation method: (a) full-sentence presentation, (b) self-paced word-by-word full-sentence construction, and (c) self-paced word-by-word presentation with each word removed with the onset of the next word. Participants also received tests of aphasia and reading severity.

RESULTS

The listening version produced higher overall mean scores than each of the reading versions. Differences were small and within 1 standard error of measurement of each version. Overall score test-retest reliability among versions for PWA ranged from r=.89 to r=.97. Correlations between the listening and reading versions ranged from r=.79 to r=.85. All versions correlated highly with aphasia and reading severity. Correlations were generally low for the HC due to restricted variability. Factor analysis yielded a 2-factor solution for PWA and a single-factor for HC.

CONCLUSIONS

Intermodality differences were small, and all 4 versions were reliable, concurrently valid, and sensitive to similar linguistic processing difficulties in PWA.

Modeling violations of the race model inequality in bimodal paradigms: co-activation from decision and non-decision components

The redundant-signals paradigm (RSP) is designed to investigate response behavior in perceptual tasks in which response-relevant targets are defined by either one or two features, or modalities. The common finding is that responses are speeded for redundantly compared to singly defined targets. This redundant-signals effect (RSE) can be accounted for by race models if the response times do not violate the race model inequality (RMI). When there are violations of the RMI, race models are effectively excluded as a viable account of the RSE. The common alternative is provided by co-activation accounts, which assume that redundant target signals are integrated at some processing stage. However, "co-activation" has mostly been only indirectly inferred and the accounts have only rarely been explicitly modeled; if they were modeled, the RSE has typically been assumed to have a decisional locus. Yet, there are also indications in the literature that the RSE might originate, at least in part, at a non-decisional or motor stage. In the present study, using a distribution analysis of sequential-sampling models (ex-Wald and Ratcliff Diffusion model), the locus of the RSE was investigated for two bimodal (audio-visual) detection tasks that strongly violated the RMI, indicative of substantial co-activation. Three model variants assuming different loci of the RSE were fitted to the quantile reaction time proportions: a decision, a non-decision, and a combined variant both to vincentized group as well as individual data. The results suggest that for the two bimodal detection tasks, co-activation has a shared decisional and non-decisional locus. These findings point to the possibility that the mechanisms underlying the RSE depend on the specifics (task, stimulus, conditions, etc.) of the experimental paradigm.

Lower pitch is larger, yet falling pitches shrink

Experiments using diverse paradigms, including speeded discrimination, indicate that pitch and visually-perceived size interact perceptually, and that higher pitch is congruent with smaller size. While nearly all of these studies used static stimuli, here we examine the interaction of dynamic pitch and dynamic size, using Garner's speeded discrimination paradigm. Experiment 1 examined the interaction of continuous rise/fall in pitch and increase/decrease in object size. Experiment 2 examined the interaction of static pitch and size (steady high/low pitches and large/small visual objects), using an identical procedure. Results indicate that static and dynamic auditory and visual stimuli interact in opposite ways. While for static stimuli (Experiment 2), higher pitch is congruent with smaller size (as suggested by earlier work), for dynamic stimuli (Experiment 1), ascending pitch is congruent with growing size, and descending pitch with shrinking size. In addition, while static stimuli (Experiment 2) exhibit both congruence and Garner effects, dynamic stimuli (Experiment 1) present congruence effects without Garner interference, a pattern that is not consistent with prevalent interpretations of Garner's paradigm. Our interpretation of these results focuses on effects of within-trial changes on processing in dynamic tasks and on the association of changes in apparent size with implied changes in distance. Results suggest that static and dynamic stimuli can differ substantially in their cross-modal mappings, and may rely on different processing mechanisms.

Audiovisual correspondence between musical timbre and visual shapes

This article investigates the cross-modal correspondences between musical timbre and shapes. Previously, such features as pitch, loudness, light intensity, visual size, and color characteristics have mostly been used in studies of audio-visual correspondences. Moreover, in most studies, simple stimuli e.g., simple tones have been utilized. In this experiment, 23 musical sounds varying in fundamental frequency and timbre but fixed in loudness were used. Each sound was presented once against colored shapes and once against grayscale shapes. Subjects had to select the visual equivalent of a given sound i.e., its shape, color (or grayscale) and vertical position. This scenario permitted studying the associations between normalized timbre and visual shapes as well as some of the previous findings for more complex stimuli. One hundred and nineteen subjects (31 females and 88 males) participated in the online experiment. Subjects included 36 claimed professional musicians, 47 claimed amateur musicians, and 36 claimed non-musicians. Thirty-one subjects have also claimed to have synesthesia-like experiences. A strong association between timbre of envelope normalized sounds and visual shapes was observed. Subjects have strongly associated soft timbres with blue, green or light gray rounded shapes, harsh timbres with red, yellow or dark gray sharp angular shapes and timbres having elements of softness and harshness together with a mixture of the two previous shapes. Color or grayscale had no effect on timbre-shape associations. Fundamental frequency was not associated with height, grayscale or color. The significant correspondence between timbre and shape revealed by the present work allows designing substitution systems which might help the blind to perceive shapes through timbre.

Why we are not all synesthetes (not even weakly so)

A little over a decade ago, Martino and Marks (Current Directions in Psychological Science 10:61-65, 2001) put forward the influential claim that cases of intuitive matchings between stimuli in different sensory modalities should be considered as a weak form of synesthesia. Over the intervening years, many other researchers have agreed-at the very least, implicitly-with this position (e.g., Bien, ten Oever, Goebel, & Sack NeuroImage 59:663-672, 2012; Eagleman Cortex 45:1266-1277, 2009; Esterman, Verstynen, Ivry, & Robertson Journal of Cognitive Neuroscience 18:1570-1576, 2006; Ludwig, Adachi, & Matzuzawa Proceedings of the National Academy of Sciences of the United States of America 108:20661-20665, 2011; Mulvenna & Walsh Trends in Cognitive Sciences 10:350-352, 2006; Sagiv & Ward 2006; Zellner, McGarry, Mattern-McClory, & Abreu Chemical Senses 33:211-222:2008). Here, though, we defend the separatist view, arguing that these cases are likely to form distinct kinds of phenomena despite their superficial similarities. We believe that crossmodal correspondences should be studied in their own right and not assimilated, either in terms of the name used or in terms of the explanation given, to synesthesia. To conflate these two phenomena is both inappropriate and potentially misleading. Below, we critically evaluate the evidence concerning the descriptive and constitutive features of crossmodal correspondences and synesthesia and highlight how they differ. Ultimately, we wish to provide a general definition of crossmodal correspondences as acquired, malleable, relative, and transitive pairings between sensory dimensions and to provide a framework in which to integrate the nonsystematic cataloguing of new cases of crossmodal correspondences, a tendency that has increased in recent years.

Crossmodal interactions during affective picture processing

"Natural" crossmodal correspondences, such as the spontaneous tendency to associate high pitches with high spatial locations, are often hypothesized to occur preattentively and independently of task instructions (top-down attention). Here, we investigate bottom-up attentional engagement by using emotional scenes that are known to naturally and reflexively engage attentional resources. We presented emotional (pleasant and unpleasant) or neutral pictures either below or above a fixation cross, while participants were required to discriminate between a high or a low pitch tone (experiment 1). Results showed that despite a robust crossmodal attentional capture of task-irrelevant emotional pictures, the general advantage in classifying the tones for congruent over incongruent visual-auditory stimuli was similar for emotional and neutral pictures. On the other hand, when picture position was task-relevant (experiment 2), task-irrelevant tones did not interact with pictures with regard to their combination of pitch and visual vertical spatial position, but instead they were effective in minimizing the interference effect of emotional picture processing on the ongoing task. These results provide constraints on our current understanding of natural crossmodal correspondences.

Brain deactivation in the outperformance in bimodal tasks: an FMRI study

While it is known that some individuals can effectively perform two tasks simultaneously, other individuals cannot. How the brain deals with performing simultaneous tasks remains unclear. In the present study, we aimed to assess which brain areas corresponded to various phenomena in task performance. Nineteen subjects were requested to sequentially perform three blocks of tasks, including two unimodal tasks and one bimodal task. The unimodal tasks measured either visual feature binding or auditory pitch comparison, while the bimodal task required performance of the two tasks simultaneously. The functional magnetic resonance imaging (fMRI) results are compatible with previous studies showing that distinct brain areas, such as the visual cortices, frontal eye field (FEF), lateral parietal lobe (BA7), and medial and inferior frontal lobe, are involved in processing of visual unimodal tasks. In addition, the temporal lobes and Brodmann area 43 (BA43) were involved in processing of auditory unimodal tasks. These results lend support to concepts of modality-specific attention. Compared to the unimodal tasks, bimodal tasks required activation of additional brain areas. Furthermore, while deactivated brain areas were related to good performance in the bimodal task, these areas were not deactivated where the subject performed well in only one of the two simultaneous tasks. These results indicate that efficient information processing does not require some brain areas to be overly active; rather, the specific brain areas need to be relatively deactivated to remain alert and perform well on two tasks simultaneously. Meanwhile, it can also offer a neural basis for biofeedback in training courses, such as courses in how to perform multiple tasks simultaneously.

Crossmodal correspondences between sounds and tastes

In this article, the rapidly growing body of research that has been published recently on the topic of crossmodal correspondences that involve auditory and gustatory/flavor stimuli is critically reviewed. The evidence demonstrates that people reliably match different tastes/flavors to auditory stimuli varying in both their psychoacoustic (e.g., pitch) and musical (e.g., timbre) properties. In order to stimulate further progress in this relatively young research field, the present article aims at consolidating prior findings concerning specific auditory-gustatory mappings, whereby special attention is given to highlighting (1) any conflicts in the existing experimental evidence and (2) any potential caveats with regard to the most appropriate interpretation of prior studies. Next, potential mechanisms underlying auditory-gustatory crossmodal correspondences are discussed. Finally, a number of potentially fruitful avenues for future research are outlined.

Cross-modality correspondence between pitch and spatial location modulates attentional orienting

The brain constantly integrates incoming signals across the senses to form a cohesive view of the world. Most studies on multisensory integration concern the roles of spatial and temporal parameters. However, recent findings suggest cross-modal correspondences (eg high-pitched sounds associated with bright, small objects located high up) also affect multisensory integration. Here, we focus on the association between auditory pitch and spatial location. Surprisingly little is known about the cognitive and perceptual roots of this phenomenon, despite its long use in ergonomic design. In a series of experiments, we explore how this cross-modal mapping affects the allocation of attention with an attentional cuing paradigm. Our results demonstrate that high and low tones induce attention shifts to upper or lower locations, depending on pitch height. Furthermore, this pitch-induced cuing effect is susceptible to contextual manipulations and volitional control. These findings suggest the cross-modal interaction between pitch and location originates from an attentional level rather than from response mapping alone. The flexible contextual mapping between pitch and location, as well as its susceptibility to top-down control, suggests the pitch-induced cuing effect is primarily mediated by cognitive processes after initial sensory encoding and occurs at a relatively late stage of voluntary attention orienting.

Re-evaluating visual and auditory dominance through modality switching costs and congruency analyses

Competition between the senses can lead to modality dominance, where one sense influences multi-modal processing to a greater degree than another. Modality dominance can be influenced by task demands, speeds of processing, contextual influence and practice. To resolve previous discrepancies in these factors, we assessed modality dominance in an audio-visual paradigm controlling for the first three factors while manipulating the fourth. Following a uni-modal task in which auditory and visual processing were equated, participants completed a pre-practice selective attention bimodal task in which the congruency relationship and task-relevant modality changed across trials. Participants were given practice in one modality prior to completing a post-practice selective attention bimodal task similar to the first. The effects of practice were non-specific as participants were speeded post-practice relative to pre-practice. Congruent stimuli relative to incongruent stimuli, also led to increased processing efficiency. RT data tended to reveal symmetric modality switching costs whereas the error rate data tended to reveal asymmetric modality switching costs in which switching from auditory to visual processing was particularly costly. The data suggest that when a number of safeguards are put in place to equate auditory and visual responding as far as possible, evidence for an auditory advantage can arise.

Implicit phonetic symbolism in voicing of consonants and visual lightness using Garner's speeded classification task

The present study examines implicit phonetic symbolism which posits that arbitrary linguistic sound is related to certain aspects of characteristics of other modalities, such as color, touch, or emotion. In consonant discrimination and lightness discrimination using Garner's speeded classification paradigm, spoken sounds (voiced/voiceless consonants) and lightness of visual stimuli (black/white squares) were systematically varied to assess cross-modal interactions. Congruent audio-visual pairs (voiced consonants and black, and between voiceless consonants and white) facilitated consonant discrimination. In lightness discrimination, no congruent facilitation or congruence effect was observed. These results indicated that cross-modal interactions in implicit phonetic symbolism can be found in correlations between linguistic spoken sounds and visual lightness.

Associations between Auditory Pitch and Visual Elevation Do Not Depend on Language: Evidence from a Remote Population

Associations between auditory pitch and visual elevation are widespread in many languages, and behavioral associations have been extensively documented between height and pitch among speakers of those languages. However, it remains unclear whether perceptual correspondences between auditory pitch and visual elevation inform these linguistic associations, or merely reflect them. We probed this cross-modal mapping in members of a remote Kreung hill tribe in northeastern Cambodia who do not use spatial language to describe pitch. Participants viewed shapes rising or falling in space while hearing sounds either rising or falling in pitch, and reported on the auditory change. Associations between pitch and vertical position in the Kreung were similar to those demonstrated in populations where pitch is described in terms of spatial height. These results suggest that associations between visual elevation and auditory pitch can arise independently of language. Thus, widespread linguistic associations between pitch and elevation may reflect universally predisposed perceptual correspondences.

Crossmodal correspondences: a tutorial review

In many everyday situations, our senses are bombarded by many different unisensory signals at any given time. To gain the most veridical, and least variable, estimate of environmental stimuli/properties, we need to combine the individual noisy unisensory perceptual estimates that refer to the same object, while keeping those estimates belonging to different objects or events separate. How, though, does the brain "know" which stimuli to combine? Traditionally, researchers interested in the crossmodal binding problem have focused on the roles that spatial and temporal factors play in modulating multisensory integration. However, crossmodal correspondences between various unisensory features (such as between auditory pitch and visual size) may provide yet another important means of constraining the crossmodal binding problem. A large body of research now shows that people exhibit consistent crossmodal correspondences between many stimulus features in different sensory modalities. For example, people consistently match high-pitched sounds with small, bright objects that are located high up in space. The literature reviewed here supports the view that crossmodal correspondences need to be considered alongside semantic and spatiotemporal congruency, among the key constraints that help our brains solve the crossmodal binding problem.

Changes in auditory frequency guide visual-spatial attention

How do the characteristics of sounds influence the allocation of visual-spatial attention? Natural sounds typically change in frequency. Here we demonstrate that the direction of frequency change guides visual-spatial attention more strongly than the average or ending frequency, and provide evidence suggesting that this cross-modal effect may be mediated by perceptual experience. We used a Go/No-Go color-matching task to avoid response compatibility confounds. Participants performed the task either with their heads upright or tilted by 90°, misaligning the head-centered and environmental axes. The first of two colored circles was presented at fixation and the second was presented in one of four surrounding positions in a cardinal or diagonal direction. Either an ascending or descending auditory-frequency sweep was presented coincident with the first circle. Participants were instructed to respond to the color match between the two circles and to ignore the uninformative sounds. Ascending frequency sweeps facilitated performance (response time and/or sensitivity) when the second circle was presented at the cardinal top position and descending sweeps facilitated performance when the second circle was presented at the cardinal bottom position; there were no effects of the average or ending frequency. The sweeps had no effects when circles were presented at diagonal locations, and head tilt entirely eliminated the effect. Thus, visual-spatial cueing by pitch change is narrowly tuned to vertical directions and dominates any effect of average or ending frequency. Because this cross-modal cueing is dependent on the alignment of head-centered and environmental axes, it may develop through associative learning during waking upright experience.

Decomposing the Garner interference paradigm: evidence for dissociations between macrolevel and microlevel performance

Three Garner interference experiments are described in which baseline, filtering, and correlated performance were assessed at both a macrolevel (condition average) and microlevel (intertrial contingency), using the pair-wise combinations of auditory pitch, loudness, and location. Discrepancies between pairs of dimensions were revealed between macro- and microlevel estimates of performance and, also, between filtering costs and correlated benefits, relative to baseline. The examination of the intertrial effects associated with filtering costs suggested that effects of increased stimulus uncertainty were mandatory, whereas effects of irrelevant variation were not. The examination of the intertrial effects associated with correlated benefits suggested that the detection of stimulus repetition took precedence over that of stimulus change. Violations of standard horse race accounts of processing did not appear to stem from differences in the absolute or relative speeds of processing between dimensions but, rather, from the special role that certain dimensions (e.g., pitch) may play in certain modalities (e.g., audition). The utility of examining repetition effects is demonstrated by revealing a level of understanding regarding stimulus processing typically hidden by aggregated measures of performance.

Trial after trial: General processing consequences as a function of repetition and change in multidimensional sound

While there are pointers relating to the consequences of repetition, a general framework regarding the cognitive implications of processing multidimensional stimuli as a function of previous stimulus history is currently lacking. Three experiments using sounds varying in location and pitch were carried out, in which the immediate consequences of repeating or changing task-relevant and task-irrelevant attributes were orthogonally examined. A consistent pattern of data was shown, in that the magnitude of selective attention failure was larger when the task-relevant value repeated across trials, while differences between dimensions were larger when the task-relevant value changed across trials. These effects of irrelevance and dimension as a function of intertrial contingency are summarized in a model depicting the dynamic allocation of processing resource.

Multisensory synesthetic interactions in the speeded classification of visual size

In the present study, we attempted to demonstrate a synesthetic relationship between auditory frequency and visual size. In Experiment 1, participants performed a speeded visual size discrimination task in which they had to judge whether a variable-sized disk was bigger or smaller than a standard reference disk. A task-irrelevant sound that was either synesthetically congruent with the relative size of the disk (e.g., a low-frequency sound presented with a bigger disk) or synesthetically incongruent with it (e.g., a low-frequency sound presented with a smaller disk) was sometimes presented together with the variable disk. Reaction times were shorter in the synesthetically congruent condition than in the incongruent condition. Verbal labeling and semantic mediation interpretations of this interaction were explored in Experiment 2, in which high- and low-frequency sounds were presented in separate blocks of trials, and in Experiment 3, in which the tones were replaced by the spoken words "high" and "low." Response priming/bias explanations were ruled out in Experiment 4, in which a synesthetic congruency effect was still reported even when participants made same-versus-different discrimination responses regarding the relative sizes of the two disks. Taken together, these results provide the first empirical demonstration that the relative frequency of an irrelevant sound can influence the speed with which participants judge the size of visual stimuli when the sound varies on a trial-by-trial basis along a synesthetically compatible dimension. The possible cognitive bases for this synesthetic association are also discussed.

Presentation format effects in working memory: The role of attention

Four experiments are reported in which participants attempted to remember three or six concrete nouns, presented as pictures, spoken words, or printed words, while also verifying the accuracy of sentences. Hypotheses meant to explain the higher recall of pictures and spoken words over printed words were tested. Increasing the difficulty and changing the type of processing task from arithmetic to a visual/spatial reasoning task did not influence recall. An examination of long-term modality effects showed that those effects were not sufficient to explain the superior performance with spoken words and pictures. Only when we manipulated the allocation of attention to the items in the storage task by requiring the participants to articulate the items and by presenting the stimulus items under a degraded condition were we able to reduce or remove the effect of presentation format. The findings suggest that the better recall of pictures and spoken words over printed words result from the fact that under normal presentation conditions, printed words receive less processing attention than pictures and spoken words do.