Cross-form conceptual relations between sounds and words: effects on the novelty P3

An early perceptual locus of absolute pitch

Absolute pitch (AP) refers to the naming of musical tone without external reference. The influential two-component model states that AP is limited by the late-emerging pitch labeling process only and not the earlier perceptual and memory processes. Over the years, however, support for this model at the neural level has been mixed with various methodological limitations. Here, the electroencephalography responses of 27 AP possessors and 27 non-AP possessors were recorded. During both name verification and passive listening, event-related potential analyses showed a difference between AP and non-AP possessors at about 200 ms in their response toward tones compared with noise stimuli. Multivariate pattern analyses suggested that pitch naming was subserved by a series of transient processes for the first 250 ms, followed by a stage-like process for both AP and non-AP possessors with no group differences between them. These findings are inconsistent with the predictions of the two-component model, and instead suggest the existence of an early perceptual locus of AP.

Multimodal feature binding in object memory retrieval using event-related potentials: Implications for models of semantic memory

To test the hypothesis that semantic processes are represented in multiple subsystems, we recorded electroencephalogram (EEG) as we elicited object memories using the modified Semantic Object Retrieval Test, during which an object feature, presented as a visual word [VW], an auditory word [AW], or a picture [Pic], was followed by a second feature always presented as a visual word. We performed both hypothesis-driven and data-driven analyses using event-related potentials (ERPs) time locked to the second stimulus. We replicated a previously reported left fronto-temporal ERP effect (750-1000 ms post-stimulus) in the VW task, and also found that this ERP component was only present during object memory retrieval in verbal (VW, AW) as opposed to non-verbal (Pic) stimulus types. We also found a right temporal ERP effect (850-1000 ms post-stimulus) that was present in auditory (AW) but not in visual (VW, Pic) stimulus types. In addition, we found an earlier left temporo-parietal ERP effect between 350 and 700 ms post-stimulus and a later midline parietal ERP effect between 700 and 1100 ms post-stimulus, present in all stimulus types, suggesting common neural mechanisms for object retrieval processes and object activation, respectively. These findings support multiple semantic subsystems that respond to varying stimulus modalities, and argue against an ultimate unitary amodal semantic analysis.

Distraction by deviant sounds: disgusting and neutral words capture attention to the same extent

Several studies have argued that words evoking negative emotions, such as disgust, grab attention more than neutral words, and leave traces in memory that are more persistent. However, these conclusions are typically based on tasks requiring participants to process the semantic content of these words in a voluntarily manner. We sought to compare the involuntary attention grabbing power of disgusting and neutral words using them as rare and unexpected auditory distractors in a cross-modal oddball task, and then probing the participants’ memory for these stimuli in a surprise recognition task. Frequentist and Bayesian analyses converged to show that, compared to a standard tone, disgusting and neutral auditory words produced significant but equivalent levels of distraction in a visual categorization task, that they elicited comparable levels of memory discriminability in the incidental recognition task, and that the participants’ individual sensitivity to disgust did not influence the results. Our results suggest that distraction by unexpected words is not modulated by their emotional valence, at least when these words are task-irrelevant and are temporally and perceptually decoupled from the target stimuli.

Distraction by deviant sounds during reading: An eye-movement study

Oddball studies have shown that sounds unexpectedly deviating from an otherwise repeated sequence capture attention away from the task at hand. While such distraction is typically regarded as potentially important in everyday life, previous work has so far not examined how deviant sounds affect performance on more complex daily tasks. In this study, we developed a new method to examine whether deviant sounds can disrupt reading performance by recording participants’ eye movements. Participants read single sentences in silence and while listening to task-irrelevant sounds. In the latter condition, a 50-ms sound was played contingent on the fixation of five target words in the sentence. On most occasions, the same tone was presented (standard sound), whereas on rare and unexpected occasions it was replaced by white noise (deviant sound). The deviant sound resulted in significantly longer fixation durations on the target words relative to the standard sound. A time-course analysis showed that the deviant sound began to affect fixation durations around 180 ms after fixation onset. Furthermore, deviance distraction was not modulated by the lexical frequency of target words. In summary, fixation durations on the target words were longer immediately after the presentation of the deviant sound, but there was no evidence that it interfered with the lexical processing of these words. The present results are in line with the recent proposition that deviant sounds yield a temporary motor suppression and suggest that deviant sounds likely inhibit the programming of the next saccade.

Food words distract the hungry: Evidence of involuntary semantic processing of task-irrelevant but biologically-relevant unexpected auditory words

Rare changes in a stream of otherwise repeated task-irrelevant sounds break through selective attention and disrupt performance in an unrelated visual task by triggering shifts of attention to and from the deviant sound (deviance distraction). Evidence indicates that the involuntary orientation of attention to unexpected sounds is followed by their semantic processing. However, past demonstrations relied on tasks in which the meaning of the deviant sounds overlapped with features of the primary task. Here we examine whether such processing is observed when no such overlap is present but sounds carry some relevance to the participants’ biological need to eat when hungry. We report the results of an experiment in which hungry and satiated participants partook in a cross-modal oddball task in which they categorized visual digits (odd/even) while ignoring task-irrelevant sounds. On most trials the irrelevant sound was a sinewave tone (standard sound). On the remaining trials, deviant sounds consisted of spoken words related to food (food deviants) or control words (control deviants). Questionnaire data confirmed state (but not trait) differences between the two groups with respect to food craving, as well as a greater desire to eat the food corresponding to the food-related words in the hungry relative to the satiated participants. The results of the oddball task revealed that food deviants produced greater distraction (longer response times) than control deviants in hungry participants while the reverse effect was observed in satiated participants. This effect was observed in the first block of trials but disappeared thereafter, reflecting semantic saturation. Our results suggest that (1) the semantic content of deviant sounds is involuntarily processed even when sharing no feature with the primary task; and that (2) distraction by deviant sounds can be modulated by the participants’ biological needs.

Conceptual priming with pictures and environmental sounds

A series of experiments was conducted to examine conceptual priming within and across modalities with pictures and environmental sounds. In Experiment 1, we developed a new multimodal stimulus set consisting of two picture and sound exemplars that represented 80 object items. In Experiments 2, we investigated whether categorization of the stimulus items would be facilitated by picture and environmental sound primes that were derived from different exemplars of the target items; and in Experiments 3 and 4, we tested the additional influence on priming when trials were consolidated within a target modality and the inter stimulus interval was lengthened. The results demonstrated that target categorization was facilitated by the advanced presentation of conceptually related exemplars, but there were differences in effectiveness when pictures and sounds appeared as primes.

Perceptual comparisons with laterally presented pictures and environmental sounds

Two experiments studied perceptual comparisons with cues that vary in one of four ways (picture, sound, spoken word, or printed word) and with targets that are either pictures or environmental sounds. The basic question probed whether modality or differences in format were factors that would influence picture and sound perception. Also of interest were cue effect differences when targets are presented on either the right or left side. Students responded to a same-different reaction time task that entailed matching cue-target pairs to determine whether the successive stimulus events represented features drawn from the same basic item. Cue type influenced reaction times to pictures and environmental sounds, but the effects were qualified by response type and with picture targets by presentation side. These results provide some additional evidence of processing asymmetry when pictures are directed to either the right or left hemisphere, as well as for some asymmetries in cross-modality cuing. Implications of these findings for theories of multisensory processing and models of object recognition are discussed.

Dissociation between the activity of the right middle frontal gyrus and the middle temporal gyrus in processing semantic priming

The aim of this event-related functional magnetic resonance imaging (fMRI) study was to test whether the right middle frontal gyrus (MFG) and middle temporal gyrus (MTG) would show differential sensitivity to the effect of prime-target association strength on repetition priming. In the experimental condition (RP), the target occurred after repetitive presentation of the prime within an oddball design. In the control condition (CTR), the target followed a single presentation of the prime with equal probability of the target as in RP. To manipulate semantic overlap between the prime and the target both conditions (RP and CTR) employed either the onomatopoeia "oink" as the prime and the referent "pig" as the target (OP) or vice-versa (PO) since semantic overlap was previously shown to be greater in OP. The results showed that the left MTG was sensitive to release of adaptation while both the right MTG and MFG were sensitive to sequence regularity extraction and its verification. However, dissociated activity between OP and PO was revealed in RP only in the right MFG. Specifically, target "pig" (OP) and the physically equivalent target in CTR elicited comparable deactivations whereas target "oink" (PO) elicited less inhibited response in RP than in CTR. This interaction in the right MFG was explained by integrating these effects into a competition model between perceptual and conceptual effects in priming processing.

Contribution of subregions of human frontal cortex to novelty processing

Novelty processing was studied in patients with lesions centered in either OFC or lateral pFC (LPFC). An auditory novelty oddball ERP paradigm was applied with environmental sounds serving as task irrelevant novel stimuli. Lesions to the LPFC as well as the OFC resulted in a reduction of the frontal Novelty P3 response, supporting a key role of both frontal subdivisions in novelty processing. The posterior P3b to target sounds was unaffected in patients with frontal lobe lesions in either location, indicating intact posterior cortical target detection mechanisms. LPFC patients displayed an enhanced sustained negative slow wave (NSW) to novel sounds not observed in OFC patients, indicating prolonged resource allocation to task-irrelevant stimuli after LPFC damage. Both patient groups displayed an enhanced NSW to targets relative to controls. However, there was no difference in behavior between patients and controls suggesting that the enhanced NSW to targets may index an increased resource allocation to response requirements enabling comparable performance in the frontal lesioned patients. The current findings indicate that the LPFC and OFC have partly shared and partly differential contributions to the cognitive subcomponents of novelty processing.

Perceptual and conceptual priming of environmental sounds

It is still unknown whether sonic environments influence the processing of individual sounds in a similar way as discourse or sentence context influences the processing of individual words. One obstacle to answering this question has been the failure to dissociate perceptual (i.e., how similar are sonic environment and target sound?) and conceptual (i.e., how related are sonic environment and target?) priming effects. In this study, we dissociate these effects by creating prime-target pairs with a purely perceptual or both a perceptual and conceptual relationship. Perceptual prime-target pairs were derived from perceptual-conceptual pairs (i.e., meaningful environmental sounds) by shuffling the spectral composition of primes and targets so as to preserve their perceptual relationship while making them unrecognizable. Hearing both original and shuffled targets elicited a more positive N1/P2 complex in the ERP when targets were related to a preceding prime as compared with unrelated. Only related original targets reduced the N400 amplitude. Related shuffled targets tended to decrease the amplitude of a late temporo-parietal positivity. Taken together, these effects indicate that sonic environments influence first the perceptual and then the conceptual processing of individual sounds. Moreover, the influence on conceptual processing is comparable to the influence linguistic context has on the processing of individual words.

Infants' recognition of meaningful verbal and nonverbal sounds

To examine how young children recognize the association between two different types of meaningful sounds and their visual referents, we compared 15-, 20-, and 25-month-old infants' looking time responses to familiar naturalistic environmental sounds, (e.g., the sound of a dog barking) and their empirically matched verbal descriptions (e.g., "Dog barking") in an intermodal preferential looking paradigm. Across all three age groups, performance was indistinguishable over the two domains. Infants with the largest vocabularies were more accurate in processing the verbal phrases than the environmental sounds. However, after taking into account each child's verbal comprehension/production and the onomatopoetic test items, all cross-domain differences disappeared. Correlational analyses revealed that the processing of environmental sounds was tied to chronological age, while the processing of speech was linked to verbal proficiency. Overall, while infants' ability to recognize the two types of sounds did not differ behaviorally, the underlying processes may differ depending on the type of auditory input.

Neurophysiological evidence for categorical perception of color

The aim of this investigation was to examine the time course and the relative contributions of perceptual and post-perceptual processes to categorical perception (CP) of color. A visual oddball task was used with standard and deviant stimuli from same (within-category) or different (between-category) categories, with chromatic separations for within- and between-category stimuli equated in Munsell Hue. CP was found on a behavioral version of the task, with faster RTs and greater accuracy for between- compared to within-category stimuli. On a neurophysiological version of the task, event-related potentials (ERPs) showed earlier latencies for P1 and N1 components at posterior locations to between- relative to within-category deviants, providing novel evidence for early perceptual processes on color CP. Enhanced P2 and P3 waves were also found for between- compared to within-category stimuli, indicating a role for later post-perceptual processes.

What is Involved and What is Necessary for Complex Linguistic and Nonlinguistic Auditory Processing: Evidence from Functional Magnetic Resonance Imaging and Lesion Data

We used functional magnetic resonance imaging (fMRI) in conjunction with a voxel-based approach to lesion symptom mapping to quantitatively evaluate the similarities and differences between brain areas involved in language and environmental sound comprehension. In general, we found that language and environmental sounds recruit highly overlapping cortical regions, with cross-domain differences being graded rather than absolute. Within language-based regions of interest, we found that in the left hemisphere, language and environmental sound stimuli evoked very similar volumes of activation, whereas in the right hemisphere, there was greater activation for environmental sound stimuli. Finally, lesion symptom maps of aphasic patients based on environmental sounds or linguistic deficits [Saygin, A. P., Dick, F., Wilson, S. W., Dronkers, N. F., & Bates, E. Shared neural resources for processing language and environmental sounds: Evidence from aphasia. Brain, 126, 928–945, 2003] were generally predictive of the extent of blood oxygenation level dependent fMRI activation across these regions for sounds and linguistic stimuli in young healthy subjects.

Age and novelty: Event-related potentials to visual stimuli within an auditory oddball—visual detection task

Age-related change of event-related potentials to novel visual stimuli was investigated while participants attended to both auditory and visual stimulation. Meaningful but irrelevant pictures (novel stimuli) were presented to younger (mean=21.8, range=18-26 years) and older (mean=70.0, range=60-78) participants (10 in each group). The participants were performing an auditory oddball task and counting silently the changes of a visually presented letter. In the younger group novel stimuli elicited a posterior positivity in the 220-255 ms range. This component habituated to the repetition of the same picture. In the older group this component had longer latency, and did not habituate. A later positivity had shorter latency and larger amplitude in the younger group, but this positivity was preceded by a negative component (N2b) only in the elderly. Results show decreased sensitivity to the content of the visual stimuli in an earlier stage of novelty processing in the elderly, and the age-related slowing of both orientation-related and task-related processes.

Age and novelty: Event-related brain potentials and autonomic activity

Our aim was to study age-related differences in the habituation of orienting reaction by using novel visual stimuli. We intended to fill a gap in habituation research by recording both autonomic and ERP components of orienting to visual stimuli in the same sample and in highly related paradigms. We report data showing that in young subjects repetition of visual novels yielded fast habituation of both skin conductance responses and ERP components (P3(novel), N2b) whereas elderly people displayed no sign of habituation. However, cardiac deceleration--thought conventionally to be part of the orienting reaction--did not habituate in either group. Overall, most of our results harmonize with those obtained by using auditory stimuli; therefore we conclude that there is no significant modality specificity in age-related deterioration of habituation processes.

Event-related potentials as an index of similarity between words and pictures

This report examines the ERP correlates of processing nontarget stimuli that are conceptually, but not perceptually, similar to a target. In two studies, the P300 component was examined in healthy adults during a multistimulus oddball paradigm. The stimuli were pictures of five objects and their five corresponding names. Participants were required to keep a mental count of number of target presentations. In Experiment 1, the target was the word "globe." Both the word target and the nontarget picture of the globe elicited large P300s, though the P300 to the picture was smaller in amplitude. No other stimulus elicited a prominent P300. In Experiment 2, the target was the picture of the globe, and the word was considered to be the related nontarget. Again, the target elicited a large P300. However, in this case the related nontarget stimulus failed to elicit a P300. The relevance of the data to theories of word/picture processing is discussed.