Sequential neural processes of tactile-visual crossmodal working memory

Crossmodal Associations and Working Memory in the Brain

Crossmodal associations between stimuli from different sensory modalities could emerge in non-synesthetic people and be stored in working memory to guide goal-directed behaviors. This chapter reviews a plethora of studies in this field to summarize where, when, and how crossmodal associations and working memory are processed. It has been found that in those brain regions that are traditionally considered as unimodal primary sensory areas, neural activity could be influenced by crossmodal sensory signals at temporally very early stage of information processing. This phenomenon could not be due to feedback projections from higher level associative areas. Sequentially, neural processes would then occur in associative cortical areas including the posterior parietal cortex and prefrontal cortex. Neural oscillations in multiple frequency bands may reflect brain activity in crossmodal associations, and it is likely that neural synchrony is related to potential neural mechanisms underlying these processes. Primary sensory areas and associative areas coordinate together through neural synchrony to fulfil crossmodal associations and to guide working memory performance.

Dorsolateral prefrontal cortex bridges bilateral primary somatosensory cortices during cross-modal working memory

Neural activity in the dorsolateral prefrontal cortex (DLPFC) has been suggested to integrate information from distinct sensory areas. However, how the DLPFC interacts with the bilateral primary somatosensory cortices (SIs) in tactile-visual cross-modal working memory has not yet been established. In the present study, we applied single-pulse transcranial magnetic stimulation (sp-TMS) over the contralateral DLPFC and bilateral SIs of human participants at various time points, while they performed a tactile-visual delayed matching-to-sample task with a 2-second delay. sp-TMS over the contralateral DLPFC or the contralateral SI at either an sensory encoding stage [i.e. 100 ms after the onset of a vibrotactile sample stimulus (200-ms duration)] or an early maintenance stage (i.e. 300 ms after the onset), significantly impaired the accuracy of task performance; sp-TMS over the contralateral DLPFC or the ipsilateral SI at a late maintenance stage (1600 ms and 1900 ms) also significantly disrupted the performance. Furthermore, at 300 ms after the onset of the vibrotactile sample stimulus, there was a significant correlation between the deteriorating effects of sp-TMS over the contralateral SI and the contralateral DLPFC. These results imply that the DLPFC and the bilateral SIs play causal roles at distinctive stages during cross-modal working memory, while the contralateral DLPFC communicates with the contralateral SI in the early delay, and cooperates with the ipsilateral SI in the late delay.

Neural correlates of visuo-tactile crossmodal paired-associate learning and memory in humans

Studies have indicated that a cortical sensory system is capable of processing information from different sensory modalities. However, it still remains unclear when and how a cortical system integrates and retains information across sensory modalities during learning. Here we investigated the neural dynamics underlying crossmodal associations and memory by recording event-related potentials (ERPs) when human participants performed visuo-tactile (crossmodal) and visuo-visual (unimodal) paired-associate (PA) learning tasks. In a trial of the tasks, the participants were required to explore and learn the relationship (paired or non-paired) between two successive stimuli. EEG recordings revealed dynamic ERP changes during participants' learning of paired-associations. Specifically, (1) the frontal N400 component showed learning-related changes in both unimodal and crossmodal tasks but did not show any significant difference between these two tasks, while the central P400 displayed both learning changes and task differences; (2) a late posterior negative slow wave (LPN) showed the learning effect only in the crossmodal task; (3) alpha-band oscillations appeared to be involved in crossmodal working memory. Additional behavioral experiments suggested that these ERP components were not relevant to the participants' familiarity with stimuli per se. Further, by shortening the delay length (from 1300ms to 400ms or 200 ms) between the first and second stimulus in the crossmodal task, declines in participants' task performance were observed accordingly. Taken together, these results provide insights into the cortical plasticity (induced by PA learning) of neural networks involved in crossmodal associations in working memory.

Neural correlates of heat-evoked pain memory in humans

The neural processes underlying pain memory are not well understood. To explore these processes, contact heat-evoked potentials (CHEPs) were recorded in humans with electroencephalography (EEG) technique during a delayed matching-to-sample task, a working memory task involving presentations of two successive painful heat stimuli (S-1 and S-2) with different intensities separated by a 2-s interval (the memorization period). At the end of the task, the subject was required to discriminate the stimuli by indicating which (S-1 or S-2) induced more pain. A control task was used, in which no active discrimination was required between stimuli. All event-related potential (ERP) analysis was aligned to the onset of S-1. EEG activity exhibited two successive CHEPs: an N2-P2 complex (∼400 ms after onset of S-1) and an ultralate component (ULC, ∼900 ms). The amplitude of the N2-P2 at vertex, but not the ULC, was significantly correlated with stimulus intensity in these two tasks, suggesting that the N2-P2 represents neural coding of pain intensity. A late negative component (LNC) in the frontal recording region was observed only in the memory task during a 500-ms period before onset of S-2. LNC amplitude differed between stimulus intensities and exhibited significant correlations with the N2-P2 complex. These indicate that the frontal LNC is involved in maintenance of intensity of pain in working memory. Furthermore, alpha-band oscillations observed in parietal recording regions during the late delay displayed significant power differences between tasks. This study provides in the temporal domain previously unidentified neural evidence showing the neural processes involved in working memory of painful stimuli.

Brain process for perception of the "out of the body" tactile illusion for virtual object interaction

"Out of the body" tactile illusion refers to the phenomenon in which one can perceive tactility as if emanating from a location external to the body without any stimulator present there. Taking advantage of such a tactile illusion is one way to provide and realize richer interaction feedback without employing and placing actuators directly at all stimulation target points. However, to further explore its potential, it is important to better understand the underlying physiological and neural mechanism. As such, we measured the brain wave patterns during such tactile illusion and mapped out the corresponding brain activation areas. Participants were given stimulations at different levels with the intention to create veridical (i.e., non-illusory) and phantom sensations at different locations along an external hand-held virtual ruler. The experimental data and analysis indicate that both veridical and illusory sensations involve, among others, the parietal lobe, one of the most important components in the tactile information pathway. In addition, we found that as for the illusory sensation, there is an additional processing resulting in the delay for the ERP (event-related potential) and involvement by the limbic lobe. These point to regarding illusion as a memory and recognition task as a possible explanation. The present study demonstrated some basic understanding; how humans process "virtual" objects and the way associated tactile illusion is generated will be valuable for HCI (Human-Computer Interaction).

Music and Language Expertise Influence the Categorization of Speech and Musical Sounds: Behavioral and Electrophysiological Measurements

In this study, we used high-density EEG to evaluate whether speech and music expertise has an influence on the categorization of expertise-related and unrelated sounds. With this purpose in mind, we compared the categorization of speech, music, and neutral sounds between professional musicians, simultaneous interpreters (SIs), and controls in response to morphed speech–noise, music–noise, and speech–music continua. Our hypothesis was that music and language expertise will strengthen the memory representations of prototypical sounds, which act as a perceptual magnet for morphed variants. This means that the prototype would “attract” variants. This so-called magnet effect should be manifested by an increased assignment of morphed items to the trained category, by a reduced maximal slope of the psychometric function, as well as by differential event-related brain responses reflecting memory comparison processes (i.e., N400 and P600 responses). As a main result, we provide first evidence for a domain-specific behavioral bias of musicians and SIs toward the trained categories, namely music and speech. In addition, SIs showed a bias toward musical items, indicating that interpreting training has a generic influence on the cognitive representation of spectrotemporal signals with similar acoustic properties to speech sounds. Notably, EEG measurements revealed clear distinct N400 and P600 responses to both prototypical and ambiguous items between the three groups at anterior, central, and posterior scalp sites. These differential N400 and P600 responses represent synchronous activity occurring across widely distributed brain networks, and indicate a dynamical recruitment of memory processes that vary as a function of training and expertise.

An Empirical Reevaluation of Absolute Pitch: Behavioral and Electrophysiological Measurements

Here, we reevaluated the “two-component” model of absolute pitch (AP) by combining behavioral and electrophysiological measurements. This specific model postulates that AP is driven by a perceptual encoding ability (i.e., pitch memory) plus an associative memory component (i.e., pitch labeling). To test these predictions, during EEG measurements AP and non-AP (NAP) musicians were passively exposed to piano tones (first component of the model) and additionally instructed to judge whether combinations of tones and labels were conceptually associated or not (second component of the model). Auditory-evoked N1/P2 potentials did not reveal differences between the two groups, thus indicating that AP is not necessarily driven by a differential pitch encoding ability at the processing level of the auditory cortex. Otherwise, AP musicians performed the conceptual association task with an order of magnitude better accuracy and shorter RTs than NAP musicians did, this result clearly pointing to distinctive conceptual associations in AP possessors. Most notably, this behavioral superiority was reflected by an increased N400 effect and accompanied by a subsequent late positive component, the latter not being distinguishable in NAP musicians.

Short term memory and working memory in blind versus sighted children

There is evidence that blind people may strengthen their memory skills to compensate for absence of vision. However, which aspects of memory are involved is open to debate and a developmental perspective is generally lacking. In the present study, we compared the short term memory (STM) and working memory (WM) of 10-year-old blind children and sighted children. STM was measured using digit span forward, name learning, and word span tasks; WM was measured using listening span and digit span backward tasks. The blind children outperformed their sighted peers on both STM and WM tasks. The enhanced capacity of the blind children on digit span and other STM tasks confirms the results of earlier research; the significantly better performance of the blind children relative to their sighted peers on verbal WM tasks is a new interesting finding. Task characteristics, including the verbal nature of the WM tasks and strategies used to perform these tasks, are discussed.

Control-display mapping in brain-computer interfaces

Event-related potential (ERP) based brain-computer interfaces (BCIs) employ differences in brain responses to attended and ignored stimuli. When using a tactile ERP-BCI for navigation, mapping is required between navigation directions on a visual display and unambiguously corresponding tactile stimuli (tactors) from a tactile control device: control-display mapping (CDM). We investigated the effect of congruent (both display and control horizontal or both vertical) and incongruent (vertical display, horizontal control) CDMs on task performance, the ERP and potential BCI performance. Ten participants attended to a target (determined via CDM), in a stream of sequentially vibrating tactors. We show that congruent CDM yields best task performance, enhanced the P300 and results in increased estimated BCI performance. This suggests a reduced availability of attentional resources when operating an ERP-BCI with incongruent CDM. Additionally, we found an enhanced N2 for incongruent CDM, which indicates a conflict between visual display and tactile control orientations.

PRACTITIONER SUMMARY

Incongruency in control-display mapping reduces task performance. In this study, brain responses, task and system performance are related to (in)congruent mapping of command options and the corresponding stimuli in a brain-computer interface (BCI). Directional congruency reduces task errors, increases available attentional resources, improves BCI performance and thus facilitates human-computer interaction.

Visual identification of haptically explored objects in high and low hypnotizable subjects

Hypnotizability is associated with peculiar characteristics of sensorimotor integration, imaginal abilities, and preferences in the sensory modality of imagery. The visual recognition of haptically explored objects involves an interaction among these processes and is a proper tool to investigate their possible hypnotizability-related modulation. Sixteen high hypnotizables and 16 lows participated in the study. Higher frequencies of correct recognition (RF) were observed in highs. RF improved across both groups. As an effect of learning, shorter recognition times were found in males among highs and in females among lows. The findings are consistent with the literature suggesting that hypnotizability levels may be associated with specific modes of sensory integration and/or imagery.

Electrophysiological Correlates of Inductive Generalization

The cognitive mechanism of inductive generalization has been studied broadly, although the neural correlates are still unclear. In the present study, participants were provided with battery quadruplets. Within each quadruplet, some batteries were charged while others were not. Participants were asked to generate a hypothesis about what kinds of batteries were charged. The results revealed that the reaction times and the latencies of frontal N1 were shorter in the Generalizable condition than in the Nongeneralizable condition. After 420 ms onset of stimuli, the late positive complex (LPC) was increased significantly in the generalizable condition, reflecting a process of hypothesis generation and subsequent updating.

Neural activities of tactile cross-modal working memory in humans: an event-related potential study

In the present study, we examined the neural mechanisms underlying cross-modal working memory by analyzing scalp-recorded event-related potentials (ERPs) from normal human subjects performing tactile-tactile unimodal or tactile-auditory cross-modal delay tasks that consisted of stimulus-1 (S-1, tactile), interval (delay), and stimulus-2 (S-2, tactile or auditory). We hypothesized that there would be sequentially discrete task-correlated changes in ERPs representing neural processes of tactile working memory, and in addition, significant differences would be observed in ERPs between the unimodal task and the cross-modal task. In comparison to the ERP components in the unimodal task, two late positive ERP components (LPC-1 and LPC-2) evoked by the tactile S-1 in the delay of the cross-modal task were enhanced by expectation of the associated auditory S-2 presented at the end of the delay. Such enhancement might represent neural activities involved in cross-modal association between the tactile stimulus and the auditory stimulus. Later in the delay, a late negative component (LNC) was observed. The amplitude of LNC depended on information retained during the delay, and when the same information was retained, this amplitude was not influenced by modality or location of S-2 (auditory S-2 through headphones, or tactile S-2 on the left index finger). LNC might represent the neural activity involved in working memory. The above results suggest that the sequential ERP changes in the present study represent temporally distinguishable neural processes, such as the cross-modal association and cross-modal working memory.

Prefrontal cortex and somatosensory cortex in tactile crossmodal association: an independent component analysis of ERP recordings

Our previous studies on scalp-recorded event-related potentials (ERPs) showed that somatosensory N140 evoked by a tactile vibration in working memory tasks was enhanced when human subjects expected a coming visual stimulus that had been paired with the tactile stimulus. The results suggested that such enhancement represented the cortical activities involved in tactile-visual crossmodal association. In the present study, we further hypothesized that the enhancement represented the neural activities in somatosensory and frontal cortices in the crossmodal association. By applying independent component analysis (ICA) to the ERP data, we found independent components (ICs) located in the medial prefrontal cortex (around the anterior cingulate cortex, ACC) and the primary somatosensory cortex (SI). The activity represented by the IC in SI cortex showed enhancement in expectation of the visual stimulus. Such differential activity thus suggested the participation of SI cortex in the task-related crossmodal association. Further, the coherence analysis and the Granger causality spectral analysis of the ICs showed that SI cortex appeared to cooperate with ACC in attention and perception of the tactile stimulus in crossmodal association. The results of our study support with new evidence an important idea in cortical neurophysiology: higher cognitive operations develop from the modality-specific sensory cortices (in the present study, SI cortex) that are involved in sensation and perception of various stimuli.

The involvement of ipsilateral temporoparietal cortex in tactile pattern working memory as reflected in beta event-related desynchronization

Cortical oscillatory activity in various frequency bands has been shown to reflect working memory processes operating on visual and auditory stimulus information. Here we use magnetoencephalography to investigate cortical oscillatory activity related to working memory for tactile patterns. Right-handed subjects made same-different judgements on two dot patterns sequentially applied with a 3-s delay to the right middle fingertip. Spectral analysis revealed beta desynchronization (17+/-2.5 Hz) at contralateral postcentral and ipsilateral temporoparietal regions preceding and during the presentation of both tactile stimuli as well as during the early and late delay periods. Whereas contralateral beta desynchronization preceding tactile stimulation may reflect anticipation of incoming stimuli, ipsilateral beta desynchronization may underlie working memory maintenance of tactile patterns. The later hypothesis is supported by a significant positive correlation between subjects' performance and the amplitude of ipsilateral beta desynchronization 800 ms to 500 ms before the onset of the second pattern stimulus. Thus, our results suggest that ipsilateral temporoparietal cortex contributes to the maintenance of tactile pattern information in working memory.

Why Cyclops could not compete with Ulysses: monocular vision and mental images

The present research demonstrates that the limitations of congenitally blind people in tasks requiring the processing of mental images are specifically related to the absence of binocular vision and not to the absence of vision per se. We contrasted three different groups of participants: sighted; visually impaired, with reduced binocular vision; monocular, with a normal visual acuity although in one eye only. Visually impaired participants (i.e. blurred vision) show a pattern of performance comparable to that of the sighted. In contrast, monocular participants show a similar pattern of performance to congenitally blind individuals despite being able to see perfectly well. These results shed new light on the relationship between perception and imagery and on the characteristics of sequential and simultaneous processes in the human brain.