Modeling implicit learning in a cross-modal audio-visual serial reaction time task

Unraveling Temporal Dynamics of Multidimensional Statistical Learning in Implicit and Explicit Systems: An X-Way Hypothesis

Statistical learning enables humans to involuntarily process and utilize different kinds of patterns from the environment. However, the cognitive mechanisms underlying the simultaneous acquisition of multiple regularities from different perceptual modalities remain unclear. A novel multidimensional serial reaction time task was developed to test 40 participants' ability to learn simple first-order and complex second-order relations between uni-modal visual and cross-modal audio-visual stimuli. Using the difference in reaction times between sequenced and random stimuli as the index of domain-general statistical learning, a significant difference and dissociation of learning occurred between the initial and final learning phases. Furthermore, we used a negative and positive occurrence-frequency-and-reaction-time correlation to indicate implicit and explicit learning, respectively, and found that learning simple uni-modal patterns involved an implicit-to-explicit segue, while acquiring complex cross-modal patterns required an explicit-to-implicit segue, resulting in a X-shape crossing of regularity learning. Thus, we propose an X-way hypothesis to elucidate the dynamic interplay between the implicit and explicit systems at two distinct stages when acquiring various regularities in a multidimensional probability space.

Effects of emotional congruency and task complexity on decision-making

The heuristic approach to decision-making holds that the selection process becomes more efficient when part of the information available is ignored. One element involved in selecting information is emotional valence. If emotional congruency is related to simplified decision-making strategies, then the interaction of this factor with task complexity should exist. The present study explored how factors of this nature influence decision-making efficiency. We hypothesized that emotional congruency would have a positive effect on task execution and that the magnitude of that effect would increase with greater task complexity because in that condition the amount of information to be processed is greater, meaning that a heuristic approach to the problem would be more efficient. We design a decision in browser decision-making task in which participants had to select emotional images to gain points. Depending on the correlation between emotional valence and in-task image value, we defined three emotional congruency conditions: direct, null, and inverse. Our results show that distinct types of emotional congruency have differential effects on behavior. While direct congruency-enhanced overall decision-making performance, inverse congruency interacted with task complexity to modify the pace at which task feedback affected behavior.

How do irrelevant stimuli from another modality influence responses to the targets in a same-different task

It remains unclear whether multisensory interaction can implicitly occur at the abstract level. To address this issue, a same-different task was used to select comparable images and sounds in Experiment 1. Then, the stimuli with various levels of discrimination difficulty were adopted in a modified same-different task in Experiments 2, 3, and 4. The resultsshowed that only when the irrelevant stimuli were easily distinguishable, aconsistency effectcould beobservedin the testing phase. Moreover, when easily distinguishableirrelevant stimuliwere simultaneously presented with difficulttarget stimuli, irrelevant auditorystimuli facilitated responses to visual targets whereas irrelevant visual stimuli interfered with responses to auditorytargetsin the training phase,indicating an asymmetry in the role of visual and auditory in abstract multisensory integration. The results suggested that abstract multisensory information could be implicitly integrated and the inverse effectiveness principle might not apply to high-level processing of abstract multisensory integration.

Temporal division of the decision-making process: An EEG study

Decision-making is a process that allows individuals to choose an option or alternative in order to maximize a subjective gain or achieve a set goal by evaluating and establishing a preference based on contextual and internal information. Ernst and Paulus proposed a three-stage temporal division of this process: 1) the assessment and formation of preferences among possible options; 2) the selection and execution of an action; and 3) the experience or evaluation of an outcome. Each stage involves the participation of several brain regions, including the prefrontal, parietal, and temporal cortices. There are reports of distinct functionalities of these cortices for each stage of decision-making, but those studies focus on individual stages and do not provide any direct comparisons among them. Therefore, using a task that allows the clear temporal separation of the three stages of decision-making, we characterized the electroencephalographic activity (EEG) of those cortices in 30 healthy right-handed men during preference changes that occurred while performing a decision-making task. As the trials progressed, the preference for the stimulus shifted towards maximizing gains on the task. Forty trials sufficed to maintain these behavioral changes. Specific EEG patterns for each stage of decision-making were obtained, and it was possible to associate them with the cognitive processes involved in each one. These EEG data support the temporal division of the decision-making process proposed by Ernest and Paulus and show that the task designed could be a useful tool for determining behavioral and cerebral changes associated with stimuli preference during decision-making.

Tones slow down visuomotor responses in a visual-spatial task

The current study examined how simple tones affect speeded visuomotor responses in a visual-spatial sequence learning task. Across the three reported experiments, participants were presented with a visual target that appeared in different locations on a touchscreen monitor and they were instructed to touch the visual targets as quickly as possible. Visual sequences were either paired with sounds that correlated with the location of the target, paired with sounds that did not correlate with the location of the target, or the sequences were presented in silence (baseline). Response times decreased across training and participants were slower to respond to the visual stimuli when the sequences were paired with tones. Moreover, these interference effects were more pronounced early in training and explicit instructions directing attention to the visual modality had little effect on eliminating auditory interference, suggesting that these interference effects may stem from bottom-up factors and do not appear to be under attentional control. These findings have implications on tasks that require the processing of simultaneously presented auditory and visual information and provide support for a proposed mechanism underlying auditory dominance on a task that is typically better suited for the visual modality.

Implicit perceptual learning of visual-auditory modality sequences

We examined perceptual modality sequence learning by presenting number words either visually (V) or auditorily (A). Manual responses were assigned to number identity, which was random, but the stimulus modalities followed a predictable 6-element sequence (e.g., VVAAVA). In two experiments, we assessed sequence-specific learning as the performance difference between the predictable sequence and a random transfer sequence. We expected learning benefits, but for visual trials we did not find any clear predictability benefits, and, surprisingly, for auditory trials we even found a general processing disadvantage (i.e., a predictability cost) for auditory trials (Experiment 1) or a cost-benefit pattern (Experiment 2, with equated shift rates in predictable and random sequences), with costs for auditory repetition trials and benefits for shifting to auditory processing. Hence, overall there was a general learning "cost" (Experiment 1) or a null net benefit of predictability for performance (Experiment 2). Together, the findings reveal a modality-specific sensitivity towards variations in shift frequency and modality predictability only for auditory trials, but there was no overall benefit of modality-specific sequence learning.

Multimodal Memory Components and Their Long-Term Dynamics Identified in Cortical Layers II/III but Not Layer V

Activity patterns of cerebral cortical regions represent the current environment in which animals receive multi-modal inputs. These patterns are also shaped by the history of activity that reflects learned information on past multimodal exposures. We studied the long-term dynamics of cortical activity patterns during the formation of multimodal memories by analyzing in vivo high-resolution 2-photon mouse brain imaging data of Immediate Early Gene (IEG) expression, resolved by cortical layers. Strikingly, in superficial layers II/III, the patterns showed similar dynamics across structurally and functionally distinct cortical areas and the consistency of dynamic patterns lasted for one to several days. By contrast, in deep layer V, the activity dynamics varied across different areas, and the current activities were sensitive to the previous activities at different time points, depending on the cortical locations, indicating that the information stored in the cortex at different time points was distributed across different cortical areas. These results suggest different roles of superficial and deep layer neurons in the long-term multimodal representation of the environment.

Lack of Automatic Vocal Response Learning While Reading Aloud

Research on implicit sequence learning with the Serial Reaction Task (SRT) has demonstrated that people automatically acquire knowledge about fixed repeating sequences of responses and can transfer response sequence knowledge to novel stimuli. Such demonstrations are, however, mostly limited to setups with visual stimuli and manual responses. Here we systematically follow up on scarce attempts to demonstrate implicit sequence learning in word reading. While the literature on implicit sequence learning can be taken to suggest that sequence knowledge is acquired and affecting performance in word reading, we show that neither is the case in a series of four experiments. Sequence knowledge was acquired and affecting performance in color naming but not in word reading. On the one hand, we observed slowing of voice-onset times in off-sequence as compared to regularly sequenced trials when people named the color of a centrally presented disk. Yet, hardly any effect was observed when the very same sequence of words was verbalized in word reading instead. Transfer of sequence knowledge to and from color naming was not observed, either. This contrasts with sequence learning studies with manual responses, which have been taken to suggest that a fixed and repeating sequence of responses is sufficient for learning to occur even in fast choice reaction tasks and to transfer across stimuli as long as the sequence of responses remains intact. Rather, in line with dimensional action accounts of task performance, the results underline the role of translation between processing streams for implicit sequence learning.