Auditory Multi-Stability: Idiosyncratic Perceptual Switching Patterns, Executive Functions and Personality Traits

Bimodal moment-by-moment coupling in perceptual multistability

Multistable perception occurs in all sensory modalities, and there is ongoing theoretical debate about whether there are overarching mechanisms driving multistability across modalities. Here we study whether multistable percepts are coupled across vision and audition on a moment-by-moment basis. To assess perception simultaneously for both modalities without provoking a dual-task situation, we query auditory perception by direct report, while measuring visual perception indirectly via eye movements. A support-vector-machine (SVM)-based classifier allows us to decode visual perception from the eye-tracking data on a moment-by-moment basis. For each timepoint, we compare visual percept (SVM output) and auditory percept (report) and quantify the co-occurrence of integrated (one-object) or segregated (two-objects) interpretations in the two modalities. Our results show an above-chance coupling of auditory and visual perceptual interpretations. By titrating stimulus parameters toward an approximately symmetric distribution of integrated and segregated percepts for each modality and individual, we minimize the amount of coupling expected by chance. Because of the nature of our task, we can rule out that the coupling stems from postperceptual levels (i.e., decision or response interference). Our results thus indicate moment-by-moment perceptual coupling in the resolution of visual and auditory multistability, lending support to theories that postulate joint mechanisms for multistable perception across the senses.

Functional coupling between auditory memory and verbal transformations

The ability to parse sound mixtures into coherent auditory objects is fundamental to cognitive functions, such as speech comprehension and language acquisition. Yet, we still lack a clear understanding of how auditory objects are formed. To address this question, we studied a speech-specific case of perceptual multistability, called verbal transformations (VTs), in which a variety of verbal forms is induced by continuous repetition of a physically unchanging word. Here, we investigated the degree to which auditory memory through sensory adaptation influences VTs. Specifically, we hypothesized that when memory persistence is longer, participants are able to retain the current verbal form longer, resulting in sensory adaptation, which in turn, affects auditory perception. Participants performed VT and auditory memory tasks on different days. In the VT task, Japanese participants continuously reported their perception while listening to a Japanese word (2- or 3-mora in length) played repeatedly for 5 min. In the auditory memory task, a different sequence of three morae, e.g., /ka/, /hi/, and /su/, was presented to each ear simultaneously. After some period (0-4 s), participants were visually cued to recall one of the sequences, i.e., in the left or right ear. We found that delayed recall accuracy was negatively correlated with the number of VTs, particularly under 2-mora conditions. This suggests that memory persistence is important for formation and selection of perceptual objects.

Cross-modal auditory priors drive the perception of bistable visual stimuli with reliable differences between individuals

It is a widely held assumption that the brain performs perceptual inference by combining sensory information with prior expectations, weighted by their uncertainty. A distinction can be made between higher- and lower-level priors, which can be manipulated with associative learning and sensory priming, respectively. Here, we simultaneously investigate priming and the differential effect of auditory vs. visual associative cues on visual perception, and we also examine the reliability of individual differences. Healthy individuals (N = 29) performed a perceptual inference task twice with a one-week delay. They reported the perceived direction of motion of dot pairs, which were preceded by a probabilistic visuo-acoustic cue. In 30% of the trials, motion direction was ambiguous, and in half of these trials, the auditory versus the visual cue predicted opposing directions. Cue-stimulus contingency could change every 40 trials. On ambiguous trials where the visual and the auditory cue predicted conflicting directions of motion, participants made more decisions consistent with the prediction of the acoustic cue. Increased predictive processing under stimulus uncertainty was indicated by slower responses to ambiguous (vs. non-ambiguous) stimuli. Furthermore, priming effects were also observed in that perception of ambiguous stimuli was influenced by perceptual decisions on the previous ambiguous and unambiguous trials as well. Critically, behavioural effects had substantial inter-individual variability which showed high test-retest reliability (intraclass correlation coefficient (ICC) > 0.78). Overall, higher-level priors based on auditory (vs. visual) information had greater influence on visual perception, and lower-level priors were also in action. Importantly, we observed large and stable differences in various aspects of task performance. Computational modelling combined with neuroimaging could allow testing hypotheses regarding the potential mechanisms causing these behavioral effects. The reliability of the behavioural differences implicates that such perceptual inference tasks could be valuable tools during large-scale biomarker and neuroimaging studies.

Ensemble modeling of auditory streaming reveals potential sources of bistability across the perceptual hierarchy

Perceptual bistability-the spontaneous, irregular fluctuation of perception between two interpretations of a stimulus-occurs when observing a large variety of ambiguous stimulus configurations. This phenomenon has the potential to serve as a tool for, among other things, understanding how function varies across individuals due to the large individual differences that manifest during perceptual bistability. Yet it remains difficult to interpret the functional processes at work, without knowing where bistability arises during perception. In this study we explore the hypothesis that bistability originates from multiple sources distributed across the perceptual hierarchy. We develop a hierarchical model of auditory processing comprised of three distinct levels: a Peripheral, tonotopic analysis, a Central analysis computing features found more centrally in the auditory system, and an Object analysis, where sounds are segmented into different streams. We model bistable perception within this system by applying adaptation, inhibition and noise into one or all of the three levels of the hierarchy. We evaluate a large ensemble of variations of this hierarchical model, where each model has a different configuration of adaptation, inhibition and noise. This approach avoids the assumption that a single configuration must be invoked to explain the data. Each model is evaluated based on its ability to replicate two hallmarks of bistability during auditory streaming: the selectivity of bistability to specific stimulus configurations, and the characteristic log-normal pattern of perceptual switches. Consistent with a distributed origin, a broad range of model parameters across this hierarchy lead to a plausible form of perceptual bistability.

Children's perception of visual and auditory ambiguity and its link to executive functions and creativity

The phenomenon of perceptual bistability provides insights into aspects of perceptual processing not normally accessible to everyday experience. However, most experiments have been conducted in adults, and it is not clear to what extent key aspects of perceptual switching change through development. The current research examined the ability of 6-, 8-, and 10-year-old children (N = 66) to switch between competing percepts of ambiguous visual and auditory stimuli and links between switching rate, executive functions, and creativity. The numbers of switches participants reported in two visual tasks (ambiguous figure and ambiguous structure from motion) and two auditory tasks (verbal transformation and auditory streaming) were measured in three 60-s blocks. In addition, inhibitory control was measured with a Stroop task, set shifting was measured with a verbal fluency task, and creativity was measured with a divergent thinking task. The numbers of perceptual switches increased in all four tasks from 6 to 10 years of age but differed across tasks in that they were higher in the verbal transformation and ambigous structure-from-motion tasks than in the ambigous figure and auditory streaming tasks for all age groups. Although perceptual switching rates differed across tasks, there were predictive relationships between switching rates in some tasks. However, little evidence for the influence of central processes on perceptual switching was found. Overall, the results support the notion that perceptual switching is largely modality and task specific and that this property is already evident when perceptual switching emerges.

Functional brain networks underlying idiosyncratic switching patterns in multi-stable auditory perception

In perceptual multi-stability, perception stochastically switches between alternative interpretations of the stimulus allowing examination of perceptual experience independent of stimulus parameters. Previous studies found that listeners show temporally stable idiosyncratic switching patterns when listening to a multi-stable auditory stimulus, such as in the auditory streaming paradigm. This inter-individual variability can be described along two dimensions, Exploration and Segregation. In the current study, we explored the functional brain networks associated with these dimensions and their constituents using electroencephalography. Results showed that Segregation and its constituents are related to brain networks operating in the theta EEG band, whereas Exploration and its constituents are related to networks in the lower and upper alpha and beta bands. Thus, the dimensions on which individuals' perception differ from each other in the auditory streaming paradigm probably reflect separate perceptual processes in the human brain. Further, the results suggest that networks mainly located in left auditory areas underlie the perception of integration, whereas perceiving the alternative patterns is accompanied by stronger interhemispheric connections.

Comparison of perceptual properties of auditory streaming between spectral and amplitude modulation domains

The two-tone sequence (ABA_), which comprises two different sounds (A and B) and a silent gap, has been used to investigate how the auditory system organizes sequential sounds depending on various stimulus conditions or brain states. Auditory streaming can be evoked by differences not only in the tone frequency ("spectral cue": ΔF_TONE, TONE condition) but also in the amplitude modulation rate ("AM cue": ΔF_AM, AM condition). The aim of the present study was to explore the relationship between the perceptual properties of auditory streaming for the TONE and AM conditions. A sequence with a long duration (400 repetitions of ABA_) was used to examine the property of the bistability of streaming. The ratio of feature differences that evoked an equivalent probability of the segregated percept was close to the ratio of the Q-values of the auditory and modulation filters, consistent with a "channeling theory" of auditory streaming. On the other hand, for values of ΔF_AM and ΔF_TONE evoking equal probabilities of the segregated percept, the number of perceptual switches was larger for the TONE condition than for the AM condition, indicating that the mechanism(s) that determine the bistability of auditory streaming are different between or sensitive to the two domains. Nevertheless, the number of switches for individual listeners was positively correlated between the spectral and AM domains. The results suggest a possibility that the neural substrates for spectral and AM processes share a common switching mechanism but differ in location and/or in the properties of neural activity or the strength of internal noise at each level.

Auditory multistability and neurotransmitter concentrations in the human brain

Multistability in perception is a powerful tool for investigating sensory-perceptual transformations, because it produces dissociations between sensory inputs and subjective experience. Spontaneous switching between different perceptual objects occurs during prolonged listening to a sound sequence of tone triplets or repeated words (termed auditory streaming and verbal transformations, respectively). We used these examples of auditory multistability to examine to what extent neurochemical and cognitive factors influence the observed idiosyncratic patterns of switching between perceptual objects. The concentrations of glutamate-glutamine (Glx) and γ-aminobutyric acid (GABA) in brain regions were measured by magnetic resonance spectroscopy, while personality traits and executive functions were assessed using questionnaires and response inhibition tasks. Idiosyncratic patterns of perceptual switching in the two multistable stimulus configurations were identified using a multidimensional scaling (MDS) analysis. Intriguingly, although switching patterns within each individual differed between auditory streaming and verbal transformations, similar MDS dimensions were extracted separately from the two datasets. Individual switching patterns were significantly correlated with Glx and GABA concentrations in auditory cortex and inferior frontal cortex but not with the personality traits and executive functions. Our results suggest that auditory perceptual organization depends on the balance between neural excitation and inhibition in different brain regions.This article is part of the themed issue 'Auditory and visual scene analysis'.

Computational Models of Auditory Scene Analysis: A Review

Auditory scene analysis (ASA) refers to the process (es) of parsing the complex acoustic input into auditory perceptual objects representing either physical sources or temporal sound patterns, such as melodies, which contributed to the sound waves reaching the ears. A number of new computational models accounting for some of the perceptual phenomena of ASA have been published recently. Here we provide a theoretically motivated review of these computational models, aiming to relate their guiding principles to the central issues of the theoretical framework of ASA. Specifically, we ask how they achieve the grouping and separation of sound elements and whether they implement some form of competition between alternative interpretations of the sound input. We consider the extent to which they include predictive processes, as important current theories suggest that perception is inherently predictive, and also how they have been evaluated. We conclude that current computational models of ASA are fragmentary in the sense that rather than providing general competing interpretations of ASA, they focus on assessing the utility of specific processes (or algorithms) for finding the causes of the complex acoustic signal. This leaves open the possibility for integrating complementary aspects of the models into a more comprehensive theory of ASA.