Auditory and visual scene analysis: an overview

Sensing and feeling: an overview

Emotional experiences are driven, in part, by the way we process and integrate information from different sensory modalities. Understanding how perceptual and emotional systems interact to give rise to subjective feelings is an important, complex and challenging issue, requiring new approaches and integrative thinking that fuses the fundamentals of low-level sensory perception with higher-level cognitive and affective processes. The Theme Issue 'Sensing and feeling: an integrative approach to sensory processing and emotional experience' showcases fifteen theoretical, empirical, and review articles from experts working at the intersection of perception and emotion, encompassing multiple sensory systems (visual, auditory, tactile and interoceptive), clinical and non-clinical perspectives (e.g. affective disorders and hearing loss), contextual and social perspectives, and complex emotional experiences in special populations. Articles in Part 1 emphasize recent advances across fields in sensory and emotion science and give insights into future directions. Each article in Part 2 provides more detailed and specific methodological approaches or theoretical models, and focuses on basic mechanisms linking sensation to emotional experience. This article is part of the theme issue 'Sensing and feeling: an integrative approach to sensory processing and emotional experience'.

The different contributions of the eight prefrontal cortex subregions to reactive responses after unpredictable slip perturbations and vibrotactile cueing

Introduction

Recent advancements in functional near-infrared spectroscopy technology have offered a portable, wireless, wearable solution to measure the activity of the prefrontal cortex (PFC) in the human neuroscience field. This study is the first to validate the different contributions made by the PFC's eight subregions in healthy young adults to the reactive recovery responses following treadmill-induced unpredictable slip perturbations and vibrotactile cueing (i.e., precues).

Methods

Our fall-inducing technology platform equipped with a split-belt treadmill provided unpredictable slip perturbations to healthy young adults while walking at their self-selected walking speed. A portable, wireless, wearable, and multi-channel (48 channels) functional near-infrared spectroscopy system evaluated the activity of PFC's eight subregions [i.e., right and left dorsolateral prefrontal cortex (DLPFC), ventrolateral prefrontal cortex (VLPFC), frontopolar prefrontal cortex (FPFC), and orbitofrontal cortex (OFC)] as quantified by oxyhemoglobin and deoxyhemoglobin concentrations. A motion capture system and two force plates beneath the split-belt treadmill were used to quantify participants' kinematic and kinetic behavior. All participants completed 6 trials: 2 consecutive trials without vibrotactile cueing and with a slip perturbation (control trials); 3 trials with vibrotactile cueing [2 trials with the slip perturbation (cueing trial) and 1 trial without the slip perturbation (catch trial)], and 1 trial without vibrotactile cueing and with a slip perturbation (post-control trial). The PFC subregions' activity and kinematic behavior were assessed during the three periods (i.e., standing, walking, and recovery periods).

Results

Compared to the walkers' standing and walking periods, recovery periods showed significantly higher and lower levels of oxyhemoglobin and deoxyhemoglobin concentrations, respectively, in the right and left DLPFC, VLPFC, and FPFC, regardless of the presence of vibrotactile cueing. However, there was no significant difference in the right and left OFC between the three periods. Kinematic analyses confirmed that vibrotactile cueing significantly improved reactive recovery responses without requiring more involvement by the PFC subregions, which suggests that the sum of attentional resources is similar in cued and non-cued motor responses.

Discussion

The results could inform the design of wearable technologies that alert their users to the risks of falling and assist with the development of new gait perturbation paradigms that prompt reactive responses.

A framework for understanding post-detection deception in predator-prey interactions

Predators and prey exist in persistent conflict that often hinges on deception-the transmission of misleading or manipulative signals-as a means for survival. Deceptive traits are widespread across taxa and sensory systems, representing an evolutionarily successful and common strategy. Moreover, the highly conserved nature of the major sensory systems often extends these traits past single species predator-prey interactions toward a broader set of perceivers. As such, deceptive traits can provide a unique window into the capabilities, constraints and commonalities across divergent and phylogenetically-related perceivers. Researchers have studied deceptive traits for centuries, but a unified framework for categorizing different types of post-detection deception in predator-prey conflict still holds potential to inform future research. We suggest that deceptive traits can be distinguished by their effect on object formation processes. Perceptual objects are composed of physical attributes (what) and spatial (where) information. Deceptive traits that operate after object formation can therefore influence the perception and processing of either or both of these axes. We build upon previous work using a perceiver perspective approach to delineate deceptive traits by whether they closely match the sensory information of another object or create a discrepancy between perception and reality by exploiting the sensory shortcuts and perceptual biases of their perceiver. We then further divide this second category, sensory illusions, into traits that distort object characteristics along either the what or where axes, and those that create the perception of whole novel objects, integrating the what/where axes. Using predator-prey examples, we detail each step in this framework and propose future avenues for research. We suggest that this framework will help organize the many forms of deceptive traits and help generate predictions about selective forces that have driven animal form and behavior across evolutionary time.

Superior Identification of Component Odors in a Mixture Is Linked to Autistic Traits in Children and Adults

Most familiar odors are complex mixtures of volatile molecules, which the olfactory system automatically synthesizes into a perceptual whole. However, odors are rarely encountered in isolation; thus, the brain must also separate distinct odor objects from complex and variable backgrounds. In vision, autistic traits are associated with superior performance in tasks that require focus on the local features of a perceptual scene. The aim of the present study was to determine whether the same advantage was observed in the analysis of olfactory scenes. To do this, we compared the ability of 1) 40 young adults (aged 16-35) with high (n = 20) and low levels of autistic traits and 2) 20 children (aged 7-11), with (n = 10) and without an autism spectrum disorder diagnosis, to identify individual odor objects presented within odor mixtures. First, we used a 4-alternative forced choice task to confirm that both adults and children were able to reliably identify 8 blended fragrances, representing food-related odors, when presented individually. We then used the same forced choice format to test participants' ability to identify the odors when they were combined in either binary or ternary mixtures. Adults with high levels of autistic traits showed superior performance on binary but not ternary mixture trials, whereas children with an autism spectrum disorder diagnosis outperformed age-matched neurotypical peers, irrespective of mixture complexity. These findings indicate that the local processing advantages associated with high levels of autistic traits in visual tasks are also apparent in a task requiring analytical processing of odor mixtures.

Excitation-inhibition balance and auditory multistable perception are correlated with autistic traits and schizotypy in a non-clinical population

Individuals with autism spectrum disorder and individuals with schizophrenia have impaired social and communication skills. They also have altered auditory perception. This study investigated autistic traits and schizotypy in a non-clinical population as well as the excitation-inhibition (EI) balance in different brain regions and their auditory multistable perception. Thirty-four healthy participants were assessed by the Autism-Spectrum Quotient (AQ) and Schizotypal Personality Questionnaire (SPQ). The EI balance was evaluated by measuring the resting-state concentrations of glutamate-glutamine (Glx) and ϒ-aminobutyric acid (GABA) in vivo by using magnetic resonance spectroscopy. To observe the correlation between their traits and perception, we conducted an auditory streaming task and a verbal transformation task, in which participants reported spontaneous perceptual switching while listening to a sound sequence. Their AQ and SPQ scores were positively correlated with the Glx/GABA ratio in the auditory cortex but not in the frontal areas. These scores were negatively correlated with the number of perceptual switches in the verbal transformation task but not in the auditory streaming task. Our results suggest that the EI balance in the auditory cortex and the perceptual formation of speech are involved in autistic traits and schizotypy.

Scale-free behaviour and metastable brain-state switching driven by human cognition, an empirical approach

We developed a framework to study brain dynamics under cognition. In particular, we investigated the spatiotemporal properties of brain state switches under cognition. The lack of electroencephalography stationarity is exploited as one of the signatures of the metastability of brain states. We correlated power law exponents in the variables that we proposed to describe brain states, and dynamical properties of non-stationarities with cognitive conditions. This framework was successfully tested with three different datasets: a working memory dataset, an Alzheimer disease dataset, and an emotions dataset. We discuss the temporal organization of switches between states, providing evidence suggesting the need to reconsider the piecewise model, in which switches appear at discrete times. Instead, we propose a more dynamically rich view, in which besides the seemingly discrete switches, switches between neighbouring states occur all the time. These micro switches are not (physical) noise, as their properties are also affected by cognition.

Auditory processing and neuropsychological profiles of children with functional hearing loss

OBJECTIVES

This paper compares structured history, auditory processing abilities and neuropsychological findings of children with functional hearing loss (FHL) to those with suspected auditory processing disorder without FHL (control). The main aim was to evaluate the value of a holistic assessment protocol for FHL used in a routine pediatric audiology clinic. The protocol incorporated a commercially available test battery for auditory processing disorder (APD), non-verbal intelligence (NVIQ) and tools to screen for common co-existing neurodevelopmental conditions such as attention deficit hyperactivity disorder (ADHD), language impairment (LI) and developmental coordination disorder (DCD). The outcome of such holistic assessment was expected to help in understanding the nature of FHL and to provide individualized support to mitigate their difficulties.

METHODS

This retrospective study compared two groups, 40 children (M = 17, F = 23) in each group between seven and sixteen years of age, one group with a history of FHL and the other with suspected APD without FHL (control). The groups were matched against age, gender, hand use, diagnosis of APD or non-APD (31 with APD and 9 without APD in each group) and non-verbal intelligence. All the children were healthy English speaking children attending mainstream schools with no middle or inner ear abnormalities. Structured history was obtained from parents regarding different nonacademic and academic concerns. The SCAN-3:C and SCAN-3:A test batteries were used to assess auditory processing abilities; Lucid Ability test for NVIQ; Children's Communication Checklist-2 (CCC-2) for language ability; Swanson Nolan and Pelham-IV Rating Scale (SNAP-IV) for ADHD; and the manual dexterity components of the Movement Assessment Battery for Children-2 (MABC-2) as a screening tool for DCD.

RESULTS

About 60% of children in both the groups had concerns regarding listening in noisy background. In the history, poor attention was reported in 45% of children in the FHL group compared to 82.5% in the control group (p < 0.01). Hyperacusis was present in 35% of children in the FHL group and in 62% of children in the control group (p < 0.05). Concerns about overall academic abilities were present in 59% of children in the FHL group and 75% of the controls (p > 0.05). Only 15% of children in the FHL group had concerns with numeracy skills in contrast to 41% of the controls (p < 0.05). Significantly fewer (p < 0.01) children in the FHL group (41%) received additional support at school than the controls (75%). Fewer children performed poorly in Filtered Words (FW) test of the SCAN-3 batteries, 30% in the FHL group and 17.5% in the control group, in contrast to Auditory Figure Ground 0 (AFG0), 85% in FHL and 80% in the control group. The number of children performing poorly in AFG0 was significantly higher compared to all the other SCAN-3 tests in FHL (P < 0.05), in contrast to FW and Competing Sentences (CS) only in the control group (p < 0.05). The control group had higher prevalence of atypical ear advantage (AEA) in left directed Competing Words (CW) (32.5%) and Time Compressed Sentences (TCS) (32.5%) compared to FW (7.5%). In contrast, FHL group had higher prevalence of AEA in AFG0 (48.7%) compared to CS (21%). High proportions of children in both the groups had LI (80% in FHL and 82.5% in the control group), with significantly lower (p < 0.05) levels of ADHD symptoms in the FHL group (39.5%) compared to the control group (72.5%). Impaired manual dexterity was present in 30.7% of children in FHL group and 47.5% in the controls.

CONCLUSIONS

The prevalences of APD and language impairment are high compared to ADHD symptoms in children with FHL, and holistic assessment is recommended. Despite some similarities in the auditory and neuropsychological profiles between children with FHL and those with suspected APD without FHL some differences were noted. The results suggest that children with FHL have genuine difficulties that need to be identified and addressed. Future research is required to identify the neural pathways which could explain the similarities and dissimilarities between the two groups.

Interaction of spatial and non-spatial cues in auditory stream segregation in the European starling

Integrating sounds from the same source and segregating sounds from different sources in an acoustic scene are an essential function of the auditory system. Naturally, the auditory system simultaneously makes use of multiple cues. Here, we investigate the interaction between spatial cues and frequency cues in stream segregation of European starlings (Sturnus vulgaris) using an objective measure of perception. Neural responses to streaming sounds were recorded, while the bird was performing a behavioural task that results in a higher sensitivity during a one-stream than a two-stream percept. Birds were trained to detect an onset time shift of a B tone in an ABA- triplet sequence in which A and B could differ in frequency and/or spatial location. If the frequency difference or spatial separation between the signal sources or both were increased, the behavioural time shift detection performance deteriorated. Spatial separation had a smaller effect on the performance compared to the frequency difference and both cues additively affected the performance. Neural responses in the primary auditory forebrain were affected by the frequency and spatial cues. However, frequency and spatial cue differences being sufficiently large to elicit behavioural effects did not reveal correlated neural response differences. The difference between the neuronal response pattern and behavioural response is discussed with relation to the task given to the bird. Perceptual effects of combining different cues in auditory scene analysis indicate that these cues are analysed independently and given different weights suggesting that the streaming percept arises consecutively to initial cue analysis.

Normal Aging Slows Spontaneous Switching in Auditory and Visual Bistability

Age-related changes in auditory and visual perception have an impact on the quality of life. It has been debated how perceptual organization is influenced by advancing age. From the neurochemical perspective, we investigated age effects on auditory and visual bistability. In perceptual bistability, a sequence of sensory inputs induces spontaneous switching between different perceptual objects. We used different modality tasks of auditory streaming and visual plaids. Young and middle-aged participants (20-60years) were instructed to indicate by a button press whenever their perception changed from one stable state to the other. The number of perceptual switches decreased with participants' ages. We employed magnetic resonance spectroscopy to measure non-invasively concentrations of the inhibitory neurotransmitter (γ-aminobutyric acid, GABA) in the brain regions of interest. When participants were asked to voluntarily modulate their perception, the amount of effective volitional control was positively correlated with the GABA concentration in the auditory and motion-sensitive areas corresponding to each sensory modality. However, no correlation was found in the prefrontal cortex and anterior cingulate cortex. In addition, effective volitional control was reduced with advancing age. Our results suggest that sequential scene analysis in auditory and visual domains is influenced by both age-related and neurochemical factors.

Interindividual variability in auditory scene analysis revealed by confidence judgements

Because musicians are trained to discern sounds within complex acoustic scenes, such as an orchestra playing, it has been hypothesized that musicianship improves general auditory scene analysis abilities. Here, we compared musicians and non-musicians in a behavioural paradigm using ambiguous stimuli, combining performance, reaction times and confidence measures. We used 'Shepard tones', for which listeners may report either an upward or a downward pitch shift for the same ambiguous tone pair. Musicians and non-musicians performed similarly on the pitch-shift direction task. In particular, both groups were at chance for the ambiguous case. However, groups differed in their reaction times and judgements of confidence. Musicians responded to the ambiguous case with long reaction times and low confidence, whereas non-musicians responded with fast reaction times and maximal confidence. In a subsequent experiment, non-musicians displayed reduced confidence for the ambiguous case when pure-tone components of the Shepard complex were made easier to discern. The results suggest an effect of musical training on scene analysis: we speculate that musicians were more likely to discern components within complex auditory scenes, perhaps because of enhanced attentional resolution, and thus discovered the ambiguity. For untrained listeners, stimulus ambiguity was not available to perceptual awareness.This article is part of the themed issue 'Auditory and visual scene analysis'.

Resolving the neural dynamics of visual and auditory scene processing in the human brain: a methodological approach

In natural environments, visual and auditory stimulation elicit responses across a large set of brain regions in a fraction of a second, yielding representations of the multimodal scene and its properties. The rapid and complex neural dynamics underlying visual and auditory information processing pose major challenges to human cognitive neuroscience. Brain signals measured non-invasively are inherently noisy, the format of neural representations is unknown, and transformations between representations are complex and often nonlinear. Further, no single non-invasive brain measurement technique provides a spatio-temporally integrated view. In this opinion piece, we argue that progress can be made by a concerted effort based on three pillars of recent methodological development: (i) sensitive analysis techniques such as decoding and cross-classification, (ii) complex computational modelling using models such as deep neural networks, and (iii) integration across imaging methods (magnetoencephalography/electroencephalography, functional magnetic resonance imaging) and models, e.g. using representational similarity analysis. We showcase two recent efforts that have been undertaken in this spirit and provide novel results about visual and auditory scene analysis. Finally, we discuss the limits of this perspective and sketch a concrete roadmap for future research.

This article is part of the themed issue ‘Auditory and visual scene analysis’.

Auditory multistability and neurotransmitter concentrations in the human brain

Multistability in perception is a powerful tool for investigating sensory-perceptual transformations, because it produces dissociations between sensory inputs and subjective experience. Spontaneous switching between different perceptual objects occurs during prolonged listening to a sound sequence of tone triplets or repeated words (termed auditory streaming and verbal transformations, respectively). We used these examples of auditory multistability to examine to what extent neurochemical and cognitive factors influence the observed idiosyncratic patterns of switching between perceptual objects. The concentrations of glutamate-glutamine (Glx) and γ-aminobutyric acid (GABA) in brain regions were measured by magnetic resonance spectroscopy, while personality traits and executive functions were assessed using questionnaires and response inhibition tasks. Idiosyncratic patterns of perceptual switching in the two multistable stimulus configurations were identified using a multidimensional scaling (MDS) analysis. Intriguingly, although switching patterns within each individual differed between auditory streaming and verbal transformations, similar MDS dimensions were extracted separately from the two datasets. Individual switching patterns were significantly correlated with Glx and GABA concentrations in auditory cortex and inferior frontal cortex but not with the personality traits and executive functions. Our results suggest that auditory perceptual organization depends on the balance between neural excitation and inhibition in different brain regions.This article is part of the themed issue 'Auditory and visual scene analysis'.