Detecting changes in dynamic and complex acoustic environments

Neural correlates of perceptual texture change during active touch

Introduction

Texture changes occur frequently during real-world haptic explorations, but the neural processes that encode perceptual texture change remain relatively unknown. The present study examines cortical oscillatory changes during transitions between different surface textures during active touch.

Methods

Participants explored two differing textures whilst oscillatory brain activity and finger position data were recorded using 129-channel electroencephalography and a purpose-built touch sensor. These data streams were fused to calculate epochs relative to the time when the moving finger crossed the textural boundary on a 3D-printed sample. Changes in oscillatory band power in alpha (8-12 Hz), beta (16-24 Hz) and theta (4-7 Hz) frequency bands were investigated.

Results

Alpha-band power reduced over bilateral sensorimotor areas during the transition period relative to ongoing texture processing, indicating that alpha-band activity is modulated by perceptual texture change during complex ongoing tactile exploration. Further, reduced beta-band power was observed in central sensorimotor areas when participants transitioned from rough to smooth relative to transitioning from smooth to rough textures, supporting previous research that beta-band activity is mediated by high-frequency vibrotactile cues.

Discussion

The present findings suggest that perceptual texture change is encoded in the brain in alpha-band oscillatory activity whilst completing continuous naturalistic movements across textures.

Pupillary dynamics reflect the impact of temporal expectation on detection strategy

Everyday life's perceptual decision-making is informed by experience. In particular, temporal expectation can ease the detection of relevant events in noisy sensory streams. Here, we investigated if humans can extract hidden temporal cues from the occurrences of probabilistic targets and utilize them to inform target detection in a complex acoustic stream. To understand what neural mechanisms implement temporal expectation influence on decision-making, we used pupillometry as a proxy for underlying neuromodulatory activity. We found that participants' detection strategy was influenced by the hidden temporal context and correlated with sound-evoked pupil dilation. A model of urgency fitted on false alarms predicted detection reaction time. Altogether, these findings suggest that temporal expectation informs decision-making and could be implemented through neuromodulatory-mediated urgency signals.

The role of temporal coherence and temporal predictability in the build-up of auditory grouping

The cochlea decomposes sounds into separate frequency channels, from which the auditory brain must reconstruct the auditory scene. To do this the auditory system must make decisions about which frequency information should be grouped together, and which should remain distinct. Two key cues for grouping are temporal coherence, resulting from coherent changes in power across frequency, and temporal predictability, resulting from regular or predictable changes over time. To test how these cues contribute to the construction of a sound scene we present listeners with a range of precursor sounds, which act to prime the auditory system by providing information about each sounds structure, followed by a fixed masker in which participants were required to detect the presence of an embedded tone. By manipulating temporal coherence and/or temporal predictability in the precursor we assess how prior sound exposure influences subsequent auditory grouping. In Experiment 1, we measure the contribution of temporal predictability by presenting temporally regular or jittered precursors, and temporal coherence by using either narrow or broadband sounds, demonstrating that both independently contribute to masking/unmasking. In Experiment 2, we measure the relative impact of temporal coherence and temporal predictability and ask whether the influence of each in the precursor signifies an enhancement or interference of unmasking. We observed that interfering precursors produced the largest changes to thresholds.

Rational arbitration between statistics and rules in human sequence processing

Detecting and learning temporal regularities is essential to accurately predict the future. A long-standing debate in cognitive science concerns the existence in humans of a dissociation between two systems, one for handling statistical regularities governing the probabilities of individual items and their transitions, and another for handling deterministic rules. Here, to address this issue, we used finger tracking to continuously monitor the online build-up of evidence, confidence, false alarms and changes-of-mind during sequence processing. All these aspects of behaviour conformed tightly to a hierarchical Bayesian inference model with distinct hypothesis spaces for statistics and rules, yet linked by a single probabilistic currency. Alternative models based either on a single statistical mechanism or on two non-commensurable systems were rejected. Our results indicate that a hierarchical Bayesian inference mechanism, capable of operating over distinct hypothesis spaces for statistics and rules, underlies the human capability for sequence processing.

Accumulation of continuously time-varying sensory evidence constrains neural and behavioral responses in human collision threat detection

Evidence accumulation models provide a dominant account of human decision-making, and have been particularly successful at explaining behavioral and neural data in laboratory paradigms using abstract, stationary stimuli. It has been proposed, but with limited in-depth investigation so far, that similar decision-making mechanisms are involved in tasks of a more embodied nature, such as movement and locomotion, by directly accumulating externally measurable sensory quantities of which the precise, typically continuously time-varying, magnitudes are important for successful behavior. Here, we leverage collision threat detection as a task which is ecologically relevant in this sense, but which can also be rigorously observed and modelled in a laboratory setting. Conventionally, it is assumed that humans are limited in this task by a perceptual threshold on the optical expansion rate-the visual looming-of the obstacle. Using concurrent recordings of EEG and behavioral responses, we disprove this conventional assumption, and instead provide strong evidence that humans detect collision threats by accumulating the continuously time-varying visual looming signal. Generalizing existing accumulator model assumptions from stationary to time-varying sensory evidence, we show that our model accounts for previously unexplained empirical observations and full distributions of detection response. We replicate a pre-response centroparietal positivity (CPP) in scalp potentials, which has previously been found to correlate with accumulated decision evidence. In contrast with these existing findings, we show that our model is capable of predicting the onset of the CPP signature rather than its buildup, suggesting that neural evidence accumulation is implemented differently, possibly in distinct brain regions, in collision detection compared to previously studied paradigms.

Neurophysiology of Human Perceptual Decision-Making

The discovery of neural signals that reflect the dynamics of perceptual decision formation has had a considerable impact. Not only do such signals enable detailed investigations of the neural implementation of the decision-making process but they also can expose key elements of the brain's decision algorithms. For a long time, such signals were only accessible through direct animal brain recordings, and progress in human neuroscience was hampered by the limitations of noninvasive recording techniques. However, recent methodological advances are increasingly enabling the study of human brain signals that finely trace the dynamics of the unfolding decision process. In this review, we highlight how human neurophysiological data are now being leveraged to furnish new insights into the multiple processing levels involved in forming decisions, to inform the construction and evaluation of mathematical models that can explain intra- and interindividual differences, and to examine how key ancillary processes interact with core decision circuits.

Computational framework for investigating predictive processing in auditory perception

BACKGROUND

The brain tracks sound sources as they evolve in time, collecting contextual information to predict future sensory inputs. Previous work in predictive coding typically focuses on the perception of predictable stimuli, leaving the implementation of these same neural processes in more complex, real-world environments containing randomness and uncertainty up for debate.

NEW METHOD

To facilitate investigation into the perception of less tightly-controlled listening scenarios, we present a computational model as a tool to ask targeted questions about the underlying predictive processes that connect complex sensory inputs to listener behavior and neural responses. In the modeling framework, observed sound features (e.g. pitch) are tracked sequentially using Bayesian inference. Sufficient statistics are inferred from past observations at multiple time scales and used to make predictions about future observation while tracking the statistical structure of the sensory input.

RESULTS

Facets of the model are discussed in terms of their application to perceptual research, and examples taken from real-world audio demonstrate the model's flexibility to capture a variety of statistical structures along various perceptual dimensions.

COMPARISON WITH EXISTING METHODS

Previous models are often targeted toward interpreting a particular experimental paradigm (e.g., oddball paradigm), perceptual dimension (e.g., pitch processing), or task (e.g., speech segregation), thus limiting their ability to generalize to other domains. The presented model is designed as a flexible and practical tool for broad application.

CONCLUSION

The model is presented as a general framework for generating new hypotheses and guiding investigation into the neural processes underlying predictive coding of complex scenes.

The role of premature evidence accumulation in making difficult perceptual decisions under temporal uncertainty

The computations and neural processes underpinning decision making have primarily been investigated using highly simplified tasks in which stimulus onsets cue observers to start accumulating choice-relevant information. Yet, in daily life we are rarely afforded the luxury of knowing precisely when choice-relevant information will appear. Here, we examined neural indices of decision formation while subjects discriminated subtle stimulus feature changes whose timing relative to stimulus onset ('foreperiod') was uncertain. Joint analysis of behavioural error patterns and neural decision signal dynamics indicated that subjects systematically began the accumulation process before any informative evidence was presented, and further, that accumulation onset timing varied systematically as a function of the foreperiod of the preceding trial. These results suggest that the brain can adjust to temporal uncertainty by strategically modulating accumulation onset timing according to statistical regularities in the temporal structure of the sensory environment with particular emphasis on recent experience.

Illusory sound texture reveals multi-second statistical completion in auditory scene analysis

Sound sources in the world are experienced as stable even when intermittently obscured, implying perceptual completion mechanisms that “fill in” missing sensory information. We demonstrate a filling-in phenomenon in which the brain extrapolates the statistics of background sounds (textures) over periods of several seconds when they are interrupted by another sound, producing vivid percepts of illusory texture. The effect differs from previously described completion effects in that 1) the extrapolated sound must be defined statistically given the stochastic nature of texture, and 2) the effect lasts much longer, enabling introspection and facilitating assessment of the underlying representation. Illusory texture biases subsequent texture statistic estimates indistinguishably from actual texture, suggesting that it is represented similarly to actual texture. The illusion appears to represent an inference about whether the background is likely to continue during concurrent sounds, providing a stable statistical representation of the ongoing environment despite unstable sensory evidence.

Auditory textures are sounds defined by a particular statistical distribution, e.g. as is produced by rain, or a swarm of insects. Here, the authors describe a striking perceptual illusion in which sound textures are heard to continue, even though they have in fact been replaced by white noise.

Talker change detection: A comparison of human and machine performance

The automatic analysis of conversational audio remains difficult, in part, due to the presence of multiple talkers speaking in turns, often with significant intonation variations and overlapping speech. The majority of prior work on psychoacoustic speech analysis and system design has focused on single-talker speech or multi-talker speech with overlapping talkers (for example, the cocktail party effect). There has been much less focus on how listeners detect a change in talker or in probing the acoustic features significant in characterizing a talker's voice in conversational speech. This study examines human talker change detection (TCD) in multi-party speech utterances using a behavioral paradigm in which listeners indicate the moment of perceived talker change. Human reaction times in this task can be well-estimated by a model of the acoustic feature distance among speech segments before and after a change in talker, with estimation improving for models incorporating longer durations of speech prior to a talker change. Further, human performance is superior to several online and offline state-of-the-art machine TCD systems.

Detecting change in stochastic sound sequences

Our ability to parse our acoustic environment relies on the brain's capacity to extract statistical regularities from surrounding sounds. Previous work in regularity extraction has predominantly focused on the brain's sensitivity to predictable patterns in sound sequences. However, natural sound environments are rarely completely predictable, often containing some level of randomness, yet the brain is able to effectively interpret its surroundings by extracting useful information from stochastic sounds. It has been previously shown that the brain is sensitive to the marginal lower-order statistics of sound sequences (i.e., mean and variance). In this work, we investigate the brain's sensitivity to higher-order statistics describing temporal dependencies between sound events through a series of change detection experiments, where listeners are asked to detect changes in randomness in the pitch of tone sequences. Behavioral data indicate listeners collect statistical estimates to process incoming sounds, and a perceptual model based on Bayesian inference shows a capacity in the brain to track higher-order statistics. Further analysis of individual subjects' behavior indicates an important role of perceptual constraints in listeners' ability to track these sensory statistics with high fidelity. In addition, the inference model facilitates analysis of neural electroencephalography (EEG) responses, anchoring the analysis relative to the statistics of each stochastic stimulus. This reveals both a deviance response and a change-related disruption in phase of the stimulus-locked response that follow the higher-order statistics. These results shed light on the brain's ability to process stochastic sound sequences.

Adaptive and Selective Time Averaging of Auditory Scenes

To overcome variability, estimate scene characteristics, and compress sensory input, perceptual systems pool data into statistical summaries. Despite growing evidence for statistical representations in perception, the underlying mechanisms remain poorly understood. One example of such representations occurs in auditory scenes, where background texture appears to be represented with time-averaged sound statistics. We probed the averaging mechanism using "texture steps"-textures containing subtle shifts in stimulus statistics. Although generally imperceptible, steps occurring in the previous several seconds biased texture judgments, indicative of a multi-second averaging window. Listeners seemed unable to willfully extend or restrict this window but showed signatures of longer integration times for temporally variable textures. In all cases the measured timescales were substantially longer than previously reported integration times in the auditory system. Integration also showed signs of being restricted to sound elements attributed to a common source. The results suggest an integration process that depends on stimulus characteristics, integrating over longer extents when it benefits statistical estimation of variable signals and selectively integrating stimulus components likely to have a common cause in the world. Our methodology could be naturally extended to examine statistical representations of other types of sensory signals.

Evidence Integration in Natural Acoustic Textures during Active and Passive Listening

Many natural sounds can be well described on a statistical level, for example, wind, rain, or applause. Even though the spectro-temporal profile of these acoustic textures is highly dynamic, changes in their statistics are indicative of relevant changes in the environment. Here, we investigated the neural representation of change detection in natural textures in humans, and specifically addressed whether active task engagement is required for the neural representation of this change in statistics. Subjects listened to natural textures whose spectro-temporal statistics were modified at variable times by a variable amount. Subjects were instructed to either report the detection of changes (active) or to passively listen to the stimuli. A subset of passive subjects had performed the active task before (passive-aware vs passive-naive). Psychophysically, longer exposure to pre-change statistics was correlated with faster reaction times and better discrimination performance. EEG recordings revealed that the build-up rate and size of parieto-occipital (PO) potentials reflected change size and change time. Reduced effects were observed in the passive conditions. While P2 responses were comparable across conditions, slope and height of PO potentials scaled with task involvement. Neural source localization identified a parietal source as the main contributor of change-specific potentials, in addition to more limited contributions from auditory and frontal sources. In summary, the detection of statistical changes in natural acoustic textures is predominantly reflected in parietal locations both on the skull and source level. The scaling in magnitude across different levels of task involvement suggests a context-dependent degree of evidence integration.

Characterization of decision commitment rule alterations during an auditory change detection task

A critical component of decision making is determining when to commit to a choice. This involves stopping rules that specify the requirements for decision commitment. Flexibility of decision stopping rules provides an important means of control over decision-making processes. In many situations, these stopping rules establish a balance between premature decisions and late decisions. In this study we use a novel change detection paradigm to examine how subjects control this balance when invoking different decision stopping rules. The task design allows us to estimate the temporal weighting of sensory information for the decisions, and we find that different stopping rules did not result in systematic differences in that weighting. We also find bidirectional post-error alterations of decision strategy that depend on the type of error and effectively reduce the probability of making consecutive mistakes of the same type. This is a generalization to change detection tasks of the widespread observation of unidirectional post-error slowing in forced-choice tasks. On the basis of these results, we suggest change detection tasks as a promising paradigm to study the neural mechanisms that support flexible control of decision rules.NEW & NOTEWORTHY Flexible decision stopping rules confer control over decision processes. Using an auditory change detection task, we found that alterations of decision stopping rules did not result in systematic changes in the temporal weighting of sensory information. We also found that post-error alterations of decision stopping rules depended on the type of mistake subjects make. These results provide guidance for understanding the neural mechanisms that control decision stopping rules, one of the critical components of decision making and behavioral flexibility.

Signal integration at spherical bushy cells enhances representation of temporal structure but limits its range

Neuronal inhibition is crucial for temporally precise and reproducible signaling in the auditory brainstem. Previously we showed that for various synthetic stimuli, spherical bushy cell (SBC) activity in the Mongolian gerbil is rendered sparser and more reliable by subtractive inhibition (Keine et al., 2016). Here, employing environmental stimuli, we demonstrate that the inhibitory gain control becomes even more effective, keeping stimulated response rates equal to spontaneous ones. However, what are the costs of this modulation? We performed dynamic stimulus reconstructions based on neural population responses for auditory nerve (ANF) input and SBC output to assess the influence of inhibition on acoustic signal representation. Compared to ANFs, reconstructions of natural stimuli based on SBC responses were temporally more precise, but the match between acoustic and represented signal decreased. Hence, for natural sounds, inhibition at SBCs plays an even stronger role in achieving sparse and reproducible neuronal activity, while compromising general signal representation.