Auditory frequency perception adapts rapidly to the immediate past

Opposing serial dependencies revealed for sequences of auditory emotional stimuli

Our percept of the world is not solely determined by what we perceive and process at a given moment in time, but also depends on what we processed recently. In the present study, we investigate whether the perceived emotion of a spoken sentence is contingent upon the emotion of an auditory stimulus on the preceding trial (i.e., serial dependence). Thereto, participants were exposed to spoken sentences that varied in emotional affect by changing the prosody that ranged from 'happy' to 'fearful'. Participants were instructed to rate the emotion. We found a positive serial dependence for emotion processing whereby the perceived emotion was biased towards the emotion on the preceding trial. When we introduced 'no-go' trials (i.e., no rating was required), we found a negative serial dependence when participants knew in advance to withhold their response on a given trial (Experiment 2) and a positive serial dependence when participants received the information to withhold their response after the stimulus presentation (Experiment 3). We therefore established a robust serial dependence for emotion processing in speech and introduce a methodology to disentangle perceptual from post-perceptual processes. This approach can be applied to the vast majority of studies investigating sequential dependencies to separate positive from negative serial dependence.

Rule-based and stimulus-based cues bias auditory decisions via different computational and physiological mechanisms

Expectations, such as those arising from either learned rules or recent stimulus regularities, can bias subsequent auditory perception in diverse ways. However, it is not well understood if and how these diverse effects depend on the source of the expectations. Further, it is unknown whether different sources of bias use the same or different computational and physiological mechanisms. We examined how rule-based and stimulus-based expectations influenced behavior and pupil-linked arousal, a marker of certain forms of expectation-based processing, of human subjects performing an auditory frequency-discrimination task. Rule-based cues consistently biased choices and response times (RTs) toward the more-probable stimulus. In contrast, stimulus-based cues had a complex combination of effects, including choice and RT biases toward and away from the frequency of recently presented stimuli. These different behavioral patterns also had: 1) distinct computational signatures, including different modulations of key components of a novel form of a drift-diffusion decision model and 2) distinct physiological signatures, including substantial bias-dependent modulations of pupil size in response to rule-based but not stimulus-based cues. These results imply that different sources of expectations can modulate auditory processing via distinct mechanisms: one that uses arousal-linked, rule-based information and another that uses arousal-independent, stimulus-based information to bias the speed and accuracy of auditory perceptual decisions.

Prior information about upcoming stimuli can bias our perception of those stimuli. Whether different sources of prior information bias perception in similar or distinct ways is not well understood. We compared the influence of two kinds of prior information on tone-frequency discrimination: rule-based cues, in the form of explicit information about the most-likely identity of the upcoming tone; and stimulus-based cues, in the form of sequences of tones presented before the to-be-discriminated tone. Although both types of prior information biased auditory decision-making, they demonstrated distinct behavioral, computational, and physiological signatures. Our results suggest that the brain processes prior information in a form-specific manner rather than utilizing a general-purpose prior. Such form-specific processing has implications for understanding decision biases real-world contexts, in which prior information comes from many different sources and modalities.

No differences between adults with and without autism in audiovisual synchrony perception

LAY ABSTRACT

It has been known for a long time that individuals diagnosed with autism spectrum disorder perceive the world differently. In this study, we investigated how people with or without autism perceive visual and auditory information. We know that an auditory and a visual stimulus do not have to be perfectly synchronous for us to perceive them as synchronous: first, when the two are within a certain time window (temporal binding window), the brain will tell us that they are synchronous. Second, the brain can also adapt quickly to audiovisual asynchronies (rapid recalibration). Although previous studies have shown that people with autism spectrum disorder have different temporal binding windows, and less rapid recalibration, we did not find these differences in our study. However, we did find that both processes develop over age, and since previous studies tested only young people (children, adolescents, and young adults), and we tested adults from 18 to 55 years, this might explain the different findings. In the end, there might be quite a complex story, where people with and without autism spectrum disorder perceive the world differently, even dependent on how old they are.

Adaptive auditory brightness perception

Perception adapts to the properties of prior stimulation, as illustrated by phenomena such as visual color constancy or speech context effects. In the auditory domain, only little is known about adaptive processes when it comes to the attribute of auditory brightness. Here, we report an experiment that tests whether listeners adapt to spectral colorations imposed on naturalistic music and speech excerpts. Our results indicate consistent contrastive adaptation of auditory brightness judgments on a trial-by-trial basis. The pattern of results suggests that these effects tend to grow with an increase in the duration of the adaptor context but level off after around 8 trials of 2 s duration. A simple model of the response criterion yields a correlation of r = .97 with the measured data and corroborates the notion that brightness perception adapts on timescales that fall in the range of auditory short-term memory. Effects turn out to be similar for spectral filtering based on linear spectral filter slopes and filtering based on a measured transfer function from a commercially available hearing device. Overall, our findings demonstrate the adaptivity of auditory brightness perception under realistic acoustical conditions.

Serial Dependence of Emotion Within and Between Stimulus Sensory Modalities

How we perceive the world is not solely determined by what we sense at a given moment in time, but also by what we processed recently. Here we investigated whether such serial dependencies for emotional stimuli transfer from one modality to another. Participants were presented a random sequence of emotional sounds and images and instructed to rate the valence and arousal of each stimulus (Experiment 1). For both ratings, we conducted an intertrial analysis, based on whether the rating on the previous trial was low or high. We found a positive serial dependence for valence and arousal regardless of the stimulus modality on two consecutive trials. In Experiment 2, we examined whether passively perceiving a stimulus is sufficient to induce a serial dependence. In Experiment 2, participants were instructed to rate the stimuli only on active trials and not on passive trials. The participants were informed that the active and passive trials were presented in alternating order, so that they were able to prepare for the task. We conducted an intertrial analysis on active trials, based on whether the rating on the previous passive trial (determined in Experiment 1) was low or high. For both ratings, we again observed positive serial dependencies regardless of the stimulus modality. We conclude that the emotional experience triggered by one stimulus affects the emotional experience for a subsequent stimulus regardless of their sensory modalities, that this occurs in a bottom-up fashion, and that this can be explained by residual activation in the emotional network in the brain.

Task-dependent audiovisual temporal sensitivity is not affected by stimulus intensity levels

Flexibility and robustness of multisensory temporal recalibration is paramount for maintaining perceptual constancy of the surrounding natural world. Different environments impart various impediments, distances and routes that alter the propagation times of sight and sound cues comprising a multimodal event. One's ability to rapidly calibrate and account for these external variations allows for maintained perception of synchrony which is crucial for coherent and consistent perception. The two common paradigms used to compare precision of temporal processing between experimental and control groups, the simultaneity judgment (SJ) and temporal order judgment (TOJ) tasks, often use supra-threshold stimuli. However, few studies have specifically examined the effects of normalizing stimulus intensities to participant's unisensory detection thresholds. The current project presented multiple combinations of auditory and visual stimulus intensity levels, based on individual detection thresholds, during a TOJ and a SJ task. While no effect of stimulus intensity was found on temporal sensitivity or perceived temporal synchrony, there was a significant difference in point of subjective simultaneity (PSS) measures between tasks. In addition, PSS estimates were audio-leading, rather than visual-leading as previously reported, suggesting that exposure to the particular combinations of stimulus intensity levels used influenced temporal synchrony perception. Overall, these results support the use of supra-threshold stimuli in TOJ and SJ tasks as a way of minimizing the confound from differences in unisensory processing.

Helmholtz Versus Haute Couture: How Horizontal Stripes and Dark Clothes Make You Look Thinner

In Helmholtz’s illusion, a square with horizontal stripes appears taller than an identical square with vertical stripes. This effect has also been observed in experiments with human stimuli, where a human figure wearing a dress with horizontal stripes appears thinner than a drawing clad in vertical stripes. These findings do not agree with the common belief that clothes with horizontal stripes make someone appear wider, neither do they disentangle whether the horizontal or vertical stripes account for the thinning effect. In the present study, we focused on the effect of horizontal stripes in clothes comparing horizontal stripes against no-stripes (not against vertical; Experiments 1 and 2), using photos of a real-life female model, and controlling for the average luminance of the stripes (Experiment 2). Results showed that horizontal stripes and lower luminance have—independently—a small-to-moderate thinning effect on the perceived size of the body, and the effect is larger when the two variables are combined. In Experiment 3, we further show that the thinning effect due to the luminance of the dress is enhanced when the general background gets darker.

Auditory Rate Perception Displays a Positive Serial Dependence

We investigated perceived timing in auditory rate perception using a reproduction task. The study aimed to test (a) whether central tendency occurs in rate perception, as shown for interval timing, and (b) whether rate is perceived independently on each trial or shows a serial dependence, as shown for other perceptual attributes. Participants were well able to indicate perceived rate as reproduced and presented rates were linearly related with a slope that approached unity, although tapping significantly overestimated presented rates. While the slopes approached unity, they were significantly less than 1, indicating a central tendency in which reproduced rates tended towards the mean of the presented range. We tested for serial dependency by seeing if current trial rate reproductions depended on the preceding rate. In two conditions, a positive dependence was observed. A third condition in which participants withheld responses on every second trial produced a negative dependency. These results suggest separate components of serial dependence linked to stimulus and response: Withholding responses reveals a negative perceptual effect, whereas making responses adds a stronger positive effect that is postperceptual and makes the combined effect positive. Together, these data show that auditory rate perception exhibits both central tendency and serial dependence effects.

Auditory and tactile frequency representations are co-embedded in modality-defined cortical sensory systems

Sensory information is represented and elaborated in hierarchical cortical systems that are thought to be dedicated to individual sensory modalities. This traditional view of sensory cortex organization has been challenged by recent evidence of multimodal responses in primary and association sensory areas. Although it is indisputable that sensory areas respond to multiple modalities, it remains unclear whether these multimodal responses reflect selective information processing for particular stimulus features. Here, we used fMRI adaptation to identify brain regions that are sensitive to the temporal frequency information contained in auditory, tactile, and audiotactile stimulus sequences. A number of brain regions distributed over the parietal and temporal lobes exhibited frequency-selective temporal response modulation for both auditory and tactile stimulus events, as indexed by repetition suppression effects. A smaller set of regions responded to crossmodal adaptation sequences in a frequency-dependent manner. Despite an extensive overlap of multimodal frequency-selective responses across the parietal and temporal lobes, representational similarity analysis revealed a cortical "regional landscape" that clearly reflected distinct somatosensory and auditory processing systems that converged on modality-invariant areas. These structured relationships between brain regions were also evident in spontaneous signal fluctuation patterns measured at rest. Our results reveal that multimodal processing in human cortex can be feature-specific and that multimodal frequency representations are embedded in the intrinsically hierarchical organization of cortical sensory systems.

Positive sequential dependency for face attractiveness perception

Recent findings from several groups have demonstrated that visual perception at a given moment can be biased toward what was recently seen. This is true both for basic visual attributes and for more complex representations, such as face identity, gender, or expression. This assimilation to the recent past is a positive serial dependency, similar to a temporal averaging process that capitalizes on short-term correlations in visual input to reduce noise and boost perceptual continuity. Here we examine serial dependencies in face perception using a simple attractiveness rating task and a rapid series of briefly presented face stimuli. In a series of three experiments, our results confirm a previous report that face attractiveness exhibits a positive serial dependency. This intertrial effect is not only determined by face attractiveness on the previous trial, but also depends on the faces shown up to five trials back. We examine the effect of stimulus presentation duration and find that stimuli as brief as 56 ms produce a significant positive dependency similar in magnitude to that produced by stimuli presented for 1,000 ms. We observed stronger positive dependencies between same-gender faces, and found a task dependency: Alternating gender discrimination trials with attractiveness rating trials produced no serial dependency. In sum, these findings show that a perception-stabilizing assimilation effect operates in face attractiveness perception that is task dependent and is acquired surprisingly quickly.

Visual serial dependence in an audiovisual stimulus

Serial dependence is a phenomenon that biases the perception of features or objects systematically toward sensory input from the recent past (Fischer & Whitney, 2014). There is an active debate whether this effect is rooted directly in perception or reflects biases in decision making. We investigated serial dependence across three experiments by manipulating the decision made on each trial. A multimodal audiovisual stimulus comprising a Gabor and a vowel sound was presented repeatedly. On each trial, participants reported either the Gabor orientation or the vowel sound. Participants either ignored one modality (Experiment 1) or attended to both modalities (Experiments 2 and 3). In Experiments 2 and 3, the response task was randomized to prevent anticipating which modality to respond to until the response phase. In Experiment 3, no-response trials were additionally interleaved. Results across the three experiments demonstrated serial dependence only when participants reported the visual modality. Serial dependence was also present in visual reports when participants completed auditory reports or made no reports on previous trials. The previous stimulus alone was enough to elicit an effect. Serial dependence is unlikely to be an effect of the previous decision on the stimulus, but rather an effect of perceiving the previous stimulus.

Multisensory perceptual interactions between higher-order temporal frequency signals

Naturally occurring signals in audition and touch can be complex and marked by temporal variations in frequency and amplitude. Auditory frequency sweep processing has been studied extensively; however, much less is known about sweep processing in touch because studies have primarily focused on the perception of simple sinusoidal vibrations. Given the extensive interactions between audition and touch in the frequency processing of pure tone signals, we reasoned that these senses might also interact in the processing of higher-order frequency representations like sweeps. In a series of psychophysical experiments, we characterized the influence of auditory distractors on the ability of participants to discriminate tactile frequency sweeps. Auditory frequency sweeps systematically biased the tactile perception of sweep direction. Importantly, auditory cues exerted little influence on tactile sweep direction perception when the sounds and vibrations occupied different absolute frequency ranges or when the sounds consisted of intensity sweeps. Thus, audition and touch interact in frequency sweep perception in a frequency- and feature-specific manner. Our results demonstrate that audio-tactile interactions are not constrained to the processing of simple sinusoids. Because higher-order frequency representations may be synthesized from simpler representations, our findings imply that multisensory interactions in the temporal frequency domain span multiple hierarchical levels in sensory processing. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

A Thickness Illusion: Horizontal Is Perceived as Thicker than Vertical

We report two psychophysical experiments that investigate a visual illusion that is considered common knowledge among type designers, but has never been studied scientifically. Specifically, the thickness of a horizontal line is overestimated in relation to that of a vertical line. Experiment 1 confirmed the existence of the illusion. In Experiment 2, we replicated the effect and showed that the illusion is closely related to the vertical-horizontal illusion, in which the length of a vertical line is overestimated in comparison to a horizontal one. Both the overestimation of thickness and length is larger when the stimulus is surrounded by a horizontally elongated frame, as opposed to a vertically elongated frame. We discuss potential explanations for the thickness illusion and its relation to the vertical-horizontal illusion.

Rapid Audiovisual Temporal Recalibration Generalises Across Spatial Location

Recent exposure to asynchronous multisensory signals has been shown to shift perceived timing between the sensory modalities, a phenomenon known as 'temporal recalibration'. Recently, Van der Burg et al. (2013, J Neurosci, 33, pp. 14633-14637) reported results showing that recalibration to asynchronous audiovisual events can happen extremely rapidly. In an extended series of variously asynchronous trials, simultaneity judgements were analysed based on the modality order in the preceding trial and showed that shifts in the point of subjective synchrony occurred almost instantaneously, shifting from one trial to the next. Here we replicate the finding that shifts in perceived timing occur following exposure to a single, asynchronous audiovisual stimulus and by manipulating the spatial location of the audiovisual events we demonstrate that recalibration occurs even when the adapting stimulus is presented in a different location. Timing shifts were also observed when the adapting audiovisual pair were defined only by temporal proximity, with the auditory component presented over headphones rather than being collocated with the visual stimulus. Combined with previous findings showing that timing shifts are independent of stimulus features such as colour and pitch, our finding that recalibration is not spatially specific provides strong evidence for a rapid recalibration process that is solely dependent on recent temporal information, regardless of feature or location. These rapid and automatic shifts in perceived synchrony may allow our sensory systems to flexibly adjust to the variation in timing of neural signals occurring as a result of delayed environmental transmission and differing neural latencies for processing vision and audition.

Sequential dependencies in pitch judgments

Studies that measure pitch discrimination relate a subject's response on each trial to the stimuli presented on that trial, but there is evidence that behavior depends also on earlier stimulation. Here, listeners heard a sequence of tones and reported after each tone whether it was higher or lower in pitch than the previous tone. Frequencies were determined by an adaptive staircase targeting 75% correct, with interleaved tracks to ensure independence between consecutive frequency changes. Responses for this specific task were predicted by a model that took into account the frequency interval on the current trial, as well as the interval and response on the previous trial. This model was superior to simpler models. The dependence on the previous interval was positive (assimilative) for all subjects, consistent with persistence of the sensory trace. The dependence on the previous response was either positive or negative, depending on the subject, consistent with a subject-specific suboptimal response strategy. It is argued that a full stimulus + response model is necessary to account for effects of stimulus history and obtain an accurate estimate of sensory noise.

A Matched Comparison Across Three Different Sensory Pairs of Cross-Modal Temporal Recalibration From Sustained and Transient Adaptation

Sustained exposure to an asynchronous multisensory signal causes perceived simultaneity to shift in the direction of the leading component of the adapting stimulus. This is known as temporal recalibration, and recent evidence suggests that it can occur very rapidly, even after a single asynchronous audiovisual (AV) stimulus. However, this form of rapid recalibration appears to be unique to AV stimuli, in contrast to recalibration following sustained asynchronies which occurs with audiotactile (AT) and visuotactile (VT) stimuli. This study examines temporal recalibration to AV, VT and AT asynchrony with spatially collocated stimuli using a design that produces both sustained and inter-trial recalibration by combining the traditional sustained adaptation approach with an inter-trial analysis of sequential dependencies in an extended test period. Thus, we compare temporal recalibration to both sustained and transient asynchrony in three crossmodal combinations using the same design, stimuli and observers. The results reveal that prolonged exposure to asynchrony produced equivalent temporal recalibration for all combinations: AV, AT and VT. The pattern for rapid, inter-trial recalibration was very different. Rapid recalibration occurred strongly for AV stimuli, weakly for AT and did not occur at all for VT. For all sensory pairings, recalibration from sustained asynchrony decayed to baseline during the test phase while inter-trial recalibration was present and stable throughout testing, suggesting different mechanisms may underlie adaptation at long and short timescales.

Prior context in audition informs binding and shapes simple features

A perceptual phenomenon is reported, whereby prior acoustic context has a large, rapid and long-lasting effect on a basic auditory judgement. Pairs of tones were devised to include ambiguous transitions between frequency components, such that listeners were equally likely to report an upward or downward 'pitch' shift between tones. We show that presenting context tones before the ambiguous pair almost fully determines the perceived direction of shift. The context effect generalizes to a wide range of temporal and spectral scales, encompassing the characteristics of most realistic auditory scenes. Magnetoencephalographic recordings show that a relative reduction in neural responsivity is correlated to the behavioural effect. Finally, a computational model reproduces behavioural results, by implementing a simple constraint of continuity for binding successive sounds in a probabilistic manner. Contextual processing, mediated by ubiquitous neural mechanisms such as adaptation, may be crucial to track complex sound sources over time.

Linear Summation of Repulsive and Attractive Serial Dependencies: Orientation and Motion Dependencies Sum in Motion Perception

Recent work from several groups has shown that perception of various visual attributes in human observers at a given moment is biased toward what was recently seen. This positive serial dependency is a kind of temporal averaging that exploits short-term correlations in visual scenes to reduce noise and stabilize perception. To date, this stabilizing "continuity field" has been demonstrated on stable visual attributes such as orientation and face identity, yet it would be counterproductive to apply it to dynamic attributes in which change sensitivity is needed. Here, we tested this using motion direction discrimination and predict a negative perceptual dependency: a contrastive relationship that enhances sensitivity to change. Surprisingly, our data showed a cubic-like pattern of dependencies with positive and negative components. By interleaving various stimulus combinations, we separated the components and isolated a positive perceptual dependency for motion and a negative dependency for orientation. A weighted linear sum of the separate dependencies described the original cubic pattern well. The positive dependency for motion shows an integrative perceptual effect and was unexpected, although it is consistent with work on motion priming. These findings suggest that a perception-stabilizing continuity field occurs pervasively, occurring even when it obscures sensitivity to dynamic stimuli.SIGNIFICANCE STATEMENT Recent studies show that visual perception at a given moment is not entirely veridical, but rather biased toward recently seen stimuli: a positive serial dependency. This temporal smoothing process helps perceptual continuity by preserving stable aspects of the visual scene over time, yet, for dynamic stimuli, temporal smoothing would blur dynamics and reduce sensitivity to change. We tested whether this process is selective for stable attributes by examining dependencies in motion perception. We found a clear positive dependency for motion, suggesting that positive perceptual dependencies are pervasive. We also found a concurrent negative (contrastive) dependency for orientation. Both dependencies combined linearly to determine perception, showing that the brain can calculate contrastive and integrative dependencies simultaneously from recent stimulus history when making perceptual decisions.

Auditory adaptation improves tactile frequency perception

Our ability to process temporal frequency information by touch underlies our capacity to perceive and discriminate surface textures. Auditory signals, which also provide extensive temporal frequency information, can systematically alter the perception of vibrations on the hand. How auditory signals shape tactile processing is unclear; perceptual interactions between contemporaneous sounds and vibrations are consistent with multiple neural mechanisms. Here we used a crossmodal adaptation paradigm, which separated auditory and tactile stimulation in time, to test the hypothesis that tactile frequency perception depends on neural circuits that also process auditory frequency. We reasoned that auditory adaptation effects would transfer to touch only if signals from both senses converge on common representations. We found that auditory adaptation can improve tactile frequency discrimination thresholds. This occurred only when adaptor and test frequencies overlapped. In contrast, auditory adaptation did not influence tactile intensity judgments. Thus auditory adaptation enhances touch in a frequency- and feature-specific manner. A simple network model in which tactile frequency information is decoded from sensory neurons that are susceptible to auditory adaptation recapitulates these behavioral results. Our results imply that the neural circuits supporting tactile frequency perception also process auditory signals. This finding is consistent with the notion of supramodal operators performing canonical operations, like temporal frequency processing, regardless of input modality.NEW & NOTEWORTHY Auditory signals can influence the tactile perception of temporal frequency. Multiple neural mechanisms could account for the perceptual interactions between contemporaneous auditory and tactile signals. Using a crossmodal adaptation paradigm, we found that auditory adaptation causes frequency- and feature-specific improvements in tactile perception. This crossmodal transfer of aftereffects between audition and touch implies that tactile frequency perception relies on neural circuits that also process auditory frequency.

Face familiarity promotes stable identity recognition: exploring face perception using serial dependence

Studies suggest that familiar faces are processed in a manner distinct from unfamiliar faces and that familiarity with a face confers an advantage in identity recognition. Our visual system seems to capitalize on experience to build stable face representations that are impervious to variation in retinal input that may occur due to changes in lighting, viewpoint, viewing distance, eye movements, etc. Emerging evidence also suggests that our visual system maintains a continuous perception of a face's identity from one moment to the next despite the retinal input variations through serial dependence. This study investigates whether interactions occur between face familiarity and serial dependence. In two experiments, participants used a continuous scale to rate attractiveness of unfamiliar and familiar faces (either experimentally learned or famous) presented in rapid sequences. Both experiments revealed robust inter-trial effects in which attractiveness ratings for a given face depended on the preceding face's attractiveness. This inter-trial attractiveness effect was most pronounced for unfamiliar faces. Indeed, when participants were familiar with a given face, attractiveness ratings showed significantly less serial dependence. These results represent the first evidence that familiar faces can resist the temporal integration seen in sequential dependencies and highlight the importance of familiarity to visual cognition.

Love at second sight: Sequential dependence of facial attractiveness in an on-line dating paradigm

Millions of people use online dating sites each day, scanning through streams of face images in search of an attractive mate. Face images, like most visual stimuli, undergo processes whereby the current percept is altered by exposure to previous visual input. Recent studies using rapid sequences of faces have found that perception of face identity is biased towards recently seen faces, promoting identity-invariance over time, and this has been extended to perceived face attractiveness. In this paper we adapt the rapid sequence task to ask a question about mate selection pertinent in the digital age. We designed a binary task mimicking the selection interface currently popular in online dating websites in which observers typically make binary decisions (attractive or unattractive) about each face in a sequence of unfamiliar faces. Our findings show that binary attractiveness decisions are not independent: we are more likely to rate a face as attractive when the preceding face was attractive than when it was unattractive.