Probing the functions of contextual modulation by adapting images rather than observers

Designing for sensory adaptation: what you see depends on what you've been looking at - Recommendations, guidelines and standards should reflect this

Sensory systems continuously recalibrate their responses according to the current stimulus environment. As a result, perception is strongly affected by the current and recent context. These adaptative changes affect both sensitivity (e.g., habituating to noise, seeing better in the dark) and appearance (e.g. how things look, what catches attention) and adjust to many perceptual properties (e.g. from light level to the characteristics of someone's face). They therefore have a profound effect on most perceptual experiences, and on how and how well the senses work in different settings. Characterizing the properties of adaptation, how it manifests, and when it influences perception in modern environments can provide insights into the diversity of human experience. Adaptation could also be leveraged both to optimize perceptual abilities (e.g. in visual inspection tasks like radiology) and to mitigate unwanted consequences (e.g. exposure to potentially unhealthy stimulus environments).

Modeling facial perception in group context from a serial perception perspective

By utilizing statistical properties and summary statistics, the visual system can efficiently integrate perception of spatially and temporally adjacent stimuli into perception of a given target. For instance, perception of a target face can either be biased positively toward previous faces (e.g. the serial dependence effect) or be biased negatively by surrounding faces in the same trial/space (e.g. spatial ensemble averaging). However, both aspects were investigated separately. As spatial and temporal processing share the same purpose to reduce redundancy in visual processing, if one statistical processing occurs, would the statistical processing in the other domain still exist or be discarded? We investigated this question by exploring whether serial dependence of face perception (of attractiveness and averageness) survives when the changed face perception in the group context occurs. The results of Markov Chain modeling and conventional methods suggested that serial dependence (the temporal aspect) co-occurs with changed face perception in the group context (the spatial aspect). We also utilized the Hidden Markov modeling, as a new mathematical method, to model statistical processing from both domains. The results confirmed the co-occurrence of temporal effect and changed face perception in the group context for both attractiveness and averageness, suggesting potentially different spatial and temporal compression mechanisms in high-level vision. Further modeling and cluster analysis further revealed that the detailed computation of spatially and temporally adjacent faces in the attractiveness and averageness processing were similar yet different among different individuals. This work builds a bridge to understanding mathematical principles underlying changed face perception in the group context from the serial perspective.

Color appearance model incorporating contrast adaptation - implications for individual differences in color vision

Color appearance models use standard color matching functions to derive colorimetric information from spectral radiometric measurements of a visual environment, and they process that information to predict color perceptual attributes such as hue, chroma and lightness. That processing is usually done by equations with fixed numerical coefficients that were predetermined to yield optimal agreement for a given standard observer. Here we address the well-known fact that, among color-normal observers, there are significant differences of color matching functions. These cause disagreements between individuals as to whether certain colors match, an important effect that is often called observer metamerism. Yet how these individual sensitivity differences translate into differences in perceptual metrics is not fully addressed by many appearance models. It might seem that appearance could be predicted by substituting an individual's color matching functions into an otherwise-unchanged color appearance model, but this is problematic because the model's coefficients were not optimized for the new observer. Here we explore a solution guided by the idea that processes of adaptation in the visual system tend to compensate color perception for differences in cone responses and consequent color matching functions. For this purpose, we developed a simple color appearance model that uses only a few numerical coefficients, yet accurately predicts the perceptual attributes of Munsell samples under a selected standard lighting condition. We then added a feedback loop to automatically adjust the model coefficients, in response to switching between cone fundamentals simulating different observers and color matching functions. This adjustment is intended to model long term contrast adaptation in the vision system by maintaining average overall color contrast levels. Incorporating this adaptation principle into color appearance models could allow better assessments of displays and illumination systems, to help improve color appearances for most observers.

The Verriest Lecture: Adventures in blue and yellow

Conventional models of color vision assume that blue and yellow (along with red and green) are the fundamental building blocks of color appearance, yet how these hues are represented in the brain and whether and why they might be special are questions that remain shrouded in mystery. Many studies have explored the visual encoding of color categories, from the statistics of the environment to neural processing to perceptual experience. Blue and yellow are tied to salient features of the natural color world, and these features have likely shaped several important aspects of color vision. However, it remains less certain that these dimensions are encoded as primary or "unique" in the visual representation of color. There are also striking differences between blue and yellow percepts that may reflect high-level inferences about the world, specifically about the colors of light and surfaces. Moreover, while the stimuli labeled as blue or yellow or other basic categories show a remarkable degree of constancy within the observer, they all vary independently of one another across observers. This pattern of variation again suggests that blue and yellow and red and green are not a primary or unitary dimension of color appearance, and instead suggests a representation in which different hues reflect qualitatively different categories rather than quantitative differences within an underlying low-dimensional "color space."

Individual differences and their implications for color perception

Individual differences are a conspicuous feature of color vision and arise from many sources, in both the observer and the world. These differences have important practical implications for comparing and correcting perception and performance, and important theoretical implications for understanding the design principles underlying color coding. Color percepts within and between individuals often vary less than the variations in spectral sensitivity might predict. This stability is achieved by a variety of processes that compensate perception for the sensitivity limits of the eye and brain. Yet judgments of color between individuals can also vary widely, and in ways that are not readily explained by differences in sensitivity or the environment. These differences are uncorrelated across different color categories, and could reflect how these categories are learned or represented.

Visual adaptation and the amplitude spectra of radiological images

We examined how visual sensitivity and perception are affected by adaptation to the characteristic amplitude spectra of X-ray mammography images. Because of the transmissive nature of X-ray photons, these images have relatively more low-frequency variability than natural images, a difference that is captured by a steeper slope of the amplitude spectrum (~ − 1.5) compared to the ~ 1/f (slope of − 1) spectra common to natural scenes. Radiologists inspecting these images are therefore exposed to a different balance of spectral components, and we measured how this exposure might alter spatial vision. Observers (who were not radiologists) were adapted to images of normal mammograms or the same images sharpened by filtering the amplitude spectra to shallower slopes. Prior adaptation to the original mammograms significantly biased judgments of image focus relative to the sharpened images, demonstrating that the images are sufficient to induce substantial after-effects. The adaptation also induced strong losses in threshold contrast sensitivity that were selective for lower spatial frequencies, though these losses were very similar to the threshold changes induced by the sharpened images. Visual search for targets (Gaussian blobs) added to the images was also not differentially affected by adaptation to the original or sharper images. These results complement our previous studies examining how observers adapt to the textural properties or phase spectra of mammograms. Like the phase spectrum, adaptation to the amplitude spectrum of mammograms alters spatial sensitivity and visual judgments about the images. However, unlike the phase spectrum, adaptation to the amplitude spectra did not confer a selective performance advantage relative to more natural spectra.

Visualizing Visual Adaptation

Many techniques have been developed to visualize how an image would appear to an individual with a different visual sensitivity: e.g., because of optical or age differences, or a color deficiency or disease. This protocol describes a technique for incorporating sensory adaptation into the simulations. The protocol is illustrated with the example of color vision, but is generally applicable to any form of visual adaptation. The protocol uses a simple model of human color vision based on standard and plausible assumptions about the retinal and cortical mechanisms encoding color and how these adjust their sensitivity to both the average color and range of color in the prevailing stimulus. The gains of the mechanisms are adapted so that their mean response under one context is equated for a different context. The simulations help reveal the theoretical limits of adaptation and generate "adapted images" that are optimally matched to a specific environment or observer. They also provide a common metric for exploring the effects of adaptation within different observers or different environments. Characterizing visual perception and performance with these images provides a novel tool for studying the functions and consequences of long-term adaptation in vision or other sensory systems.

Adaptation and visual search in mammographic images

Radiologists face the visually challenging task of detecting suspicious features within the complex and noisy backgrounds characteristic of medical images. We used a search task to examine whether the salience of target features in x-ray mammograms could be enhanced by prior adaptation to the spatial structure of the images. The observers were not radiologists, and thus had no diagnostic training with the images. The stimuli were randomly selected sections from normal mammograms previously classified with BIRADS Density scores of "fatty" versus "dense," corresponding to differences in the relative quantities of fat versus fibroglandular tissue. These categories reflect conspicuous differences in visual texture, with dense tissue being more likely to obscure lesion detection. The targets were simulated masses corresponding to bright Gaussian spots, superimposed by adding the luminance to the background. A single target was randomly added to each image, with contrast varied over five levels so that they varied from difficult to easy to detect. Reaction times were measured for detecting the target location, before or after adapting to a gray field or to random sequences of a different set of dense or fatty images. Observers were faster at detecting the targets in either dense or fatty images after adapting to the specific background type (dense or fatty) that they were searching within. Thus, the adaptation led to a facilitation of search performance that was selective for the background texture. Our results are consistent with the hypothesis that adaptation allows observers to more effectively suppress the specific structure of the background, thereby heightening visual salience and search efficiency.

Visual Adaptation

Sensory systems continuously mold themselves to the widely varying contexts in which they must operate. Studies of these adaptations have played a long and central role in vision science. In part this is because the specific adaptations remain a powerful tool for dissecting vision, by exposing the mechanisms that are adapting. That is, "if it adapts, it's there." Many insights about vision have come from using adaptation in this way, as a method. A second important trend has been the realization that the processes of adaptation are themselves essential to how vision works, and thus are likely to operate at all levels. That is, "if it's there, it adapts." This has focused interest on the mechanisms of adaptation as the target rather than the probe. Together both approaches have led to an emerging insight of adaptation as a fundamental and ubiquitous coding strategy impacting all aspects of how we see.