What #theDress reveals about the role of illumination priors in color perception and color constancy

Color constancy in real-world settings

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

Explaining individual differences

Most psychological research aims to uncover generalizations about the mind that hold across subjects. Philosophical discussions of scientific explanation have focused on such generalizations, but in doing so, have often overlooked an important phenomenon: variation. Variation is ubiquitous in psychology and many other domains, and an important target of explanation in its own right. Here I characterize explananda that concern individual differences and formulate an account of what it takes to explain them. I argue that the notion of actual difference making, the only causal concept in the literature that explicitly addresses variation, cannot be used to ground such an account. Instead, I propose a view on which explaining individual differences involves identifying causes that could be intervened on to reduce the variability in the population. This account provides criteria of success for explaining variation and deepens our understanding of causal explanation.

Computational modeling of color perception with biologically plausible spiking neural networks

Biologically plausible computational modeling of visual perception has the potential to link high-level visual experiences to their underlying neurons' spiking dynamic. In this work, we propose a neuromorphic (brain-inspired) Spiking Neural Network (SNN)-driven model for the reconstruction of colorful images from retinal inputs. We compared our results to experimentally obtained V1 neuronal activity maps in a macaque monkey using voltage-sensitive dye imaging and used the model to demonstrate and critically explore color constancy, color assimilation, and ambiguous color perception. Our parametric implementation allows critical evaluation of visual phenomena in a single biologically plausible computational framework. It uses a parametrized combination of high and low pass image filtering and SNN-based filling-in Poisson processes to provide adequate color image perception while accounting for differences in individual perception.

Do You See What I See? Diversity in Human Color Perception

In our tendency to discuss the objective properties of the external world, we may fail to notice that our subjective perceptions of those properties differ between individuals. Variability at all levels of the color vision system creates diversity in color perception, from discrimination to color matching, appearance, and subjective experience, such that each of us lives in a unique perceptual world. In this review, I discuss what is known about individual differences in color perception and its determinants, particularly considering genetically mediated variability in cone photopigments and the paradoxical effects of visual environments in both contributing to and counteracting individual differences. I make the case that, as well as being of interest in their own right and crucial for a complete account of color vision, individual differences can be used as a methodological tool in color science for the insights that they offer about the underlying mechanisms of perception. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

What colour are your eyes? Teaching the genetics of eye colour & colour vision. Edridge Green Lecture RCOphth Annual Congress Glasgow May 2019

Eye colour and colour perception are excellent examples to use when teaching genetics as they encompass not simply the basic Mendelian genetics of dominant, recessive and X-linked disorders, but also many of the new concepts such as non-allelic diseases, polygenic disease, phenocopies, genome-wide association study (GWAS), founder effects, gene-environment interaction, evolutionary drivers for variations, copy number variation, insertions deletions, methylation and gene inactivation. Beyond genetics, colour perception touches on concepts involving optics, physics, physiology and psychology and can capture the imagination of the population, as we saw with social media trend of "#the dress". Television shows such as Game of Thrones focused attention on the eye colour of characters, as well as their Dire-wolves and Dragons. These themes in popular culture can be leveraged as tools to teach and engage everyone in genetics, which is now a key component in all eye diseases. As the explosion of data from genomics, big data and artificial intelligence transforms medicine, ophthalmologists need to be genetically literate. Genetics is relevant, not just for Inherited Retinal Diseases and congenital abnormalities but also for the leading causes of blindness: age-related macular degeneration, glaucoma, myopia, diabetic retinopathy and cataract. Genetics should be part of the armamentarium of every practicing ophthalmologist. We need to ask every patient about their family history. In the near future, patients will attend eye clinics with genetic results showing they are at high risk of certain eye diseases and ophthalmologists will need to know how to screen, follow and treat these patients.

Crystal or jelly? Effect of color on the perception of translucent materials with photographs of real-world objects

Translucent materials are ubiquitous in nature (e.g. teeth, food, and wax), but our understanding of translucency perception is limited. Previous work in translucency perception has mainly used monochromatic rendered images as stimuli, which are restricted by their diversity and realism. Here, we measure translucency perception with photographs of real-world objects. Specifically, we use three behavior tasks: binary classification of "translucent" versus "opaque," semantic attribute rating of perceptual qualities (see-throughness, glossiness, softness, glow, and density), and material categorization. Two different groups of observers finish the three tasks with color or grayscale images. We find that observers' agreements depend on the physical material properties of the objects such that translucent materials generate more interobserver disagreements. Further, there are more disagreements among observers in the grayscale condition in comparison to that in the color condition. We also discover that converting images to grayscale substantially affects the distributions of attribute ratings for some images. Furthermore, ratings of see-throughness, glossiness, and glow could predict individual observers' binary classification of images in both grayscale and color conditions. Last, converting images to grayscale alters the perceived material categories for some images such that observers tend to misjudge images of food as non-food and vice versa. Our result demonstrates that color is informative about material property estimation and recognition. Meanwhile, our analysis shows that mid-level semantic estimation of material attributes might be closely related to high-level material recognition. We also discuss individual differences in our results and highlight the importance of such consideration in material perception.

Individual Differences in Colour Perception: The Role of Low-Saturated and Complementary Colours in Ambiguous Images

Individual differences in colour perception, as evidenced by the popular debate of “The Dress” picture, have garnered additional interest with the popularisation of additional, similar photographs. We investigated which colorimetric characteristics were responsible for individual differences in colour perception. All objects of the controversial photographs are composed of two representative colours, which are low in saturation and are either complementary to each other or reminiscent of complementary colours. Due to these colorimetric characteristics, we suggest that one of the two complementary pixel clusters should be estimated as the illuminant hue depending on assumed brightness. Thus, people perceive the object's colours as being biased toward complementarily different colour directions and perceive different pixel clusters as chromatic and achromatic. Even though the distance between colours that people perceive differently is small in colour space, people perceive the object's colour as differently categorized colours in these ambiguous photographs, thereby causing debate. We suggest that people perceive the object's colours using different “modes of colour appearance” between surface-colour and self-luminous modes.

The Certainty of Ambiguity in Visual Neural Representations

Some images evoke bistable percepts: two different visual experiences seen in alternation while continuously viewing an unchanged stimulus. The Necker Cube and Rubin's Vase are classic examples, each of which gives alternating percepts of different shapes. Other bistable percepts are alternating colors or directions of motion. Although stimuli that result in salient bistability are rare and sometimes cleverly constructed to emphasize ambiguity, they have been influential for over 150 years, since the work of von Helmholtz, who considered them to be evidence for perceptual visual processes that interpret retinal stimuli. While bistability in natural viewing is uncommon, the main point of this review is that implicit ambiguity in visual neural representations is pervasive. Resolving ambiguity, therefore, is a fundamental and ubiquitous process of vision that routinely affects what we see, not an oddity arising from cleverly crafted images. This review focuses on the causes of widespread ambiguity, historical perspectives on it, and modern knowledge and theory about resolving it.

Color Judgments of #The Dress and #The Jacket in a Sample of Different Cultures

Two viral photographs, #The Dress and #The Jacket, have received recent attention in research on perception as the colors in these photos are ambiguous. In the current study, we examined perception of these photographs across three different cultural samples: Sweden (Western culture), China (Eastern culture), and India (between Western and Eastern cultures). Participants also answered questions about gender, age, morningness, and previous experience of the photographs. Analyses revealed that only age was a significant predictor for the perception of The Dress, as older people were more likely to perceive the colors as blue and black than white and gold. In contrast, multiple factors predicted perception of The Jacket, including age, previous experience, and country. Consistent with some previous research, this suggests that the perception of The Jacket is a different phenomenon from perception of The Dress and is influenced by additional factors, most notably culture.

Explaining #theShoe based on the optimal color hypothesis: The role of chromaticity vs. luminance distribution in an ambiguous image

The image of #theShoe is a derivative image of #theDress which induces vastly different color experiences across individuals. The majority of people perceive that the shoe has grey leather with turquoise laces, but others report pink leather with white laces. We hypothesized #theShoe presents a problem of color constancy, where different people estimate different illuminants falling onto the shoe. The present study specifically aimed to understand what cues in the shoe image caused the ambiguity based on the optimal color hypothesis: our visual system knows the gamut of surface colors under various illuminants and applies the knowledge for illuminant estimation. The analysis showed that estimated illuminant chromaticity largely changes according to the assumed intensity of the illuminant. When the illuminant intensity was assumed to be low, a high color temperature was estimated. In contrast, assuming high illuminant intensity led to the estimation of low color temperature. A simulation based on a von Kries correction showed that the subtraction of estimated illuminants from the original image shifts the appearance of the shoe towards the reported states (i.e. gray-turquoise or pink-white). These results suggest that the optimal color hypothesis provides a theoretical interpretation to #theShoe phenomenon. Moreover, this luminance-dependent color-shift was observed in #theDress phenomenon, supporting the notion that the same trigger induces #theShoe.

Specular highlights improve color constancy when other cues are weakened

Previous studies suggest that to achieve color constancy, the human visual system makes use of multiple cues, including a priori assumptions about the illumination ("daylight priors"). Specular highlights have been proposed to aid constancy, but the evidence for their usefulness is mixed. Here, we used a novel cue-combination approach to test whether the presence of specular highlights or the validity of a daylight prior improves illumination chromaticity estimates, inferred from achromatic settings, to determine whether and under which conditions either cue contributes to color constancy. Observers made achromatic settings within three-dimensional rendered scenes containing matte or glossy shapes, illuminated by either daylight or nondaylight illuminations. We assessed both the variability of these settings and their accuracy, in terms of the standard color constancy index (CCI). When a spectrally uniform background was present, neither CCIs nor variability improved with specular highlights or daylight illuminants (Experiment 1). When a Mondrian background was introduced, CCIs decreased overall but were higher for scenes containing glossy, as opposed to matte, shapes (Experiments 2 and 3). There was no overall reduction in variability of settings and no benefit for scenes illuminated by daylights. Taken together, these results suggest that the human visual system indeed uses specular highlights to improve color constancy but only when other cues, such as from the local surround, are weakened.

Exploring the Determinants of Color Perception Using #Thedress and Its Variants: The Role of Spatio-Chromatic Context, Chromatic Illumination, and Material-Light Interaction

The colors that people see depend not only on the surface properties of objects but also on how these properties interact with light as well as on how light reflected from objects interacts with an individual's visual system. Because individual visual systems vary, the same visual stimulus may elicit different perceptions from different individuals. #thedress phenomenon drove home this point: different individuals viewed the same image and reported it to be widely different colors: blue and black versus white and gold. This phenomenon inspired a collection of demonstrations presented at the Vision Sciences Society 2015 Meeting which showed how spatial and temporal manipulations of light spectra affect people's perceptions of material colors and illustrated the variability in individual color perception. The demonstrations also explored the effects of temporal alterations in metameric lights, including Maxwell's Spot, an entoptic phenomenon. Crucially, the demonstrations established that #thedress phenomenon occurs not only for images of the dress but also for the real dress under real light sources of different spectral composition and spatial configurations.

How to make a #theDress

If we completely understand how a phenomenon works, we should be able to produce it ourselves. However, the individual differences in color appearance observed with #theDress seem to be a peculiarity of that photo, and it remains unclear how the proposed mechanisms underlying #theDress can be generalized to other images. Here, we developed a simple algorithm that transforms any image with bicolored objects into an image with the properties of #theDress. We measured the colors perceived in such images and compared them to those perceived in #theDress. Color adjustments confirmed that observers strongly differ in how they perceive the colors of the new images in a similar way as for #theDress. Most importantly, these differences were not unsystematic, but correlated with how observers perceive #theDress. These results imply that the color distribution is sufficient to produce the striking individual differences in color perception originally observed with #theDress-at least as long as the image appears realistic and hence compels the viewer to make assumptions about illuminations and surfaces. The algorithm can be used for stimulus production beyond this study.

Musical interaction is influenced by underlying predictive models and musical expertise

Musical interaction is a unique model for understanding humans’ ability to align goals, intentions, and actions, which also allows for the manipulation of participants’ internal predictive models of upcoming events. Here we used polyrhythms to construct two joint finger tapping tasks that even when rhythmically dissimilar resulted in equal inter-tap intervals (ITIs). Thus, behaviourally a dyad of two musicians tap isochronously at the same rate, yet with their own distinct rhythmical context model (RCM). We recruited 22 highly skilled musicians (in 11 dyads) and contrasted the effect of having a shared versus non-shared RCM on dyads’ synchronization behaviour. As expected, tapping synchronization was significantly worse at the start of trials with non-shared models compared to trials with a shared model. However, the musicians were able to quickly recover when holding dissimilar predictive models. We characterised the directionality in the tapping behaviour of the dyads and found patterns mostly of mutual adaptation. Yet, in a subset of dyads primarily consisting of drummers, we found significantly different synchronization patterns, suggesting that instrument expertise can significantly affect synchronization strategies. Overall, this demonstrates that holding different predictive models impacts synchronization in musicians performing joint finger tapping.

Stripping #The Dress: the importance of contextual information on inter-individual differences in colour perception

In 2015, a picture of a Dress (henceforth the Dress) triggered popular and scientific interest; some reported seeing the Dress in white and gold (W&G) and others in blue and black (B&B). We aimed to describe the phenomenon and investigate the role of contextualization. Few days after the Dress had appeared on the Internet, we projected it to 240 students on two large screens in the classroom. Participants reported seeing the Dress in B&B (48%), W&G (38%), or blue and brown (B&Br; 7%). Amongst numerous socio-demographic variables, we only observed that W&G viewers were most likely to have always seen the Dress as W&G. In the laboratory, we tested how much contextual information is necessary for the phenomenon to occur. Fifty-seven participants selected colours most precisely matching predominant colours of parts or the full Dress. We presented, in this order, small squares (a), vertical strips (b), and the full Dress (c). We found that (1) B&B, B&Br, and W&G viewers had selected colours differing in lightness and chroma levels for contextualized images only (b, c conditions) and hue for fully contextualized condition only (c) and (2) B&B viewers selected colours most closely matching displayed colours of the Dress. Thus, the Dress phenomenon emerges due to inter-individual differences in subjectively perceived lightness, chroma, and hue, at least when all aspects of the picture need to be integrated. Our results support the previous conclusions that contextual information is key to colour perception; it should be important to understand how this actually happens.

Color Perception: Objects, Constancy, and Categories

Color has been scientifically investigated by linking color appearance to colorimetric measurements of the light that enters the eye. However, the main purpose of color perception is not to determine the properties of incident light, but to aid the visual perception of objects and materials in our environment. We review the state of the art on object colors, color constancy, and color categories to gain insight into the functional aspects of color perception. The common ground between these areas of research is that color appearance is tightly linked to the identification of objects and materials and the communication across observers. In conclusion, we argue that research should focus on how color processing is adapted to the surface properties of objects in the natural environment in order to bridge the gap between the known early stages of color perception and the subjective appearance of color.

The perception of colour and material in naturalistic tasks

Perceived object colour and material help us to select and interact with objects. Because there is no simple mapping between the pattern of an object's image on the retina and its physical reflectance, our perceptions of colour and material are the result of sophisticated visual computations. A long-standing goal in vision science is to describe how these computations work, particularly as they act to stabilize perceived colour and material against variation in scene factors extrinsic to object surface properties, such as the illumination. If we take seriously the notion that perceived colour and material are useful because they help guide behaviour in natural tasks, then we need experiments that measure and models that describe how they are used in such tasks. To this end, we have developed selection-based methods and accompanying perceptual models for studying perceived object colour and material. This focused review highlights key aspects of our work. It includes a discussion of future directions and challenges, as well as an outline of a computational observer model that incorporates early, known, stages of visual processing and that clarifies how early vision shapes selection performance.

Age, pupil size differences, and color choices for the "dress" and the "jacket"

The color identification responses to photographs of #thedress (white/gold and blue/black) and a jacket (white/blue and green/black, and teal) reveal obvious individual differences in color perception. To explore possible association between pupil size/retinal illuminance and color perception, we recorded the pupil diameters of participants shown 22 uniformly colored (generated from the RGB values using a laptop LCD display) screens followed by photographs of #thedress and jacket. We analyzed (a) pupil size difference in the color groups and (b) age-related pupil size and/or reflex change and its influence on color perception. The data confirms that the average pupil size of the white/gold group was significantly less than the blue/black group for the dress. The pupil size difference between the color groups is slightly higher in the 21-30-year and 31-55-year age groups but not in the 18-20-year age group, while a similar variance was not observed for the jacket color groups. Interestingly, the average pupil size of both color groups was smaller for the dress compared to the baseline (collected with a gray hue displayed on the screen), whereas an opposite effect was observed for the jacket. The contrasting results for the two photographs do not allow for a strong inference of only pupil size change principal for differences in color perception. But, a probable explanation of the pupil size difference could be the subjective variation in the perceptual interpretation of illumination cues in the photographs.

Assessment of #TheDress With Traditional Color Vision Tests: Perception Differences Are Associated With Blueness

Based on known color vision theories, there is no complete explanation for the perceptual dichotomy of #TheDress in which most people see either white-and-gold (WG) or blue-and-black (BK). We determined whether some standard color vision tests (i.e., color naming, color matching, anomaloscope settings, unique white settings, and color preferences), as well as chronotypes, could provide information on the color perceptions of #TheDress. Fifty-two young observers were tested. Fifteen of the observers (29%) reported the colors as BK, 21 (40%) as WG, and 16 (31%) reported a different combination of colors. Observers who perceived WG required significantly more blue in their unique white settings than those who perceived BK. The BK, blue-and-gold, and WG observer groups had significantly different color preferences for the light cyan chip. Moreland equation anomaloscope matching showed a significant difference between WG and BK observers. In addition, #TheDress color perception categories, color preference outcomes, and unique white settings had a common association. For both the bright and dark regions of #TheDress, the color matching chromaticities formed a continuum, approximately following the daylight chromaticity locus. Color matching to the bright region of #TheDress showed two nearly distinct clusters (WG vs. BK) along the daylight chromaticity locus and there was a clear cutoff for reporting WG versus BK. All results showing a significant difference involved blue percepts, possibly due to interpretations of the illuminant interactions with the dress material. This suggests that variations in attributing blueness to the #TheDress image may be significant variables determining color perception of #TheDress.