Efficient integration across spatial frequencies for letter identification in foveal and peripheral vision

The perception of a face can be greater than the sum of its parts

Holistic processing is often used as a construct to characterize face recognition. An important recent study by Gold, Mundy, and Tjan (2012) quantified holistic processing by computing a facial-feature integration index derived from an ideal observer model. This index was mathematically defined as the ratio of the psychophysical contrast sensitivities squared for recognizing a whole face versus the sum of contrast sensitivities squared for individual face parts (left eye, right eye, nose, and mouth). They observed that this index was not significantly different from 1, leading to the provocative conclusion that the perception of a face is no more than the sum of its parts. What may not be obvious to all readers of this work is that these conclusions were based on a collection of faces that shared essentially the same configuration of face parts. We tested whether the facial-feature integration index would also equal 1 when faces have a range of configurations mirroring the range of variability in real-world faces, using the same experimental procedure and calculating the same integration index as Gold et al. When tested on faces with the same configuration, we also observed an integration index similar to what Gold et al. reported. But when tested on faces with variable configurations, we observed an integration index significantly greater than 1. Combing our results with those of Gold et al. further clarifies the theoretical construct of holistic processing in face recognition and what it means for the whole to be greater than the sum of its parts.

The perception of a familiar face is no more than the sum of its parts

Why do faces become easier to recognize with repeated exposure? Previous research has suggested that familiarity may induce a qualitative shift in visual processing from an independent analysis of individual facial features to analysis that includes information about the relationships among features (Farah, Wilson, Drain, & Tanaka Psychological Review, 105, 482-498, 1998; Maurer, Grand, & Mondloch Trends in Cognitive Science, 6, 255-260, 2002). We tested this idea by using a "summation-at-threshold" technique (Gold, Mundy, & Tjan Psychological Science, 23, 427-434, 2012; Nandy & Tjan Journal of Vision, 8, 3.1-20, 2008), in which an observer's ability to recognize each individual facial feature shown independently is used to predict their ability to recognize all of the features shown in combination. We find that, although people are better overall at recognizing familiar as opposed to unfamiliar faces, their ability to integrate information across features is similar for unfamiliar and highly familiar faces and is well predicted by their ability to recognize each of the facial features shown in isolation. These results are consistent with the idea that familiarity has a quantitative effect on the efficiency with which information is extracted from individual features, rather than a qualitative effect on the process by which features are combined.

A perceptually completed whole is less than the sum of its parts

How efficiently do people integrate the disconnected image fragments that fall on their eyes when they view partly occluded objects? In the present study, I used a psychophysical summation-at-threshold technique to address this question by measuring discrimination performance with both isolated and combined features of physically fragmented but perceptually complete objects. If visual completion promotes superior integration efficiency, performance with a visually completed object should exceed what would be expected from performance with the individual object parts shown in isolation. Contrary to this prediction, results showed that discrimination performance with both static and moving versions of physically fragmented but perceptually complete objects was significantly worse than would be expected from performance with their constituent parts. These results present a challenge for future theories of visual completion.

Information processing correlates of a size-contrast illusion

Perception is often influenced by context. A well-known class of perceptual context effects is perceptual contrast illusions, in which proximate stimulus regions interact to alter the perception of various stimulus attributes, such as perceived brightness, color and size. Although the phenomenal reality of contrast effects is well documented, in many cases the connection between these illusions and how information is processed by perceptual systems is not well understood. Here, we use noise as a tool to explore the information processing correlates of one such contrast effect: the Ebbinghaus–Titchener size-contrast illusion. In this illusion, the perceived size of a central dot is significantly altered by the sizes of a set of surrounding dots, such that the presence of larger surrounding dots tends to reduce the perceived size of the central dot (and vise versa). In our experiments, we first replicated previous results that have demonstrated the subjective reality of the Ebbinghaus–Titchener illusion. We then used visual noise in a detection task to probe the manner in which observers processed information when experiencing the illusion. By correlating the noise with observers' classification decisions, we found that the sizes of the surrounding contextual elements had a direct influence on the relative weight observers assigned to regions within and surrounding the central element. Specifically, observers assigned relatively more weight to the surrounding region and less weight to the central region in the presence of smaller surrounding contextual elements. These results offer new insights into the connection between the subjective experience of size-contrast illusions and their associated information processing correlates.

Perceptual integration for qualitatively different 3-D cues in the human brain

The visual system's flexibility in estimating depth is remarkable: We readily perceive 3-D structure under diverse conditions from the seemingly random dots of a "magic eye" stereogram to the aesthetically beautiful, but obviously flat, canvasses of the Old Masters. Yet, 3-D perception is often enhanced when different cues specify the same depth. This perceptual process is understood as Bayesian inference that improves sensory estimates. Despite considerable behavioral support for this theory, insights into the cortical circuits involved are limited. Moreover, extant work tested quantitatively similar cues, reducing some of the challenges associated with integrating computationally and qualitatively different signals. Here we address this challenge by measuring fMRI responses to depth structures defined by shading, binocular disparity, and their combination. We quantified information about depth configurations (convex "bumps" vs. concave "dimples") in different visual cortical areas using pattern classification analysis. We found that fMRI responses in dorsal visual area V3B/KO were more discriminable when disparity and shading concurrently signaled depth, in line with the predictions of cue integration. Importantly, by relating fMRI and psychophysical tests of integration, we observed a close association between depth judgments and activity in this area. Finally, using a cross-cue transfer test, we found that fMRI responses evoked by one cue afford classification of responses evoked by the other. This reveals a generalized depth representation in dorsal visual cortex that combines qualitatively different information in line with 3-D perception.

Seeing and hearing a word: combining eye and ear is more efficient than combining the parts of a word

To understand why human sensitivity for complex objects is so low, we study how word identification combines eye and ear or parts of a word (features, letters, syllables). Our observers identify printed and spoken words presented concurrently or separately. When researchers measure threshold (energy of the faintest visible or audible signal) they may report either sensitivity (one over the human threshold) or efficiency (ratio of the best possible threshold to the human threshold). When the best possible algorithm identifies an object (like a word) in noise, its threshold is independent of how many parts the object has. But, with human observers, efficiency depends on the task. In some tasks, human observers combine parts efficiently, needing hardly more energy to identify an object with more parts. In other tasks, they combine inefficiently, needing energy nearly proportional to the number of parts, over a 60∶1 range. Whether presented to eye or ear, efficiency for detecting a short sinusoid (tone or grating) with few features is a substantial 20%, while efficiency for identifying a word with many features is merely 1%. Why? We show that the low human sensitivity for words is a cost of combining their many parts. We report a dichotomy between inefficient combining of adjacent features and efficient combining across senses. Joining our results with a survey of the cue-combination literature reveals that cues combine efficiently only if they are perceived as aspects of the same object. Observers give different names to adjacent letters in a word, and combine them inefficiently. Observers give the same name to a word’s image and sound, and combine them efficiently. The brain’s machinery optimally combines only cues that are perceived as originating from the same object. Presumably such cues each find their own way through the brain to arrive at the same object representation.

Detecting shapes in noise: tuning characteristics of global shape mechanisms

The proportion of signal elements embedded in noise needed to detect a signal is a standard tool for investigating motion perception. This paradigm was applied to the shape domain to determine how local information is pooled into a global percept. Stimulus arrays consisted of oriented Gabor elements that sampled the circumference of concentric radial frequency (RF) patterns. Individual Gabors were oriented tangentially to the shape (signal) or randomly (noise). In different conditions, signal elements were located randomly within the entire array or constrained to fall along one of the concentric contours. Coherence thresholds were measured for RF patterns with various frequencies (number of corners) and amplitudes (“sharpness” of corners). Coherence thresholds (about 10% = 15 elements) were lowest for circular shapes. Manipulating shape frequency or amplitude showed a range where thresholds remain unaffected (frequency ≤ RF4; amplitude ≤ 0.05). Increasing either parameter caused thresholds to rise. Compared to circles, thresholds increased by approximately four times for RF13 and five times for amplitudes of 0.3. Confining the signals to individual contours significantly reduced the number of elements needed to reach threshold (between 4 and 6), independent of the total number of elements on the contour or contour shape. Finally, adding external noise to the orientation of the elements had a greater effect on detection thresholds than adding noise to their position. These results provide evidence for a series of highly sensitive, shape-specific analysers which sum information globally but only from within specific annuli. These global mechanisms are tuned to position and orientation of local elements from which they pool information. The overall performance for arrays of elements can be explained by the sensitivity of multiple, independent concentric shape detectors rather than a single detector integrating information widely across space (e.g. Glass pattern detector).

From regular text to artistic writing and artworks: Fourier statistics of images with low and high aesthetic appeal

The spatial characteristics of letters and their influence on readability and letter identification have been intensely studied during the last decades. There have been few studies, however, on statistical image properties that reflect more global aspects of text, for example properties that may relate to its aesthetic appeal. It has been shown that natural scenes and a large variety of visual artworks possess a scale-invariant Fourier power spectrum that falls off linearly with increasing frequency in log-log plots. We asked whether images of text share this property. As expected, the Fourier spectrum of images of regular typed or handwritten text is highly anisotropic, i.e., the spectral image properties in vertical, horizontal, and oblique orientations differ. Moreover, the spatial frequency spectra of text images are not scale-invariant in any direction. The decline is shallower in the low-frequency part of the spectrum for text than for aesthetic artworks, whereas, in the high-frequency part, it is steeper. These results indicate that, in general, images of regular text contain less global structure (low spatial frequencies) relative to fine detail (high spatial frequencies) than images of aesthetics artworks. Moreover, we studied images of text with artistic claim (ornate print and calligraphy) and ornamental art. For some measures, these images assume average values intermediate between regular text and aesthetic artworks. Finally, to answer the question of whether the statistical properties measured by us are universal amongst humans or are subject to intercultural differences, we compared images from three different cultural backgrounds (Western, East Asian, and Arabic). Results for different categories (regular text, aesthetic writing, ornamental art, and fine art) were similar across cultures.

Object recognition in clutter: cortical responses depend on the type of learning

Theoretical studies suggest that the visual system uses prior knowledge of visual objects to recognize them in visual clutter, and posit that the strategies for recognizing objects in clutter may differ depending on whether or not the object was learned in clutter to begin with. We tested this hypothesis using functional magnetic resonance imaging (fMRI) of human subjects. We trained subjects to recognize naturalistic, yet novel objects in strong or weak clutter. We then tested subjects' recognition performance for both sets of objects in strong clutter. We found many brain regions that were differentially responsive to objects during object recognition depending on whether they were learned in strong or weak clutter. In particular, the responses of the left fusiform gyrus (FG) reliably reflected, on a trial-to-trial basis, subjects' object recognition performance for objects learned in the presence of strong clutter. These results indicate that the visual system does not use a single, general-purpose mechanism to cope with clutter. Instead, there are two distinct spatial patterns of activation whose responses are attributable not to the visual context in which the objects were seen, but to the context in which the objects were learned.

The perception of a face is no more than the sum of its parts

When you see a person's face, how do you go about combining his or her facial features to make a decision about who that person is? Most current theories of face perception assert that the ability to recognize a human face is not simply the result of an independent analysis of individual features, but instead involves a holistic coding of the relationships among features. This coding is thought to enhance people's ability to recognize a face beyond what would be expected if each feature were shown in isolation. In the study reported here, we explicitly tested this idea by comparing human performance on facial-feature integration with that of an optimal Bayesian integrator. Contrary to the predictions of most current notions of face perception, our findings showed that human observers integrate facial features in a manner that is no better than would be predicted by their ability to use each individual feature when shown in isolation. That is, a face is perceived no better than the sum of its individual parts.

The integration of motion and disparity cues to depth in dorsal visual cortex

Humans exploit a range of visual depth cues to estimate three-dimensional structure. For example, the slant of a nearby tabletop can be judged by combining information from binocular disparity, texture and perspective. Behavioral tests show humans combine cues near-optimally, a feat that could depend on discriminating the outputs from cue-specific mechanisms or on fusing signals into a common representation. Although fusion is computationally attractive, it poses a substantial challenge, requiring the integration of quantitatively different signals. We used functional magnetic resonance imaging (fMRI) to provide evidence that dorsal visual area V3B/KO meets this challenge. Specifically, we found that fMRI responses are more discriminable when two cues (binocular disparity and relative motion) concurrently signal depth, and that information provided by one cue is diagnostic of depth indicated by the other. This suggests a cortical node important when perceiving depth, and highlights computations based on fusion in the dorsal stream.

Ideal observer analysis of crowding and the reduction of crowding through learning

Crowding is a prominent phenomenon in peripheral vision where nearby objects impede one's ability to identify a target of interest. The precise mechanism of crowding is not known. We used ideal observer analysis and a noise-masking paradigm to identify the functional mechanism of crowding. We tested letter identification in the periphery with and without flanking letters and found that crowding increases equivalent input noise and decreases sampling efficiency. Crowding effectively causes the signal from the target to be noisier and at the same time reduces the visual system's ability to make use of a noisy signal. After practicing identification of flanked letters without noise in the periphery for 6 days, subjects' performance for identifying flanked letters improved (reduction of crowding). Across subjects, the improvement was attributable to either a decrease in crowding-induced equivalent input noise or an increase in sampling efficiency, but seldom both. This pattern of results is consistent with a simple model whereby learning reduces crowding by adjusting the spatial extent of a perceptual window used to gather relevant input features. Following learning, subjects with inappropriately large windows reduced their window sizes; while subjects with inappropriately small windows increased their window sizes. The improvement in equivalent input noise and sampling efficiency persists for at least 6 months.

Scale dependence and channel switching in letter identification

Letters are broadband visual stimuli with information useful for discrimination over a wide range of spatial frequencies. Yet, recent evidence suggests that observers use only a single, fixed spatial-frequency channel to identify letters and that the scale of that channel, in units of letter size, is determined by the size of the letter (scale dependence). We report two letter-identification experiments using critical-band masking. With sufficiently high-amplitude, low- or high-pass masking noise, observers switched to a different range of spatial frequencies for the task. Thus, letter channels are not fixed for a given letter size. When an additional white-noise masker was added to the stimulus to flatten the contrast-sensitivity function, the letter channel used by the observer still depended on letter size, further supporting the hypothesis that letter identification is scale dependent.

Integration of Facial Information is Sub-Optimal

How efficiently do we combine information across facial features when recognizing a face? Previous studies have suggested that the perception of a face is not simply the result of an independent analysis of individual facial features, but instead involves a coding of the relationships amongst features. This additional coding of the relationships amongst features is thought to enhance our ability to recognize a face. In our experiments, we tested whether an observer's ability to recognize a face is in fact better than what one would expect from their ability to recognize the individual facial features in isolation. We tested this by using a psychophysical summation-at-threshold technique that has been used extensively to measure how efficiently observers integrate information across spatial locations and spatial frequencies. Surprisingly, we found that observers integrated information across facial features less efficiently than would be predicted by their ability to recognize the individual parts.