Recognition alters the spatial pattern of FMRI activation in early retinotopic cortex

Early retinotopic cortex has traditionally been viewed as containing a veridical representation of the low-level properties of the image, not imbued by high-level interpretation and meaning. Yet several recent results indicate that neural representations in early retinotopic cortex reflect not just the sensory properties of the image, but also the perceived size and brightness of image regions. Here we used functional magnetic resonance imaging pattern analyses to ask whether the representation of an object in early retinotopic cortex changes when the object is recognized compared with when the same stimulus is presented but not recognized. Our data confirmed this hypothesis: the pattern of response in early retinotopic visual cortex to a two-tone "Mooney" image of an object was more similar to the response to the full grayscale photo version of the same image when observers knew what the two-tone image represented than when they did not. Further, in a second experiment, high-level interpretations actually overrode bottom-up stimulus information, such that the pattern of response in early retinotopic cortex to an identified two-tone image was more similar to the response to the photographic version of that stimulus than it was to the response to the identical two-tone image when it was not identified. Our findings are consistent with prior results indicating that perceived size and brightness affect representations in early retinotopic visual cortex and, further, show that even higher-level information--knowledge of object identity--also affects the representation of an object in early retinotopic cortex.

The M170 is selective for faces, not for expertise

Are the mechanisms for face perception selectively involved in processing faces per se, or do they also participate in the processing of any class of visual stimuli that share the same basic configuration and for which the observer has gained substantial visual expertise? Here we tested the effects of visual expertise on the face-selective "M170", a magnetoencephalography (MEG) response component that occurs 170 ms after stimulus onset and is involved in the identification of individual faces. In Experiment 1, cars did not elicit a higher M170 response (relative to control objects) in car experts compared to controls subjects. In Experiment 2, the M170 amplitude was correlated with successful face identification, but not with successful car identification in car experts. These results indicate that the early face processing mechanisms marked by the M170 are involved in the identification of faces in particular, not in the identification of any objects of expertise.

Pop-out without awareness: unseen feature singletons capture attention only when top-down attention is available

Visual pop-out occurs when a unique visual target (e.g., a feature singleton) is present in a set of homogeneous distractors. However, the role of visual awareness in this process remains unclear. In the experiments reported here, we showed that even though subjects were not aware of a suppressed pop-out display, their subsequent performance on an orientation-discrimination task was significantly better at the pop-out location than at a control location. These results indicate that conscious visual awareness of a feature singleton is not necessary for it to attract attention. Furthermore, the subliminal pop-out effect disappeared when subjects diverted their attention toward a rapid sequential visual presentation task while presented with the same subliminal pop-out display. These results suggest that top-down attention is necessary for the subliminal pop-out effect and that the cognitive processes underlying attention and awareness are somewhat independent.

Determinants of repetition blindness

Repetition blindness (RB) for letters presented in rapid serial visual presentation (RSVP) was shown to be highly dependent on absolute stimulus presentation rate (Experiment 1) and on the duration of the 2nd occurrence (C2) but not the 1st occurrence (C1) of the repeated item (Experiment 2). In Experiments 3-6, RB was strongly affected by C1-C2 interstimulus interval but not by response requirements or total memory load. In Experiment 7, sensitivity for detection of the 2nd vowel in an RSVP sequence of consonants was lower when it was identical to the 1st vowel (mean d' = .40) than when it was different (mean d' = .70). It is argued that RB is not due to (a) temporal overlap of C1 and C2 recognition, (b) forgetting of C2, (c) item-specific output interference, (d) response bias, or (e) guessing strategies; rather, RB arises because of the failure to bind recognized types to individuated tokens.

Temporal selection is suppressed, delayed, and diffused during the attentional blink

How does temporal selection work, and along what dimensions does it vary from one instance to the next? We explored these questions using a phenomenon in which temporal selection goes awry. In the attentional blink, subjects fail to report the second of a pair of targets (T1 and T2) when they are presented at stimulus onset asynchronies (SOAs) of roughly 200 to 500 ms. We directly tested the properties of temporal selection during the blink by analyzing distractor intrusions at a fast rate of item presentation. Our analysis shows that attentional selection is (a) suppressed, (b) delayed, and (c) diffused in time during the attentional blink. These effects are dissociated by their time course: The measure of each effect returns to the baseline value at a different SOA. Our results constrain theories of the attentional blink and indicate that temporal selection varies along at least three dissociable dimensions: efficacy, latency, and precision.

Brain-like functional specialization emerges spontaneously in deep neural networks

The human brain contains multiple regions with distinct, often highly specialized functions, from recognizing faces to understanding language to thinking about what others are thinking. However, it remains unclear why the cortex exhibits this high degree of functional specialization in the first place. Here, we consider the case of face perception using artificial neural networks to test the hypothesis that functional segregation of face recognition in the brain reflects a computational optimization for the broader problem of visual recognition of faces and other visual categories. We find that networks trained on object recognition perform poorly on face recognition and vice versa and that networks optimized for both tasks spontaneously segregate themselves into separate systems for faces and objects. We then show functional segregation to varying degrees for other visual categories, revealing a widespread tendency for optimization (without built-in task-specific inductive biases) to lead to functional specialization in machines and, we conjecture, also brains.

How fMRI Can Inform Cognitive Theories

How can functional magnetic resonance imaging (fMRI) advance cognitive theory? Some have argued that fMRI can do little beyond localizing brain regions that carry out certain cognitive functions (and may not even be able to do that). However, in this article, we argue that fMRI can inform theories of cognition by helping to answer at least four distinct kinds of questions. Which mental functions are performed in brain regions specialized for just that function (and which are performed in more general-purpose brain machinery)? When fMRI markers of a particular Mental Process X are found, is Mental Process X engaged when people perform Task Y? How distinct are the representations of different stimulus classes? Do specific pairs of tasks engage common or distinct processing mechanisms? Thus, fMRI data can be used to address theoretical debates that have nothing to do with where in the brain a particular process is carried out.

Unimpaired Attentional Disengagement and Social Orienting in Children with Autism

Visual attention is often hypothesized to play a causal role in the development of autism spectrum disorder (ASD). Because attention shapes perception, learning, and social interaction, early deficits in attention could substantially affect the development of other perceptual and cognitive abilities. Here we test two key attentional phenomena thought to be disrupted in autism: attentional disengagement and social orienting. We find in a free-viewing paradigm that both phenomena are present in high-functioning children with ASD (n = 44, ages 5–12 years) and are identical in magnitude to those in age- and IQ-matched typical children (n = 40). Although these attentional processes may malfunction in other circumstances, our data indicate that high-functioning children with ASD do not suffer from across-the-board disruptions of either attentional disengagement or social orienting. Combined with mounting evidence that other attentional abilities are largely intact, it seems increasingly unlikely that disruptions of core attentional abilities lie at the root of ASD.

Common neural mechanisms for response selection and perceptual processing

Behavioral evidence supports a dissociation between response selection (RS; stimulus-to-response [S-R] mapping) and perceptual discrimination (PD): The former may be subject to a central processing bottleneck, whereas the latter is not (Pashler, 1994). We previously (Jiang & Kanwisher, 2003) identified a set of frontal and parietal regions involved in RS as those that produce a stronger signal when subjects follow a difficult S-R mapping rule than an easy mapping rule. Here, we test whether any of these regions are selectively activated by RS and not perceptual processing, as predicted by the central bottleneck view. In Experiment 1, subjects indicated which of four parallel lines was unique in length; PD was indexed by a higher BOLD response when the discrimination was difficult versus easy. Stimuli and responses were closely matched across conditions. We found that all regions-of-interest (ROIs) engaged by RS were also engaged by perceptual processing, arguing against the existence of mechanisms exclusively involved in RS. In Experiments 2 and 3, we asked what processes might go on in these ROIs, such that they could be recruited by both RS and perceptual processing. Our data argue against an account of this common activation in terms of spatial processing or general task difficulty. Thus, PD may recruit the same central processes that are engaged by RS.

Computational models of category-selective brain regions enable high-throughput tests of selectivity

Cortical regions apparently selective to faces, places, and bodies have provided important evidence for domain-specific theories of human cognition, development, and evolution. But claims of category selectivity are not quantitatively precise and remain vulnerable to empirical refutation. Here we develop artificial neural network-based encoding models that accurately predict the response to novel images in the fusiform face area, parahippocampal place area, and extrastriate body area, outperforming descriptive models and experts. We use these models to subject claims of category selectivity to strong tests, by screening for and synthesizing images predicted to produce high responses. We find that these high-response-predicted images are all unambiguous members of the hypothesized preferred category for each region. These results provide accurate, image-computable encoding models of each category-selective region, strengthen evidence for domain specificity in the brain, and point the way for future research characterizing the functional organization of the brain with unprecedented computational precision.

Attention as inference: selection is probabilistic; responses are all-or-none samples

Theories of probabilistic cognition postulate that internal representations are made up of multiple simultaneously held hypotheses, each with its own probability of being correct (henceforth, "probability distributions"). However, subjects make discrete responses and report the phenomenal contents of their mind to be all-or-none states rather than graded probabilities. How can these 2 positions be reconciled? Selective attention tasks, such as those used to study crowding, the attentional blink, rapid serial visual presentation, and so forth, were recast as probabilistic inference problems and used to assess how graded, probabilistic representations may produce discrete subjective states. The authors asked subjects to make multiple guesses per trial and used 2nd-order statistics to show that (a) visual selective attention operates in a graded fashion in time and space, selecting multiple targets to varying degrees on any given trial; and (b) responses are generated by a process of sampling from the probabilistic states that result from graded selection. The authors concluded that although people represent probability distributions, their discrete responses and conscious states are products of a process that samples from these probabilistic representations.

What's in a face? Effects of stimulus duration and inversion on face processing in schizophrenia

A number of studies show deficits in early-stage visual processing in schizophrenia. Deficits are also seen at more complex levels, such as ability to discriminate faces. This study investigated the "face inversion" effect, which reflects intrinsic cortical processing within the ventral visual stream, as well as contrast sensitivity, which reflects low-level visual processing, in order to evaluate integrity of specific stages of face processing in schizophrenia. Patients with schizophrenia and controls discriminated between pairs of upright or inverted faces or houses that had been manipulated to differ in the shape of the parts or the spatial distance among parts. The duration threshold for above chance performance on upright stimuli was obtained for patients using a house discrimination task. Contrast sensitivity was assessed for gratings of three spatial frequencies ranging from 0.5 to 21 cycles/degree. Patients needed significantly longer time to obtain 70% correct for upright stimuli and showed decreased contrast sensitivity. Increased duration threshold correlated with reduced contrast sensitivity to low (magnocellular-biased) but not medium or high spatial frequency stimuli. Using increased durations, patients showed significant inversion effects that were equivalent to those of controls on the face part and spacing tasks. Like controls, patients did not show inversion effects on the house tasks. These findings show that patients have difficulty integrating visual information as shown by increased duration thresholds. However, when faces were presented at these longer duration thresholds, patients showed the same relative processing ability for upright vs. inverted faces as controls, suggesting preserved intrinsic processing within cortical face processing regions. Similar inversion effects for face part and spacing for both groups suggest that they are using the same holistic face processing mechanism.

Sensitivity to musical structure in the human brain

Evidence from brain-damaged patients suggests that regions in the temporal lobes, distinct from those engaged in lower-level auditory analysis, process the pitch and rhythmic structure in music. In contrast, neuroimaging studies targeting the representation of music structure have primarily implicated regions in the inferior frontal cortices. Combining individual-subject fMRI analyses with a scrambling method that manipulated musical structure, we provide evidence of brain regions sensitive to musical structure bilaterally in the temporal lobes, thus reconciling the neuroimaging and patient findings. We further show that these regions are sensitive to the scrambling of both pitch and rhythmic structure but are insensitive to high-level linguistic structure. Our results suggest the existence of brain regions with representations of musical structure that are distinct from high-level linguistic representations and lower-level acoustic representations. These regions provide targets for future research investigating possible neural specialization for music or its associated mental processes.

Individual differences in face-looking behavior generalize from the lab to the world

Recent laboratory studies have found large, stable individual differences in the location people first fixate when identifying faces, ranging from the brows to the mouth. Importantly, this variation is strongly associated with differences in fixation-specific identification performance such that individuals' recognition ability is maximized when looking at their preferred location (Mehoudar, Arizpe, Baker, & Yovel, 2014; Peterson & Eckstein, 2013). This finding suggests that face representations are retinotopic and individuals enact gaze strategies that optimize identification, yet the extent to which this behavior reflects real-world gaze behavior is unknown. Here, we used mobile eye trackers to test whether individual differences in face gaze generalize from lab to real-world vision. In-lab fixations were measured with a speeded face identification task, while real-world behavior was measured as subjects freely walked around the Massachusetts Institute of Technology campus. We found a strong correlation between the patterns of individual differences in face gaze in the lab and real-world settings. Our findings support the hypothesis that individuals optimize real-world face identification by consistently fixating the same location and thus strongly constraining the space of retinotopic input. The methods developed for this study entailed collecting a large set of high-definition, wide field-of-view natural videos from head-mounted cameras and the viewer's fixation position, allowing us to characterize subjects' actually experienced real-world retinotopic images. These images enable us to ask how vision is optimized not just for the statistics of the "natural images" found in web databases, but of the truly natural, retinotopic images that have landed on actual human retinae during real-world experience.

Spatial repetition blindness is modulated by selective attention to color or shape

Subjects reported either the colors or shapes of two simultaneous masked letters. Our first study found that they were less accurate when the reported features were identical ("repetition blindness," or RB), while repetition along the unreported dimension had no effect. Three follow-up studies confirmed that when the same dimension was judged (overtly or covertly) for both stimuli, performance was only affected by repetition along that dimension. However, when different dimensions were judged for the two stimuli, performance was affected by repetition on both dimensions. These findings support new conclusions about both RB and visual attention. First, RB depends critically on visual attention, rather than simply on the stimulus presented or the overt response required. Second, while attention can be restricted to a single visual dimension, this is efficient only when the same dimension is selected for both objects. Selecting the color of one object and the shape of another simultaneous object results in both dimensions being accessed for both objects.

Face recognition deficits in autism spectrum disorders are both domain specific and process specific

Although many studies have reported face identity recognition deficits in autism spectrum disorders (ASD), two fundamental question remains: 1) Is this deficit "process specific" for face memory in particular, or does it extend to perceptual discrimination of faces as well? And 2) Is the deficit "domain specific" for faces, or is it found more generally for other social or even nonsocial stimuli? The answers to these questions are important both for understanding the nature of autism and its developmental etiology, and for understanding the functional architecture of face processing in the typical brain. Here we show that children with ASD are impaired (compared to age and IQ-matched typical children) in face memory, but not face perception, demonstrating process specificity. Further, we find no deficit for either memory or perception of places or cars, indicating domain specificity. Importantly, we further showed deficits in both the perception and memory of bodies, suggesting that the relevant domain of deficit may be social rather than specifically facial. These results provide a more precise characterization of the cognitive phenotype of autism and further indicate a functional dissociation between face memory and face perception.

Selective responses to faces, scenes, and bodies in the ventral visual pathway of infants

Three of the most robust functional landmarks in the human brain are the selective responses to faces in the fusiform face area (FFA), scenes in the parahippocampal place area (PPA), and bodies in the extrastriate body area (EBA). Are the selective responses of these regions present early in development or do they require many years to develop? Prior evidence leaves this question unresolved. We designed a new 32-channel infant magnetic resonance imaging (MRI) coil and collected high-quality functional MRI (fMRI) data from infants (2-9 months of age) while they viewed stimuli from four conditions-faces, bodies, objects, and scenes. We find that infants have face-, scene-, and body-selective responses in the location of the adult FFA, PPA, and EBA, respectively, powerfully constraining accounts of cortical development.

A neural population selective for song in human auditory cortex

How is music represented in the brain? While neuroimaging has revealed some spatial segregation between responses to music versus other sounds, little is known about the neural code for music itself. To address this question, we developed a method to infer canonical response components of human auditory cortex using intracranial responses to natural sounds, and further used the superior coverage of fMRI to map their spatial distribution. The inferred components replicated many prior findings, including distinct neural selectivity for speech and music, but also revealed a novel component that responded nearly exclusively to music with singing. Song selectivity was not explainable by standard acoustic features, was located near speech- and music-selective responses, and was also evident in individual electrodes. These results suggest that representations of music are fractionated into subpopulations selective for different types of music, one of which is specialized for the analysis of song.

Feature-binding errors after eye movements and shifts of attention

When people move their eyes, the eye-centered (retinotopic) locations of objects must be updated to maintain world-centered (spatiotopic) stability. Here, we demonstrated that the attentional-updating process temporarily distorts the fundamental ability to bind object locations with their features. Subjects were simultaneously presented with four colors after a saccade-one in a precued spatiotopic target location-and were instructed to report the target's color using a color wheel. Subjects' reports were systematically shifted in color space toward the color of the distractor in the retinotopic location of the cue. Probabilistic modeling exposed both crude swapping errors and subtler feature mixing (as if the retinotopic color had blended into the spatiotopic percept). Additional experiments conducted without saccades revealed that the two types of errors stemmed from different attentional mechanisms (attention shifting vs. splitting). Feature mixing not only reflects a new perceptual phenomenon, but also provides novel insight into how attention is remapped across saccades.

Syntactic processing in the human brain: what we know, what we don't know, and a suggestion for how to proceed

For every claim in the neuroimaging literature about a particular brain region supporting syntactic processing, there exist other claims implicating the target region in different linguistic processes, and, in many cases, in non-linguistic cognitive processes (e.g., Blumstein, 2009). We argue that traditional group analysis methods in neuroimaging may obscure functional specificity because of inter-subject anatomical variability (Fedorenko & Kanwisher, 2009). In Fedorenko, Hsieh, Nieto-Castanon, Whitfield-Gabrieli, and Kanwisher (2010) we presented a functional localizer that allows quick and reliable identification of key language-sensitive regions in each individual brain. This approach enables pooling data from corresponding functional regions across subjects rather than from the same locations in stereotaxic space that may differ functionally due to inter-subject anatomical variability. In the current paper we demonstrate that the individual-subjects functional localization approach is superior to the traditional methods in its ability to distinguish among conditions in a brain region's response. This ability is at the core of all neuroimaging research and is critical for answering questions of functional specialization (e.g., does a brain region specialize for processing syntactic aspects of the linguistic signal), which is in turn essential for making inferences about the precise computations conducted in each brain region. Based on our results, we argue that supplementing existing methods with an individual-subjects functional localization approach may lead to a clearer picture of the neural basis of syntactic processing, as it has in some other domains, such as high-level vision (e.g., Kanwisher, 2010) and social cognition (e.g., Saxe & Kanwisher, 2003).

Repetition blindness: the effects of stimulus modality and spatial displacement

Repetition blindness (Kanwisher, 1986, 1987) is the failure to detect repetitions of words in lists presented in rapid serial visual presentation (RSVP). Two questions were investigated in the present study. First, if repetition blindness is not found with auditory presentation, it would support a specifically visual account of the effect. Second, if displacement of the two instances in visual space eliminates repetition blindness, it would suggest that repetition blindness is restricted to instances in which identical stimuli are distinguished soley by temporal differences. In Experiment 1, the subjects omitted second occurrences of repeated words in verbatim recall of rapid sentences presented visually (in RSVP), but not auditorily (using compressed speech), indicating that repetition blindness is a modality-specific phenomenon. In Experiments 2 and 3, repetition blindness was observed even when two occurrences of a written word were presented in different locations, showing that distinct locations do not guarantee token individuation. The results are discussed within a model that distinguishes between processes of type recognition and token individuation.

Visual experience is not necessary for the development of face-selectivity in the lateral fusiform gyrus

The fusiform face area responds selectively to faces and is causally involved in face perception. How does face-selectivity in the fusiform arise in development, and why does it develop so systematically in the same location across individuals? Preferential cortical responses to faces develop early in infancy, yet evidence is conflicting on the central question of whether visual experience with faces is necessary. Here, we revisit this question by scanning congenitally blind individuals with fMRI while they haptically explored 3D-printed faces and other stimuli. We found robust face-selective responses in the lateral fusiform gyrus of individual blind participants during haptic exploration of stimuli, indicating that neither visual experience with faces nor fovea-biased inputs is necessary for face-selectivity to arise in the lateral fusiform gyrus. Our results instead suggest a role for long-range connectivity in specifying the location of face-selectivity in the human brain.