What is changing when: Decoding visual information in movies from human intracranial recordings

The majority of visual recognition studies have focused on the neural responses to repeated presentations of static stimuli with abrupt and well-defined onset and offset times. In contrast, natural vision involves unique renderings of visual inputs that are continuously changing without explicitly defined temporal transitions. Here we considered commercial movies as a coarse proxy to natural vision. We recorded intracranial field potential signals from 1,284 electrodes implanted in 15 patients with epilepsy while the subjects passively viewed commercial movies. We could rapidly detect large changes in the visual inputs within approximately 100 ms of their occurrence, using exclusively field potential signals from ventral visual cortical areas including the inferior temporal gyrus and inferior occipital gyrus. Furthermore, we could decode the content of those visual changes even in a single movie presentation, generalizing across the wide range of transformations present in a movie. These results present a methodological framework for studying cognition during dynamic and natural vision.

Individual differences in face-looking behavior generalize from the lab to the world

Recent laboratory studies have found large, stable individual differences in the location people first fixate when identifying faces, ranging from the brows to the mouth. Importantly, this variation is strongly associated with differences in fixation-specific identification performance such that individuals' recognition ability is maximized when looking at their preferred location (Mehoudar, Arizpe, Baker, & Yovel, 2014; Peterson & Eckstein, 2013). This finding suggests that face representations are retinotopic and individuals enact gaze strategies that optimize identification, yet the extent to which this behavior reflects real-world gaze behavior is unknown. Here, we used mobile eye trackers to test whether individual differences in face gaze generalize from lab to real-world vision. In-lab fixations were measured with a speeded face identification task, while real-world behavior was measured as subjects freely walked around the Massachusetts Institute of Technology campus. We found a strong correlation between the patterns of individual differences in face gaze in the lab and real-world settings. Our findings support the hypothesis that individuals optimize real-world face identification by consistently fixating the same location and thus strongly constraining the space of retinotopic input. The methods developed for this study entailed collecting a large set of high-definition, wide field-of-view natural videos from head-mounted cameras and the viewer's fixation position, allowing us to characterize subjects' actually experienced real-world retinotopic images. These images enable us to ask how vision is optimized not just for the statistics of the "natural images" found in web databases, but of the truly natural, retinotopic images that have landed on actual human retinae during real-world experience.

Connectivity precedes function in the development of the visual word form area

What determines the cortical location at which a given functionally specific region will arise in development? We tested the hypothesis that functionally specific regions develop in their characteristic locations because of pre-existing differences in the extrinsic connectivity of that region to the rest of the brain. We exploited the visual word form area (VWFA) as a test case, scanning children with diffusion and functional imaging at age 5, before they learned to read, and at age 8, after they learned to read. We found the VWFA developed functionally in this interval and that its location in a particular child at age 8 could be predicted from that child's connectivity fingerprints (but not functional responses) at age 5. These results suggest that early connectivity instructs the functional development of the VWFA, possibly reflecting a general mechanism of cortical development.

No Evidence for Automatic Remapping of Stimulus Features or Location Found with fMRI

The input to our visual system shifts every time we move our eyes. To maintain a stable percept of the world, visual representations must be updated with each saccade. Near the time of a saccade, neurons in several visual areas become sensitive to the regions of visual space that their receptive fields occupy after the saccade. This process, known as remapping, transfers information from one set of neurons to another, and may provide a mechanism for visual stability. However, it is not clear whether remapping transfers information about stimulus features in addition to information about stimulus location. To investigate this issue, we recorded blood-oxygen-level dependent (BOLD) functional magnetic resonance imaging (fMRI) responses while human subjects viewed images of faces and houses (two visual categories with many feature differences). Immediately after some image presentations, subjects made a saccade that moved the previously stimulated location to the opposite side of the visual field. We then used a combination of univariate analyses and multivariate pattern analyses to test whether information about stimulus location and stimulus features were remapped to the ipsilateral hemisphere after the saccades. We found no reliable indication of stimulus feature remapping in any region. However, we also found no reliable indication of stimulus location remapping, despite the fact that our paradigm was highly similar to previous fMRI studies of remapping. The absence of location remapping in our study precludes strong conclusions regarding feature remapping. However, these results also suggest that measurement of location remapping with fMRI depends strongly on the details of the experimental paradigm used. We highlight differences in our approach from the original fMRI studies of remapping, discuss potential reasons for the failure to generalize prior location remapping results, and suggest directions for future research.

The occipital place area represents the local elements of scenes

Neuroimaging studies have identified three scene-selective regions in human cortex: parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA). However, precisely what scene information each region represents is not clear, especially for the least studied, more posterior OPA. Here we hypothesized that OPA represents local elements of scenes within two independent, yet complementary scene descriptors: spatial boundary (i.e., the layout of external surfaces) and scene content (e.g., internal objects). If OPA processes the local elements of spatial boundary information, then it should respond to these local elements (e.g., walls) themselves, regardless of their spatial arrangement. Indeed, we found that OPA, but not PPA or RSC, responded similarly to images of intact rooms and these same rooms in which the surfaces were fractured and rearranged, disrupting the spatial boundary. Next, if OPA represents the local elements of scene content information, then it should respond more when more such local elements (e.g., furniture) are present. Indeed, we found that OPA, but not PPA or RSC, responded more to multiple than single pieces of furniture. Taken together, these findings reveal that OPA analyzes local scene elements - both in spatial boundary and scene content representation - while PPA and RSC represent global scene properties.

What is the Bandwidth of Perceptual Experience?

Although our subjective impression is of a richly detailed visual world, numerous empirical results suggest that the amount of visual information observers can perceive and remember at any given moment is limited. How can our subjective impressions be reconciled with these objective observations? Here, we answer this question by arguing that, although we see more than the handful of objects, claimed by prominent models of visual attention and working memory, we still see far less than we think we do. Taken together, we argue that these considerations resolve the apparent conflict between our subjective impressions and empirical data on visual capacity, while also illuminating the nature of the representations underlying perceptual experience.

Unimpaired attentional disengagement in toddlers with autism spectrum disorder

A prominent hypothesis holds that 'sticky' attention early in life in children with autism spectrum disorder (ASD) limits their ability to explore and learn about the world. Under this hypothesis, the core clinical symptoms of ASD - restricted interests, repetitive behaviors and impaired social/communication abilities - could all result from impaired attentional disengagement during development. However, the existence of disengagement deficits in children with ASD is controversial, and a recent study found no deficit in 5- to 12-year-olds with ASD. Nonetheless, the possibility remains that disengagement is impaired earlier in development in children with ASD, altering their developmental trajectory even if the attentional deficit itself is remediated or compensated for by the time children with ASD reach school age. Here, we tested this possibility by characterizing attentional disengagement in a group of toddlers just diagnosed with ASD (age 21 to 37 months). We found strikingly similar performance between the ASD and age-matched typically developing (TD) toddlers, and no evidence of impaired attentional disengagement. These results show that even at a young age when the clinical symptoms of ASD are first emerging, disengagement abilities are intact. Sticky attention is not a fundamental characteristic of ASD, and probably does not play a causal role in its etiology.

Reduced GABAergic Action in the Autistic Brain

An imbalance between excitatory/inhibitory neurotransmission has been posited as a central characteristic of the neurobiology of autism [1], inspired in part by the striking prevalence of seizures among individuals with the disorder [2]. Evidence supporting this hypothesis has specifically implicated the signaling pathway of the inhibitory neurotransmitter, γ-aminobutyric acid (GABA), in this putative imbalance: GABA receptor genes have been associated with autism in linkage and copy number variation studies [3-7], fewer GABA receptor subunits have been observed in the post-mortem tissue of autistic individuals [8, 9], and GABAergic signaling is disrupted across heterogeneous mouse models of autism [10]. Yet, empirical evidence supporting this hypothesis in humans is lacking, leaving a gulf between animal and human studies of the condition. Here, we present a direct link between GABA signaling and autistic perceptual symptomatology. We first demonstrate a robust, replicated autistic deficit in binocular rivalry [11], a basic visual function that is thought to rely on the balance of excitation/inhibition in visual cortex [12-15]. Then, using magnetic resonance spectroscopy, we demonstrate a tight linkage between binocular rivalry dynamics in typical participants and both GABA and glutamate levels in the visual cortex. Finally, we show that the link between GABA and binocular rivalry dynamics is completely and specifically absent in autism. These results suggest a disruption in inhibitory signaling in the autistic brain and forge a translational path between animal and human models of the condition.