Neuronal representation of occluded objects in the human brain

Amodal completion across the brain: The impact of structure and knowledge

This study investigates the phenomenon of amodal completion within the context of naturalistic objects, employing a repetition suppression paradigm to disentangle the influence of structure and knowledge cues on how objects are completed. The research focuses on early visual cortex (EVC) and lateral occipital complex (LOC), shedding light on how these brain regions respond to different completion scenarios. In LOC, we observed suppressed responses to structure and knowledge-compatible stimuli, providing evidence that both cues influence neural processing in higher-level visual areas. However, in EVC, we did not find evidence for differential responses to completions compatible or incompatible with either structural or knowledge-based expectations. Together, our findings suggest that the interplay between structure and knowledge cues in amodal completion predominantly impacts higher-level visual processing, with less pronounced effects on the early visual cortex. This study contributes to our understanding of the complex mechanisms underlying visual perception and highlights the distinct roles played by different brain regions in amodal completion.

Seeing and extrapolating motion trajectories share common informative activation patterns in primary visual cortex

The natural environment is dynamic and moving objects become constantly occluded, engaging the brain in a challenging completion process to estimate where and when the object might reappear. Although motion extrapolation is critical in daily life-imagine crossing the street while an approaching car is occluded by a larger standing vehicle-its neural underpinnings are still not well understood. While the engagement of low-level visual cortex during dynamic occlusion has been postulated, most of the previous group-level fMRI-studies failed to find evidence for an involvement of low-level visual areas during occlusion. In this fMRI-study, we therefore used individually defined retinotopic maps and multivariate pattern analysis to characterize the neural basis of visible and occluded changes in motion direction in humans. To this end, participants learned velocity-direction change pairings (slow motion-upwards; fast motion-downwards or vice versa) during a training phase without occlusion and judged the change in stimulus direction, based on its velocity, during a following test phase with occlusion. We find that occluded motion direction can be predicted from the activity patterns during visible motion within low-level visual areas, supporting the notion of a mental representation of motion trajectory in these regions during occlusion.

The nature of neural object representations during dynamic occlusion

Objects disappearing briefly from sight due to occlusion is an inevitable occurrence in everyday life. Yet we generally have a strong experience that occluded objects continue to exist, despite the fact that they objectively disappear. This indicates that neural object representations must be maintained during dynamic occlusion. However, it is unclear what the nature of such representation is and in particular whether it is perception-like or more abstract, for example, reflecting limited features such as position or movement direction only. In this study, we address this question by examining how different object features such as object shape, luminance, and position are represented in the brain when a moving object is dynamically occluded. We apply multivariate decoding methods to Magnetoencephalography (MEG) data to track how object representations unfold over time. Our methods allow us to contrast the representations of multiple object features during occlusion and enable us to compare the neural responses evoked by visible and occluded objects. The results show that object position information is represented during occlusion to a limited extent while object identity features are not maintained through the period of occlusion. Together, this suggests that the nature of object representations during dynamic occlusion is different from visual representations during perception.

Watching the Effects of Gravity. Vestibular Cortex and the Neural Representation of "Visual" Gravity

Gravity is a physical constraint all terrestrial species have adapted to through evolution. Indeed, gravity effects are taken into account in many forms of interaction with the environment, from the seemingly simple task of maintaining balance to the complex motor skills performed by athletes and dancers. Graviceptors, primarily located in the vestibular otolith organs, feed the Central Nervous System with information related to the gravity acceleration vector. This information is integrated with signals from semicircular canals, vision, and proprioception in an ensemble of interconnected brain areas, including the vestibular nuclei, cerebellum, thalamus, insula, retroinsula, parietal operculum, and temporo-parietal junction, in the so-called vestibular network. Classical views consider this stage of multisensory integration as instrumental to sort out conflicting and/or ambiguous information from the incoming sensory signals. However, there is compelling evidence that it also contributes to an internal representation of gravity effects based on prior experience with the environment. This a priori knowledge could be engaged by various types of information, including sensory signals like the visual ones, which lack a direct correspondence with physical gravity. Indeed, the retinal accelerations elicited by gravitational motion in a visual scene are not invariant, but scale with viewing distance. Moreover, the "visual" gravity vector may not be aligned with physical gravity, as when we watch a scene on a tilted monitor or in weightlessness. This review will discuss experimental evidence from behavioral, neuroimaging (connectomics, fMRI, TMS), and patients' studies, supporting the idea that the internal model estimating the effects of gravity on visual objects is constructed by transforming the vestibular estimates of physical gravity, which are computed in the brainstem and cerebellum, into internalized estimates of virtual gravity, stored in the vestibular cortex. The integration of the internal model of gravity with visual and non-visual signals would take place at multiple levels in the cortex and might involve recurrent connections between early visual areas engaged in the analysis of spatio-temporal features of the visual stimuli and higher visual areas in temporo-parietal-insular regions.

Resolving visual motion through perceptual gaps

Perceptual gaps can be caused by objects in the foreground temporarily occluding objects in the background or by eyeblinks, which briefly but frequently interrupt visual information. Resolving visual motion across perceptual gaps is particularly challenging, as object position changes during the gap. We examine how visual motion is maintained and updated through externally driven (occlusion) and internally driven (eyeblinks) perceptual gaps. Focusing on both phenomenology and potential mechanisms such as suppression, extrapolation, and integration, we present a framework for how perceptual gaps are resolved over space and time. We finish by highlighting critical questions and directions for future work.

Effectiveness of "rescue saccades" on the accuracy of tracking multiple moving targets: An eye-tracking study on the effects of target occlusions

Occlusion is one of the main challenges in tracking multiple moving objects. In almost all real-world scenarios, a moving object or a stationary obstacle occludes targets partially or completely for a short or long time during their movement. A previous study (Zelinsky & Todor, 2010) reported that subjects make timely saccades toward the object in danger of being occluded. Observers make these so-called “rescue saccades” to prevent target swapping. In this study, we examined whether these saccades are helpful. To this aim, we used as the stimuli recorded videos from natural movement of zebrafish larvae swimming freely in a circular container. We considered two main types of occlusion: object-object occlusions that naturally exist in the videos, and object-occluder occlusions created by adding a stationary doughnut-shape occluder in some videos. Four different scenarios were studied: (1) no occlusions, (2) only object-object occlusions, (3) only object-occluder occlusion, or (4) both object-object and object-occluder occlusions. For each condition, two set sizes (two and four) were applied. Participants’ eye movements were recorded during tracking, and rescue saccades were extracted afterward. The results showed that rescue saccades are helpful in handling object-object occlusions but had no reliable effect on tracking through object-occluder occlusions. The presence of occlusions generally increased visual sampling of the scenes; nevertheless, tracking accuracy declined due to occlusion.

Ocular tracking of occluded ballistic trajectories: Effects of visual context and of target law of motion

In tracking a moving target, the visual context may provide cues for an observer to interpret the causal nature of the target motion and extract features to which the visual system is weakly sensitive, such as target acceleration. This information could be critical when vision of the target is temporarily impeded, requiring visual motion extrapolation processes. Here we investigated how visual context influences ocular tracking of motion either congruent or not with natural gravity. To this end, 28 subjects tracked computer-simulated ballistic trajectories either perturbed in the descending segment with altered gravity effects (0g/2g) or retaining natural-like motion (1g). Shortly after the perturbation (550 ms), targets disappeared for either 450 or 650 ms and became visible again until landing. Target motion occurred with either quasi-realistic pictorial cues or a uniform background, presented in counterbalanced order. We analyzed saccadic and pursuit movements after 0g and 2g target-motion perturbations and for corresponding intervals of unperturbed 1g trajectories, as well as after corresponding occlusions. Moreover, we considered the eye-to-target distance at target reappearance. Tracking parameters differed significantly between scenarios: With a neutral background, eye movements did not depend consistently on target motion, whereas with pictorial background they showed significant dependence, denoting better tracking of accelerated targets. These results suggest that oculomotor control is tuned to realistic properties of the visual scene.

Neuroimaging Findings on Amodal Completion: A Review

Amodal completion is the phenomenon of perceiving completed objects even though physically they are partially occluded. In this review, we provide an extensive overview of the results obtained from a variety of neuroimaging studies on the neural correlates of amodal completion. We discuss whether low-level and high-level cortical areas are implicated in amodal completion; provide an overview of how amodal completion unfolds over time while dissociating feedforward, recurrent, and feedback processes; and discuss how amodal completion is represented at the neuronal level. The involvement of low-level visual areas such as V1 and V2 is not yet clear, while several high-level structures such as the lateral occipital complex and fusiform face area seem invariant to occlusion of objects and faces, respectively, and several motor areas seem to code for object permanence. The variety of results on the timing of amodal completion hints to a mixture of feedforward, recurrent, and feedback processes. We discuss whether the invisible parts of the occluded object are represented as if they were visible, contrary to a high-level representation. While plenty of questions on amodal completion remain, this review presents an overview of the neuroimaging findings reported to date, summarizes several insights from computational models, and connects research of other perceptual completion processes such as modal completion. In all, it is suggested that amodal completion is the solution to deal with various types of incomplete retinal information, and highly depends on stimulus complexity and saliency, and therefore also give rise to a variety of observed neural patterns.

Timing in the absence of a clock reset

Prominent models of time perception assume a reset of the timing mechanism with an explicit onset of the interval to be timed. Here we investigated the accuracy and precision of temporal estimations when the duration does not have such an explicit onset. Participants were tracking a disc moving on a circular path with varying speeds, and estimated the duration of one full revolution before the stimulus stopped. The onset of that revolution was either cued (explicit), or undetermined until the stimulus stopped (implicit). Reproduced duration was overestimated for short and underestimated for long durations, and variability of the estimates scaled with the duration in both temporal conditions. However, the bias was more pronounced in the implicit condition. In addition, if the stimulus path was partially occluded, duration of the occluded motion was correctly estimated. In a second experiment, we compared the precision in the explicit and implicit conditions by asking participants to discriminate the duration of one revolution before the stimulus stopped to that of a static stimulus presentation in a forced-choice task. Sensitivity of discrimination was worse in the implicit onset condition, but surprisingly, still comparable to the explicit condition. In summary, the estimates follow principles described in prospective timing paradigms, although not knowing beforehand when to start timing decreases sensitivity of temporal estimations. Since in naturalistic contexts, we often do not know in advance which durations might be relevant to estimate, the simple task presented here could become a valuable tool for testing models of temporal estimation.

Differential contributions to the interception of occluded ballistic trajectories by the temporoparietal junction, area hMT/V5+, and the intraparietal cortex

The ability to catch objects when transiently occluded from view suggests their motion can be extrapolated. Intraparietal cortex (IPS) plays a major role in this process along with other brain structures, depending on the task. For example, interception of objects under Earth's gravity effects may depend on time-to-contact predictions derived from integration of visual signals processed by hMT/V5+ with a priori knowledge of gravity residing in the temporoparietal junction (TPJ). To investigate this issue further, we disrupted TPJ, hMT/V5+, and IPS activities with transcranial magnetic stimulation (TMS) while subjects intercepted computer-simulated projectile trajectories perturbed randomly with either hypo- or hypergravity effects. In experiment 1, trajectories were occluded either 750 or 1,250 ms before landing. Three subject groups underwent triple-pulse TMS (tpTMS, 3 pulses at 10 Hz) on one target area (TPJ | hMT/V5+ | IPS) and on the vertex (control site), timed at either trajectory perturbation or occlusion. In experiment 2, trajectories were entirely visible and participants received tpTMS on TPJ and hMT/V5+ with same timing as experiment 1 tpTMS of TPJ, hMT/V5+, and IPS affected differently the interceptive timing. TPJ stimulation affected preferentially responses to 1-g motion, hMT/V5+ all response types, and IPS stimulation induced opposite effects on 0-g and 2-g responses, being ineffective on 1-g responses. Only IPS stimulation was effective when applied after target disappearance, implying this area might elaborate memory representations of occluded target motion. Results are compatible with the idea that IPS, TPJ, and hMT/V5+ contribute to distinct aspects of visual motion extrapolation, perhaps through parallel processing.NEW & NOTEWORTHY Visual extrapolation represents a potential neural solution to afford motor interactions with the environment in the face of missing information. We investigated relative contributions by temporoparietal junction (TPJ), hMT/V5+, and intraparietal cortex (IPS), cortical areas potentially involved in these processes. Parallel organization of visual extrapolation processes emerged with respect to the target's motion causal nature: TPJ was primarily involved for visual motion congruent with gravity effects, IPS for arbitrary visual motion, whereas hMT/V5+ contributed at earlier processing stages.

Representing dynamic stimulus information during occlusion

Human observers maintain a representation of the visual features of objects when they become occluded. This representation facilitates the interpretation of occluded events and allows us to quickly identify objects upon reappearing. Here we investigated whether visual features that change over time are also represented during occlusion. To answer this question we used an illusion from the time perception domain in which the perceived duration of an event increases as its temporal frequency content increases. In the first experiment we demonstrate temporal frequency induced modulation of duration both when the object remains visible as well as when it becomes temporarily occluded. Additionally, we demonstrate that time dilation for temporarily occluded objects cannot be explained by modulations of duration as a result of pre- and post-occlusion presentation of the object. In a second experiment, we corroborate this finding by demonstrating that modulation of the perceived duration of occluded events depends on the expected temporal frequency content of the object during occlusion. Together these results demonstrate that the dynamic properties of an object are represented during occlusion. We conclude that the representations of occluded objects contain a wide range of features derived from the period when the object was still visible, including information about both the static and dynamic properties of the object.

Fast moving texture has opposite effects on the perceived speed of visible and occluded object trajectories

In a series of psychophysical experiments, we altered the perceived speed of a spot (target) using a grayscale texture moving in the same (iso-motion) or opposite (anti-motion) direction of the target. In Experiment 1, using a velocity discrimination task (2IFC), the target moved in front of the texture and was perceived faster with anti-motion than iso-motion texture. The integration and segregation of motion signals in high-level motion areas may have accounted for the illusion. In Experiment 2, by asking observers to estimate the time-to-contact (TTC) with a bar indicating the end of the invisible trajectory, we showed that this illusory visible speed, due to anti- (iso-) texture, reduced (increased) the subsequent estimated duration of occluded target trajectory. However, in Experiment 3, when the target disappeared behind the iso-motion texture, the TTC was estimated shorter than anti- and static textures. In Experiment 4, using an interruption paradigm, we found negative Point of Subjective Equalities (PSEs) with iso-motion but not static texture, suggesting that iso-motion led to overestimation of the hidden speed. However, sensitivity to target speed differences, as assessed by JNDs and d'values was not affected. Results of Experiments 3 and 4 indicate that only the iso-texture affected the estimated target speed, but with opposite polarity compared to visible motion, suggesting a different origin of speed bias. Because our results show that visuospatial tracking was facilitated by the fast iso-motion, we conclude that motion of the occluded target was tracked by shifting visuospatial attention.

Decoding information about dynamically occluded objects in visual cortex

During dynamic occlusion, an object passes behind an occluding surface and then later reappears. Even when completely occluded from view, such objects are experienced as continuing to exist or persist behind the occluder even though they are no longer visible. The contents and neural basis of this persistent representation remain poorly understood. Questions remain as to whether there is information maintained about the object itself (i.e. its shape or identity) or non-object-specific information such as its position or velocity as it is tracked behind an occluder, as well as which areas of visual cortex represent such information. Recent studies have found that early visual cortex is activated by "invisible" objects during visual imagery and by unstimulated regions along the path of apparent motion, suggesting that some properties of dynamically occluded objects may also be neurally represented in early visual cortex. We applied functional magnetic resonance imaging in human subjects to examine representations within visual cortex during dynamic occlusion. For gradually occluded, but not for instantly disappearing objects, there was an increase in activity in early visual cortex (V1, V2, and V3). This activity was spatially-specific, corresponding to the occluded location in the visual field. However, the activity did not encode enough information about object identity to discriminate between different kinds of occluded objects (circles vs. stars) using MVPA. In contrast, object identity could be decoded in spatially-specific subregions of higher-order, topographically organized areas such as ventral, lateral, and temporal occipital areas (VO, LO, and TO) as well as the functionally defined LOC and hMT+. These results suggest that early visual cortex may only represent the dynamically occluded object's position or motion path, while later visual areas represent object-specific information.

The neural representation of objects formed through the spatiotemporal integration of visual transients

Oftentimes, objects are only partially and transiently visible as parts of them become occluded during observer or object motion. The visual system can integrate such object fragments across space and time into perceptual wholes or spatiotemporal objects. This integrative and dynamic process may involve both ventral and dorsal visual processing pathways, along which shape and spatial representations are thought to arise. We measured fMRI BOLD response to spatiotemporal objects and used multi-voxel pattern analysis (MVPA) to decode shape information across 20 topographic regions of visual cortex. Object identity could be decoded throughout visual cortex, including intermediate (V3A, V3B, hV4, LO1-2,) and dorsal (TO1-2, and IPS0-1) visual areas. Shape-specific information, therefore, may not be limited to early and ventral visual areas, particularly when it is dynamic and must be integrated. Contrary to the classic view that the representation of objects is the purview of the ventral stream, intermediate and dorsal areas may play a distinct and critical role in the construction of object representations across space and time.

Filling gaps in visual motion for target capture

A remarkable challenge our brain must face constantly when interacting with the environment is represented by ambiguous and, at times, even missing sensory information. This is particularly compelling for visual information, being the main sensory system we rely upon to gather cues about the external world. It is not uncommon, for example, that objects catching our attention may disappear temporarily from view, occluded by visual obstacles in the foreground. Nevertheless, we are often able to keep our gaze on them throughout the occlusion or even catch them on the fly in the face of the transient lack of visual motion information. This implies that the brain can fill the gaps of missing sensory information by extrapolating the object motion through the occlusion. In recent years, much experimental evidence has been accumulated that both perceptual and motor processes exploit visual motion extrapolation mechanisms. Moreover, neurophysiological and neuroimaging studies have identified brain regions potentially involved in the predictive representation of the occluded target motion. Within this framework, ocular pursuit and manual interceptive behavior have proven to be useful experimental models for investigating visual extrapolation mechanisms. Studies in these fields have pointed out that visual motion extrapolation processes depend on manifold information related to short-term memory representations of the target motion before the occlusion, as well as to longer term representations derived from previous experience with the environment. We will review recent oculomotor and manual interception literature to provide up-to-date views on the neurophysiological underpinnings of visual motion extrapolation.

The role of areas MT+/V5 and SPOC in spatial and temporal control of manual interception: an rTMS study

Manual interception, such as catching or hitting an approaching ball, requires the hand to contact a moving object at the right location and at the right time. Many studies have examined the neural mechanisms underlying the spatial aspects of goal-directed reaching, but the neural basis of the spatial and temporal aspects of manual interception are largely unknown. Here, we used repetitive transcranial magnetic stimulation (rTMS) to investigate the role of the human middle temporal visual motion area (MT+/V5) and superior parieto-occipital cortex (SPOC) in the spatial and temporal control of manual interception. Participants were required to reach-to-intercept a downward moving visual target that followed an unpredictably curved trajectory, presented on a screen in the vertical plane. We found that rTMS to MT+/V5 influenced interceptive timing and positioning, whereas rTMS to SPOC only tended to increase the spatial variance in reach end points for selected target trajectories. These findings are consistent with theories arguing that distinct neural mechanisms contribute to spatial, temporal, and spatiotemporal control of manual interception.

The role of areas MT+/V5 and SPOC in spatial and temporal control of manual interception: an rTMS study

Manual interception, such as catching or hitting an approaching ball, requires the hand to contact a moving object at the right location and at the right time. Many studies have examined the neural mechanisms underlying the spatial aspects of goal-directed reaching, but the neural basis of the spatial and temporal aspects of manual interception are largely unknown. Here, we used repetitive transcranial magnetic stimulation (rTMS) to investigate the role of the human middle temporal visual motion area (MT+/V5) and superior parieto-occipital cortex (SPOC) in the spatial and temporal control of manual interception. Participants were required to reach-to-intercept a downward moving visual target that followed an unpredictably curved trajectory, presented on a screen in the vertical plane. We found that rTMS to MT+/V5 influenced interceptive timing and positioning, whereas rTMS to SPOC only tended to increase the spatial variance in reach end points for selected target trajectories. These findings are consistent with theories arguing that distinct neural mechanisms contribute to spatial, temporal, and spatiotemporal control of manual interception.

Electrophysiological correlates of biological motion permanence in humans

Spatiotemporal discontinuity of visual input is a common occurrence in daily life. For example, when a walking person disappears temporarily behind a wall, observers have a clear sense of his physical presence despite the absence of any visual information (movement permanence). To investigate the neural substrates of biological motion permanence, we recorded scalp EEG activity of sixteen subjects while they passively observed either biological or scrambled motion disappearing behind an occluder and reappearing. The moment of the occluder's appearance was either fixed or randomized. The statistical comparison between the biological and scrambled motion ERP waveforms revealed a modulation of activity in centro-parietal and right occipito-temporal regions during the occlusion phase when the biological motion disappearance was time-locked, possibly reflecting the recall of sensorimotor representations. These representations might allow the prediction of moving organisms in occlusion conditions. When the appearance of the occluder was unpredictable there was no difference between biological and scrambled motion either before or during occlusion, indicating that temporal prediction is relevant to the processing of biological motion permanence.

Learning to sample: eye tracking and fMRI indices of changes in object perception

We used an fMRI/eye-tracking approach to examine the mechanisms involved in learning to segment a novel, occluded object in a scene. Previous research has suggested a role for effective visual sampling and prior experience in the development of mature object perception. However, it remains unclear how the naive system integrates across variable sampled experiences to induce perceptual change. We generated a Target Scene in which a novel occluded Target Object could be perceived as either "disconnected" or "complete." We presented one group of participants with this scene in alternating sequence with variable visual experience: three Paired Scenes consisting of the same Target Object in variable rotations and states of occlusion. A second control group was presented with similar Paired Scenes that did not incorporate the Target Object. We found that, relative to the Control condition, participants in the Training condition were significantly more likely to change their percept from "disconnected" to "connected," as indexed by pretraining and posttraining test performance. In addition, gaze patterns during Target Scene inspection differed as a function of variable object exposure. We found increased looking to the Target Object in the Training compared with the Control condition. This pattern was not restricted to participants who changed their initial "disconnected" object percept. Neuroimaging data suggest an involvement of the hippocampus and BG, as well as visual cortical and fronto-parietal regions, in using ongoing regular experience to enable changes in amodal completion.

Covert tracking: a combined ERP and fixational eye movement study

Attention can be directed to particular spatial locations, or to objects that appear at anticipated points in time. While most work has focused on spatial or temporal attention in isolation, we investigated covert tracking of smoothly moving objects, which requires continuous coordination of both. We tested two propositions about the neural and cognitive basis of this operation: first that covert tracking is a right hemisphere function, and second that pre-motor components of the oculomotor system are responsible for driving covert spatial attention during tracking. We simultaneously recorded event related potentials (ERPs) and eye position while participants covertly tracked dots that moved leftward or rightward at 12 or 20°/s. ERPs were sensitive to the direction of target motion. Topographic development in the leftward motion was a mirror image of the rightward motion, suggesting that both hemispheres contribute equally to covert tracking. Small shifts in eye position were also lateralized according to the direction of target motion, implying covert activation of the oculomotor system. The data addresses two outstanding questions about the nature of visuospatial tracking. First, covert tracking is reliant upon a symmetrical frontoparietal attentional system, rather than being right lateralized. Second, this same system controls both pursuit eye movements and covert tracking.

Dynamic Object Representations in Infants with and without Fragile X Syndrome

Our visual world is dynamic in nature. The ability to encode, mentally represent, and track an object's identity as it moves across time and space is critical for integrating and maintaining a complete and coherent view of the world. Here we investigated dynamic object processing in typically developing (TD) infants and infants with fragile X syndrome (FXS), a single-gene disorder associated with deficits in dorsal stream functioning. We used the violation of expectation method to assess infants' visual response to expected versus unexpected outcomes following a brief dynamic (dorsal stream) or static (ventral stream) occlusion event. Consistent with previous reports of deficits in dorsal stream-mediated functioning in individuals with this disorder, these results reveal that, compared to mental age-matched TD infants, infants with FXS could maintain the identity of static, but not dynamic, object information during occlusion. These findings are the first to experimentally evaluate visual object processing skills in infants with FXS, and further support the hypothesis of dorsal stream difficulties in infants with this developmental disorder.

Representational momentum in children born preterm and at term

The term representational momentum (RM) refers to the idea that our memory representations for moving objects incorporate information about movement - a fact that can lead us to make errors when judging an object's location (the RM effect). In this study, we explored the RM effect in a sample of children born very prematurely and a sample born at term. Because preterm children are known to be at risk for problems with motion perception, we anticipated that they would show a weaker or absent RM effect. This prediction was confirmed. In addition, we found that, in both samples of children, 5-6year olds showed a reduced RM effect compared to 7-9year olds. These results demonstrate that the ability to represent motion information in memory shows continued development over this age range, and may help to elucidate factors contributing to problems with fine and gross motor planning and execution that have been observed in the preterm population. We propose that problems affecting the formation, maintenance, or use of predictive models, or motion extrapolation skills, may have cascading effects on the development of other abilities.

Neuroanatomical distribution of five semantic components of verbs: evidence from fMRI

The Simulation Framework, also known as the Embodied Cognition Framework, maintains that conceptual knowledge is grounded in sensorimotor systems. To test several predictions that this theory makes about the neural substrates of verb meanings, we used functional magnetic resonance imaging (fMRI) to scan subjects' brains while they made semantic judgments involving five classes of verbs-specifically, Running verbs (e.g., run, jog, walk), Speaking verbs (e.g., shout, mumble, whisper), Hitting verbs (e.g., hit, poke, jab), Cutting verbs (e.g., cut, slice, hack), and Change of State verbs (e.g., shatter, smash, crack). These classes were selected because they vary with respect to the presence or absence of five distinct semantic components-specifically, ACTION, MOTION, CONTACT, CHANGE OF STATE, and TOOL USE. Based on the Simulation Framework, we hypothesized that the ACTION component depends on the primary motor and premotor cortices, that the MOTION component depends on the posterolateral temporal cortex, that the CONTACT component depends on the intraparietal sulcus and inferior parietal lobule, that the CHANGE OF STATE component depends on the ventral temporal cortex, and that the TOOL USE component depends on a distributed network of temporal, parietal, and frontal regions. Virtually all of the predictions were confirmed. Taken together, these findings support the Simulation Framework and extend our understanding of the neuroanatomical distribution of different aspects of verb meaning.

Spatiotemporal object continuity in human ventral visual cortex

Coherent visual experience requires that objects be represented as the same persisting individuals over time and motion. Cognitive science research has identified a powerful principle that guides such processing: Objects must trace continuous paths through space and time. Little is known, however, about how neural representations of objects, typically defined by visual features, are influenced by spatiotemporal continuity. Here, we report the consequences of spatiotemporally continuous vs. discontinuous motion on perceptual representations in human ventral visual cortex. In experiments using both dynamic occlusion and apparent motion, face-selective cortical regions exhibited significantly less activation when faces were repeated in continuous vs. discontinuous trajectories, suggesting that discontinuity caused featurally identical objects to be represented as different individuals. These results indicate that spatiotemporal continuity modulates neural representations of object identity, influencing judgments of object persistence even in the most staunchly "featural" areas of ventral visual cortex.

Naming dynamic and static actions: neuropsychological evidence

There has been considerable interest in identifying the neural correlates of action naming, but the bulk of previous work on this topic has utilized static stimuli. Recent research comparing the visual processing of dynamic versus static actions suggests that these two types of stimuli engage largely overlapping neural systems, raising the possibility that the higher-order processing requirements for naming dynamic and static actions might not be very different. To explore this issue in greater depth, we developed the Dynamic Action Naming Test (DANT), which consists of 158 video clips 3-5s in length, for each of which the participant is asked to produce the most appropriate verb. We administered the DANT to 78 brain-damaged patients drawn from our Patient Registry, and to a demographically matched group of 50 normal participants. Out of the 16 patients who performed defectively on the DANT, nearly all (15/16) had damage in the left hemisphere. Lesion analysis indicated that the frontal operculum was the most frequent area of damage in the 15 patients; also, damage to the posterolateral temporal-occipital sector (in and near MT) was specifically related to defective dynamic action naming. Most of the brain-damaged participants (n=71) also received our Static Action Naming Test (SANT), and we found that performances on verb items that were common across the DANT and SANT were highly correlated (R=.91). Moreover, patients who failed the DANT almost invariably also failed the SANT. These findings lend further support to the hypothesis that there is considerable commonality in the neural systems underlying the use of verbs to orally name dynamic and static actions, a conclusion that is in turn compatible with the concept of "representational momentum". Our results also contribute more generally to the rapidly growing field of research on embodied cognition.

Lateralization of the Posterior Parietal Cortex for Internal Monitoring of Self- versus Externally Generated Movements

Internal monitoring or state estimation of movements is essential for human motor control to compensate for inherent delays and noise in sensorimotor loops. Two types of internal estimation of movements exist: self-generated movements, and externally generated movements. We used functional magnetic resonance imaging to investigate differences in brain activity for internal monitoring of self- versus externally generated movements during visual occlusion. Participants tracked a sinusoidally moving target with a mouse cursor. On some trials, vision of either target (externally generated) or cursor (self-generated) movement was transiently occluded, during which subjects continued tracking by estimating current position of either the invisible target or cursor on screen. Analysis revealed that both occlusion conditions were associated with increased activity in the presupplementary motor area and decreased activity in the right lateral occipital cortex compared to a control condition with no occlusion. Moreover, the right and left posterior parietal cortex (PPC) showed greater activation during occlusion of target and cursor movements, respectively. This study suggests lateralization of the PPC for internal monitoring of internally versus externally generated movements, fully consistent with previously reported clinical findings.

Predictive tracking over occlusions by 4-month-old infants

Two experiments investigated how 16-20-week-old infants visually tracked an object that oscillated on a horizontal trajectory with a centrally placed occluder. To determine the principles underlying infants' tendency to shift gaze to the exiting side before the object arrives, occluder width, oscillation frequency, and motion amplitude were manipulated resulting in occlusion durations between 0.20 and 1.66 s. Through these manipulations, we were able to distinguish between several possible modes of behavior underlying 'predictive' actions at occluders. Four such modes were tested. First, if passage-of-time determines when saccades are made, the tendency to shift gaze over the occluder is expected to be a function of time since disappearance. Second, if visual salience of the exiting occluder edge determines when saccades are made, occluder width would determine the pre-reappearance gaze shifts but not oscillation frequency, amplitude, or velocity. Third, if memory of the duration of the previous occlusion determines when the subjects shift gaze over the occluder, it is expected that the gaze will shift after the same latency at the next occlusion irrespective of whether occlusion duration is changed or not. Finally, if infants base their pre-reappearance gaze shifts on their ability to represent object motion (cognitive mode), it is expected that the latency of the gaze shifts over the occluder is scaled to occlusion duration. Eye and head movements as well as object motion were measured at 240 Hz. In 49% of the passages, the infants shifted gaze to the opposite side of the occluder before the object arrived there. The tendency to make such gaze shifts could not be explained by the passage of time since disappearance. Neither could it be fully explained in terms of visual information present during occlusion, i.e. occluder width. On the contrary, it was found that the latency of the pre-reappearance gaze shifts was determined by the time of object reappearance and that it was a function of all three factors manipulated. The results suggest that object velocity is represented during occlusion and that infants track the object behind the occluder in their 'mind's eye'.

Neural substrates of dynamic object occlusion

In everyday environments, objects frequently go out of sight as they move and our view of them becomes obstructed by nearer objects, yet we perceive these objects as continuous and enduring entities. Here, we used functional magnetic resonance imaging with an attentive tracking paradigm to clarify the nature of perceptual and cognitive mechanisms subserving this ability to fill in the gaps in perception of dynamic object occlusion. Imaging data revealed distinct regions of cortex showing increased activity during periods of occlusion relative to full visibility. These regions may support active maintenance of a representation of the target's spatiotemporal properties ensuring that the object is perceived as a persisting entity when occluded. Our findings may shed light on the neural substrates involved in object tracking that give rise to the phenomenon of object permanence.

Predicting point-light actions in real-time

There is convincing evidence for a mirror system in humans which simulates actions of conspecifics. One possible purpose of such a simulation system is to support action prediction in real-time. Our goal was to study whether the prediction of actions involves a real-time simulation process. We motion-captured a number of human actions and rendered them as point-light action sequences. Observers perceived brief videos of these actions, followed by an occluder and a static test posture. We independently varied the occluder time and the movement gap (i.e., the time between the endpoint of the action and the test posture). Observers were required to judge whether the test stimulus depicted a continuation of the action in the same depth orientation. Prediction performance was best when occluder time and movement gap corresponded, i.e., when the test posture was a continuation of the sequence that matched the occluder duration (Experiments 1, 2 and 4). This pattern of results was destroyed when the sequences and test images were flipped around the horizontal axis (Experiment 3). Overall, our findings suggest that action prediction involves a simulation process that operates in real-time. This process can break down when the actions are presented under viewing conditions for which observers have little experience.

A new response-time measure of object persistence in the tunnel effect

The recognition of information about an object is facilitated by a preview of the information concerning that object. This facilitation is regarded as evidence for the representational persistence of the object. It is not known, however, if such facilitation is obtained even under the tunnel effect, in which a moving object is temporarily occluded. This facilitation may be a new way to measure the representational persistence of a moving object in the tunnel effect. We addressed this question by a "same-different" judgment task of a target symbol (" composite function" or "+"), drawn within the moving object, before and after encountering the occluder. Response times (RTs) were shorter when the object reappeared with spatial continuity at the proper place than it reappeared at the improper place, as in Experiments 1 and 3. Thus, facilitation was obtained even in the tunnel effect. When the occluder was invisible and deletion/accretion cues along the contour of the occluder were either removed (Experiment 2) or given improperly (Experiment 4), no facilitation was found. These results clearly indicate that the facilitated recognition was caused by amodal integration of the persisting representation from the unoccluded and modal phases. The present study demonstrates that the facilitated recognition (RT measurement) can be used to investigate the representational persistence in the tunnel effect.

The predictive power of trajectory motion

When the central region of an obliquely oriented line is bisected by a wide, vertical opaque occluder, observers misperceive the two line segments as being misaligned (the Poggendorff illusion). If the oblique line segment is replaced with a spot moving on an oblique trajectory, little if any misalignment is perceived. This accurate alignment of oblique segments depends upon the consistent motion of the dot along the oblique trajectory and not other temporal or spatial characteristics of the motion-defined segments since random plotting of the dot along each oblique segment resulted in robust misalignment. The nullification of the Poggendorff illusion was also obtained if only one of the segments was defined by a moving spot so long as the spot moved in a direction that 'pointed' to the static segment. Moreover, if the occluder boundary was defined by rows of vertically moving dots, was filled with vertically moving dots or was a real (cardboard) occluder, the motion-defined oblique segments were still perceived to be aligned with little error, consistent with the unimpaired detection of a trajectory dot in noise interrupted by similar occluders [Watamaniuk, S. N. J. & McKee, S. P. (1995). 'Seeing' motion behind occluders. Nature, 377, 729-730]. The results are interpreted as evidence that trajectory motion produces a cascade of activity in appropriately aligned motion detectors, in the direction of motion, that continues after the moving object has been occluded to produce a prediction of where the moving object should reappear.

Combined smooth and saccadic ocular pursuit during the transient occlusion of a moving visual object

Accurate ocular pursuit during a transient occlusion interval would minimize retinal position and velocity error, and could provide an advantage when discriminating object characteristics at reappearance. This study was designed to examine how the smooth and saccadic response extrapolates the trajectory of a moving visual object during a transient occlusion. We confirmed that subjects could not maintain unity gain smooth pursuit during the transient occlusion. Eye velocity decayed significantly without visual feedback but then in the majority of subjects, there was a recovery that brought eye velocity back towards object velocity. However, eye velocity did not increase to a level that eliminated the developing position error. Subjects corrected for the resulting error in eye position by releasing saccades that generally placed the eye ahead of the occluded object's extrapolated position. The majority of saccadic correction occurred between 220 and 600 ms of the occlusion interval, and when combined with the smooth response enabled accurate pursuit of a 10 degrees/s object for up to 1,200 ms of occlusion. The lack of saccadic correction after 600 ms of occlusion combined with the reduced eye velocity resulted in significant undershoot of eye position at the moment of object reappearance when pursuing an 18 degrees/s object. We suggest that extra-retinal information regarding eye velocity and smooth eye displacement could be available from a continually updating efference copy of eye motion in MST, whereas a veridical representation of extrapolated object velocity and displacement could be obtained from persistent activity in FEF.

The Case for Motor Involvement in Perceiving Conspecifics

Perceiving other people's behaviors activates imitative motor plans in the perceiver, but there is disagreement as to the function of this activation. In contrast to other recent proposals (e.g., that it subserves overt imitation, identification and understanding of actions, or working memory), here it is argued that imitative motor activation feeds back into the perceptual processing of conspecifics' behaviors, generating top-down expectations and predictions of the unfolding action. Furthermore, this account incorporates recent ideas about emulators in the brain-mental simulations that run in parallel to the external events they simulate-to provide a mechanism by which motoric involvement could contribute to perception. Evidence from a variety of literatures is brought to bear to support this account of perceiving human body movement.

Changes in functional connectivity of human MT/V5 with visual motion input

The neural basis of human mental function is characterized by interactions between brain regions. Temporal correlations in MR signals between areas may provide one method for investigating these interactions. This approach was used to examine functional connectivity in the motion processing system of the human brain. Correlations between MT/V5 and other brain regions were examined in a resting state (without visual stimulation) and in an active state produced by viewing moving concentric circles. A network of regions consistent with the known functional anatomy of visual processing was correlated with MT/V5 during rest. When subjects were viewing motion, a more limited network was correlated with MT/V5, suggesting MT/V5 was acting in concert with a smaller network specific to the task.