Movement trajectories as a window into the dynamics of emerging neural representations

The rapid transformation of sensory inputs into meaningful neural representations is critical to adaptive human behaviour. While non-invasive neuroimaging methods are the de-facto method for investigating neural representations, they remain expensive, not widely available, time-consuming, and restrictive. Here we show that movement trajectories can be used to measure emerging neural representations with fine temporal resolution. By combining online computer mouse-tracking and publicly available neuroimaging data via representational similarity analysis (RSA), we show that movement trajectories track the unfolding of stimulus- and category-wise neural representations along key dimensions of the human visual system. We demonstrate that time-resolved representational structures derived from movement trajectories overlap with those derived from M/EEG (albeit delayed) and those derived from fMRI in functionally-relevant brain areas. Our findings highlight the richness of movement trajectories and the power of the RSA framework to reveal and compare their information content, opening new avenues to better understand human perception.

Visual Representations: Insights from Neural Decoding

Patterns of brain activity contain meaningful information about the perceived world. Recent decades have welcomed a new era in neural analyses, with computational techniques from machine learning applied to neural data to decode information represented in the brain. In this article, we review how decoding approaches have advanced our understanding of visual representations and discuss efforts to characterize both the complexity and the behavioral relevance of these representations. We outline the current consensus regarding the spatiotemporal structure of visual representations and review recent findings that suggest that visual representations are at once robust to perturbations, yet sensitive to different mental states. Beyond representations of the physical world, recent decoding work has shone a light on how the brain instantiates internally generated states, for example, during imagery and prediction. Going forward, decoding has remarkable potential to assess the functional relevance of visual representations for human behavior, reveal how representations change across development and during aging, and uncover their presentation in various mental disorders.

Statistical learning of distractor co-occurrences facilitates visual search

Visual search is facilitated by knowledge of the relationship between the target and the distractors, including both where the target is likely to be among the distractors and how it differs from the distractors. Whether the statistical structure among distractors themselves, unrelated to target properties, facilitates search is less well understood. Here, we assessed the benefit of distractor structure using novel shapes whose relationship to each other was learned implicitly during visual search. Participants searched for target items in arrays of shapes that comprised either four pairs of co-occurring distractor shapes (structured scenes) or eight distractor shapes randomly partitioned into four pairs on each trial (unstructured scenes). Across five online experiments (N = 1,140), we found that after a period of search training, participants were more efficient when searching for targets in structured than unstructured scenes. This structure benefit emerged independently of whether the position of the shapes within each pair was fixed or variable and despite participants having no explicit knowledge of the structured pairs they had seen. These results show that implicitly learned co-occurrence statistics between distractor shapes increases search efficiency. Increased efficiency in the rejection of regularly co-occurring distractors may contribute to the efficiency of visual search in natural scenes, where such regularities are abundant.

Critical information thresholds underlying generic and familiar face categorisation at the same face encounter

Seeing a face in the real world provokes a host of automatic categorisations related to sex, emotion, identity, and more. Such individual facets of human face recognition have been extensively examined using overt categorisation judgements, yet their relative informational dependencies during the same face encounter are comparatively unknown. Here we used EEG to assess how increasing access to sensory input governs two ecologically relevant brain functions elicited by seeing a face: Distinguishing faces and nonfaces, and recognising people we know. Observers viewed a large set of natural images that progressively increased in either image duration (experiment 1) or spatial frequency content (experiment 2). We show that in the absence of an explicit categorisation task, the human brain requires less sensory input to categorise a stimulus as a face than it does to recognise whether that face is familiar. Moreover, where sensory thresholds for distinguishing faces/nonfaces were remarkably consistent across observers, there was high inter-individual variability in the lower informational bound for familiar face recognition, underscoring the neurofunctional distinction between these categorisation functions. By i) indexing a form of face recognition that goes beyond simple low-level differences between categories, and ii) tapping multiple recognition functions elicited by the same face encounters, the information minima we report bear high relevance to real-world face encounters, where the same stimulus is categorised along multiple dimensions at once. Thus, our finding of lower informational requirements for generic vs. familiar face recognition constitutes some of the strongest evidence to date for the intuitive notion that sensory input demands should be lower for recognising face category than face identity.

Visual mismatch responses index surprise signalling but not expectation suppression

The ability to distinguish between commonplace and unusual sensory events is critical for efficient learning and adaptive behaviour. This has been investigated using oddball designs in which sequences of often-appearing (i.e., expected) stimuli are interspersed with rare (i.e., surprising) deviants. Resulting differences in electrophysiological responses following surprising compared to expected stimuli are known as visual mismatch responses (VMRs). VMRs are thought to index co-occurring contributions of stimulus repetition effects, expectation suppression (that occurs when one's expectations are fulfilled), and expectation violation (i.e., surprise) responses; however, these different effects have been conflated in existing oddball designs. To better isolate and quantify effects of expectation suppression and surprise, we adapted an oddball design based on Fast Periodic Visual Stimulation (FPVS) that controls for stimulus repetition effects. We recorded electroencephalography (EEG) while participants (N = 48) viewed stimulation sequences in which a single face identity was periodically presented at 6 Hz. Critically, one of two different face identities (termed oddballs) appeared as every 7th image throughout the sequence. The presentation probabilities of each oddball image within a sequence varied between 10 and 90%, such that participants could form expectations about which oddball face identity was more likely to appear within each sequence. We also included 'expectation neutral' 50% probability sequences, whereby consistently biased expectations would not be formed for either oddball face identity. We found that VMRs indexed surprise responses, and effects of expectation suppression were absent. That is, ERPs were more negative-going at occipitoparietal electrodes for surprising compared to neutral oddballs, but did not differ between expected and neutral oddballs. Surprising oddball-evoked ERPs were also highly similar across the 10-40% appearance probability conditions. Our findings indicate that VMRs which are not accounted for by repetition effects are best described as an all-or-none surprise response, rather than a minimisation of prediction error responses associated with expectation suppression.

Contextual and Spatial Associations Between Objects Interactively Modulate Visual Processing

Much of what we know about object recognition arises from the study of isolated objects. In the real world, however, we commonly encounter groups of contextually associated objects (e.g., teacup and saucer), often in stereotypical spatial configurations (e.g., teacup above saucer). Here we used electroencephalography to test whether identity-based associations between objects (e.g., teacup–saucer vs. teacup–stapler) are encoded jointly with their typical relative positioning (e.g., teacup above saucer vs. below saucer). Observers viewed a 2.5-Hz image stream of contextually associated object pairs intermixed with nonassociated pairs as every fourth image. The differential response to nonassociated pairs (measurable at 0.625 Hz in 28/37 participants) served as an index of contextual integration, reflecting the association of object identities in each pair. Over right occipitotemporal sites, this signal was larger for typically positioned object streams, indicating that spatial configuration facilitated the extraction of the objects’ contextual association. This high-level influence of spatial configuration on object identity integration arose ~ 320 ms post-stimulus onset, with lower-level perceptual grouping (shared with inverted displays) present at ~ 130 ms. These results demonstrate that contextual and spatial associations between objects interactively influence object processing. We interpret these findings as reflecting the high-level perceptual grouping of objects that frequently co-occur in highly stereotyped relative positions.

Object Vision in a Structured World

In natural vision, objects appear at typical locations, both with respect to visual space (e.g., an airplane in the upper part of a scene) and other objects (e.g., a lamp above a table). Recent studies have shown that object vision is strongly adapted to such positional regularities. In this review we synthesize these developments, highlighting that adaptations to positional regularities facilitate object detection and recognition, and sharpen the representations of objects in visual cortex. These effects are pervasive across various types of high-level content. We posit that adaptations to real-world structure collectively support optimal usage of limited cortical processing resources. Taking positional regularities into account will thus be essential for understanding efficient object vision in the real world.

Fast periodic visual stimulation to study tool-selective processing in the human brain

Because tools are manipulated for the purpose of action, they are often considered to be a specific object category that associates perceptual and motor properties. Their neural processing has been studied extensively by comparing the cortical activity elicited by the separate presentation of tool and non-tool objects, assuming that observed differences are solely due to activity selective for processing tools. Here, using a fast periodic visual stimulation (FPVS) paradigm, we isolated EEG activity selectively related to the processing of tool objects embedded in a stream of non-tool objects. Participants saw a continuous sequence of tool and non-tool images at a 3.7 Hz presentation rate, arranged as a repeating pattern of four non-tool images followed by one tool image. We expected the stimulation to generate an EEG response at the frequency of image presentation (3.7 Hz) and its harmonics, reflecting activity common to the processing of tool and non-tool images. Most importantly, if tool and non-tool images evoked different neural responses, we expected this differential activity to generate an additional response at the frequency of tool images (3.7 Hz/5 = 0.74 Hz). To ensure that this response was not due to unaccounted for systematic differences in low-level visual features, we also tested a phase-scrambled version of the sequence. The periodic insertion of tool stimuli within a stream of non-tool stimuli elicited a significant EEG response at the tool-selective frequency and its harmonics. This response was reduced when the images were phase-scrambled. We conclude that FPVS is a promising technique to selectively measure tool-related activity.

Gaining the upper hand: evidence of vertical asymmetry in sex-categorisation of human hands

Visual perception is characterised by asymmetries arising from the brain's preferential response to particular stimulus types at different retinal locations. Where the lower visual field (LVF) holds an advantage over the upper visual field (UVF) for many tasks (e.g., hue discrimination, contrast sensitivity, motion processing), face-perception appears best supported at above-fixation locations (Quek & Finkbeiner, 2014a). This finding is consistent with Previc's (1990) suggestion that vision in the UVF has become specialised for object recognition processes often required in "extrapersonal" space. Outside of faces, however, there have been very few investigations of vertical asymmetry effects for higher-level objects. Our aim in the present study was, thus, to determine whether the UVF advantage reported for face-perception would extend to a nonface object - human hands. Participants classified the sex of hand images presented above or below central fixation by reaching out to touch a left or right response panel. On each trial, a briefly presented spatial cue captured the participant's spatial attention to either the location where the hand was about to appear (valid cue) or the opposite location (invalid cue). We observed that cue validity only modulated the efficiency of the sex-categorisation response for targets in the LVLVF and not the UVF, just as we have reported previously for face-sex categorisation (Quek & Finkbeiner, 2014a). Taken together, the data from these studies provide some empirical support for Previc's (1990) speculation that object recognition processes may enjoy an advantage in the upper-hemifield.