Flexible visual processing of spatial relationships

A Phone in a Basket Looks Like a Knife in a Cup: Role-Filler Independence in Visual Processing

When a piece of fruit is in a bowl, and the bowl is on a table, we appreciate not only the individual objects and their features, but also the relations containment and support, which abstract away from the particular objects involved. Independent representation of roles (e.g., containers vs. supporters) and "fillers" of those roles (e.g., bowls vs. cups, tables vs. chairs) is a core principle of language and higher-level reasoning. But does such role-filler independence also arise in automatic visual processing? Here, we show that it does, by exploring a surprising error that such independence can produce. In four experiments, participants saw a stream of images containing different objects arranged in force-dynamic relations-e.g., a phone contained in a basket, a marker resting on a garbage can, or a knife sitting in a cup. Participants had to respond to a single target image (e.g., a phone in a basket) within a stream of distractors presented under time constraints. Surprisingly, even though participants completed this task quickly and accurately, they false-alarmed more often to images matching the target's relational category than to those that did not-even when those images involved completely different objects. In other words, participants searching for a phone in a basket were more likely to mistakenly respond to a knife in a cup than to a marker on a garbage can. Follow-up experiments ruled out strategic responses and also controlled for various confounding image features. We suggest that visual processing represents relations abstractly, in ways that separate roles from fillers.

Neural dynamic foundations of a theory of higher cognition: the case of grounding nested phrases

Because cognitive competences emerge in evolution and development from the sensory-motor domain, we seek a neural process account for higher cognition in which all representations are necessarily grounded in perception and action. The challenge is to understand how hallmarks of higher cognition, productivity, systematicity, and compositionality, may emerge from such a bottom-up approach. To address this challenge, we present key ideas from Dynamic Field Theory which postulates that neural populations are organized by recurrent connectivity to create stable localist representations. Dynamic instabilities enable the autonomous generation of sequences of mental states. The capacity to apply neural circuitry across broad sets of inputs that emulates the function call postulated in symbolic computation emerges through coordinate transforms implemented in neural gain fields. We show how binding localist neural representations through a shared index dimension enables conceptual structure, in which the interdependence among components of a representation is flexibly expressed. We demonstrate these principles in a neural dynamic architecture that represents and perceptually grounds nested relational and action phrases. Sequences of neural processing steps are generated autonomously to attentionally select the referenced objects and events in a manner that is sensitive to their interdependencies. This solves the problem of 2 and the massive binding problem in expressions such as "the small tree that is to the left of the lake which is to the left of the large tree". We extend earlier work by incorporating new types of grammatical constructions and a larger vocabulary. We discuss the DFT framework relative to other neural process accounts of higher cognition and assess the scope and challenges of such neural theories.

Language systematizes attention: How relational language enhances relational representation by guiding attention

Language can affect cognition, but through what mechanism? Substantial past research has focused on how labeling can elicit categorical representation during online processing. We focus here on a particularly powerful type of language-relational language-and show that relational language can enhance relational representation in children through an embodied attention mechanism. Four-year-old children were given a color-location conjunction task, in which they were asked to encode a two-color square, split either vertically or horizontally (e.g., red on the left, blue on the right), and later recall the same configuration from its mirror reflection. During the encoding phase, children in the experimental condition heard relational language (e.g., "Red is on the left of blue"), while those in the control condition heard generic non-relational language (e.g., "Look at this one, look at it closely"). At recall, children in the experimental condition were more successful at choosing the correct relational representation between the two colors compared to the control group. Moreover, they exhibited different attention patterns as predicted by the attention shift account of relational representation (Franconeri et al., 2012). To test the sustained effect of language and the role of attention, during the second half of the study, the experimental condition was given generic non-relational language. There was a sustained advantage in the experimental condition for both behavioral accuracies and signature attention patterns. Overall, our findings suggest that relational language enhances relational representation by guiding learners' attention, and this facilitative effect persists over time even in the absence of language. Implications for the mechanism of how relational language can enhance the learning of relational systems (e.g., mathematics, spatial cognition) by guiding attention will be discussed.

What is moving where? Infants' visual attention to dynamic objects may assist with processing of spatial relations

Introduction

A central question in infant spatial cognition concerns how infants form abstract categories of spatial relations such as support (on) and containment (in). Prior work suggests two different possibilities regarding the role of attention to objects in infants' formation of abstract categories of spatial relations: Attention to objects may compete with (and thus hamper) attention to the spatial relations between them, or assist with encoding of the spatial relation information. Using eye-tracking, we examined how infants' visual attention to objects related to their successful formation of an abstract category of support relations (i.e., an object on another).

Methods

Thirty-eight 8-month-old infants' eye movements were recorded during a support categorization task, where infants were habituated to four dynamic events depicting support relations (e.g., resting a block on a box) and then presented with test events that depicted either a support or containment relation with objects that they had seen or not seen in the habituation phase. Based on their looking time to the familiar versus novel spatial relation in the test, infants were classified into two groups: categorizers, who formed an abstract category of a support relation, and non-categorizers, who did not do so.

Results

During their initial phase of learning (i.e., the first habituation trial), categorizers paid greater attention to the object moved by a hand (i.e., the dynamic object) in comparison to non-categorizers, whereas their attention to the static object or their gaze shifts between the two objects did not differ. In addition, when presented with novel objects in a novel spatial relation after habituation, only categorizers displayed asymmetric attention between the objects, attending to the dynamic object more than the static object. Gaze shifts and attention to the concave area (i.e., hole) of the container did not differ between categorizers and non-categorizers.

Discussion

These findings suggest that infants' focused attention to an object in motion may play a key role in young infants' spatial category learning, and support the idea that attention to objects can assist with encoding of the spatial relational information.

Variability of dot spread is overestimated

Previous research has demonstrated that individuals exhibit a tendency to overestimate the variability of both low-level features (e.g., color, orientation) and mid-level features (e.g., size) when items are presented dynamically in a sequential order, a finding we will refer to as the variability overestimation effect. Because previous research on this bias used sequential displays, an open question is whether the effect is due to a memory-related bias or a vision-related bias. To assess whether the bias would also be apparent with static, simultaneous displays, and to examine whether the bias generalizes to spatial properties, we tested participants' perception of the variability of a cluster of dots. Results showed a consistent overestimation bias: Participants judged the dots as being more spread than they actually were. The variability overestimation effect was observed when there were 10 or 20 dots but not when there were 50 dots. Taken together, the results of the current study contribute to the ensemble perception literature by providing evidence that simultaneously presented stimuli are also susceptible to the variability overestimation effect. The use of static displays further demonstrates that this bias is present in both dynamic and static contexts, suggesting an inherent bias existent in the human visual system. A potential theoretical account-boundary effect-is discussed as a potential underlying mechanism. Moreover, the present study has implications for common visual tasks carried out in real-world scenarios, such as a radiologist making judgments about distribution of calcification in breast cancer diagnoses.

Spatial Alignment Facilitates Visual Comparison in Children

Visual comparison is a key process in everyday learning and reasoning. Recent research has discovered the spatial alignment principle, based on the broader framework of structure-mapping theory in comparison. According to the spatial alignment principle, visual comparison is more efficient when the figures being compared are arranged in direct placement-that is, juxtaposed with parallel structural axes. In this placement, (1) the intended relational correspondences are readily apparent, and (2) the influence of potential competing correspondences is minimized. There is evidence for the spatial alignment principle in adults' visual comparison (Matlen et al., 2020). Here, we test whether it holds for children. Six- and eight-year-old children performed a same-different task over visual pairs. The results indicated that direct placement led to faster and more accurate comparison, both for concrete same-different matches (matches of both objects and relations) and for purely relational matches-evidence that the same structural alignment process holds for visual comparison in 6- and 8-year-olds as in adults. These findings have implications for learning and education.

Visual Arrangements of Bar Charts Influence Comparisons in Viewer Takeaways

Well-designed data visualizations can lead to more powerful and intuitive processing by a viewer. To help a viewer intuitively compare values to quickly generate key takeaways, visualization designers can manipulate how data values are arranged in a chart to afford particular comparisons. Using simple bar charts as a case study, we empirically tested the comparison affordances of four common arrangements: vertically juxtaposed, horizontally juxtaposed, overlaid, and stacked. We asked participants to type out what patterns they perceived in a chart and we coded their takeaways into types of comparisons. In a second study, we asked data visualization design experts to predict which arrangement they would use to afford each type of comparison and found both alignments and mismatches with our findings. These results provide concrete guidelines for how both human designers and automatic chart recommendation systems can make visualizations that help viewers extract the "right" takeaway.

The Science of Visual Data Communication: What Works

Effectively designed data visualizations allow viewers to use their powerful visual systems to understand patterns in data across science, education, health, and public policy. But ineffectively designed visualizations can cause confusion, misunderstanding, or even distrust-especially among viewers with low graphical literacy. We review research-backed guidelines for creating effective and intuitive visualizations oriented toward communicating data to students, coworkers, and the general public. We describe how the visual system can quickly extract broad statistics from a display, whereas poorly designed displays can lead to misperceptions and illusions. Extracting global statistics is fast, but comparing between subsets of values is slow. Effective graphics avoid taxing working memory, guide attention, and respect familiar conventions. Data visualizations can play a critical role in teaching and communication, provided that designers tailor those visualizations to their audience.

A Neural Dynamic Model of the Perceptual Grounding of Spatial and Movement Relations

How does the human brain link relational concepts to perceptual experience? For example, a speaker may say "the cup to the left of the computer" to direct the listener's attention to one of two cups on a desk. We provide a neural dynamic account for both perceptual grounding, in which relational concepts enable the attentional selection of objects in the visual array, and for the generation of descriptions of the visual array using relational concepts. In the model, activation in neural populations evolves dynamically under the influence of both inputs and strong interaction as formalized in dynamic field theory. Relational concepts are modeled as patterns of connectivity to perceptual representations. These generalize across the visual array through active coordinate transforms that center the representation of target objects in potential reference objects. How the model perceptually grounds or generates relational descriptions is probed in 104 simulations that systematically vary the spatial and movement relations employed, the number of feature dimensions used, and the number of matching and nonmatching objects. We explain how sequences of decisions emerge from the time- and state-continuous neural dynamics, how relational hypotheses are generated and either accepted or rejected, followed by the selection of new objects or the generation of new relational hypotheses. Its neural realism distinguishes the model from information processing accounts, its capacity to autonomously generate sequences of processing steps distinguishes it from deep neural network accounts. The model points toward a neural dynamic theory of higher cognition.

Three Perceptual Tools for Seeing and Understanding Visualized Data

The visual system evolved and develops to process the scenes, faces, and objects of the natural world, but people adapt this powerful system to process data within an artificial world of visualizations. To extract patterns in data from these artificial displays, viewers appear to use at least three perceptual tools, including a tool that extracts global statistics, one that extracts shapes within the data, and one that produces sentence-like comparisons. A better understanding of the power, limits, and deployment of these tools would lead to better guidelines for designing effective data displays.

The Perception of Relations

The world contains not only objects and features (red apples, glass bowls, wooden tables), but also relations holding between them (apples contained in bowls, bowls supported by tables). Representations of these relations are often developmentally precocious and linguistically privileged; but how does the mind extract them in the first place? Although relations themselves cast no light onto our eyes, a growing body of work suggests that even very sophisticated relations display key signatures of automatic visual processing. Across physical, eventive, and social domains, relations such as support, fit, cause, chase, and even socially interact are extracted rapidly, are impossible to ignore, and influence other perceptual processes. Sophisticated and structured relations are not only judged and understood, but also seen - revealing surprisingly rich content in visual perception itself.

The Language of Vision

The descriptions of surfaces, objects, and events computed by visual processes are not solely for consumption in the visual system but are meant to be passed on to other brain centers. Clearly, the description of the visual scene cannot be sent in its entirety, like a picture or movie, to other centers, as that would require that each of them have their own visual system to decode the description. Some very compressed, annotated, or labeled version must be constructed that can be passed on in a format that other centers—memory, language, planning—can understand. If this is a “visual language,” what is its grammar? In a first pass, we see, among other things, differences in processing of visual “nouns,” visual “verbs,” and visual “prepositions.” Then we look at recursion and errors of visual grammar. Finally, the possibility of a visual language also raises the question of the acquisition of its grammar from the visual environment and the chance that this acquisition process was borrowed and adapted for spoken language.

Differential Involvement of EEG Oscillatory Components in Sameness versus Spatial-Relation Visual Reasoning Tasks

The development of deep convolutional neural networks (CNNs) has recently led to great successes in computer vision, and CNNs have become de facto computational models of vision. However, a growing body of work suggests that they exhibit critical limitations on tasks beyond image categorization. Here, we study one such fundamental limitation, concerning the judgment of whether two simultaneously presented items are the same or different (SD) compared with a baseline assessment of their spatial relationship (SR). In both human subjects and artificial neural networks, we test the prediction that SD tasks recruit additional cortical mechanisms which underlie critical aspects of visual cognition that are not explained by current computational models. We thus recorded electroencephalography (EEG) signals from human participants engaged in the same tasks as the computational models. Importantly, in humans the two tasks were matched in terms of difficulty by an adaptive psychometric procedure; yet, on top of a modulation of evoked potentials (EPs), our results revealed higher activity in the low β (16–24 Hz) band in the SD compared with the SR conditions. We surmise that these oscillations reflect the crucial involvement of additional mechanisms, such as working memory and attention, which are missing in current feed-forward CNNs.

Beyond the feedforward sweep: feedback computations in the visual cortex

Visual perception involves the rapid formation of a coarse image representation at the onset of visual processing, which is iteratively refined by late computational processes. These early versus late time windows approximately map onto feedforward and feedback processes, respectively. State-of-the-art convolutional neural networks, the main engine behind recent machine vision successes, are feedforward architectures. Their successes and limitations provide critical information regarding which visual tasks can be solved by purely feedforward processes and which require feedback mechanisms. We provide an overview of recent work in cognitive neuroscience and machine vision that highlights the possible role of feedback processes for both visual recognition and beyond. We conclude by discussing important open questions for future research.

Measures of the Benefit of Direct Encoding of Data Deltas for Data Pair Relation Perception

The power of data visualization is not to convey absolute values of individual data points, but to allow the exploration of relations (increases or decreases in a data value) among them. One approach to highlighting these relations is to explicitly encode the numeric differences (deltas) between data values. Because this approach removes the context of the individual data values, it is important to measure how much of a performance improvement it actually offers, especially across differences in encodings and tasks, to ensure that it is worth adding to a visualization design. Across 3 different tasks, we measured the increase in visual processing efficiency for judging the relations between pairs of data values, from when only the values were shown, to when the deltas between the values were explicitly encoded, across position and length visual feature encodings (and slope encodings in Experiments 1 & 2). In Experiment 1, the participant's task was to locate a pair of data values with a given relation (e.g., Find the 'small bar to the left of a tall bar' pair) among pairs of the opposite relation, and we measured processing efficiency from the increase in response times as the number of pairs increased. In Experiment 2, the task was to judge which of two relation types was more prevalent in a briefly presented display of 10 data pairs (e.g., Are there more 'small bar to the left of a tall bar' pairs or more 'tall bar to the left of a small bar' pairs?). In the final experiment, the task was to estimate the average delta within a briefly presented display of 6 data pairs (e.g., What is the average bar height difference across all 'small bar to the left of a tall bar' pairs?). Across all three experiments, visual processing of relations between data value pairs was significantly better when directly encoded as deltas rather than implicitly between individual data points, and varied substantially depending on the task (improvement ranged from 25% to 95%). Considering the ubiquity of bar charts and dot plots, relation perception for individual data values is highly inefficient, and confirms the need for alternative designs that provide not only absolute values, but also direct encoding of critical relationships between those values.

Focusing on what matters: Modulation of the human hippocampus by relational attention

Hippocampal episodic memory is fundamentally relational, comprising links between events and the spatiotemporal contexts in which they occurred. Such relations are also important over shorter timescales, during online perception. For example, how do we assess the relative spatial positions of objects, their temporal order, or the relationship between their features? Here, we investigate the role of the hippocampus in online relational processing by manipulating attention to different kinds of relations. While undergoing fMRI, participants viewed two images in rapid succession on each trial and performed one of three relational tasks, judging the images' relative: spatial positions, temporal onsets, or sizes. Additionally, they sometimes judged whether one image was tilted, irrespective of the other. This served as a baseline item task with no demands on relational processing. The hippocampus showed reliable deactivation when participants attended to relational vs. item information. Attention to temporal relations was associated with the most robust deactivation. One interpretation of such deactivation is that it reflects hippocampal disengagement. If true, there should be reduced information content and noisier activity patterns for the temporal vs. other tasks. Instead, multivariate pattern analysis revealed more stable hippocampal representations in the temporal task. This increased pattern similarity was not simply a reflection of lower univariate activity. Thus, the hippocampus differentiates between relational and item processing even during online perception, and its representations of temporal relations are particularly robust. These findings suggest that the relational computations of the hippocampus extend beyond long-term memory, enabling rapid extraction of relational information in perception.

Computer mouse tracking reveals motor signatures in a cognitive task of spatial language grounding

In a novel computer mouse tracking paradigm, participants read a spatial phrase such as "The blue item to the left of the red one" and then see a scene composed of 12 visual items. The task is to move the mouse cursor to the target item (here, blue), which requires perceptually grounding the spatial phrase. This entails visually identifying the reference item (here, red) and other relevant items through attentional selection. Response trajectories are attracted toward distractors that share the target color but match the spatial relation less well. Trajectories are also attracted toward items that share the reference color. A competing pair of items that match the specified colors but are in the inverse spatial relation increases attraction over-additively compared to individual items. Trajectories are also influenced by the spatial term itself. While the distractor effect resembles deviation toward potential targets in previous studies, the reference effect suggests that the relevance of the reference item for the relational task, not its role as a potential target, was critical. This account is supported by the strengthened effect of a competing pair. We conclude, therefore, that the attraction effects in the mouse trajectories reflect the neural processes that operate on sensorimotor representations to solve the relational task. The paradigm thus provides an experimental window through motor behavior into higher cognitive function and the evolution of activation in modal substrates, a longstanding topic in the area of embodied cognition.

Similarity Grouping as Feature-Based Selection

Across the natural world as well as the artificial worlds of maps, diagrams, and data visualizations, feature similarity (e.g., color and shape) links spatially separate areas into sets. Despite a century of study, it is yet unclear what mechanism underlies this gestalt similarity grouping. One recent proposal is that similarity grouping-for example, seeing a red, vertical, or square group-is just global selection of those features. Although parsimonious, this account makes the counterintuitive prediction that similarity grouping is strictly serial: A green group cannot be constructed at the same time as a red group. We tested this prediction with a novel measure-a grouping illusion within number-estimation tasks that should work only if participants simultaneously construct groups-and found the strongest evidence yet in favor of serial feature-based attention ( Ns = 14, 12, and 12 for Experiment 1, Experiment 2, and Experiment 3, respectively).

The Graph Rubric: Development of a Teaching, Learning, and Research Tool

As undergraduate biology curricula increasingly aim to provide students with access to courses and experiences that engage them in the practices of science, tools are needed for instruction, evaluation, and research around student learning. One of the important skills for undergraduate biology students to master is the selection and creation of appropriate graphs to summarize data they acquire through investigations in their course work and research experiences. Graphing is a complex skill, and there are few, discipline-informed tools available for instructors, students, and researchers to use. Here, we describe the development of a graph rubric informed by literature from the learning sciences, statistics, representations literature, and feedback and use of the rubric by a variety of users. The result is an evidence-based, analytic rubric that consists of categories essential for graph choice and construction: graph mechanics, graph communication, and graph choice. Each category of the rubric can be evaluated at three levels of achievement. Our analysis demonstrates the potential for the rubric to provide formative feedback to students and allow instructors to gauge and guide learning and instruction. We further discuss and identify potentially interesting research targets for science education researchers.

Not-So-CLEVR: learning same-different relations strains feedforward neural networks

The advent of deep learning has recently led to great successes in various engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural network, now approach human accuracy on visual recognition tasks like image classification and face recognition. However, here we will show that feedforward neural networks struggle to learn abstract visual relations that are effortlessly recognized by non-human primates, birds, rodents and even insects. We systematically study the ability of feedforward neural networks to learn to recognize a variety of visual relations and demonstrate that same-different visual relations pose a particular strain on these networks. Networks fail to learn same-different visual relations when stimulus variability makes rote memorization difficult. Further, we show that learning same-different problems becomes trivial for a feedforward network that is fed with perceptually grouped stimuli. This demonstration and the comparative success of biological vision in learning visual relations suggests that feedback mechanisms such as attention, working memory and perceptual grouping may be the key components underlying human-level abstract visual reasoning.

Lateralized pointing does not cause a cognitive bias

Lateralized pointing has been shown to cause not only a shift in visuo-motor midline, but also a shift in non-lateralized spatial attention. Non-lateralized cognitive consequences of lateralized pointing have been reported for local and global visuospatial processing. Here, we evaluate these findings and examine this effect for categorical and coordinate spatial relation processing, for which the attentional processes are thought to be highly similar to local and global visuospatial processing, respectively. Participants performed a commonly used working memory task to assess categorical and coordinate spatial relation processing. Lateralized pointing with either the left or the right hand, to either the left or the right side was introduced as a manipulation, as well as a new control condition without any pointing. Performance on the spatial relation task was measured before and after pointing. The results suggest that non-lateralized consequences of lateralized pointing cannot be generalized to other cognitive tasks relying on attentional processing. Further examination of lateralized pointing is recommended before drawing further conclusions concerning its impact on non-lateralized cognition.

Audio-visual synchrony and spatial attention enhance processing of dynamic visual stimulation independently and in parallel: A frequency-tagging study

The neural processing of a visual stimulus can be facilitated by attending to its position or by a co-occurring auditory tone. Using frequency-tagging, we investigated whether facilitation by spatial attention and audio-visual synchrony rely on similar neural processes. Participants attended to one of two flickering Gabor patches (14.17 and 17 Hz) located in opposite lower visual fields. Gabor patches further "pulsed" (i.e. showed smooth spatial frequency variations) at distinct rates (3.14 and 3.63 Hz). Frequency-modulating an auditory stimulus at the pulse-rate of one of the visual stimuli established audio-visual synchrony. Flicker and pulsed stimulation elicited stimulus-locked rhythmic electrophysiological brain responses that allowed tracking the neural processing of simultaneously presented Gabor patches. These steady-state responses (SSRs) were quantified in the spectral domain to examine visual stimulus processing under conditions of synchronous vs. asynchronous tone presentation and when respective stimulus positions were attended vs. unattended. Strikingly, unique patterns of effects on pulse- and flicker driven SSRs indicated that spatial attention and audiovisual synchrony facilitated early visual processing in parallel and via different cortical processes. We found attention effects to resemble the classical top-down gain effect facilitating both, flicker and pulse-driven SSRs. Audio-visual synchrony, in turn, only amplified synchrony-producing stimulus aspects (i.e. pulse-driven SSRs) possibly highlighting the role of temporally co-occurring sights and sounds in bottom-up multisensory integration.

Visual routines are associated with specific graph interpretations

We argue that people compare values in graphs with a visual routine – attending to data values in an ordered pattern over time. Do these visual routines exist to manage capacity limitations in how many values can be encoded at once, or do they actually affect the relations that are extracted? We measured eye movements while people judged configurations of a two-bar graph based on size only (“[short tall] or [tall short]?”) and contrast only (“[light dark] or [dark light]?”). Participants exhibited visual routines in which they systematically attended to a specific feature (or “anchor point”) in the graph; in the size task, most participants inspected the taller bar first, and in the contrast task, most participants attended to the darker bar first. Participants then judged configurations that varied in both size and contrast (e.g., [short-light tall-dark]); however, only one dimension was task-relevant (varied between subjects). During this orthogonal task, participants overwhelmingly relied on the same anchor point used in the single-dimension version, but only for the task-relevant dimension (e.g., taller bar for the size-relevant task). These results suggest that visual routines are associated with specific graph interpretations. Responses were also faster when task-relevant and task-irrelevant anchor points appeared on the same object (congruent) than on different objects (incongruent). This interference from the task-irrelevant dimension suggests that top-down control may be necessary to extract relevant relations from graphs. The effect of visual routines on graph comprehension has implications for both science, technology, engineering, and mathematics pedagogy and graph design.

Are Categorical Spatial Relations Encoded by Shifting Visual Attention between Objects?

Perceiving not just values, but relations between values, is critical to human cognition. We tested the predictions of a proposed mechanism for processing categorical spatial relations between two objects-the shift account of relation processing-which states that relations such as 'above' or 'below' are extracted by shifting visual attention upward or downward in space. If so, then shifts of attention should improve the representation of spatial relations, compared to a control condition of identity memory. Participants viewed a pair of briefly flashed objects and were then tested on either the relative spatial relation or identity of one of those objects. Using eye tracking to reveal participants' voluntary shifts of attention over time, we found that when initial fixation was on neither object, relational memory showed an absolute advantage for the object following an attention shift, while identity memory showed no advantage for either object. This result is consistent with the shift account of relation processing. When initial fixation began on one of the objects, identity memory strongly benefited this fixated object, while relational memory only showed a relative benefit for objects following an attention shift. This result is also consistent, although not as uniquely, with the shift account of relation processing. Taken together, we suggest that the attention shift account provides a mechanistic explanation for the overall results. This account can potentially serve as the common mechanism underlying both linguistic and perceptual representations of spatial relations.

Visual routines for extracting magnitude relations

Linking relations described in text with relations in visualizations is often difficult. We used eye tracking to measure the optimal way to extract such relations in graphs, college students, and young children (6- and 8-year-olds). Participants compared relational statements ("Are there more blueberries than oranges?") with simple graphs, and two systematic patterns emerged: eye movements that followed the verbal order of the question (inspecting the "blueberry" value first) versus those that followed a left-first bias (regardless of the left value's identity). Question-order patterns led substantially to faster responses and increased in prevalence with age, whereas the left-first pattern led to far slower responses and was the dominant strategy for younger children. We argue that the optimal way to verify a verbally expressed relation's consistency with visualization is for the eyes to mimic the verbal ordering but that this strategy requires executive control and coordination with language.

Working memory for relations among objects

Across many areas of study in cognition, the capacity of working memory (WM) is widely agreed to be roughly three to five items: three to five objects (i.e., bound collections of object features) in the literature on visual WM or three to five role bindings (i.e., objects in specific relational roles) in the literature on memory and reasoning. Three experiments investigated the capacity of observers' WM for the spatial relations among objects in a visual display, and the results suggest that the "items" in WM are neither simply objects nor simply role bindings. The results of Experiment 1 are most consistent with a model that treats an "item" in visual WM as an object, along with the roles of all its relations to one other object. Experiment 2 compared observers' WM for object size with their memory for relative size and provided evidence that observers compute and store objects' relations per se (rather than just absolute size) in WM. Experiment 3 tested and confirmed several more nuanced predictions of the model supported by Experiment 1. Together, these findings suggest that objects are stored in visual WM in pairs (along with all the relations between the objects in a pair) and that, from the perspective of WM, a given object in one pair is not the same "item" as that same object in a different pair.

Visual attention during spatial language comprehension

Spatial terms such as “above”, “in front of”, and “on the left of” are all essential for describing the location of one object relative to another object in everyday communication. Apprehending such spatial relations involves relating linguistic to object representations by means of attention. This requires at least one attentional shift, and models such as the Attentional Vector Sum (AVS) predict the direction of that attention shift, from the sausage to the box for spatial utterances such as “The box is above the sausage”. To the extent that this prediction generalizes to overt gaze shifts, a listener’s visual attention should shift from the sausage to the box. However, listeners tend to rapidly look at referents in their order of mention and even anticipate them based on linguistic cues, a behavior that predicts a converse attentional shift from the box to the sausage. Four eye-tracking experiments assessed the role of overt attention in spatial language comprehension by examining to which extent visual attention is guided by words in the utterance and to which extent it also shifts “against the grain” of the unfolding sentence. The outcome suggests that comprehenders’ visual attention is predominantly guided by their interpretation of the spatial description. Visual shifts against the grain occurred only when comprehenders had some extra time, and their absence did not affect comprehension accuracy. However, the timing of this reverse gaze shift on a trial correlated with that trial’s verification time. Thus, while the timing of these gaze shifts is subtly related to the verification time, their presence is not necessary for successful verification of spatial relations.

The relation between gray matter morphology and divergent thinking in adolescents and young adults

Adolescence and early adulthood are developmental time periods during which creative cognition is highly important for adapting to environmental changes. Divergent thinking, which refers to generating novel and useful solutions to open-ended problems, has often been used as a measure of creative cognition. The first goal of this structural neuroimaging study was to elucidate the relationship between gray matter morphology and performance in the verbal (AUT; alternative uses task) and visuo-spatial (CAT; creative ability test) domain of divergent thinking in adolescents and young adults. The second goal was to test if gray matter morphology is related to brain activity during AUT performance. Neural and behavioral data were combined from a cross-sectional study including 25 adolescents aged 15-17 and 20 young adults aged 25-30. Brain-behavior relationships were assessed without a priori location assumptions and within areas that were activated during an AUT-scanner task. Gray matter volume and cortical thickness were not significantly associated with verbal divergent thinking. However, visuo-spatial divergent thinking (CAT originality and fluency) was positively associated with cortical thickness of the right middle temporal gyrus and left brain areas including the superior frontal gyrus and various occipital, parietal, and temporal areas, independently of age. AUT brain activity was not associated with cortical thickness. The results support an important role of a widespread brain network involved in flexible visuo-spatial divergent thinking, providing evidence for a relation between cortical thickness and visuo-spatial divergent thinking in adolescents and young adults. However, studies including visuo-spatial divergent thinking tasks in the scanner are warranted.

Focusing narrowly or broadly attention when judging categorical and coordinate spatial relations: a MEG study

We measured activity in the dorsal system of the human cortex with magnetoencephalography (MEG) during a matching-to-sample plus cueing paradigm, where participants judged the occurrence of changes in either categorical or coordinate spatial relations (e.g., exchanges of left versus right positions or changes in the relative distances) between images of pairs of animals. The attention window was primed in each trial to be either small or large by using cues that immediately preceded the matching image. In this manner, we could assess the modulatory effects of the scope of attention on the activity of the dorsal system of the human cortex during spatial relations processing. The MEG measurements revealed that large spatial cues yielded greater activations and longer peak latencies in the right inferior parietal lobe for coordinate trials, whereas small cues yielded greater activations and longer peak latencies in the left inferior parietal lobe for categorical trials. The activity in the superior parietal lobe, middle frontal gyrus, and visual cortex, was also modulated by the size of the spatial cues and by the type of spatial relation change. The present results support the theory that the lateralization of each kind of spatial processing hinges on differences in the sizes of regions of space attended to by the two hemispheres. In addition, the present findings are inconsistent with the idea of a right-hemispheric dominance for all kinds of challenging spatial tasks, since response times and accuracy rates showed that the categorical spatial relation task was more difficult than the coordinate task and the cortical activations were overall greater in the left hemisphere than in the right hemisphere.

Perception of average value in multiclass scatterplots

The visual system can make highly efficient aggregate judgements about a set of objects, with speed roughly independent of the number of objects considered. While there is a rich literature on these mechanisms and their ramifications for visual summarization tasks, this prior work rarely considers more complex tasks requiring multiple judgements over long periods of time, and has not considered certain critical aggregation types, such as the localization of the mean value of a set of points. In this paper, we explore these questions using a common visualization task as a case study: relative mean value judgements within multi-class scatterplots. We describe how the perception literature provides a set of expected constraints on the task, and evaluate these predictions with a large-scale perceptual study with crowd-sourced participants. Judgements are no harder when each set contains more points, redundant and conflicting encodings, as well as additional sets, do not strongly affect performance, and judgements are harder when using less salient encodings. These results have concrete ramifications for the design of scatterplots.

The strategic retention of task-relevant objects in visual working memory

The serial and spatially extended nature of many real-world visual tasks suggests the need for control over the content of visual working memory (VWM). We examined the management of VWM in a task that required participants to prioritize individual objects for retention during scene viewing. There were 5 principal findings: (a) Strategic retention of task-relevant objects was effective and was dissociable from the current locus of visual attention; (b) strategic retention was implemented by protection from interference rather than by preferential encoding; (c) this prioritization was flexibly transferred to a new object as task demands changed; (d) no-longer-relevant items were efficiently eliminated from VWM; and (e) despite this level of control, attended and fixated objects were consolidated into VWM regardless of task relevance. These results are consistent with a model of VWM control in which each fixated object is automatically encoded into VWM, replacing a portion of the content in VWM. However, task-relevant objects can be selectively protected from replacement.

Asymmetric coding of categorical spatial relations in both language and vision

Describing certain types of spatial relationships between a pair of objects requires that the objects are assigned different “roles” in the relation, e.g., “A is above B” is different than “B is above A.” This asymmetric representation places one object in the “target” or “figure” role and the other in the “reference” or “ground” role. Here we provide evidence that this asymmetry may be present not just in spatial language, but also in perceptual representations. More specifically, we describe a model of visual spatial relationship judgment where the designation of the target object within such a spatial relationship is guided by the location of the “spotlight” of attention. To demonstrate the existence of this perceptual asymmetry, we cued attention to one object within a pair by briefly previewing it, and showed that participants were faster to verify the depicted relation when that object was the linguistic target. Experiment 1 demonstrated this effect for left-right relations, and Experiment 2 for above-below relations. These results join several other types of demonstrations in suggesting that perceptual representations of some spatial relations may be asymmetrically coded, and further suggest that the location of selective attention may serve as the mechanism that guides this asymmetry.

The head of the table: marking the "front" of an object is tightly linked with selection

Objects in the world do not have a surface that can be objectively labeled the "front." We impose this designation on one surface of an object according to several cues, including which surface is associated with the most task-relevant information or the direction of motion of an object. However, when these cues are competing, weak, or absent, we can also flexibly assign one surface as the front. One possibility is that this assignment is guided by the location of the "spotlight" of selection, where the selected region becomes the front. Here we used an electrophysiological correlate to show a direct temporal link between object structure assignments and the spatial locus of selection. We found that when human participants viewed a shape whose front and back surfaces were ambiguous, seeing a given surface as front was associated with selectively attending to that location. In Experiment 1, this pattern occurred during directed rapid (every 1 s) switches in structural percepts. In Experiment 2, this pattern occurred during spontaneous reversals, from 900 ms before to 600 ms after the reported percept. These results suggest that the distribution of selective attention might guide the organization of object structure.