When Natural Behavior Engages Working Memory

Information Access Effort: The Role of Head Movements for Information Presented at Increasing Eccentricity on Flat Panel and Head-Mounted Displays

OBJECTIVE

This experiment examined performance costs when processing two sources of information positioned at increasing distances using a flat panel display and an augmented reality head-mounted display (AR-HMD).

BACKGROUND

The AR-HMD enables positioning virtual information at various distances in space. However, the proximity compatibility principle suggests that closer separation when two sources of information require mental integration assists performance, whereas increased separation between two sources hurts integration performance more than when a single source requires focused attention. Previous studies have provided inconsistent findings regarding costs associated with increased separation. Few of these experiments have examined separation for both focused and integration tasks, compared vertical and lateral separation, or measured head movements.

METHOD

Three experiments collectively examined these issues using a flat panel display and a virtual display presented with an HMD, where the separation of information varied laterally or vertically during a focused attention (digit reading) task and an information integration (mental subtraction) task.

RESULTS

There was no performance cost for either display when information was increasingly separated. However, head movements mitigated performance costs by preserving accuracy at larger separations without increasing response time.

CONCLUSION

Head movements appear to mitigate performance costs associated with presenting information increasingly far apart on flat panel displays and HMDs. Both eye scanning and head movements appear to be less effortful than expected.

APPLICATION

These findings have important implications for design guidelines regarding the placement of information presented on flat panel displays and, more specifically, HMDs, which can present information 360° around the user.

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Multifaceted consequences of visual distraction during natural behaviour

Visual distraction is a ubiquitous aspect of everyday life. Studying the consequences of distraction during temporally extended tasks, however, is not tractable with traditional methods. Here we developed a virtual reality approach that segments complex behaviour into cognitive subcomponents, including encoding, visual search, working memory usage, and decision-making. Participants copied a model display by selecting objects from a resource pool and placing them into a workspace. By manipulating the distractibility of objects in the resource pool, we discovered interfering effects of distraction across the different cognitive subcomponents. We successfully traced the consequences of distraction all the way from overall task performance to the decision-making processes that gate memory usage. Distraction slowed down behaviour and increased costly body movements. Critically, distraction increased encoding demands, slowed visual search, and decreased reliance on working memory. Our findings illustrate that the effects of visual distraction during natural behaviour can be rather focal but nevertheless have cascading consequences.

Just-in-Time Encoding Into Visual Working Memory Is Contingent Upon Constant Availability of External Information

Humans maintain an intricate balance between storing information in visual working memory (VWM) and just-in-time sampling of the external world, rooted in a trade-off between the cost of maintaining items in VWM versus retrieving information as it is needed. Previous studies have consistently shown that one prerequisite of just-in-time sampling is a high degree of availability of external information, and that introducing a delay before being able to access information led participants to rely less on the external world and more on VWM. However, these studies manipulated availability in such a manner that the cost of sampling was stable and predictable. It is yet unclear whether participants become less reliant on external information when it is more difficult to factor in the cost of sampling that information. In two experiments, participants copied an example layout from the left to the right side of the screen. In Experiment 1, intermittent occlusion of the example layout led participants to attempt to encode more items per inspection than when the layout was constantly available, but this did not consistently result in more correct placements. However, these findings could potentially be explained by inherent differences in how long the example layout could be viewed. Therefore in Experiment 2, the example layout only became available after a gaze-contingent delay, which could be constant or variable. Here, the introduction of any delay led to increased VWM load compared to no delay, although the degree of variability in the delay did not alter behaviour. These results reaffirm that the nature of when we engage VWM is dynamical, and suggest that any disruption to the continuous availability of external information is the main driver of increased VWM usage relative to whether availability is predictable or not.

The voluntary utilization of visual working memory

While a vast amount of research has focused on understanding the capacity limits of visual working memory (VWM), little is known about how VWM resources are employed in unforced behavior and how they correlate with individual capacity constraints. We present a novel, openly available, and easy-to-administer paradigm enabling participants to freely utilize their VWM capacity. Participants had to reconstruct an array of colored squares. In each trial, they were allowed to alternate between the memory array and the reconstruction screen as many times as they wished, each time choosing how many items to reconstruct. This approach allowed us to estimate the number of utilized items, as well as the accuracy of the reconstruction. In addition, VWM capacity was measured using a change detection task. In two experiments, we show that participants tend to under-utilize their VWM resources, performing well below their capacity limits. Surprisingly, while the extent to which participants utilized their VWM was highly reliable, it was uncorrelated with VWM capacity, suggesting that VWM utilization is limited due to strategic considerations rather than capacity limits.

Effects of optogenetic silencing the anterior cingulate cortex in a delayed non-match to trajectory task

Working memory is a fundamental cognitive ability, allowing us to keep information in memory for the time needed to perform a given task. A complex neural circuit fulfills these functions, among which is the anterior cingulate cortex (CG). Functionally and anatomically connected to the medial prefrontal, retrosplenial, midcingulate and hippocampus, as well as motor cortices, CG has been implicated in retrieving appropriate information when needed to select and control appropriate behavior. The role of cingulate cortex in working memory-guided behaviors remains unclear due to the lack of studies reversibly interfering with its activity during specific epochs of working memory. We used eNpHR3.0 to silence cingulate neurons while animals perform a standard delayed non-match to trajectory task, and found that, while not causing an absolute impairment in working memory, silencing cingulate neurons during retrieval decreases the mean performance if compared to silencing during encoding. Such retrieval-associated changes are accompanied by longer delays observed when light is delivered to control animals, when compared to eNpHR3.0+ ones, consistent with an adaptive recruitment of additional cognitive resources.

Task-adaptive changes to the target template in response to distractor context: Separability versus similarity

Theories of attention hypothesize the existence of an attentional template that contains target features in working or long-term memory. It is frequently assumed that the template contains a veridical copy of the target, but recent studies suggest that this is not true when the distractors are linearly separable from the target. In such cases, target representations shift "off-veridical" in response to the distractor context, presumably because doing so is adaptive and increases the representational distinctiveness of targets from distractors. However, some have argued that the shifts may be entirely explained by perceptual biases created by simultaneous color contrast. Here we address this debate and test the more general hypothesis that the target template is adaptively shaped by elements of the distractor context needed to distinguish targets from distractors. We used a two-dimensional target and separately manipulated the linear separability of one dimension (color) and the visual similarity of the other (orientation). We found that target shifting along the linearly separable color dimension was dependent on the similarity of targets-to-distractors along the other dimension. The target representations were consistent with a postexperiment strategy questionnaire in which participants reported using color more when orientation was hard to use, and orientation more when it was easier to use. We conclude that the target template is task-adaptive and exploit features in the distractor context that most predictably distinguish targets from distractors to increase visual search efficiency. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Perceived image size modulates visual memory

Previous studies have demonstrated that visual memory is improved when stimuli are processed by larger cortical regions. For example, a physically large stimulus that recruits larger areas of the retinotopic cortex is better remembered. However, the spatial extent of neural responses in the visual cortex is not only modulated by the retinal size of a stimulus, but also by the perceived size of the stimulus. In this online study, we modulated the perceived size of the visual stimuli using the Ebbinghaus illusion and asked participants to remember the stimuli. The results showed that perceptually larger images were remembered better than perceptually smaller but physically same-sized images. Our finding supports the idea that visual memory is modulated by top-down feedback from higher visual regions to the early visual cortex.

Visual short-term memory-related EEG components in a virtual reality setup

Virtual reality (VR) offers a powerful tool for investigating cognitive processes, as it allows researchers to gauge behaviors and mental states in complex, yet highly controlled, scenarios. The use of VR head-mounted displays in combination with physiological measures such as EEG presents new challenges and raises the question whether established findings also generalize to a VR setup. Here, we used a VR headset to assess the spatial constraints underlying two well-established EEG correlates of visual short-term memory: the amplitude of the contralateral delay activity (CDA) and the lateralization of induced alpha power during memory retention. We tested observers' visual memory in a change detection task with bilateral stimulus arrays of either two or four items while varying the horizontal eccentricity of the memory arrays (4, 9, or 14 degrees of visual angle). The CDA amplitude differed between high and low memory load at the two smaller eccentricities, but not at the largest eccentricity. Neither memory load nor eccentricity significantly influenced the observed alpha lateralization. We further fitted time-resolved spatial filters to decode memory load from the event-related potential as well as from its time-frequency decomposition. Classification performance during the retention interval was above-chance level for both approaches and did not vary significantly across eccentricities. We conclude that commercial VR hardware can be utilized to study the CDA and lateralized alpha power, and we provide caveats for future studies targeting these EEG markers of visual memory in a VR setup.

Keeping track of reality: embedding visual memory in natural behaviour

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings.

Using XR (Extended Reality) for Behavioral, Clinical, and Learning Sciences Requires Updates in Infrastructure and Funding

Extended reality (XR, including augmented and virtual reality) creates a powerful intersection between information technology and cognitive, clinical, and education sciences. XR technology has long captured the public imagination, and its development is the focus of major technology companies. This article demonstrates the potential of XR to (1) deliver behavioral insights, (2) transform clinical treatments, and (3) improve learning and education. However, without appropriate policy, funding, and infrastructural investment, many research institutions will struggle to keep pace with the advances and opportunities of XR. To realize the full potential of XR for basic and translational research, funding should incentivize (1) appropriate training, (2) open software solutions, and (3) collaborations between complementary academic and industry partners. Bolstering the XR research infrastructure with the right investments and incentives is vital for delivering on the potential for transformative discoveries, innovations, and applications.

Measuring ancient technological complexity and its cognitive implications using Petri nets

We implement a method from computer sciences to address a challenge in Paleolithic archaeology: how to infer cognition differences from material culture. Archaeological material culture is linked to cognition, and more complex ancient technologies are assumed to have required complex cognition. We present an application of Petri net analysis to compare Neanderthal tar production technologies and tie the results to cognitive requirements. We applied three complexity metrics, each relying on their own unique definitions of complexity, to the modeled production processes. Based on the results, we propose that Neanderthal technical cognition may have been analogous to that of contemporary modern humans. This method also enables us to distinguish the high-order cognitive functions combining traits like planning, inhibitory control, and learning that were likely required by different ancient technological processes. The Petri net approach can contribute to our understanding of technology and cognitive evolution as it can be used on different materials and technologies, across time and species.

Scaling Palaeolithic tar production processes exponentially increases behavioural complexity

Technological processes, reconstructed from the archaeological record, are used to study the evolution of behaviour and cognition of Neanderthals and early modern humans. In comparisons, technologies that are more complex infer more complex behaviour and cognition. The manufacture of birch bark tar adhesives is regarded as particularly telling and often features in debates about Neanderthal cognition. One method of tar production, the 'condensation technique', demonstrates a pathway for Neanderthals to have discovered birch bark tar. However, to improve on the relatively low yield, and to turn tar into a perennial innovation, this method likely needed to be scaled up. Yet, it is currently unknown how scaling Palaeolithic technological processes influences their complexity. We used Petri net models and the Extended Cyclomatic Metric to measure system complexity of birch tar production with a single and three concurrent condensation assemblies. Our results show that changing the number of concurrent tar production assemblies substantially increases the measured complexity. This has potential implications on the behavioural and cognitive capacities required by Neanderthals, such as an increase in cooperation or inhibition control.

Viewpoint dependence and scene context effects generalize to depth rotated three-dimensional objects

Viewpoint effects on object recognition interact with object-scene consistency effects. While recognition of objects seen from "noncanonical" viewpoints (e.g., a cup from below) is typically impeded compared to processing of objects seen from canonical viewpoints (e.g., the string-side of a guitar), this effect is reduced by meaningful scene context information. In the present study we investigated if these findings established by using photographic images, generalize to strongly noncanonical orientations of three-dimensional (3D) models of objects. Using 3D models allowed us to probe a broad range of viewpoints and empirically establish viewpoints with very strong noncanonical and canonical orientations. In Experiment 1, we presented 3D models of objects from six different viewpoints (0°, 60°, 120°, 180° 240°, 300°) in color (1a) and grayscaled (1b) in a sequential matching task. Viewpoint had a significant effect on accuracy and response times. Based on the viewpoint effect in Experiments 1a and 1b, we could empirically determine the most canonical and noncanonical viewpoints from our set of viewpoints to use in Experiment 2. In Experiment 2, participants again performed a sequential matching task, however now the objects were paired with scene backgrounds which could be either consistent (e.g., a cup in the kitchen) or inconsistent (e.g., a guitar in the bathroom) to the object. Viewpoint interacted significantly with scene consistency in that object recognition was less affected by viewpoint when consistent scene information was provided, compared to inconsistent information. Our results show that scene context supports object recognition even when using extremely noncanonical orientations of depth rotated 3D objects. This supports the important role object-scene processing plays for object constancy especially under conditions of high uncertainty.

Capacity and selection in immersive visual working memory following naturalistic object disappearance

Visual working memory-holding past visual information in mind for upcoming behavior-is commonly studied following the abrupt removal of visual objects from static two-dimensional (2D) displays. In everyday life, visual objects do not typically vanish from the environment in front of us. Rather, visual objects tend to enter working memory following self or object motion: disappearing from view gradually and changing the spatial relation between memoranda and observer. Here, we used virtual reality (VR) to investigate whether two classic findings from visual working memory research-a capacity of around three objects and the reliance on space for object selection-generalize to more naturalistic modes of object disappearance. Our static reference condition mimicked traditional laboratory tasks whereby visual objects were held static in front of the participant and removed from view abruptly. In our critical flow condition, the same visual objects flowed by participants, disappearing from view gradually and behind the observer. We considered visual working memory performance and capacity, as well as space-based mnemonic selection, indexed by directional biases in gaze. Despite vastly distinct modes of object disappearance and altered spatial relations between memoranda and observer, we found comparable capacity and comparable gaze signatures of space-based mnemonic selection. This finding reveals how classic findings from visual working memory research generalize to immersive situations with more naturalistic modes of object disappearance and with dynamic spatial relations between memoranda and observer.

Don't hide the instruction manual: A dynamic trade-off between using internal and external templates during visual search

Visual search is typically studied by requiring participants to memorize a template initially, for which they subsequently search in a crowded display. Search in daily life, however, often involves templates that remain accessible externally, and may therefore be (re)attended for just-in-time encoding or to refresh internal template representations. Here, we show that participants indeed use external templates during search when given the chance. This behavior was observed during both simple and complex search, scaled with task difficulty, and was associated with improved performance. Furthermore, we show that participants used external sampling not only to offload memory, but also as a means of verifying whether the template was remembered correctly at the end of trials. We conclude that the external world may not only provide the challenge (e.g., distractors), but may dynamically ease search. These results argue for extensions of state-of-the-art models of search, because external sampling seems to be used frequently, in at least two ways and is actually beneficial for task performance. Our findings support a model of visual working memory that emphasizes a resource-efficient trade-off between storing and (re)attending external information.

Attentional selection is a sufficient cause for visual working memory interference

Visual attention and visual working memory (VWM) are intertwined processes that allow navigation of the visual world. These systems can compete for highly limited cognitive resources, creating interference effects when both operate in tandem. Performing an attentional task while maintaining a VWM load often leads to a loss of memory information. These losses are seen even with very simple visual search tasks. Previous research has argued that this may be due to the attentional selection process, of choosing the target item out of surrounding nontarget items. Over two experiments, the current study disentangles the roles of search and selection in visual search and their influence on a retained VWM load. Experiment 1 revealed that, when search stimuli were relatively simple, target-absent searches (which did not require attentional selection) did not provoke memory interference, whereas target-present search did. In Experiment 2, the number of potential targets was varied in the search displays. In one condition, participants were required to select any one of the items displayed, requiring an attentional selection but no need to search for a specific item. Importantly, this condition led to memory interference to the same extent as a condition where a single target was presented among nontargets. Together, these results show that the process of attentional selection is a sufficient cause for interference with a concurrently maintained VWM load.

Modeling Oldowan tool transport from a primate perspective

Living nonhuman primates have long served as a referential framework for understanding various aspects of hominin biological and cultural evolution. Comparing the cognitive, social, and ecological contexts of nonhuman primate and hominin tool use has allowed researchers to identify key adaptations relevant to the evolution of hominin behavior. Although the Oldowan is often considered to be a major evolutionary milestone, it has been argued that the Oldowan is rather an extension of behaviors already present in the ape lineage. This is based on the fact that while apes move tools through repeated, unplanned, short-distance transport bouts, they produce material patterning often associated with long-distance transport, planning, and foresight in the Oldowan. Nevertheless, remain fundamental differences in how Oldowan core and flake technology and nonhuman primate tools are used. The goal of the Oldowan hominins is to produce sharp-edged flakes, whereas nonhuman primates use stone tools primarily as percussors. Here, we present an agent-based model that investigates the explanatory power of the ape tool transport model in light of these differences. The model simulates the formation of the Oldowan record under the conditions of an accumulated short-distance transport pattern, as seen in extant chimpanzees. Our results show that while ape tool transport can account for some of the variation observed in the archaeological record, factors related to use-life duration severely limit how far an Oldowan core can be moved through repeated short-distance transport bouts. Thus, the ape tool transport has limitations in its ability to explain patterns in the Oldowan. These results provide a basis for discussing adaptive processes that would have facilitated the development of the Oldowan.

Eye Movements as Proxy for Visual Working Memory Usage: Increased Reliance on the External World in Korsakoff Syndrome

In the assessment of visual working memory, estimating the maximum capacity is currently the gold standard. However, traditional tasks disregard that information generally remains available in the external world. Only when to-be-used information is not readily accessible, memory is taxed. Otherwise, people sample information from the environment as a form of cognitive offloading. To investigate how memory deficits impact the trade-off between sampling externally or storing internally, we compared gaze behaviour of individuals with Korsakoff amnesia (n = 24, age range 47-74 years) and healthy controls (n = 27, age range 40-81 years) on a copy task that provoked different strategies by having information freely accessible (facilitating sampling) or introducing a gaze-contingent waiting time (provoking storing). Indeed, patients sampled more often and longer, compared to controls. When sampling became time-consuming, controls reduced sampling and memorised more. Patients also showed reduced and longer sampling in this condition, suggesting an attempt at memorisation. Importantly, however, patients sampled disproportionately more often than controls, whilst accuracy dropped. This finding suggests that amnesia patients sample frequently and do not fully compensate for increased sampling costs by memorising more at once. In other words, Korsakoff amnesia resulted in a heavy reliance on the world as 'external memory'.

Mountains of memory in a sea of uncertainty: Sampling the external world despite useful information in visual working memory

A large part of research on visual working memory (VWM) has traditionally focused on estimating its maximum capacity. Yet, humans rarely need to load up their VWM maximally during natural behavior, since visual information often remains accessible in the external world. Recent work, using paradigms that take into account the accessibility of information in the outside world, has indeed shown that observers utilize only one or two items in VWM before sampling from the external world again. One straightforward interpretation of this finding is that, in daily behavior, much fewer items are memorized than the typically reported capacity limits. Here, we first investigate whether this lower reliance on VWM when information is externally accessible might instead reflect resampling before VWM is actually depleted. To this aim we devised an online task, in which participants copied a model (six items in a 4x4 grid; always accessible) in an adjacent empty 4x4 grid. A key aspect of our paradigm is that we (unpredictably) interrupted participants just before inspection of the model with a 2-alternative-forced-choice (2-AFC) question, probing their VWM content. Critically, we observed above-chance performance on probes appearing just before model inspection. This finding shows that the external world was resampled, despite VWM still containing relevant information. We then asked whether increasing the cost of sampling causes participants to load up more information in VWM or, alternatively, to squeeze out more information from VWM (at the cost of making more errors). To manipulate the cost of resampling, we made it more difficult (specifically, more time-consuming) to access the model. We show that with increased cost of accessing the model (which lead to fewer, but longer model inspections), participants could place more items correctly immediately after sampling, and they kept attempting to place items for longer after their first error. These findings demonstrate that participants both encoded more information in VWM and made attempts to squeeze out more information from VWM when sampling became more costly. We argue that human observers constantly evaluate how certain they are of their VWM contents, and only use that VWM content of which their certainty exceeds a context-dependent "action threshold". This threshold, in turn, depends on the trade-off between the cost of resampling and the benefits of making an action. We argue that considering the interplay between the available VWM contents and a context-dependent action threshold, is key for reconciling the traditional VWM literature with VWM use in our day-to-day behavior.

The influence of music liking on episodic memory for rich spatiotemporal contexts

It is thought that the presence of music influences episodic memory encoding. However, no studies have isolated the influence of music liking - the hedonic value listeners attribute to a musical stimulus - from that of the core affect induced by the presence of that music. In an online survey, participants rated musical excerpts in terms of how much they liked them, as well as in terms of felt valence, felt arousal and familiarity. These ratings were then used to inform the stimuli presented in an online episodic memory task which, across different scenarios, involved dragging cued objects to cued locations and then recalling details of what was moved, where they were moved to and the order of movements made. Our results showed an influence of liking and music-reward sensitivity on memory for what was moved, as well as a detrimental effect of arousing musical stimuli on memory for un-cued scenario details. Taken together, our study showcases the importance of episodic memory paradigms that involve rich spatiotemporal contexts and provides insights into how different aspects of episodic memory may be influenced by the presence of music.

Attention in flux

Selective attention comprises essential infrastructural functions supporting cognition-anticipating, prioritizing, selecting, routing, integrating, and preparing signals to guide adaptive behavior. Most studies have examined its consequences, systems, and mechanisms in a static way, but attention is at the confluence of multiple sources of flux. The world advances, we operate within it, our minds change, and all resulting signals progress through multiple pathways within the dynamic networks of our brains. Our aim in this review is to raise awareness of and interest in three important facets of how timing impacts our understanding of attention. These include the challenges posed to attention by the timing of neural processing and psychological functions, the opportunities conferred to attention by various temporal structures in the environment, and how tracking the time courses of neural and behavioral modulations with continuous measures yields surprising insights into the workings and principles of attention.

Priming of probabilistic attentional templates

Attentional priming has a dominating influence on vision, speeding visual search, releasing items from crowding, reducing masking effects, and during free-choice, primed targets are chosen over unprimed ones. Many accounts postulate that templates stored in working memory control what we attend to and mediate the priming. But what is the nature of these templates (or representations)? Analyses of real-world visual scenes suggest that tuning templates to exact color or luminance values would be impractical since those can vary greatly because of changes in environmental circumstances and perceptual interpretation. Tuning templates to a range of the most probable values would be more efficient. Recent evidence does indeed suggest that the visual system represents such probability, gradually encoding statistical variation in the environment through repeated exposure to input statistics. This is consistent with evidence from neurophysiology and theoretical neuroscience as well as computational evidence of probabilistic representations in visual perception. I argue that such probabilistic representations are the unit of attentional priming and that priming of, say, a repeated single-color value simply involves priming of a distribution with no variance. This "priming of probability" view can be modelled within a Bayesian framework where priming provides contextual priors. Priming can therefore be thought of as learning of the underlying probability density function of the target or distractor sets in a given continuous task.

vexptoolbox: A software toolbox for human behavior studies using the Vizard virtual reality platform

Virtual reality (VR) is a powerful tool for researchers due to its potential to study dynamic human behavior in highly naturalistic environments while retaining full control over the presented stimuli. Due to advancements in consumer hardware, VR devices are now very affordable and have also started to include technologies such as eye tracking, further extending potential research applications. Rendering engines such as Unity, Unreal, or Vizard now enable researchers to easily create complex VR environments. However, implementing the experimental design can still pose a challenge, and these packages do not provide out-of-the-box support for trial-based behavioral experiments. Here, we present a Python toolbox, designed to facilitate common tasks when developing experiments using the Vizard VR platform. It includes functionality for common tasks like creating, randomizing, and presenting trial-based experimental designs or saving results to standardized file formats. Moreover, the toolbox greatly simplifies continuous recording of eye and body movements using any hardware supported in Vizard. We further implement and describe a simple goal-directed reaching task in VR and show sample data recorded from five volunteers. The toolbox, example code, and data are all available on GitHub under an open-source license. We hope that our toolbox can simplify VR experiment development, reduce code duplication, and aid reproducibility and open-science efforts.

Turning Attention Inside Out: How Working Memory Serves Behavior

Flexible behavior requires guidance not only by sensations that are available immediately but also by relevant mental contents carried forward through working memory. Therefore, selective-attention functions that modulate the contents of working memory to guide behavior (inside-out) are just as important as those operating on sensory signals to generate internal contents (outside-in). We review the burgeoning literature on selective attention in the inside-out direction and underscore its functional, flexible, and future-focused nature. We discuss in turn the purpose (why), targets (what), sources (when), and mechanisms (how) of selective attention inside working memory, using visual working memory as a model. We show how the study of internal selective attention brings new insights concerning the core cognitive processes of attention and working memory and how considering selective attention and working memory together paves the way for a rich and integrated understanding of how mind serves behavior.

Active visual search in naturalistic environments reflects individual differences in classic visual search performance

Visual search is a ubiquitous activity in real-world environments. Yet, traditionally, visual search is investigated in tightly controlled paradigms, where head-restricted participants locate a minimalistic target in a cluttered array that is presented on a computer screen. Do traditional visual search tasks predict performance in naturalistic settings, where participants actively explore complex, real-world scenes? Here, we leverage advances in virtual reality technology to test the degree to which classic and naturalistic search are limited by a common factor, set size, and the degree to which individual differences in classic search behavior predict naturalistic search behavior in a large sample of individuals (N = 75). In a naturalistic search task, participants looked for an object within their environment via a combination of head-turns and eye-movements using a head-mounted display. Then, in a classic search task, participants searched for a target within a simple array of colored letters using only eye-movements. In each task, we found that participants' search performance was impacted by increases in set size-the number of items in the visual display. Critically, we observed that participants' efficiency in classic search tasks-the degree to which set size slowed performance-indeed predicted efficiency in real-world scenes. These results demonstrate that classic, computer-based visual search tasks are excellent models of active, real-world search behavior.

VR for Cognition and Memory

This chapter will provide a review of research into human cognition through the lens of VR-based paradigms for studying memory. Emphasis is placed on why VR increases the ecological validity of memory research and the implications of such enhancements.

Opening Questions in Visual Working Memory

In this reflective piece on visual working memory, I depart from the laboriously honed skills of writing a review. Instead of integrating approaches, synthesizing evidence, and building a cohesive perspective, I scratch my head and share niggles and puzzlements. I expose where my scholarship and understanding are stumped by findings and standard views in the literature.

Updating, Fast and Slow: Items, but Not Item-Context Bindings, are Quickly Updated Into Working Memory as Part of Response Selection

It is commonly held that attending to items facilitates their encoding into working memory (WM). This implies that the content of WM is updated with new input as a consequence of directing attention to it. On the other hand, abundant research shows that WM updating is rather slow and effortful, suggesting that shielding WM representation against incoming input, rather than its updating, is the default. To resolve this discrepancy, we suggest that while updating item-to-context associations is costly, updating a single item is fast and is automatically carried out as part of directing attention to items, for example as part of response selection. Participants performed a choice-RT task, in which stimuli appeared within frames, and needed to update their WM with the most recent red item that appeared in each frame. The need for updating was manipulated, so that some trials required updating and others did not. Experiment 1 (N = 25) showed that updating was slower than not updating with a set-size of two items, that required item-context binding, but faster when the set-size only involved one item. Experiment 2 (N = 28) replicated this finding. Experiment 3 (N = 20) showed that the slower no-update RTs are due to the removal of erroneously updated information. In contrast to previous findings, these results suggest that updating can be effortless and obligatory.

Auxiliary Scene-Context Information Provided by Anchor Objects Guides Attention and Locomotion in Natural Search Behavior

Successful adaptive behavior requires efficient attentional and locomotive systems. Previous research has thoroughly investigated how we achieve this efficiency during natural behavior by exploiting prior knowledge related to targets of our actions (e.g., attending to metallic targets when looking for a pot) and to the environmental context (e.g., looking for the pot in the kitchen). Less is known about whether and how individual nontarget components of the environment support natural behavior. In our immersive virtual reality task, 24 adult participants searched for objects in naturalistic scenes in which we manipulated the presence and arrangement of large, static objects that anchor predictions about targets (e.g., the sink provides a prediction for the location of the soap). Our results show that gaze and body movements in this naturalistic setting are strongly guided by these anchors. These findings demonstrate that objects auxiliary to the target are incorporated into the representations guiding attention and locomotion.

Multiple spatial frames for immersive working memory

As we move around, relevant information that disappears from sight can still be held in working memory to serve upcoming behaviour. How we maintain and select visual information as we move through the environment remains poorly understood because most laboratory tasks of working memory rely on removing visual material while participants remain still. We used virtual reality to study visual working memory following self-movement in immersive environments. Directional biases in gaze revealed the recruitment of more than one spatial frame for maintaining and selecting memoranda following self-movement. The findings bring the important realization that multiple spatial frames support working memory in natural behaviour. The results also illustrate how virtual reality can be a critical experimental tool to characterize this core memory system.

Gaze dynamics are sensitive to target orienting for working memory encoding in virtual reality

Numerous studies have demonstrated that visuospatial attention is a requirement for successful working memory encoding. It is unknown, however, whether this established relationship manifests in consistent gaze dynamics as people orient their visuospatial attention toward an encoding target when searching for information in naturalistic environments. To test this hypothesis, participants' eye movements were recorded while they searched for and encoded objects in a virtual apartment (Experiment 1). We decomposed gaze into 61 features that capture gaze dynamics and a trained sliding window logistic regression model that has potential for use in real-time systems to predict when participants found target objects for working memory encoding. A model trained on group data successfully predicted when people oriented to a target for encoding for the trained task (Experiment 1) and for a novel task (Experiment 2), where a new set of participants found objects and encoded an associated nonword in a cluttered virtual kitchen. Six of these features were predictive of target orienting for encoding, even during the novel task, including decreased distances between subsequent fixation/saccade events, increased fixation probabilities, and slower saccade decelerations before encoding. This suggests that as people orient toward a target to encode new information at the end of search, they decrease task-irrelevant, exploratory sampling behaviors. This behavior was common across the two studies. Together, this research demonstrates how gaze dynamics can be used to capture target orienting for working memory encoding and has implications for real-world use in technology and special populations.

Dynamic in-flight shifts of working memory resources across saccades

Little is known about how memory resources are allocated in natural vision across sequential eye movements and fixations, as people actively extract information from the visual environment. Here, we used gaze-contingent eye tracking to examine how such resources are dynamically reallocated from old to new information entering working memory. As participants looked sequentially at items, we interrupted the process at different times by extinguishing the display as a saccade was initiated. After a brief interval, participants were probed on one of the items that had been presented. Paradoxically, across all experiments, the final (unfixated) saccade target was recalled more precisely when more items had previously been fixated, that is, with longer rather than shorter saccade sequences. This result is difficult to explain on current models of working memory because recall error, even for the final item, is typically higher as memory load increases. The findings could however be accounted for by a model that describes how resources are dynamically reallocated on a moment-by-moment basis. During each saccade, the target is encoded by consuming a proportion of currently available resources from a limited working memory, as well as by reallocating resources away from previously encoded items. These findings reveal how working memory resources are shifted across memoranda in active vision. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Elements of exogenous attentional cueing preserved during optokinetic motion of the visual scene

Navigating through our environment raises challenges for perception by generating salient background visual motion, and eliciting prominent eye movements to stabilise the retinal image. It remains unclear if exogenous spatial attentional orienting is possible during background motion and the eye movements it causes, and whether this compromises the underlying neural processing. To test this, we combined exogenous orienting, visual scene motion, and EEG. 26 participants viewed a background of moving black and grey bars (optokinetic stimulation). We tested for effects of non-spatially predictive peripheral cueing on visual motion discrimination of a target dot, presented either at the same (valid) or opposite (invalid) location as the preceding cue. Valid cueing decreased reaction times not only when participants kept their gaze fixed on a central point (fixation blocks), but even when there was no fixation point, so that participants performed intensive, repetitive tracking eye movements (eye movements blocks). Overall, manual response reaction times were slower during eye movements. Cueing also produced reliable effects on neural activity on either block, including within the first 120 milliseconds of neural processing of the target. The key pattern with larger ERP amplitudes on invalid versus valid trials showed that the neural substrate of exogenous cueing was highly similar during eye movements or fixation. Exogenous peripheral cueing and its neural correlates are robust against distraction from the moving visual scene, important for perceptual cognition during navigation.

Working Memory: How Much Is It Used in Natural Behavior?

A new study suggests that visual working memory usage is interestingly low during a more naturalistic virtual reality paradigm, compared to capacity estimates from traditional lab studies. This raises new questions about the use of working memory in everyday tasks.

Adapting the coordination of eyes and head to differences in task and environment during fully-mobile visual exploration

How are eyes and head adapted to meet the demands of visual exploration in different tasks and environments? In two studies, we measured the horizontal movements of the eyes (using mobile eye tracking in Studies 1 and 2) and the head (using inertial sensors in Study 2) while participants completed a walking task and a search and retrieval task in a large, outdoor environment. We found that the spread of visual exploration was greater while searching compared with walking, and this was primarily driven by increased movement of the head as opposed to the eyes. The contributions of the head to gaze shifts of different eccentricities was greater when searching compared to when walking. Findings are discussed with respect to understanding visual exploration as a motor action with multiple degrees of freedom.

The Cognitive Science of Technology

Technology is central to human life but hard to define and study. This review synthesizes advances in fields from anthropology to evolutionary biology and neuroscience to propose an interdisciplinary cognitive science of technology. The foundation of this effort is an evolutionarily motivated definition of technology that highlights three key features: material production, social collaboration, and cultural reproduction. This broad scope respects the complexity of the subject but poses a challenge for theoretical unification. Addressing this challenge requires a comparative approach to reduce the diversity of real-world technological cognition to a smaller number of recurring processes and relationships. To this end, a synthetic perceptual-motor hypothesis (PMH) for the evolutionary-developmental-cultural construction of technological cognition is advanced as an initial target for investigation.

How context changes the neural basis of perception and language

Cognitive processes-from basic sensory analysis to language understanding-are typically contextualized. While the importance of considering context for understanding cognition has long been recognized in psychology and philosophy, it has not yet had much impact on cognitive neuroscience research, where cognition is often studied in decontextualized paradigms. Here, we present examples of recent studies showing that context changes the neural basis of diverse cognitive processes, including perception, attention, memory, and language. Within the domains of perception and language, we review neuroimaging results showing that context interacts with stimulus processing, changes activity in classical perception and language regions, and recruits additional brain regions that contribute crucially to naturalistic perception and language. We discuss how contextualized cognitive neuroscience will allow for discovering new principles of the mind and brain.

Keeping it real: Looking beyond capacity limits in visual cognition

Research within visual cognition has made tremendous strides in uncovering the basic operating characteristics of the visual system by reducing the complexity of natural vision to artificial but well-controlled experimental tasks and stimuli. This reductionist approach has for example been used to assess the basic limitations of visual attention, visual working memory (VWM) capacity, and the fidelity of visual long-term memory (VLTM). The assessment of these limits is usually made in a pure sense, irrespective of goals, actions, and priors. While it is important to map out the bottlenecks our visual system faces, we focus here on selected examples of how such limitations can be overcome. Recent findings suggest that during more natural tasks, capacity may be higher than reductionist research suggests and that separable systems subserve different actions, such as reaching and looking, which might provide important insights about how pure attentional or memory limitations could be circumvented. We also review evidence suggesting that the closer we get to naturalistic behavior, the more we encounter implicit learning mechanisms that operate "for free" and "on the fly." These mechanisms provide a surprisingly rich visual experience, which can support capacity-limited systems. We speculate whether natural tasks may yield different estimates of the limitations of VWM, VLTM, and attention, and propose that capacity measurements should also pass the real-world test within naturalistic frameworks. Our review highlights various approaches for this and suggests that our understanding of visual cognition will benefit from incorporating the complexities of real-world cognition in experimental approaches.