The Treachery of Images: How Realism Influences Brain and Behavior

The medium modulates the medusa effect: Perceived mind in analogue and digital images

We effortlessly attribute mental states to other people. We also attribute minds to people depicted in pictures, albeit at a reduced strength. Intriguingly, this reduction in intensity continues for images of people within a photograph itself-a phenomenon known as the Medusa effect. The present study replicates the Medusa effect for images shown digitally and on paper. Crucially, we demonstrate that we can reduce the magnitude of the Medusa effect by depicting people digitally within a computer screen (e.g., as if one were interacting with a person on a Zoom call). As well as modulating the quantity of the Medusa effect, changes in pictorial medium can affect the quality of the perceived mind. Specifically, the dimension of Experience-what a depicted person can feel-reflected participants' observations that they could interact with an onscreen person embedded in a digital image. This combination of a robust Medusa effect and the ability to control it both quantitatively and qualitatively opens many avenues for its future application, such as manipulating and measuring mind in immersive media.

Haptic and visuo-haptic impairments for object recognition in children with autism spectrum disorder: focus on the sensory and multisensory processing dysfunctions

Dysfunctions in sensory processing are widely described in individuals with autism spectrum disorder (ASD), although little is known about the developmental course and the impact of these difficulties on the learning processes during the preschool and school ages of ASD children. Specifically, as regards the interplay between visual and haptic information in ASD during developmental age, knowledge is very scarce and controversial. In this study, we investigated unimodal (visual and haptic) and cross-modal (visuo-haptic) processing skills aimed at object recognition through a behavioural paradigm already used in children with typical development (TD), with cerebral palsy and with peripheral visual impairments. Thirty-five children with ASD (age range: 5-11 years) and thirty-five age-matched and gender-matched typically developing peers were recruited. The procedure required participants to perform an object-recognition task relying on only the visual modality (black-and-white photographs), only the haptic modality (manipulation of real objects) and visuo-haptic transfer of these two types of information. Results are consistent with the idea that visuo-haptic transfer may be significantly worse in ASD children than in TD peers, leading to significant impairment in multisensory interactions for object recognition facilitation. Furthermore, ASD children tended to show a specific deficit in haptic information processing, while a similar trend of maturation of visual modality between the two groups is reported. This study adds to the current literature by suggesting that ASD differences in multisensory processes also regard visuo-haptic abilities necessary to identify and recognise objects of daily life.

The importance of multisensory-motor learning on subsequent visual recognition

Speed of visual object recognition is facilitated after active manual exploration of objects relative to passive visual processing alone. Manual exploration allows viewers to select important information about object structure that may facilitate recognition. Viewpoints where the objects' axis of elongation is perpendicular or parallel to the line of sight are selected more during exploration, recognized faster than other viewpoints, and afford the most information about structure when object movement is controlled by the viewer. Prior work used virtual object exploration in active and passive viewing conditions, limiting multisensory structural object information. Adding multisensory information to encoding may change accuracy of overall recognition, viewpoint selection, and viewpoint recognition. We tested whether the known active advantage for object recognition would change when real objects were studied, affording visual and haptic information. Participants interacted with 3D novel objects during manual exploration or passive viewing of another's object interactions. Object recognition was tested using several viewpoints of rendered objects. We found that manually explored objects were recognized more accurately than objects studied through passive exploration and that recognition of viewpoints differed from previous work.

Orthogonal neural representations support perceptual judgements of natural stimuli

In natural behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on simple backgrounds. Natural viewing, however, carries a set of challenges that are inaccessible using artificial stimuli, including neural responses to background objects that are task-irrelevant. An emerging body of evidence suggests that the visual abilities of humans and animals can be modeled through the linear decoding of task-relevant information from visual cortex. This idea suggests the hypothesis that irrelevant features of a natural scene should impair performance on a visual task only if their neural representations intrude on the linear readout of the task relevant feature, as would occur if the representations of task-relevant and irrelevant features are not orthogonal in the underlying neural population. We tested this hypothesis using human psychophysics and monkey neurophysiology, in response to parametrically variable naturalistic stimuli. We demonstrate that 1) the neural representation of one feature (the position of a central object) in visual area V4 is orthogonal to those of several background features, 2) the ability of human observers to precisely judge object position was largely unaffected by task-irrelevant variation in those background features, and 3) many features of the object and the background are orthogonally represented by V4 neural responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of objects and features despite the tremendous richness of natural visual scenes.

Taking consciousness for real: Increasing the ecological validity of the study of conscious vs. unconscious processes

The study of consciousness has developed well-controlled, rigorous methods for manipulating and measuring consciousness. Yet, in the process, experimental paradigms grew farther away from everyday conscious and unconscious processes, which raises the concern of ecological validity. In this review, we suggest that the field can benefit from adopting a more ecological approach, akin to other fields of cognitive science. There, this approach challenged some existing hypotheses, yielded stronger effects, and enabled new research questions. We argue that such a move is critical for studying consciousness, where experimental paradigms tend to be artificial and small effect sizes are relatively prevalent. We identify three paths for doing so-changing the stimuli and experimental settings, changing the measures, and changing the research questions themselves-and review works that have already started implementing such approaches. While acknowledging the inherent challenges, we call for increasing ecological validity in consciousness studies.

Electrophysiological correlates of face and object perception: A comparative analysis of 2D laboratory and virtual reality conditions

Human face perception is a specialized visual process with inherent social significance. The neural mechanisms reflecting this intricate cognitive process have evolved in spatially complex and emotionally rich environments. Previous research using VR to transfer an established face perception paradigm to realistic conditions has shown that the functional properties of face-sensitive neural correlates typically observed in the laboratory are attenuated outside the original modality. The present study builds on these results by comparing the perception of persons and objects under conventional laboratory (PC) and realistic conditions in VR. Adhering to established paradigms, the PC- and VR modalities both featured images of persons and cars alongside standard control images. To investigate the individual stages of realistic face processing, response times, the typical face-sensitive N170 component, and relevant subsequent components (L1, L2; pre-, post-response) were analyzed within and between modalities. The between-modality comparison of response times and component latencies revealed generally faster processing under realistic conditions. However, the obtained N170 latency and amplitude differences showed reduced discriminative capacity under realistic conditions during this early stage. These findings suggest that the effects commonly observed in the lab are specific to monitor-based presentations. Analyses of later and response-locked components showed specific neural mechanisms for identification and evaluation are employed when perceiving the stimuli under realistic conditions, reflected in discernible amplitude differences in response to faces and objects beyond the basic perceptual features. Conversely, the results do not provide evidence for comparable stimulus-specific perceptual processing pathways when viewing pictures of the stimuli under conventional laboratory conditions.

Neural Mechanisms Determining the Duration of Task-free, Self-paced Visual Perception

Humans spend hours each day spontaneously engaging with visual content, free from specific tasks and at their own pace. Currently, the brain mechanisms determining the duration of self-paced perceptual behavior remain largely unknown. Here, participants viewed naturalistic images under task-free settings and self-paced each image's viewing duration while undergoing EEG and pupillometry recordings. Across two independent data sets, we observed large inter- and intra-individual variability in viewing duration. However, beyond an image's presentation order and category, specific image content had no consistent effects on spontaneous viewing duration across participants. Overall, longer viewing durations were associated with sustained enhanced posterior positivity and anterior negativity in the ERPs. Individual-specific variations in the spontaneous viewing duration were consistently correlated with evoked EEG activity amplitudes and pupil size changes. By contrast, presentation order was selectively correlated with baseline alpha power and baseline pupil size. Critically, spontaneous viewing duration was strongly predicted by the temporal stability in neural activity patterns starting as early as 350 msec after image onset, suggesting that early neural stability is a key predictor for sustained perceptual engagement. Interestingly, neither bottom-up nor top-down predictions about image category influenced spontaneous viewing duration. Overall, these results suggest that individual-specific factors can influence perceptual processing at a surprisingly early time point and influence the multifaceted ebb and flow of spontaneous human perceptual behavior in naturalistic settings.

Naturalistic Object Representations Depend on Distance and Size Cues

Egocentric distance and real-world size are important cues for object perception and action. Nevertheless, most studies of human vision rely on two-dimensional pictorial stimuli that convey ambiguous distance and size information. Here, we use fMRI to test whether pictures are represented differently in the human brain from real, tangible objects that convey unambiguous distance and size cues. Participants directly viewed stimuli in two display formats (real objects and matched printed pictures of those objects) presented at different egocentric distances (near and far). We measured the effects of format and distance on fMRI response amplitudes and response patterns. We found that fMRI response amplitudes in the lateral occipital and posterior parietal cortices were stronger overall for real objects than for pictures. In these areas and many others, including regions involved in action guidance, responses to real objects were stronger for near vs. far stimuli, whereas distance had little effect on responses to pictures-suggesting that distance determines relevance to action for real objects, but not for pictures. Although stimulus distance especially influenced response patterns in dorsal areas that operate in the service of visually guided action, distance also modulated representations in ventral cortex, where object responses are thought to remain invariant across contextual changes. We observed object size representations for both stimulus formats in ventral cortex but predominantly only for real objects in dorsal cortex. Together, these results demonstrate that whether brain responses reflect physical object characteristics depends on whether the experimental stimuli convey unambiguous information about those characteristics.

Significance Statement

Classic frameworks of vision attribute perception of inherent object characteristics, such as size, to the ventral visual pathway, and processing of spatial characteristics relevant to action, such as distance, to the dorsal visual pathway. However, these frameworks are based on studies that used projected images of objects whose actual size and distance from the observer were ambiguous. Here, we find that when object size and distance information in the stimulus is less ambiguous, these characteristics are widely represented in both visual pathways. Our results provide valuable new insights into the brain representations of objects and their various physical attributes in the context of naturalistic vision.

Developmental trajectories of picture-based object representations during the first year of life

Experience with an object's photograph changes 9-month-olds' preference for the referent object, confirming they can represent objects from pictures. However, picture-based representations appear weaker than object-based representations. The current study's first objective was to investigate age differences in object recognition memory after familiarization with objects' pictures. The second objective was to test whether age differences in object permanence sensitivity with picture-based representations match those found with object-based representations, whereby 7-month-olds search more for familiar hidden objects but 11-month-olds search more for novel ones. Six- and 11-month-olds were familiarized with an object's photo and tested on their representation of the real object by comparing their reaching for it versus a novel object. Objects were visible under conditions testing recognition memory and hidden under conditions testing object permanence. Like 9-month-olds, 6- and 11-month-olds preferred novelty with visible objects, showing early object recognition after picture familiarization, as well as developmental continuity. Unlike 9-month-olds, who switched to preferring familiarity with hidden objects, 6- and 11-month-olds switched to null preference. This pattern fails to match 7- and 11-month-olds' hidden-object preferences after familiarization with real objects, revealing discontinuity in sensitivity to object permanence after picture familiarization, and suggesting that picture-based representations are weaker than object-based ones.

Internal representations of the canonical real-world distance of objects

In the real world, every object has its canonical distance from observers. For example, airplanes are usually far away from us, whereas eyeglasses are close to us. Do we have an internal representation of the canonical real-world distance of objects in our cognitive system? If we do, does the canonical distance influence the perceived size of an object? Here, we conducted two experiments to address these questions. In Experiment 1, we first asked participants to rate the canonical distance of objects. Participants gave consistent ratings to each object. Then, pairs of object images were presented one by one in a trial, and participants were asked to rate the distance of the second object (i.e., a priming paradigm). We found that the rating of the perceived distance of the target object was modulated by the canonical real-world distance of the prime. In Experiment 2, participants were asked to judge the perceived size of canonically near or far objects that were presented at the converging end (i.e., far location) or the opening end (i.e., near location) of a background image with converging lines. We found that regardless of the presentation location, participants perceived the canonically near object as smaller than the canonically far object even though their retinal and real-world sizes were matched. In all, our results suggest that we have an internal representation of the canonical real-world distance of objects, which affects the perceived distance of subsequent objects and the perceived size of the objects themselves.

Comparing eye-hand coordination between controller-mediated virtual reality, and a real-world object interaction task

Virtual reality (VR) technology has advanced significantly in recent years, with many potential applications. However, it is unclear how well VR simulations mimic real-world experiences, particularly in terms of eye-hand coordination. This study compares eye-hand coordination from a previously validated real-world object interaction task to the same task re-created in controller-mediated VR. We recorded eye and body movements and segmented participants' gaze data using the movement data. In the real-world condition, participants wore a head-mounted eye tracker and motion capture markers and moved a pasta box into and out of a set of shelves. In the VR condition, participants wore a VR headset and moved a virtual box using handheld controllers. Unsurprisingly, VR participants took longer to complete the task. Before picking up or dropping off the box, participants in the real world visually fixated the box about half a second before their hand arrived at the area of action. This 500-ms minimum fixation time before the hand arrived was preserved in VR. Real-world participants disengaged their eyes from the box almost immediately after their hand initiated or terminated the interaction, but VR participants stayed fixated on the box for much longer after it was picked up or dropped off. We speculate that the limited haptic feedback during object interactions in VR forces users to maintain visual fixation on objects longer than in the real world, altering eye-hand coordination. These findings suggest that current VR technology does not replicate real-world experience in terms of eye-hand coordination.

How to construct liquid-crystal spectacles to control vision of real-world objects and environments

A major challenge in studying naturalistic vision lies in controlling stimulus and scene viewing time. This is especially the case for studies using real-world objects as stimuli (rather than computerized images) because real objects cannot be "onset" and "offset" in the same way that images can be. Since the late 1980s, one solution to this problem has been to have the observer wear electro-optic spectacles with computer-controlled liquid-crystal lenses that switch between transparent ("open") and translucent ("closed") states. Unfortunately, the commercially available glasses (PLATO Visual Occlusion Spectacles) command a high price tag, the hardware is fragile, and the glasses cannot be customized. This led us to explore how to manufacture liquid-crystal occlusion glasses in our own laboratory. Here, we share the products of our work by providing step-by-step instructions for researchers to design, build, operate, and test liquid-crystal glasses for use in experimental contexts. The glasses can be assembled with minimal technical knowledge using readily available components, and they can be customized for different populations and applications. The glasses are robust, and they can be produced at a fraction of the cost of commercial alternatives. Tests of reliability and temporal accuracy show that the performance of our laboratory prototype was comparable to that of the PLATO glasses. We discuss the results of our work with respect to implications for promoting rigor and reproducibility, potential use cases, comparisons with other liquid-crystal shutter glasses, and how users can find information regarding future updates and developments.

Temporal features of size constancy for perception and action in a real-world setting: A combined EEG-kinematics study

A stable representation of object size, in spite of continuous variations in retinal input due to changes in viewing distance, is critical for perceiving and acting in a real 3D world. In fact, our perceptual and visuo-motor systems exhibit size and grip constancies in order to compensate for the natural shrinkage of the retinal image with increased distance. The neural basis of this size-distance scaling remains largely unknown, although multiple lines of evidence suggest that size-constancy operations might take place remarkably early, already at the level of the primary visual cortex. In this study, we examined for the first time the temporal dynamics of size constancy during perception and action by using a combined measurement of event-related potentials (ERPs) and kinematics. Participants were asked to maintain their gaze steadily on a fixation point and perform either a manual estimation or a grasping task towards disks of different sizes placed at different distances. Importantly, the physical size of the target was scaled with distance to yield a constant retinal angle. Meanwhile, we recorded EEG data from 64 scalp electrodes and hand movements with a motion capture system. We focused on the first positive-going visual evoked component peaking at approximately 90 ms after stimulus onset. We found earlier latencies and greater amplitudes in response to bigger than smaller disks of matched retinal size, regardless of the task. In line with the ERP results, manual estimates and peak grip apertures were larger for the bigger targets. We also found task-related differences at later stages of processing from a cluster of central electrodes, whereby the mean amplitude of the P2 component was greater for manual estimation than grasping. Taken together, these findings provide novel evidence that size constancy for real objects at real distances occurs at the earliest cortical stages and that early visual processing does not change as a function of task demands.

Neural networks need real-world behavior

Bowers et al. propose to use controlled behavioral experiments when evaluating deep neural networks as models of biological vision. We agree with the sentiment and draw parallels to the notion that "neuroscience needs behavior." As a promising path forward, we suggest complementing image recognition tasks with increasingly realistic and well-controlled task environments that engage real-world object recognition behavior.

Human Neuroimaging Reveals Differences in Activation and Connectivity between Real and Pantomimed Tool Use

Because the sophistication of tool use is vastly enhanced in humans compared with other species, a rich understanding of its neural substrates requires neuroscientific experiments in humans. Although functional magnetic resonance imaging (fMRI) has enabled many studies of tool-related neural processing, surprisingly few studies have examined real tool use. Rather, because of the many constraints of fMRI, past research has typically used proxies such as pantomiming despite neuropsychological dissociations between pantomimed and real tool use. We compared univariate activation levels, multivariate activation patterns, and functional connectivity when participants used real tools (a plastic knife or fork) to act on a target object (scoring or poking a piece of putty) or pantomimed the same actions with similar movements and timing. During the Execute phase, we found higher activation for real versus pantomimed tool use in sensorimotor regions and the anterior supramarginal gyrus, and higher activation for pantomimed than real tool use in classic tool-selective areas. Although no regions showed significant differences in activation magnitude during the Plan phase, activation patterns differed between real versus pantomimed tool use and motor cortex showed differential functional connectivity. These results reflect important differences between real tool use, a closed-loop process constrained by real consequences, and pantomimed tool use, a symbolic gesture that requires conceptual knowledge of tools but with limited consequences. These results highlight the feasibility and added value of employing natural tool use tasks in functional imaging, inform neuropsychological dissociations, and advance our theoretical understanding of the neural substrates of natural tool use.SIGNIFICANCE STATEMENT The study of tool use offers unique insights into how the human brain synthesizes perceptual, cognitive, and sensorimotor functions to accomplish a goal. We suggest that the reliance on proxies, such as pantomiming, for real tool use has (1) overestimated the contribution of cognitive networks, because of the indirect, symbolic nature of pantomiming; and (2) underestimated the contribution of sensorimotor networks necessary for predicting and monitoring the consequences of real interactions between hand, tool, and the target object. These results enhance our theoretical understanding of the full range of human tool functions and inform our understanding of neuropsychological dissociations between real and pantomimed tool use.

A review of the impairments, preserved visual functions, and neuropathology in 21 patients with visual form agnosia - A unique defect with line drawings

We present a comprehensive review of the rare syndrome visual form agnosia (VFA). We begin by documenting its history, including the origins of the term, and the first case study labelled as VFA. The defining characteristics of the syndrome, as others have previously defined it, are then described. The impairments, preserved aspects of visual perception, and areas of brain damage in 21 patients who meet these defining characteristics are described in detail, including which tests were used to verify the presence or absence of key symptoms. From this, we note important similarities along with notable areas of divergence between patients. Damage to the occipital lobe (20/21), an inability to recognise line drawings (19/21), preserved colour vision (14/21), and visual field defects (16/21) were areas of consistency across most cases. We found it useful to distinguish between shape and form as distinct constructs when examining perceptual abilities in VFA patients. Our observations suggest that these patients often exhibit difficulties in processing simplified versions of form. Deficits in processing orientation and size were uncommon. Motion perception and visual imagery were not widely tested for despite being typically cited as defining features of the syndrome - although in the sample described, motion perception was never found to be a deficit. Moreover, problems with vision (e.g., poor visual acuity and the presence of hemianopias/scotomas in the visual fields) are more common than we would have thought and may also contribute to perceptual impairments in patients with VFA. We conclude that VFA is a perceptual disorder where the visual system has a reduced ability to synthesise lines together for the purposes of making sense of what images represent holistically.

Massive body-brain disconnection consequent to spinal cord injuries drives profound changes in higher-order cognitive and emotional functions: A PRISMA scoping review

Spinal cord injury (SCI) leads to a massive disconnection between the brain and the body parts below the lesion level representing a unique opportunity to explore how the body influences a person's mental life. We performed a systematic scoping review of 59 studies on higher-order cognitive and emotional changes after SCI. The results suggest that fluid abilities (e.g. attention, executive functions) and emotional regulation (e.g. emotional reactivity and discrimination) are impaired in people with SCI, with progressive deterioration over time. Although not systematically explored, the factors that are directly (e.g. the severity and level of the lesion) and indirectly associated (e.g. blood pressure, sleeping disorders, medication) with the damage may play a role in these deficits. The inconsistency which was found in the results may derive from the various methods used and the heterogeneity of samples (i.e. the lesion completeness, the time interval since lesion onset). Future studies which are specifically controlled for methods, clinical and socio-cultural dimensions are needed to better understand the role of the body in cognition.

Human memory for real-world solid objects is not predicted by responses to image displays

In experimental psychology and neuroscience, computerized image stimuli are typically used as artificial proxies for real-world objects to understand brain and behavior. Here, in a series of five experiments (n = 165), we studied human memory for objects presented as tangible solids versus computerized images. We found that recall for solids was superior to images, both immediately after learning, and after a 24-hr delay. A "realness advantage" was also evident relative to three-dimensional (3-D) stereoscopic images, and when solids were viewed monocularly, arguing against explanations based on the presence of binocular depth cues in the stimulus. Critically, memory for solids was modulated by physical distance, with superior recall for objects positioned within versus outside of observers' reach, whereas recall for images was unaffected by distance. We conclude that solids are processed quantitatively and qualitatively differently in episodic memory than are images, suggesting caution in assuming that artifice can always substitute for reality. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Depth from motion parallax: Deictic consistency, eye contact, and a serious problem with Zoom

The dynamics of head and eye gaze between two or more individuals displayed during verbal and nonverbal face-to-face communication contains a wealth of information and is used for both volitionary and unconscious signaling. Current video communication systems convey visual signals about gaze behavior and other directional cues, but the information they carry is often spurious and potentially misleading. I discuss the consequences of this situation, identify the source of the problem as a more general lack of deictic consistency, and demonstrate that using display technologies that simulate motion parallax are both necessary and sufficient to alleviate it. I then devise an avatar-based remote communication solution that achieves deictic consistency and provides natural, dynamic eye contact for computer-mediated audiovisual communication.

Affordances and curvature preference: The case of real objects and spaces

Chuquichambi and colleagues recently questioned the prevailing belief that a universal human visual preference exists for curved shapes and lines. Their comprehensive meta-analysis demonstrated that while curvature preference is widespread, it is not universally constant or invariant. By revisiting their dataset, we made an intriguing discovery: a negative relationship between curvature preference and an object's "affordances." Taking an embodiment perspective into account, we propose an explanation for this phenomenon, suggesting that the diminished curvature preference in objects with abundant affordances can be understood through the lens of embodied cognition.

Testing cognitive theories with multivariate pattern analysis of neuroimaging data

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

Reach Corrections Toward Moving Objects are Faster Than Reach Corrections Toward Instantaneously Switching Targets

Visually guided reaching is a common motor behavior that engages subcortical circuits to mediate rapid corrections. Although these neural mechanisms have evolved for interacting with the physical world, they are often studied in the context of reaching toward virtual targets on a screen. These targets often change position by disappearing from one place reappearing in another instantaneously. In this study, we instructed participants to perform rapid reaches to physical objects that changed position in different ways. In one condition, the objects moved very rapidly from one place to another. In the other condition, illuminated targets instantaneously switched position by being extinguished in one position and illuminating in another. Participants were consistently faster in correcting their reach trajectories when the object moved continuously.

Capacity and selection in immersive visual working memory following naturalistic object disappearance

Visual working memory-holding past visual information in mind for upcoming behavior-is commonly studied following the abrupt removal of visual objects from static two-dimensional (2D) displays. In everyday life, visual objects do not typically vanish from the environment in front of us. Rather, visual objects tend to enter working memory following self or object motion: disappearing from view gradually and changing the spatial relation between memoranda and observer. Here, we used virtual reality (VR) to investigate whether two classic findings from visual working memory research-a capacity of around three objects and the reliance on space for object selection-generalize to more naturalistic modes of object disappearance. Our static reference condition mimicked traditional laboratory tasks whereby visual objects were held static in front of the participant and removed from view abruptly. In our critical flow condition, the same visual objects flowed by participants, disappearing from view gradually and behind the observer. We considered visual working memory performance and capacity, as well as space-based mnemonic selection, indexed by directional biases in gaze. Despite vastly distinct modes of object disappearance and altered spatial relations between memoranda and observer, we found comparable capacity and comparable gaze signatures of space-based mnemonic selection. This finding reveals how classic findings from visual working memory research generalize to immersive situations with more naturalistic modes of object disappearance and with dynamic spatial relations between memoranda and observer.

Understanding the encoding of object locations in small-scale spaces during free exploration using eye tracking

Navigation is instrumental to daily life and is often used to encode and locate objects, such as keys in one's house. Yet, little is known about how navigation works in more ecologically valid situations such as finding objects within a room. Specifically, it is not clear how vision vs. body movements contribute differentially to spatial memory in such small-scale spaces. In the current study, participants encoded object locations by viewing them while standing (stationary condition) or by additionally being guided by the experimenter while blindfolded (walking condition) after viewing the objects. They then retrieved the objects from the same or different viewpoint, creating a 2 × 2 within subject design. We simultaneously recorded participant eye movements throughout the experiment using mobile eye tracking. The results showed no statistically significant differences among our four conditions (stationary, same viewpoint as encoding; stationary, different viewpoint; walking, same viewpoint; walking, different viewpoint), suggesting that in a small real-world space, vision may be sufficient to remember object locations. Eye tracking analyses revealed that object locations were better remembered next to landmarks and that participants encoded items on one wall together, suggesting the use of local wall coordinates rather than global room coordinates. A multivariate regression analysis revealed that the only significant predictor of object placement accuracy was average looking time. These results suggest that vision may be sufficient for encoding object locations in a small-scale environment and that such memories may be formed largely locally rather than globally.

Enhanced temporal resolution of vision in action video game players

Video game play has been suggested to improve visual and attention processing. Nevertheless, while action video game play is highly dynamic, there is scarce research on how information is temporally discriminated at the millisecond level. This cross-sectional study investigates whether temporal discrimination at the millisecond level in vision varies across action video game players (VGPs; N = 23) and non-video game players (NVGPs; N = 23). Participants discriminated synchronous from asynchronous onsets of two visual targets in virtual reality, while their EEG and oculomotor movements were recorded. Results show an increased sensitivity to short asynchronies (11, 33 and 66 ms) in VGPs compared with NVGPs, which was especially marked at the start of the task, suggesting better temporal discrimination abilities. Pre-targets oculomotor freezing - the inhibition of small fixational saccades - was associated with correct temporal discrimination, probably revealing attentional preparation. However, this parameter did not differ between groups. EEG and reconstruction analyses suggest that the enhancement of temporal discrimination in VGPs during temporal discrimination is related to parieto-occipital processing, and a reduction of alpha-band (8-14 Hz) power and inter-trial phase coherence. Overall, the study reveals an enhanced ability in action video game players to discriminate in time visual events in close temporal proximity combined with reduced alpha-band oscillatory activities. Consequently, playing action video games is associated with an improved temporal resolution of vision.

Real-life relevant face perception is not captured by the N170 but reflected in later potentials: A comparison of 2D and virtual reality stimuli

The perception of faces is one of the most specialized visual processes in the human brain and has been investigated by means of the early event-related potential component N170. However, face perception has mostly been studied in the conventional laboratory, i.e., monitor setups, offering rather distal presentation of faces as planar 2D-images. Increasing spatial proximity through Virtual Reality (VR) allows to present 3D, real-life-sized persons at personal distance to participants, thus creating a feeling of social involvement and adding a self-relevant value to the presented faces. The present study compared the perception of persons under conventional laboratory conditions (PC) with realistic conditions in VR. Paralleling standard designs, pictures of unknown persons and standard control images were presented in a PC- and a VR-modality. To investigate how the mechanisms of face perception differ under realistic conditions from those under conventional laboratory conditions, the typical face-specific N170 and subsequent components were analyzed in both modalities. Consistent with previous laboratory research, the N170 lost discriminatory power when translated to realistic conditions, as it only discriminated faces and controls under laboratory conditions. Most interestingly, analysis of the later component [230–420 ms] revealed more differentiated face-specific processing in VR, as indicated by distinctive, stimulus-specific topographies. Complemented by source analysis, the results on later latencies show that face-specific neural mechanisms are applied only under realistic conditions (A video abstract is available in the Supplementary material and via YouTube: https://youtu.be/TF8wiPUrpSY).

Eye movement characteristics in a mental rotation task presented in virtual reality

Introduction

Eye-tracking technology provides a reliable and cost-effective approach to characterize mental representation according to specific patterns. Mental rotation tasks, referring to the mental representation and transformation of visual information, have been widely used to examine visuospatial ability. In these tasks, participants visually perceive three-dimensional (3D) objects and mentally rotate them until they identify whether the paired objects are identical or mirrored. In most studies, 3D objects are presented using two-dimensional (2D) images on a computer screen. Currently, visual neuroscience tends to investigate visual behavior responding to naturalistic stimuli rather than image stimuli. Virtual reality (VR) is an emerging technology used to provide naturalistic stimuli, allowing the investigation of behavioral features in an immersive environment similar to the real world. However, mental rotation tasks using 3D objects in immersive VR have been rarely reported.

Methods

Here, we designed a VR mental rotation task using 3D stimuli presented in a head-mounted display (HMD). An eye tracker incorporated into the HMD was used to examine eye movement characteristics during the task synchronically. The stimuli were virtual paired objects oriented at specific angular disparities (0, 60, 120, and 180°). We recruited thirty-three participants who were required to determine whether the paired 3D objects were identical or mirrored.

Results

Behavioral results demonstrated that the response times when comparing mirrored objects were longer than identical objects. Eye-movement results showed that the percent fixation time, the number of within-object fixations, and the number of saccades for the mirrored objects were significantly lower than that for the identical objects, providing further explanations for the behavioral results.

Discussion

In the present work, we examined behavioral and eye movement characteristics during a VR mental rotation task using 3D stimuli. Significant differences were observed in response times and eye movement metrics between identical and mirrored objects. The eye movement data provided further explanation for the behavioral results in the VR mental rotation task.

Tool use acquisition induces a multifunctional interference effect during object processing: evidence from the sensorimotor mu rhythm

A fundamental characteristic of human development is acquiring and accumulating tool use knowledge through observation and sensorimotor experience. Recent studies showed that, in children and adults, different action possibilities to grasp-to-move and grasp-to-use objects generate a conflict that extinguishes neural motor resonance phenomena during visual object processing. In this study, a training protocol coupled with EEG recordings was administered in virtual reality to healthy adults to evaluate whether a similar conflict occurs between novel tool use knowledge. Participants perceived and manipulated two novel 3D tools trained beforehand with either single or double-usage. A weaker reduction of mu-band (10-13 Hz) power, accompanied by a reduced inter-trial phase coherence, was recorded during the perception of the tool associated with the double-usage. These effects started within the first 200 ms of visual object processing and were predominantly recorded over the left motor system. Furthermore, interacting with the double usage tool delayed grasp-to-reach movements. The results highlight a multifunctional interference effect, such as tool use acquisition reduces the neural motor resonance phenomenon and inhibits the activation of the motor system during subsequent object recognition. These results imply that learned tool use information guides sensorimotor processes of objects.

Statistical Learning Within Objects

Research has recently shown that efficient selection relies on the implicit extraction of environmental regularities, known as statistical learning. Although this has been demonstrated for scenes, similar learning arguably also occurs for objects. To test this, we developed a paradigm that allowed us to track attentional priority at specific object locations irrespective of the object's orientation in three experiments with young adults (all Ns = 80). Experiments 1a and 1b established within-object statistical learning by demonstrating increased attentional priority at relevant object parts (e.g., hammerhead). Experiment 2 extended this finding by demonstrating that learned priority generalized to viewpoints in which learning never took place. Together, these findings demonstrate that as a function of statistical learning, the visual system not only is able to tune attention relative to specific locations in space but also can develop preferential biases for specific parts of an object independently of the viewpoint of that object.

Predictive Processing of Scene Layout Depends on Naturalistic Depth of Field

Boundary extension is a classic memory illusion in which observers remember more of a scene than was presented. According to predictive-processing accounts, boundary extension reflects the integration of visual input and expectations of what is beyond a scene's boundaries. According to normalization accounts, boundary extension rather reflects one end of a normalization process toward a scene's typically experienced viewing distance, such that close-up views give boundary extension but distant views give boundary contraction. Here, across four experiments (N = 125 adults), we found that boundary extension strongly depends on depth of field, as determined by the aperture settings on a camera. Photographs with naturalistic depth of field led to larger boundary extension than photographs with unnaturalistic depth of field, even when distant views were shown. We propose that boundary extension reflects a predictive mechanism with adaptive value that is strongest for naturalistic views of scenes. The current findings indicate that depth of field is an important variable to consider in the study of scene perception and memory.

Contribution of the stereoscopic representation of motion-in-depth during visually guided feedback control

Considerable studies have focused on the neural basis of visually guided tracking movement in the frontoparallel plane, whereas the neural process in real-world circumstances regarding the influence of binocular disparity and motion-in-depth (MID) perception is less understood. Although the role of stereoscopic versus monoscopic MID information has been extensively described for visual processing, its influence on top-down regulation for motor execution has not received much attention. Here, we orthogonally varied the visual representation (stereoscopic versus monoscopic) and motion direction (depth motion versus bias depth motion versus frontoparallel motion) during visually guided tracking movements, with simultaneous functional near-infrared spectroscopy recordings. Results show that the stereoscopic representation of MID could lead to more accurate movements, which was supported by specific neural activity pattern. More importantly, we extend prior evidence about the role of frontoparietal network in brain-behavior relationship, showing that occipital area, more specifically, visual area V2/V3 was also robustly involved in the association. Furthermore, by using the stereoscopic representation of MID, it is plausible to detect robust brain-behavior relationship even with small sample size at low executive task demand. Taken together, these findings highlight the importance of the stereoscopic representation of MID for investigating neural correlates of visually guided feedback control.

Familiar size affects perception differently in virtual reality and the real world

The promise of virtual reality (VR) as a tool for perceptual and cognitive research rests on the assumption that perception in virtual environments generalizes to the real world. Here, we conducted two experiments to compare size and distance perception between VR and physical reality (Maltz et al. 2021 J. Vis. 21, 1-18). In experiment 1, we used VR to present dice and Rubik's cubes at their typical sizes or reversed sizes at distances that maintained a constant visual angle. After viewing the stimuli binocularly (to provide vergence and disparity information) or monocularly, participants manually estimated perceived size and distance. Unlike physical reality, where participants relied less on familiar size and more on presented size during binocular versus monocular viewing, in VR participants relied heavily on familiar size regardless of the availability of binocular cues. In experiment 2, we demonstrated that the effects in VR generalized to other stimuli and to a higher quality VR headset. These results suggest that the use of binocular cues and familiar size differs substantially between virtual and physical reality. A deeper understanding of perceptual differences is necessary before assuming that research outcomes from VR will generalize to the real world. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Comparing encoding mechanisms in realistic virtual reality and conventional 2D laboratory settings: Event-related potentials in a repetition suppression paradigm

Although the human brain is adapted to function within three-dimensional environments, conventional laboratory research commonly investigates cognitive mechanisms in a reductionist approach using two-dimensional stimuli. However, findings regarding mnemonic processes indicate that realistic experiences in Virtual Reality (VR) are stored in richer and more intertwined engrams than those obtained from the conventional laboratory. Our study aimed to further investigate the generalizability of laboratory findings and to differentiate whether the processes underlying memory formation differ between VR and the conventional laboratory already in early encoding stages. Therefore, we investigated the Repetition Suppression (RS) effect as a correlate of the earliest instance of mnemonic processes under conventional laboratory conditions and in a realistic virtual environment. Analyses of event-related potentials (ERPs) indicate that the ERP deflections at several electrode clusters were lower in VR compared to the PC condition. These results indicate an optimized distribution of cognitive resources in realistic contexts. The typical RS effect was replicated under both conditions at most electrode clusters for a late time window. Additionally, a specific RS effect was found in VR at anterior electrodes for a later time window, indicating more extensive encoding processes in VR compared to the laboratory. Specifically, electrotomographic results (VARETA) indicate multimodal integration involving a broad cortical network and higher cognitive processes during the encoding of realistic objects. Our data suggest that object perception under realistic conditions, in contrast to the conventional laboratory, requires multisensory integration involving an interconnected functional system, facilitating the formation of intertwined memory traces in realistic environments.

Computational meaningfulness as the source of beneficial cognitive biases

The human brain has evolved to solve the problems it encounters in multiple environments. In solving these challenges, it forms mental simulations about multidimensional information about the world. These processes produce context-dependent behaviors. The brain as overparameterized modeling organ is an evolutionary solution for producing behavior in a complex world. One of the most essential characteristics of living creatures is that they compute the values of information they receive from external and internal contexts. As a result of this computation, the creature can behave in optimal ways in each environment. Whereas most other living creatures compute almost exclusively biological values (e.g., how to get food), the human as a cultural creature computes meaningfulness from the perspective of one's activity. The computational meaningfulness means the process of the human brain, with the help of which an individual tries to make the respective situation comprehensible to herself to know how to behave optimally. This paper challenges the bias-centric approach of behavioral economics by exploring different possibilities opened up by computational meaningfulness with insight into wider perspectives. We concentrate on confirmation bias and framing effect as behavioral economics examples of cognitive biases. We conclude that from the computational meaningfulness perspective of the brain, the use of these biases are indispensable property of an optimally designed computational system of what the human brain is like. From this perspective, cognitive biases can be rational under some conditions. Whereas the bias-centric approach relies on small-scale interpretable models which include only a few explanatory variables, the computational meaningfulness perspective emphasizes the behavioral models, which allow multiple variables in these models. People are used to working in multidimensional and varying environments. The human brain is at its best in such an environment and scientific study should increasingly take place in such situations simulating the real environment. By using naturalistic stimuli (e.g., videos and VR) we can create more realistic, life-like contexts for research purposes and analyze resulting data using machine learning algorithms. In this manner, we can better explain, understand and predict human behavior and choice in different contexts.

Does memory research have a realistic future?

How do we remember our past experiences? This question remains stubbornly resistant to resolution. The next 25 years may see significant traction on this and other outstanding issues if memory researchers capitalise on exciting technological developments that allow embodied cognition to be studied in ways that closely approximate real life.

The choices hidden in photography

Photography is often understood as an objective recording of light measurements, in contrast with the subjective nature of painting. This article argues that photography entails making the same kinds of choices of color, tone, and perspective as in painting, and surveys examples from film photography and smartphone cameras. Hence, understanding picture perception requires treating photography as just one way to make pictures. More research is needed to understand the effects of these choices on pictorial perception, which in turn could lead to the design of new imaging techniques.

The role of the anterior temporal cortex in action: evidence from fMRI multivariate searchlight analysis during real object grasping

Intelligent manipulation of handheld tools marks a major discontinuity between humans and our closest ancestors. Here we identified neural representations about how tools are typically manipulated within left anterior temporal cortex, by shifting a searchlight classifier through whole-brain real action fMRI data when participants grasped 3D-printed tools in ways considered typical for use (i.e., by their handle). These neural representations were automatically evocated as task performance did not require semantic processing. In fact, findings from a behavioural motion-capture experiment confirmed that actions with tools (relative to non-tool) incurred additional processing costs, as would be suspected if semantic areas are being automatically engaged. These results substantiate theories of semantic cognition that claim the anterior temporal cortex combines sensorimotor and semantic content for advanced behaviours like tool manipulation.

The role of low-frequency oscillations in three-dimensional perception with depth cues in virtual reality

Currently, vision-related neuroscience studies are undergoing a trend from simplified image stimuli toward more naturalistic stimuli. Virtual reality (VR), as an emerging technology for visual immersion, provides more depth cues for three-dimensional (3D) presentation than two-dimensional (2D) image. It is still unclear whether the depth cues used to create 3D visual perception modulate specific cortical activation. Here, we constructed two visual stimuli presented by stereoscopic vision in VR and graphical projection with 2D image, respectively, and used electroencephalography to examine neural oscillations and their functional connectivity during 3D perception. We find that neural oscillations are specific to delta and theta bands in stereoscopic vision and the functional connectivity in the both bands increase in cortical areas related to visual pathways. These findings indicate that low-frequency oscillations play an important role in 3D perception with depth cues.

Revealing Whole-Brain Causality Networks During Guided Visual Searching

In our daily lives, we use eye movements to actively sample visual information from our environment ("active vision"). However, little is known about how the underlying mechanisms are affected by goal-directed behavior. In a study of 31 participants, magnetoencephalography was combined with eye-tracking technology to investigate how interregional interactions in the brain change when engaged in two distinct forms of active vision: freely viewing natural images or performing a guided visual search. Regions of interest with significant fixation-related evoked activity (FRA) were identified with spatiotemporal cluster permutation testing. Using generalized partial directed coherence, we show that, in response to fixation onset, a bilateral cluster consisting of four regions (posterior insula, transverse temporal gyri, superior temporal gyrus, and supramarginal gyrus) formed a highly connected network during free viewing. A comparable network also emerged in the right hemisphere during the search task, with the right supramarginal gyrus acting as a central node for information exchange. The results suggest that all four regions are vital to visual processing and guiding attention. Furthermore, the right supramarginal gyrus was the only region where activity during fixations on the search target was significantly negatively correlated with search response times. Based on our findings, we hypothesize that, following a fixation, the right supramarginal gyrus supplies the right supplementary eye field (SEF) with new information to update the priority map guiding the eye movements during the search task.

Do graspable objects always leave a motor signature? A study on memory traces

Several studies have reported the existence of reciprocal interactions between the type of motor activity physically performed on objects and the conceptual knowledge that is retained of them. Whether covert motor activity plays a similar effect is less clear. Certainly, objects are strong triggers for actions, and motor components can make the associated concepts more memorable. However, addition of an action-related memory trace may not always be automatic and could rather depend on 'how' objects are encountered. To test this hypothesis, we compared memory for objects that passive observers experienced as verbal labels (the word describing them), visual images (color photographs) and actions (pantomimes of object use). We predicted that the more direct the involvement of action-related representations the more effective would be the addition of a motor code to the experience and the more accurate would be the recall. Results showed that memory for objects presented as words i.e., a format that might only indirectly prime the sensorimotor system, was generally less accurate compared to memory for objects presented as photographs or pantomimes, which are more likely to directly elicit motor simulation processes. In addition, free recall of objects experienced as pantomimes was more accurate when these items afforded actions performed towards one's body than actions directed away from the body. We propose that covert motor activity can contribute to objects' memory, but the beneficial addition of a motor code to the experience is not necessarily automatic. An advantage is more likely to emerge when the observer is induced to take a first-person stance during the encoding phase, as may happen for objects affording actions directed towards the body, which obviously carry more relevance for the actor.

Preparatory attention incorporates contextual expectations

Humans are remarkably proficient at finding objects within complex visual scenes. According to current theories of attention,^1-3 visual processing of an object of interest is favored through the preparatory activation of object-specific representations in visual cortex.^4-15 One key problem that is inherent to real-world visual search but is not accounted for by current theories is that a given object will produce a dramatically different retinal image depending on its location, which is unknown in advance. For instance, the color of the retinal image depends on the illumination on the object, its shape depends on the viewpoint, and (most critically) its size can vary by several orders of magnitude, depending on the distance to the observer. In order to benefit search, preparatory activity thus needs to incorporate contextual expectations. In the current study, we measured fMRI blood-oxygen-level-dependent (BOLD) activity in human observers while they prepared to search for objects at different distances in indoor-scene photographs. First, we established that observers instantiated preparatory object representations: activity patterns in object-selective cortex evoked during search preparation (while no objects were presented) resembled activity patterns evoked by viewing those objects in isolation. Second, we demonstrated that these preparatory object representations were systematically modulated by expectations derived from scene context: activity patterns reflected the predicted retinal image of the object at each distance (i.e., distant search evoking smaller object representations and nearby search evoking larger object representations). These findings reconcile current theories of attentional selection with the challenges of real-world vision.

Graspability Modulates the Stronger Neural Signature of Motor Preparation for Real Objects vs. Pictures

The cognitive and neural bases of visual perception are typically studied using pictures rather than real-world stimuli. Unlike pictures, real objects are actionable solids that can be manipulated with the hands. Recent evidence from human brain imaging suggests that neural responses to real objects differ from responses to pictures; however, little is known about the neural mechanisms that drive these differences. Here, we tested whether brain responses to real objects versus pictures are differentially modulated by the "in-the-moment" graspability of the stimulus. In human dorsal cortex, electroencephalographic responses show a "real object advantage" in the strength and duration of mu (μ) and low beta (β) rhythm desynchronization-well-known neural signatures of visuomotor action planning. We compared desynchronization for real tools versus closely matched pictures of the same objects, when the stimuli were positioned unoccluded versus behind a large transparent barrier that prevented immediate access to the stimuli. We found that, without the barrier in place, real objects elicited stronger μ and β desynchronization compared to pictures, both during stimulus presentation and after stimulus offset, replicating previous findings. Critically, however, with the barrier in place, this real object advantage was attenuated during the period of stimulus presentation, whereas the amplification in later periods remained. These results suggest that the "real object advantage" is driven initially by immediate actionability, whereas later differences perhaps reflect other, more inherent properties of real objects. The findings showcase how the use of richer multidimensional stimuli can provide a more complete and ecologically valid understanding of object vision.

Learning Faces as Concepts Improves Face Recognition by Engaging the Social Brain Network

Face recognition benefits from associating social information to faces during learning. This has been demonstrated by better recognition for faces that underwent social than perceptual evaluations. Two hypotheses were proposed to account for this effect. According to the feature-elaboration hypothesis, social-evaluations encourage elaborated processing of perceptual information from faces (Winograd, 1981). According to a social-representation hypothesis, social-evaluations convert faces from a perceptual representation to a socially meaningful representation of a person. To decide between these two hypotheses, we ran a functional MRI study in which we functionally localized the posterior face-selective brain areas and social processing brain areas. Participants watched video-clips of young adults and were asked to study them for a recognition test, while making either perceptual evaluations or social evaluations about them. During the fMRI scan, participants performed an old/new recognition test. Behavioural findings replicated better recognition for faces that underwent social then perceptual evaluations. fMRI results showed higher response during the recognition phase for the faces that were learned socially than perceptually, in the social-brain network but not in posterior face-selective network. These results support the social-representation hypothesis and highlight the important role that social processing mechanisms, rather than purely perceptual processes, play in face recognition.

Grasping performance depends upon the richness of hand feedback

Although visual feedback of the hand allows fast and accurate grasping actions, little is known about whether the nature of feedback of the hand affects performance. We investigated kinematics during precision grasping (with the index finger and thumb) when participants received different levels of hand feedback, with or without visual feedback of the target. Specifically, we compared performance when participants saw (1) no hand feedback; (2) only the two critical points on the index finger and thumb tips; (3) 21 points on all digit tips and hand joints; (4) 21 points connected by a "skeleton", or (5) full feedback of the hand wearing a glove. When less hand feedback was available, participants took longer to execute the movement because they allowed more time to slow the reach and close the hand. When target feedback was unavailable, participants took longer to plan the movement and reached with higher velocity. We were particularly interested in investigating maximum grip aperture (MGA), which can reflect the margin of error that participants allow to compensate for uncertainty. A trend suggested that MGA was smallest when ample feedback was available (skeleton and full hand feedback, regardless of target feedback) and when only essential information about hand and target was provided (2-point hand feedback + target feedback) but increased when non-essential points were included (21-point feedback). These results suggest that visual feedback of the hand affects grasping performance and that, while more feedback is usually beneficial, this is not necessarily always the case.