The time course of visual processing: from early perception to decision-making

The Role of Edge-Based and Surface-Based Information in Incidental Category Learning: Evidence From Behavior and Event-Related Potentials

Although it has been demonstrated that edge-based information is more important than surface-based information in incidental category learning, it remains unclear how the two types of information play different roles in incidental category learning. To address this issue, the present study combined behavioral and event-related potential (ERP) techniques in an incidental category learning task in which the categories were defined by either edge- or surface-based features. The results from Experiment 1 showed that participants could simultaneously learn both edge- and surface-based information in incidental category learning, and importantly, there was a larger learning effect for the edge-based category than for the surface-based category. The behavioral results from Experiment 2 replicated those from Experiment 1, and the ERP results further revealed that the stimuli from the edge-based category elicited larger anterior and posterior P2 components than those from the surface-based category, whereas the stimuli from the surface-based category elicited larger anterior N1 and P3 components than those from the edge-based category. Taken together, the results suggest that, although surface-based information might attract more attention during feature detection, edge-based information plays more important roles in evaluating the relevance of information in making a decision in categorization.

Attentional priorities drive effects of time pressure on altruistic choice

Dual-process models of altruistic choice assume that automatic responses give way to deliberation over time, and are a popular way to conceptualize how people make generous choices and why those choices might change under time pressure. However, these models have led to conflicting interpretations of behaviour and underlying psychological dynamics. Here, we propose that flexible, goal-directed deployment of attention towards information priorities provides a more parsimonious account of altruistic choice dynamics. We demonstrate that time pressure tends to produce early gaze-biases towards a person’s own outcomes, and that individual differences in this bias explain how individuals’ generosity changes under time pressure. Our gaze-informed drift-diffusion model incorporating moment-to-moment eye-gaze further reveals that underlying social preferences both drive attention, and interact with it to shape generosity under time pressure. These findings help explain existing inconsistencies in the field by emphasizing the role of dynamic attention-allocation during altruistic choice.

Forcing people to choose quickly often changes pro-social behavior, but it is unclear why. Here, the authors show that under time pressure, people engage in incomplete information searches biased by concern (or lack thereof) for others, explaining effects often attributed to automatic processing.

Disentangling the Independent Contributions of Visual and Conceptual Features to the Spatiotemporal Dynamics of Scene Categorization

Human scene categorization is characterized by its remarkable speed. While many visual and conceptual features have been linked to this ability, significant correlations exist between feature spaces, impeding our ability to determine their relative contributions to scene categorization. Here, we used a whitening transformation to decorrelate a variety of visual and conceptual features and assess the time course of their unique contributions to scene categorization. Participants (both sexes) viewed 2250 full-color scene images drawn from 30 different scene categories while having their brain activity measured through 256-channel EEG. We examined the variance explained at each electrode and time point of visual event-related potential (vERP) data from nine different whitened encoding models. These ranged from low-level features obtained from filter outputs to high-level conceptual features requiring human annotation. The amount of category information in the vERPs was assessed through multivariate decoding methods. Behavioral similarity measures were obtained in separate crowdsourced experiments. We found that all nine models together contributed 78% of the variance of human scene similarity assessments and were within the noise ceiling of the vERP data. Low-level models explained earlier vERP variability (88 ms after image onset), whereas high-level models explained later variance (169 ms). Critically, only high-level models shared vERP variability with behavior. Together, these results suggest that scene categorization is primarily a high-level process, but reliant on previously extracted low-level features.SIGNIFICANCE STATEMENT In a single fixation, we glean enough information to describe a general scene category. Many types of features are associated with scene categories, ranging from low-level properties, such as colors and contours, to high-level properties, such as objects and attributes. Because these properties are correlated, it is difficult to understand each property's unique contributions to scene categorization. This work uses a whitening transformation to remove the correlations between features and examines the extent to which each feature contributes to visual event-related potentials over time. We found that low-level visual features contributed first but were not correlated with categorization behavior. High-level features followed 80 ms later, providing key insights into how the brain makes sense of a complex visual world.

Rapid scene categorization: From coarse peripheral vision to fine central vision

Studies on scene perception have shown that the rapid extraction of low spatial frequencies (LSF) allows a coarse parsing of the scene, prior to the analysis of high spatial frequencies (HSF) containing details. Many studies suggest that scene gist recognition can be achieved with only the low resolution of peripheral vision. Our study investigated the advantage of peripheral vision on central vision during a scene categorization task (indoor vs. outdoor). In Experiment 1, we used large scene photographs from which we built one central disk and four circular rings of different eccentricities. The central disk either contained or not an object semantically related to the scene category. Results showed better categorization performances for the peripheral rings, despite the presence of an object in central vision that was semantically related to the scene category that significantly improved categorization performances. In Experiment 2, the central disk and rings were assembled from Central to Peripheral vision (CtP sequence) or from Peripheral to Central vision (PtC sequence). Results revealed better performances for PtC than CtP sequences, except when no central object was present under rapid categorization constraints. As Experiment 3 suggested that the PtC advantage was not explained by a reduction of the visibility of the object in the central disk by the surrounding peripheral rings (CtP sequence), results are interpreted in the context of a predominant coarse-to-fine processing during scene categorization, with greater efficiency and utility of coarse peripheral vision relative to fine central vision during rapid scene categorization.

How visual information influences dual-task driving and tracking

The study examined the impact of visual predictability on dual-task performance in driving and tracking tasks. Participants (N = 27) performed a simulated driving task and a pursuit tracking task. In either task, visual predictability was manipulated by systematically varying the amount of advance visual information: in the driving task, participants drove at night with low beam, at night with high beam, or in daylight; in the tracking task, participants saw a white line that specified the future target trajectory for 200, 400 or 800 ms. Concurrently with driving or tracking, participants performed an auditory task. They had to discriminate between two sounds and press a pedal upon hearing the higher sound. Results show that in general, visual predictability benefited driving and tracking; however, dual-task driving performance was best with highest visual predictability (daylight), dual-task tracking performance was best with medium visual predictability (400 ms). Braking/reaction times were higher in dual tasks compared to single tasks, but were unaffected by visual predictability, showing that its beneficial effects did not transfer to the auditory task. In both tasks, manual accuracy decreased around the moment the foot pressed the pedal, indicating interference between tasks. We, therefore, conclude that despite a general beneficial impact of predictability, the integration of visual information seems to be rather task specific, and that interference between driving and audiomotor tasks, and tracking and audiomotor tasks, seems comparable.

The parallel programming of landing position in saccadic eye movement sequences

Saccadic eye movements occur in sequences, gathering new information about the visual environment to support successful task completion. Here, we examine the control of these saccadic sequences and specifically the extent to which the spatial aspects of the saccadic responses are programmed in parallel. We asked participants to saccade to a series of visual targets and, while they shifted their gaze around the display, we displaced select targets. We found that saccade landing position was deviated toward the previous location of the target suggesting that partial parallel programming of target location information was occurring. The saccade landing position was also affected by the new target location, which demonstrates that the saccade landing position was also partially updated following the shift. This pattern was present even for targets that were the subject of the next fixation. Having a greater preview about the sequence path influenced saccade accuracy with saccades being less affected by relocations when there is less preview information. The results demonstrate that landing positions from a saccade sequence are programmed in parallel and combined with more immediate visual signals.

A real-time marker of object-based attention in the human brain. A possible component of a "gate-keeping mechanism" performing late attentional selection in the Ventro-Lateral Prefrontal Cortex

The decision to process an incoming stimulus attentively - and to trigger a follow-up cascade of high-level processes - is strategic for the human brain as it becomes transiently unavailable to subsequent stimulus processing. In this study, we set to identify brain networks that carry out such evaluations. We therefore assessed the time-course of neural responses with intracerebral EEG in human patients during an attentional reading task, contrasting to-be-attended vs. to-be-ignored items. We measured High-Frequency Activity [50-150 ​Hz] as a proxy of population-level spiking activity and we identified a crucial component of a Gate-Keeping Mechanism bilateral in the mid-Ventro-Lateral Prefrontal Cortex (VLPFC), at the interplay of the Ventral and Dorsal Attention Networks, that selectively reacts before domain specialized cortical regions that engage in full stimulus analysis according to task demands.

Category-based attentional capture can be influenced by color- and shape-dimensions independently in the conjunction search task

Attention can be involuntarily attracted by a distractor that matches the current attentional control settings (ACSs). However, it remains unclear whether two category-specific ACSs can operate independently. By defining a target as a combination of two prototype-based categories, the present event-related potential (ERP) study investigated how color-category and shape-category ACSs operate within a search task paradigm and the effects of temporal task demands on these ACSs. The matching level between target and distractor was manipulated to separate the effects of each ACS. The relative position between target and distractor was employed to isolate the attentional processing of the distractor from the target. Furthermore, two display durations were used to manipulate the temporal task demands, including a short fixed window (800 ms) and a dynamic window extended until the user responded. Our results support a two-stage selection scenario. In early stage, the color- and shape-ACS independently guided attention to task-relevant property (N2pc components) and suppressed attention toward task-irrelevant properties (P_D components). In late stage, these two independent ACSs were integrated into a holistic ACS to interfere with the consolidation (contralateral delay activity components) and behavioral performance (accuracy and RTs) of target identification. Moreover, an early N1/P1 component might reflect a preattentive enhancement of relevant information or a preattentive suppression of irrelevant objects. These two category-specific ACSs weights differently in varied temporal task demands. These findings support the idea that independent early processing is followed by integrated late processing, which can be applied to category-based attentional capture with different temporal task demands.

Right frontal eye field has perceptual and oculomotor functions during optokinetic stimulation and nystagmus

The right frontal eye field (rFEF) is associated with visual perception and eye movements. rFEF is activated during optokinetic nystagmus (OKN), a reflex that moves the eye in response to visual motion (optokinetic stimulation, OKS). It remains unclear whether rFEF plays causal perceptual and/or oculomotor roles during OKS and OKN. To test this, participants viewed a leftward-moving visual scene of vertical bars and judged whether a flashed dot was moving. Single pulses of transcranial magnetic stimulation (TMS) were applied to rFEF on half of trials. In half of blocks, to explore oculomotor control, participants performed an OKN in response to the OKS. rFEF TMS, during OKN, made participants more accurate on trials when the dot was still, and it slowed eye movements. In separate blocks, participants fixated during OKS. This not only controlled for eye movements but also allowed the use of EEG to explore the FEF's role in visual motion discrimination. In these blocks, by contrast, leftward dot motion discrimination was impaired, associated with a disruption of the frontal-posterior balance in alpha-band oscillations. None of these effects occurred in a control site (M1) experiment. These results demonstrate multiple related yet dissociable causal roles of the right FEF during optokinetic stimulation.NEW & NOTEWORTHY This study demonstrates causal roles of the right frontal eye field (FEF) in motion discrimination and eye movement control during visual scene motion: previous work had only examined other stimuli and eye movements such as saccades. Using combined transcranial magnetic stimulation and EEG and a novel optokinetic stimulation motion-discrimination task, we find evidence for multiple related yet dissociable causal roles within the FEF: perceptual processing during optokinetic stimulation, generation of the optokinetic nystagmus, and the maintenance of alpha oscillations.

Ultrafast Object Detection in Naturalistic Vision Relies on Ultrafast Distractor Suppression

People are quicker to detect examples of real-world object categories in natural scenes than is predicted by classic attention theories. One explanation for this puzzle suggests that experience renders the visual system sensitive to midlevel features diagnosing target presence. These are detected without the need for spatial attention, much as occurs for targets defined by low-level features like color or orientation. The alternative is that naturalistic search relies on spatial attention but is highly efficient because global scene information can be used to quickly reject nontarget objects and locations. Here, we use ERPs to differentiate between these possibilities. Results show that hallmark evidence of ultrafast target detection in frontal brain activity is preceded by an index of spatially specific distractor suppression in visual cortex. Naturalistic search for heterogenous targets therefore appears to rely on spatial operations that act on neural object representations, as predicted by classic attention theory. People appear able to rapidly reject nontarget objects and locations, consistent with the idea that global scene information is used to constrain naturalistic search and increase search efficiency.

Feed-forward visual processing suffices for coarse localization but fine-grained localization in an attention-demanding context needs feedback processing

It is well known that simple visual tasks, such as object detection or categorization, can be performed within a short period of time, suggesting the sufficiency of feed-forward visual processing. However, more complex visual tasks, such as fine-grained localization may require high-resolution information available at the early processing levels in the visual hierarchy. To access this information using a top-down approach, feedback processing would need to traverse several stages in the visual hierarchy and each step in this traversal takes processing time. In the present study, we compared the processing time required to complete object categorization and localization by varying presentation duration and complexity of natural scene stimuli. We hypothesized that performance would be asymptotic at shorter presentation durations when feed-forward processing suffices for visual tasks, whereas performance would gradually improve as images are presented longer if the tasks rely on feedback processing. In Experiment 1, where simple images were presented, both object categorization and localization performance sharply improved until 100 ms of presentation then it leveled off. These results are a replication of previously reported rapid categorization effects but they do not support the role of feedback processing in localization tasks, indicating that feed-forward processing enables coarse localization in relatively simple visual scenes. In Experiment 2, the same tasks were performed but more attention-demanding and ecologically valid images were used as stimuli. Unlike in Experiment 1, both object categorization performance and localization precision gradually improved as stimulus presentation duration became longer. This finding suggests that complex visual tasks that require visual scrutiny call for top-down feedback processing.

The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives

Understanding how people comprehend visual narratives (including picture stories, comics, and film) requires the combination of traditionally separate theories that span the initial sensory and perceptual processing of complex visual scenes, the perception of events over time, and comprehension of narratives. Existing piecemeal approaches fail to capture the interplay between these levels of processing. Here, we propose the Scene Perception & Event Comprehension Theory (SPECT), as applied to visual narratives, which distinguishes between front‐end and back‐end cognitive processes. Front‐end processes occur during single eye fixations and are comprised of attentional selection and information extraction. Back‐end processes occur across multiple fixations and support the construction of event models, which reflect understanding of what is happening now in a narrative (stored in working memory) and over the course of the entire narrative (stored in long‐term episodic memory). We describe relationships between front‐ and back‐end processes, and medium‐specific differences that likely produce variation in front‐end and back‐end processes across media (e.g., picture stories vs. film). We describe several novel research questions derived from SPECT that we have explored. By addressing these questions, we provide greater insight into how attention, information extraction, and event model processes are dynamically coordinated to perceive and understand complex naturalistic visual events in narratives and the real world.

Comprehension of visual narratives like comics, picture stories, and films involves both decoding the visual content and construing the meaningful events they represent. The Scene Perception & Event Comprehension Theory (SPECT) proposes a framework for understanding how a comprehender perceptually negotiates the surface of a visual representation and integrates its meaning into a growing mental model.

Dissociable sensitivity and bias mechanisms mediate behavioral effects of exogenous attention

Attention can be directed endogenously, based on task-relevant goals, or captured exogenously, by salient stimuli. While recent studies have shown that endogenous attention can facilitate behavior through dissociable sensitivity (sensory) and choice bias (decisional) mechanisms, it is unknown if exogenous attention also operates through dissociable sensitivity and bias mechanisms. We tested human participants on a multialternative change detection task with exogenous attention cues, which preceded or followed change events in close temporal proximity. Analyzing participants’ behavior with a multidimensional signal detection model revealed clear dissociations between exogenous cueing effects on sensitivity and bias. While sensitivity was, overall, lower at the cued location compared to other locations, bias was highest at the cued location. With an appropriately designed post-cue control condition, we discovered that the attentional effect of exogenous pre-cueing was to enhance sensitivity proximal to the cue. In contrast, exogenous attention enhanced bias even for distal stimuli in the cued hemifield. Reaction time effects of exogenous cueing could be parsimoniously explained with a diffusion-decision model, in which drift rate was determined by independent contributions from sensitivity and bias at each location. The results suggest a mechanistic schema of how exogenous attention engages dissociable sensitivity and bias mechanisms to shape behavior.

Neurophysiological correlates of collective perceptual decision-making

Humans frequently perform tasks collaboratively in daily life. Collaborating with others may or may not result in higher task performance than if one were to complete the task alone (i.e., a collective benefit). A recent study on collective benefits in perceptual decision-making showed that dyad members with similar individual performances attain collective benefit. However, little is known about the physiological basis of these results. Here, we replicate this earlier work and also investigate the neurophysiological correlates of decision-making using EEG. In a two-interval forced-choice task, co-actors individually indicated presence of a target stimulus with a higher contrast and then indicated their confidence on a rating scale. Viewing the individual ratings, dyads made a joint decision. Replicating earlier work, we found a positive correlation between the similarity of individual performances and collective benefit. We analyzed event-related potentials (ERPs) in three phases (i.e., stimulus onset, response and feedback) using explorative cluster mass permutation tests. At stimulus onset, ERPs were significantly linearly related to our manipulation of contrast differences, validating our manipulation of task difficulty. For individual and joint responses, we found a significant centro-parietal error-related positivity for correct versus incorrect responses, which suggests that accuracy is already evaluated at the response level. At feedback presentation, we found a significant late positive fronto-central potential elicited by incorrect joint responses. In sum, these results demonstrate that response- and feedback-related components elicited by an error-monitoring system differentially integrate conflicting information exchanged during the joint decision-making process.

The spatiotemporal neural dynamics underlying perceived similarity for real-world objects

The degree to which we perceive real-world objects as similar or dissimilar structures our perception and guides categorization behavior. Here, we investigated the neural representations enabling perceived similarity using behavioral judgments, fMRI and MEG. As different object dimensions co-occur and partly correlate, to understand the relationship between perceived similarity and brain activity it is necessary to assess the unique role of multiple object dimensions. We thus behaviorally assessed perceived object similarity in relation to shape, function, color and background. We then used representational similarity analyses to relate these behavioral judgments to brain activity. We observed a link between each object dimension and representations in visual cortex. These representations emerged rapidly within 200 ms of stimulus onset. Assessing the unique role of each object dimension revealed partly overlapping and distributed representations: while color-related representations distinctly preceded shape-related representations both in the processing hierarchy of the ventral visual pathway and in time, several dimensions were linked to high-level ventral visual cortex. Further analysis singled out the shape dimension as neither fully accounted for by supra-category membership, nor a deep neural network trained on object categorization. Together our results comprehensively characterize the relationship between perceived similarity of key object dimensions and neural activity.

Detecting modular brain states in rest and task

The human brain is a dynamic networked system that continually reconfigures its functional connectivity patterns over time. Thus, developing approaches able to adequately detect fast brain dynamics is critical. Of particular interest are the methods that analyze the modular structure of brain networks, that is, the presence of clusters of regions that are densely interconnected. In this paper, we propose a novel framework to identify fast modular states that dynamically fluctuate over time during rest and task. We started by demonstrating the feasibility and relevance of this framework using simulated data. Compared with other methods, our algorithm was able to identify the simulated networks with high temporal and spatial accuracies. We further applied the proposed framework on MEG data recorded during a finger movement task, identifying modular states linking somatosensory and primary motor regions. The algorithm was also performed on dense-EEG data recorded during a picture naming task, revealing the subsecond transition between several modular states that relate to visual processing, semantic processing, and language. Next, we tested our method on a dataset of resting-state dense-EEG signals recorded from 124 patients with Parkinson's disease. Results disclosed brain modular states that differentiate cognitively intact patients, patients with moderate cognitive deficits, and patients with severe cognitive deficits. Our new approach complements classical methods, offering a new way to track the brain modular states, in healthy subjects and patients, on an adequate task-specific timescale.

Electrophysiological indexes of option characteristic processing

Decision making is vital to human behavior and can be divided into multiple stages including option assessment, behavioral output, and feedback evaluation. Studying how people evaluate option characteristics in the option assessment stage would provide important knowledge on human decision making. Using the event-related potential (ERP) method, the present study investigated the neural mechanism of evaluating two types of option characteristics (i.e., reward magnitude and degree of uncertainty) in the temporal dimension. Thirty-five volunteers participated in a monetary gambling task, where they either accepted or rejected gambles. The ERP results showed a double dissociation pattern, with the early P1 component being sensitive to magnitude but insensitive to degree of uncertainty, while both the N2 and P3 components showed the opposite pattern. The results suggest that these two fundamental option features are assessed rapidly and separately in the human brain. Specifically, small magnitude elicited a larger P1 than did large magnitude, indicating that the perceptual and attentional processing of options is modulated by magnitude. Both the N2 and P3 amplitudes evoked by the risky context were larger than those evoked by the ambiguous one, reflecting that more cognitive conflicts and resources are involved in the former condition. Furthermore, the P1, but not the N2 or P3, amplitude was sensitive to decisions, suggesting that early attentional processes may contribute to human decision making. These findings may provide insight into the temporal mechanisms of option characteristic processing.

Amygdala activation during unconscious visual processing of food

Hedonic or emotional responses to food have important positive and negative effects on human life. Behavioral studies have shown that hedonic responses to food images are elicited rapidly, even in the absence of conscious awareness of food. Although a number of previous neuroimaging studies investigated neural activity during conscious processing of food images, the neural mechanisms underlying unconscious food processing remain unknown. To investigate this issue, we measured neural activity using functional magnetic resonance imaging while participants viewed food and mosaic images presented subliminally and supraliminally. Conjunction analyses revealed that the bilateral amygdala was more strongly activated in response to food images than to mosaic images under both subliminal and supraliminal conditions. Interaction analyses revealed that the broad bilateral posterior regions, peaking at the posterior fusiform gyrus, were particularly active when participants viewed food versus mosaic images under the supraliminal compared with the subliminal condition. Dynamic causal modeling analyses supported the model in which the subcortical visual pathway from the pulvinar to the amygdala was modulated by food under the subliminal condition; in contrast, the model in which both subcortical and cortical (connecting the primary visual cortex, fusiform gyrus, and the amygdala) visual pathways were modulated by food received the most support under the supraliminal condition. These results suggest the possibility that unconscious hedonic responses to food may exert an effect through amygdala activation via the subcortical visual pathway.

Rapid Extraction of Emotion Regularities from Complex Scenes in the Human Brain

Adaptive behavior requires the rapid extraction of behaviorally relevant information in the environment, with particular emphasis on emotional cues. However, the speed of emotional feature extraction from complex visual environments is largely undetermined. Here we use objective electrophysiological recordings in combination with frequency tagging to demonstrate that the extraction of emotional information from neutral, pleasant, or unpleasant naturalistic scenes can be completed at a presentation speed of 167 ms (i.e., 6 Hz) under high perceptual load. Emotional compared to neutral pictures evoked enhanced electrophysiological responses with distinct topographical activation patterns originating from different neural sources. Cortical facilitation in early visual cortex was also more pronounced for scenes with pleasant compared to unpleasant or neutral content, suggesting a positivity offset mechanism dominating under conditions of rapid scene processing. These results significantly advance our knowledge of complex scene processing in demonstrating rapid integrative content identification, particularly for emotional cues relevant for adaptive behavior in complex environments.

The nature of visual awareness at stimulus energy and feature levels: A backward masking study

The level of processing (LoP) hypothesis proposes that low-level stimulus perception (i.e., stimulus energy and features) is a graded process whereas high-level (i.e., letters, words, meaning) stimulus perception is all-or-none. In the present study, we set up a visual masking design in order to examine the nature of visual awareness at stimulus energy (i.e., detection task) and feature levels (identification task) at specific individual target durations (13, 27, 40, 53, and 80 ms). We manipulated the strength of the masking to produce different visibility conditions and gathered participants' subjective (across a 4-point awareness scale) and objective (accuracy levels) awareness performances. We found that intermediate ratings (i.e., ratings 2 and 3, which index graded awareness experiences) were used in more than 50% of the trials for target presentations of 27, 40, 53, and 80 ms. In addition, objective accuracy performances for target presentations of 27 and 80 ms produced linearly increasing detection and identification accuracies across the awareness scale categories, respectively. Overall, our results suggest that visual awareness at energy and feature levels of stimulus perception may be graded. Furthermore, we found a divergence in detection and identification performance results, which emphasizes the need for an adequate election of target durations when studying different perceptual processes such as detection versus more complex stimulus identification processes. Finally, "clarity" in the perceptual awareness scale should be exhaustively defined depending on the level of processing of the stimulus, as participants may recalibrate the meaning of the different awareness categories depending on task demands.

The perceptual neural trace of memorable unseen scenes

Some scenes are more memorable than others: they cement in minds with consistencies across observers and time scales. While memory mechanisms are traditionally associated with the end stages of perception, recent behavioral studies suggest that the features driving these memorability effects are extracted early on, and in an automatic fashion. This raises the question: is the neural signal of memorability detectable during early perceptual encoding phases of visual processing? Using the high temporal resolution of magnetoencephalography (MEG), during a rapid serial visual presentation (RSVP) task, we traced the neural temporal signature of memorability across the brain. We found an early and prolonged memorability related signal under a challenging ultra-rapid viewing condition, across a network of regions in both dorsal and ventral streams. This enhanced encoding could be the key to successful storage and recognition.

Masking the Integration of Complementary Shape Cues

Retinal and cortical mechanisms provide for persistence of visual information across intervals of many hundreds of milliseconds, which supports the integration of partial shape cues. The present experiments displayed unknown shapes in a match recognition task, wherein a target shape was quickly followed by a comparison shape; the task was to specify whether the comparison shape was the same or different from the target. The target and comparison shapes were displayed as sparse dots that marked boundary locations. The first experiment successively displayed the target shape as two complementary subsets and found that the probability of correct match remained above chance with up to 500 ms of subset separation. The second experiment demonstrated masking of the target by a random pattern of dots when the target and mask were displayed simultaneously, but with much less or no masking when the two were separated by 100 ms. The third experiment displayed the target subsets with 200 ms of separation and found that match recognition was disrupted when the random-dot mask was displayed midway between the two subsets. Much less masking of an intact target was produced with that amount of temporal separation, which suggests that mechanisms for integration of shape cues have a special vulnerability to masking. The third experiment also found very little impairment of match recognition when the mask was displayed simultaneous with one of the subsets. We hypothesize that there is embedding of the subset pattern within the mask pattern, but additional display of the other subset effectively disembeds the buried partial shape cues.

Reliability and Generalizability of Similarity-Based Fusion of MEG and fMRI Data in Human Ventral and Dorsal Visual Streams

To build a representation of what we see, the human brain recruits regions throughout the visual cortex in cascading sequence. Recently, an approach was proposed to evaluate the dynamics of visual perception in high spatiotemporal resolution at the scale of the whole brain. This method combined functional magnetic resonance imaging (fMRI) data with magnetoencephalography (MEG) data using representational similarity analysis and revealed a hierarchical progression from primary visual cortex through the dorsal and ventral streams. To assess the replicability of this method, we here present the results of a visual recognition neuro-imaging fusion experiment and compare them within and across experimental settings. We evaluated the reliability of this method by assessing the consistency of the results under similar test conditions, showing high agreement within participants. We then generalized these results to a separate group of individuals and visual input by comparing them to the fMRI-MEG fusion data of Cichy et al (2016), revealing a highly similar temporal progression recruiting both the dorsal and ventral streams. Together these results are a testament to the reproducibility of the fMRI-MEG fusion approach and allows for the interpretation of these spatiotemporal dynamic in a broader context.

Integrated Cognitive Assessment: Speed and Accuracy of Visual Processing as a Reliable Proxy to Cognitive Performance

Various mental disorders are accompanied by some degree of cognitive impairment. Particularly in neurodegenerative disorders, cognitive impairment is the phenotypical hallmark of the disease. Effective, accurate and timely cognitive assessment is key to early diagnosis of this family of mental disorders. Current standard-of-care techniques for cognitive assessment are primarily paper-based, and need to be administered by a healthcare professional; they are additionally language and education-dependent and typically suffer from a learning bias. These tests are thus not ideal for large-scale pro-active cognitive screening and disease progression monitoring. We developed the Integrated Cognitive Assessment (referred to as CGN_ICA), a 5-minute computerized cognitive assessment tool based on a rapid visual categorization task, in which a series of carefully selected natural images of varied difficulty are presented to participants. Overall 448 participants, across a wide age-range with different levels of education took the CGN_ICA test. We compared participants' CGN_ICA test results with a variety of standard pen-and-paper tests, such as Symbol Digit Modalities Test (SDMT) and Montreal Cognitive Assessment (MoCA), that are routinely used to assess cognitive performance. CGN_ICA had excellent test-retest reliability, showed convergent validity with the standard-of-care cognitive tests used here, and demonstrated to be suitable for micro-monitoring of cognitive performance.

Dynamic Construction of Reduced Representations in the Brain for Perceptual Decision Behavior

Over the past decade, extensive studies of the brain regions that support face, object, and scene recognition suggest that these regions have a hierarchically organized architecture that spans the occipital and temporal lobes [1-14], where visual categorizations unfold over the first 250 ms of processing [15-19]. This same architecture is flexibly involved in multiple tasks that require task-specific representations-e.g. categorizing the same object as "a car" or "a Porsche." While we partly understand where and when these categorizations happen in the occipito-ventral pathway, the next challenge is to unravel how these categorizations happen. That is, how does high-dimensional input collapse in the occipito-ventral pathway to become low dimensional representations that guide behavior? To address this, we investigated what information the brain processes in a visual perception task and visualized the dynamic representation of this information in brain activity. To do so, we developed stimulus information representation (SIR), an information theoretic framework, to tease apart stimulus information that supports behavior from that which does not. We then tracked the dynamic representations of both in magneto-encephalographic (MEG) activity. Using SIR, we demonstrate that a rapid (∼170 ms) reduction of behaviorally irrelevant information occurs in the occipital cortex and that representations of the information that supports distinct behaviors are constructed in the right fusiform gyrus (rFG). Our results thus highlight how SIR can be used to investigate the component processes of the brain by considering interactions between three variables (stimulus information, brain activity, behavior), rather than just two, as is the current norm.

Categorization Goals Modulate the Use of Natural Scene Statistics

Understanding natural scenes involves the contribution of bottom–up analysis and top–down modulatory processes. However, the interaction of these processes during the categorization of natural scenes is not well understood. In the current study, we approached this issue using ERPs and behavioral and computational data. We presented pictures of natural scenes and asked participants to categorize them in response to different questions (Is it an animal/vehicle? Is it indoors/outdoors? Are there one/two foreground elements?). ERPs for target scenes requiring a “yes” response began to differ from those of nontarget scenes, beginning at 250 msec from picture onset, and this ERP difference was unmodulated by the categorization questions. Earlier ERPs showed category-specific differences (e.g., between animals and vehicles), which were associated with the processing of scene statistics. From 180 msec after scene onset, these category-specific ERP differences were modulated by the categorization question that was asked. Categorization goals do not modulate only later stages associated with target/nontarget decision but also earlier perceptual stages, which are involved in the processing of scene statistics.

We see the orange not the lemon: typicality effects in ultra-rapid categorization in adults with and without autism spectrum disorder

Semantic meaning can be extracted from pictures presented very briefly, in the order of tens of milliseconds. This ultra-rapid categorization processing appears to respect a coarse-to-fine path where lower level representations of concepts, or more detailed information, need additional time. We question whether variations in the levels of typicality of the target-item would implicate additional processing for correct classification, both in neurotypical (NT) individuals and with autism spectrum disorder (ASD). Previous research in ASD points out that atypical exemplars of a category might be abnormally processed (e.g., longer times in identifying a penguin as a bird), an observation that we further tested with a rapid serial visual presentation (RSVP) task. In this study, we applied a RSVP task, with four different presentation times (13, 27, 50, and 80 ms) and with typical and atypical exemplars to a group of NT individuals and a sample of individuals with ASD. We found, overall, a strong effect of typicality with a higher detection rate for typical items. In addition, we observed a group × typicality × duration interaction. We interpret these findings in the light of the competences of the feedforward sweep of information through our visual system.

Scene complexity modulates degree of feedback activity during object detection in natural scenes

Selective brain responses to objects arise within a few hundreds of milliseconds of neural processing, suggesting that visual object recognition is mediated by rapid feed-forward activations. Yet disruption of neural responses in early visual cortex beyond feed-forward processing stages affects object recognition performance. Here, we unite these discrepant findings by reporting that object recognition involves enhanced feedback activity (recurrent processing within early visual cortex) when target objects are embedded in natural scenes that are characterized by high complexity. Human participants performed an animal target detection task on natural scenes with low, medium or high complexity as determined by a computational model of low-level contrast statistics. Three converging lines of evidence indicate that feedback was selectively enhanced for high complexity scenes. First, functional magnetic resonance imaging (fMRI) activity in early visual cortex (V1) was enhanced for target objects in scenes with high, but not low or medium complexity. Second, event-related potentials (ERPs) evoked by target objects were selectively enhanced at feedback stages of visual processing (from ~220 ms onwards) for high complexity scenes only. Third, behavioral performance for high complexity scenes deteriorated when participants were pressed for time and thus less able to incorporate the feedback activity. Modeling of the reaction time distributions using drift diffusion revealed that object information accumulated more slowly for high complexity scenes, with evidence accumulation being coupled to trial-to-trial variation in the EEG feedback response. Together, these results suggest that while feed-forward activity may suffice to recognize isolated objects, the brain employs recurrent processing more adaptively in naturalistic settings, using minimal feedback for simple scenes and increasing feedback for complex scenes.

A gateway to consumers' minds: Achievements, caveats, and prospects of electroencephalography-based prediction in neuromarketing

In the last decade, the field of consumer neuroscience, or neuromarketing, has been flourishing, with numerous publications, academic programs, initiatives, and companies. The demand for objective neural measures to quantify consumers' preferences and predict responses to marketing campaigns is ever on the rise, particularly due to the limitations of traditional marketing techniques, such as questionnaires, focus groups, and interviews. However, research has yet to converge on a unified methodology or conclusive results that can be applied in the industry. In this review, we present the potential of electroencephalography (EEG)-based preference prediction. We summarize previous EEG research and propose features which have shown promise in capturing the consumers' evaluation process, including components acquired from an event-related potential design, inter-subject correlations, hemispheric asymmetry, and various spectral band powers. Next, we review the latest findings on attempts to predict preferences based on various features of the EEG signal. Finally, we conclude with several recommended guidelines for prediction. Chiefly, we stress the need to demonstrate that neural measures contribute to preference prediction beyond what traditional measures already provide. Second, prediction studies in neuromarketing should adopt the standard practices and methodology used in data science and prediction modeling that is common in other fields such as computer science and engineering. This article is categorized under: Economics > Interactive Decision-Making Economics > Individual Decision-Making Psychology > Prediction Neuroscience > Cognition.

Context-Sensitive Computational Mechanisms of Decision Making

Real-world information is primarily sensory in nature, and understandably people attach value to the sensory information to prepare for appropriate behavioral responses. This review presents research from value-based, perceptual, and social decision-making domains, so far studied using isolated paradigms and their corresponding computational models. For example, in perceptual decision making, the sensory evidence accumulation rather than value computation becomes central to choice behavior. Furthermore, we identify cross-linkages between the perceptual and value-based domains to help us better understand generic processes pertaining to individual decision making. The purpose of this review is 2-fold. First, we identify the need for integrated study of different domains of decision making. Second, given that both our perception and valuation are influenced by the surrounding context, we suggest the integration of different types of information in decision making could be done by studying contextual influences in decision making. Future research needs to attempt toward a system-level understanding of various subprocesses involved in decision making.

Early sensitivity of evoked potentials to surface and volumetric structure during the visual perception of three-dimensional object shape

This study used event-related potentials (ERPs) to elucidate how the human visual system processes three-dimensional (3-D) object shape structure. In particular, we examined whether the perceptual mechanisms that support the analysis of 3-D shape are differentially sensitive to higher order surface and volumetric part structure. Observers performed a whole-part novel object matching task in which part stimuli comprised sub-regions of closed edge contour, surfaces or volumetric parts. Behavioural response latency data showed an advantage in matching surfaces and volumetric parts to whole objects over contours, but no difference between surfaces and volumes. ERPs were analysed using a convergence of approaches based on stimulus dependent amplitude modulations of evoked potentials, topographic segmentation, and spatial frequency oscillations. The results showed early differential perceptual processing of contours, surfaces, and volumetric part stimuli. This was first reliably observed over occipitoparietal electrodes during the N1 (140-200 ms) with a mean peak latency of 170 ms, and continued on subsequent P2 (220-260 ms) and N2 (260-320 ms) components. The differential sensitivity in perceptual processing during the N1 was accompanied by distinct microstate patterns that distinguished among contours, surfaces and volumes, and predominant theta band activity around 4-7 Hz over right occipitoparietal and orbitofrontal sites. These results provide the first evidence of early differential perceptual processing of higher order surface and volumetric shape structure within the first 200 ms of stimulus processing. The findings challenge theoretical models of object recognition that do not attribute functional significance to surface and volumetric object structure during visual perception.

Tracking feature-based attention

OBJECTIVE

Feature-based attention (FBA) helps one detect objects with a particular color, motion, or orientation. FBA works globally; the attended feature is enhanced at all positions in the visual field. This global property of FBA lets one use stimuli presented in the peripheral visual field to track attention in a task presented centrally. The present study explores the use of SSVEPs, generated by flicker presented peripherally, to track attention in a visual search task presented centrally. We evaluate whether this use of EEG to track FBA is robust enough to track attention when performing visual search within a dynamic 3D environment presented with a head-mounted display (HMD).

APPROACH

Observers first performed a visual search task presented in the central visual field within a stationary virtual environment. The purpose of this first experiment was to establish whether flicker presented peripherally can produce SSVEPs during HMD use. The second experiment placed observers in a dynamic virtual environment in which observers moved around a racetrack. Peripheral flicker was again used to track attention to the color of the target in the visual search task.

MAIN RESULTS

SSVEPs produced by flicker in the peripheral visual field are influenced strongly by attention in observers with stationary or moving viewpoints. Offline classification results show that one can track an observer's attended color, which suggests that these methods may provide a viable means for tracking FBA in a real-time task.

SIGNIFICANCE

Current FBA and brain-computer interface (BCI) studies primarily use foveal flicker to produce SSVEP responses. The present study's finding that one can use peripherally-presented flicker to track attention in dynamic virtual environments promises a more flexible and practical approach to BCIs based on FBA.

Briefly Flashed Scenes Can Be Stored in Long-Term Memory

The capacity of human memory is impressive. Previous reports have shown that when asked to memorize images, participants can recognize several thousands of visual objects in great details even with a single viewing of a few seconds per image. In this experiment, we tested recognition performance for natural scenes that participants saw for 20 ms only once (untrained group) or 22 times over many days (trained group) in an unrelated task. 400 images (200 previously viewed and 200 novel images) were flashed one at a time and participants were asked to lift their finger from a pad whenever they thought they had already seen the image (go/no-go paradigm). Compared to previous reports of excellent recognition performance with only single presentations of a few seconds, untrained participants were able to recognize only 64% of the 200 images they had seen few minutes before. On the other hand, trained participants, who had processed the flashed images (20 ms) several times, could correctly recognize 89% of them. EEG recordings confirmed these behavioral results. As early as 230 ms after stimulus onset, a significant event-related-potential (ERP) difference between familiar and new images was observed for the trained but not for the untrained group. These results show that briefly flashed unmasked scenes can be incidentally stored in long-term memory when repeated.

Paving the Way to Eureka-Introducing "Dira" as an Experimental Paradigm to Observe the Process of Creative Problem Solving

"Dira" is a novel experimental paradigm to record combinations of behavioral and metacognitive measures for the creative process. This task allows assessing chronological and chronometric aspects of the creative process directly and without a detour through creative products or proxy phenomena. In a study with 124 participants we show that (a) people spend more time attending to selected vs. rejected potential solutions, (b) there is a clear connection between behavioral patterns and self-reported measures, (c) the reported intensity of Eureka experiences is a function of interaction time with potential solutions, and (d) experiences of emerging solutions can happen immediately after engaging with a problem, before participants explore all potential solutions. The conducted study exemplifies how "Dira" can be used as an instrument to narrow down the moment when solutions emerge. We conclude that the "Dira" experiment is paving the way to study the process, as opposed to the product, of creative problem solving.

Spatial frequency supports the emergence of categorical representations in visual cortex during natural scene perception

In navigating our environment, we rapidly process and extract meaning from visual cues. However, the relationship between visual features and categorical representations in natural scene perception is still not well understood. Here, we used natural scene stimuli from different categories and filtered at different spatial frequencies to address this question in a passive viewing paradigm. Using representational similarity analysis (RSA) and cross-decoding of magnetoencephalography (MEG) data, we show that categorical representations emerge in human visual cortex at ∼180 ms and are linked to spatial frequency processing. Furthermore, dorsal and ventral stream areas reveal temporally and spatially overlapping representations of low and high-level layer activations extracted from a feedforward neural network. Our results suggest that neural patterns from extrastriate visual cortex switch from low-level to categorical representations within 200 ms, highlighting the rapid cascade of processing stages essential in human visual perception.