Single-trial analysis of neuroimaging data: inferring neural networks underlying perceptual decision-making in the human brain

Saccade size predicts onset time of object processing during visual search of an open world virtual environment

OBJECTIVE

To date the vast majority of research in the visual neurosciences have been forced to adopt a highly constrained perspective of the vision system in which stimuli are processed in an open-loop reactive fashion (i.e., abrupt stimulus presentation followed by an evoked neural response). While such constraints enable high construct validity for neuroscientific investigation, the primary outcomes have been a reductionistic approach to isolate the component processes of visual perception. In electrophysiology, of the many neural processes studied under this rubric, the most well-known is, arguably, the P300 evoked response. There is, however, relatively little known about the real-world corollary of this component in free-viewing paradigms where visual stimuli are connected to neural function in a closed-loop. While growing evidence suggests that neural activity analogous to the P300 does occur in such paradigms, it is an open question when this response occurs and what behavioral or environmental factors could be used to isolate this component.

APPROACH

The current work uses convolutional networks to decode neural signals during a free-viewing visual search task in a closed-loop paradigm within an open-world virtual environment. From the decoded activity we construct fixation-locked response profiles that enable estimations of the variable latency of any P300 analogue around the moment of fixation. We then use these estimates to investigate which factors best reduce variable latency and, thus, predict the onset time of the response. We consider measurable, search-related factors encompassing top-down (i.e., goal driven) and bottom-up (i.e., stimulus driven) processes, such as fixation duration and salience. We also consider saccade size as an intermediate factor reflecting the integration of these two systems.

MAIN RESULTS

The results show that of these factors only saccade size reliably determines the onset time of P300 analogous activity for this task. Specifically, we find that for large saccades the variability in response onset is small enough to enable analysis using traditional ensemble averaging methods.

SIGNIFICANCE

The results show that P300 analogous activity does occur during closed-loop, free-viewing visual search while highlighting distinct differences between the open-loop version of this response and its real-world analogue. The results also further establish saccades, and saccade size, as a key factor in real-world visual processing.

Modality-specific impacts of distractors on visual and auditory categorical decision-making: an evidence accumulation perspective

Our brain constantly processes multisensory inputs to make decisions and guide behaviors, but how goal-relevant processes are influenced by irrelevant information is unclear. Here, we investigated the effects of intermodal and intramodal task-irrelevant information on visual and auditory categorical decision-making. In both visual and auditory tasks, we manipulated the modality of irrelevant inputs (visual vs. auditory vs. none) and used linear discrimination analysis of EEG and hierarchical drift-diffusion modeling (HDDM) to identify when and how task-irrelevant information affected decision-relevant processing. The results revealed modality-specific impacts of irrelevant inputs on visual and auditory categorical decision-making. The distinct effects on the visual task were shown on the neural components, with auditory distractors amplifying the sensory processing whereas visual distractors amplifying the post-sensory process. Conversely, the distinct effects on the auditory task were shown in behavioral performance and underlying cognitive processes. Visual distractors facilitate behavioral performance and affect both stages, but auditory distractors interfere with behavioral performance and impact on the sensory processing rather than the post-sensory decision stage. Overall, these findings suggested that auditory distractors affect the sensory processing stage of both tasks while visual distractors affect the post-sensory decision stage of visual categorical decision-making and both stages of auditory categorical decision-making. This study provides insights into how humans process information from multiple sensory modalities during decision-making by leveraging modality-specific impacts.

Distinct spatiotemporal brainstem pathways of outcome valence during reward- and punishment-based learning

Learning to seek rewards and avoid punishments, based on positive and negative choice outcomes, is essential for human survival. Yet, the neural underpinnings of outcome valence in the human brainstem and the extent to which they differ in reward and punishment learning contexts remain largely elusive. Here, using simultaneously acquired electroencephalography and functional magnetic resonance imaging data, we show that during reward learning the substantia nigra (SN)/ventral tegmental area (VTA) and locus coeruleus are initially activated following negative outcomes, while the VTA subsequently re-engages exhibiting greater responses for positive than negative outcomes, consistent with an early arousal/avoidance response and a later value-updating process, respectively. During punishment learning, we show that distinct raphe nucleus and SN subregions are activated only by negative outcomes with a sustained post-outcome activity across time, supporting the involvement of these brainstem subregions in avoidance behavior. Finally, we demonstrate that the coupling of these brainstem structures with other subcortical and cortical areas helps to shape participants' serial choice behavior in each context.

Human behavior in free search online shopping scenarios can be predicted from EEG activation using Hjorth parameters

The present work investigates whether and how decisions in real-world online shopping scenarios can be predicted based on brain activation. Potential customers were asked to search through product pages on e-commerce platforms and decide, which products to buy, while their EEG signal was recorded. Machine learning algorithms were then trained to distinguish between EEG activation when viewing products that are later bought or put into the shopping card as opposed to products that are later discarded. We find that Hjorth parameters extracted from the raw EEG can be used to predict purchase choices to a high level of accuracy. Above-chance predictions based on Hjorth parameters are achieved via different standard machine learning methods with random forest models showing the best performance of above 80% prediction accuracy in both 2-class (bought or put into card vs. not bought) and 3-class (bought vs. put into card vs. not bought) classification. While conventional EEG signal analysis commonly employs frequency domain features such as alpha or theta power and phase, Hjorth parameters use time domain signals, which can be calculated rapidly with little computational cost. Given the presented evidence that Hjorth parameters are suitable for the prediction of complex behaviors, their potential and remaining challenges for implementation in real-time applications are discussed.

Conceptualizing and Examining Change in Communication Research

Communication research often focuses on processes of communication, such as how messages impact individuals over time or how interpersonal relationships develop and change. Despite their importance, these change processes are often implicit in much theoretical and empirical work in communication. Intensive longitudinal data are becoming increasingly feasible to collect and, when coupled with appropriate analytic frameworks, enable researchers to better explore and articulate the types of change underlying communication processes. To facilitate the study of change processes, we (a) describe advances in data collection and analytic methods that allow researchers to articulate complex change processes of phenomena in communication research, (b) provide an overview of change processes and how they may be captured with intensive longitudinal methods, and (c) discuss considerations of capturing change when designing and implementing studies. We are excited about the future of studying processes of change in communication research, and we look forward to the iterations between empirical tests and theory revision that will occur as researchers delve into studying change within communication processes.

A Common Neural Account for Social and Nonsocial Decisions

To date, social and nonsocial decisions have been studied largely in isolation. Consequently, the extent to which social and nonsocial forms of decision uncertainty are integrated using shared neurocomputational resources remains elusive. Here, we address this question using simultaneous electroencephalography (EEG)-functional magnetic resonance imaging (fMRI) in healthy human participants (young adults of both sexes) and a task in which decision evidence in social and nonsocial contexts varies along comparable scales. First, we identify time-resolved build-up of activity in the EEG, akin to a process of evidence accumulation (EA), across both contexts. We then use the endogenous trial-by-trial variability in the slopes of these accumulating signals to construct parametric fMRI predictors. We show that a region of the posterior-medial frontal cortex (pMFC) uniquely explains trial-wise variability in the process of evidence accumulation in both social and nonsocial contexts. We further demonstrate a task-dependent coupling between the pMFC and regions of the human valuation system in dorso-medial and ventro-medial prefrontal cortex across both contexts. Finally, we report domain-specific representations in regions known to encode the early decision evidence for each context. These results are suggestive of a domain-general decision-making architecture, whereupon domain-specific information is likely converted into a "common currency" in medial prefrontal cortex and accumulated for the decision in the pMFC.SIGNIFICANCE STATEMENT Little work has directly compared social-versus-nonsocial decisions to investigate whether they share common neurocomputational origins. Here, using combined electroencephalography (EEG)-functional magnetic resonance imaging (fMRI) and computational modeling, we offer a detailed spatiotemporal account of the neural underpinnings of social and nonsocial decisions. Specifically, we identify a comparable mechanism of temporal evidence integration driving both decisions and localize this integration process in posterior-medial frontal cortex (pMFC). We further demonstrate task-dependent coupling between the pMFC and regions of the human valuation system across both contexts. Finally, we report domain-specific representations in regions encoding the early, domain-specific, decision evidence. These results suggest a domain-general decision-making architecture, whereupon domain-specific information is converted into a common representation in the valuation system and integrated for the decision in the pMFC.

At-risk alcohol users have disrupted valence discrimination during reward anticipation

Alcohol use disorder is characterised by disrupted reward learning, underpinned by dysfunctional cortico-striatal reward pathways, although relatively little is known about the biology of reward processing in populations who engage in risky alcohol use. Cues that trigger reward anticipation can be categorized according to their learnt valence (i.e., positive vs. negative outcomes) and motivational salience (i.e., incentive vs. neutral cues). Separating EEG signals associated with these dimensions is challenging because of their inherent collinearity, but the recent application of machine learning methods to single EEG trials affords a solution. Here, the Alcohol Use Disorders Identification Test (AUDIT) was used to quantify risky alcohol use, with participants split into high alcohol (HA) (n = 22, mean AUDIT score: 13.82) and low alcohol (LA) (n = 22, mean AUDIT score: 5.77) groups. We applied machine learning multivariate single-trial classification to the electroencephalography (EEG) data collected during reward anticipation. The LA group demonstrated significant valence discrimination in the early stages of reward anticipation within the cue-P3 time window (400-550 ms), whereas the HA group was insensitive to valence within this time window. Notably, the LA, but not the HA group demonstrated a relationship between single-trial variability in the early valence component and reaction times for gain and loss trials. This study evidences disrupted hypoactive valence sensitivity in the HA group, revealing potential neurophysiological markers for risky drinking behaviours which place individuals at-risk of adverse health events.

Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

When exposed to complementary features of information across sensory modalities, our brains formulate cross-modal associations between features of stimuli presented separately to multiple modalities. For example, auditory pitch-visual size associations map high-pitch tones with small-size visual objects, and low-pitch tones with large-size visual objects. Preferential, or congruent, cross-modal associations have been shown to affect behavioural performance, i.e. choice accuracy and reaction time (RT) across multisensory decision-making paradigms. However, the neural mechanisms underpinning such influences in perceptual decision formation remain unclear. Here, we sought to identify when perceptual improvements from associative congruency emerge in the brain during decision formation. In particular, we asked whether such improvements represent ‘early’ sensory processing benefits, or ‘late’ post-sensory changes in decision dynamics. Using a modified version of the Implicit Association Test (IAT), coupled with electroencephalography (EEG), we measured the neural activity underlying the effect of auditory stimulus-driven pitch-size associations on perceptual decision formation. Behavioural results showed that participants responded significantly faster during trials when auditory pitch was congruent, rather than incongruent, with its associative visual size counterpart. We used multivariate Linear Discriminant Analysis (LDA) to characterise the spatiotemporal dynamics of EEG activity underpinning IAT performance. We found an ‘Early’ component (∼100–110 ms post-stimulus onset) coinciding with the time of maximal discrimination of the auditory stimuli, and a ‘Late’ component (∼330–340 ms post-stimulus onset) underlying IAT performance. To characterise the functional role of these components in decision formation, we incorporated a neurally-informed Hierarchical Drift Diffusion Model (HDDM), revealing that the Late component decreases response caution, requiring less sensory evidence to be accumulated, whereas the Early component increased the duration of sensory-encoding processes for incongruent trials. Overall, our results provide a mechanistic insight into the contribution of ‘early’ sensory processing, as well as ‘late’ post-sensory neural representations of associative congruency to perceptual decision formation.

Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making

Despite recent progress in understanding multisensory decision-making, a conclusive mechanistic account of how the brain translates the relevant evidence into a decision is lacking. Specifically, it remains unclear whether perceptual improvements during rapid multisensory decisions are best explained by sensory (i.e., ‘Early’) processing benefits or post-sensory (i.e., ‘Late’) changes in decision dynamics. Here, we employ a well-established visual object categorisation task in which early sensory and post-sensory decision evidence can be dissociated using multivariate pattern analysis of the electroencephalogram (EEG). We capitalize on these distinct neural components to identify when and how complementary auditory information influences the encoding of decision-relevant visual evidence in a multisensory context. We show that it is primarily the post-sensory, rather than the early sensory, EEG component amplitudes that are being amplified during rapid audiovisual decision-making. Using a neurally informed drift diffusion model we demonstrate that a multisensory behavioral improvement in accuracy arises from an enhanced quality of the relevant decision evidence, as captured by the post-sensory EEG component, consistent with the emergence of multisensory evidence in higher-order brain areas.

A conclusive account on how the brain translates audiovisual evidence into a rapid decision is still lacking. Here, using a neurally-informed modelling approach, the authors show that sounds amplify visual evidence later in the decision process, in line with higher-order multisensory effects.

The Decision Decoding ToolBOX (DDTBOX) - A Multivariate Pattern Analysis Toolbox for Event-Related Potentials

In recent years, neuroimaging research in cognitive neuroscience has increasingly used multivariate pattern analysis (MVPA) to investigate higher cognitive functions. Here we present DDTBOX, an open-source MVPA toolbox for electroencephalography (EEG) data. DDTBOX runs under MATLAB and is well integrated with the EEGLAB/ERPLAB and Fieldtrip toolboxes (Delorme and Makeig 2004; Lopez-Calderon and Luck 2014; Oostenveld et al. 2011). It trains support vector machines (SVMs) on patterns of event-related potential (ERP) amplitude data, following or preceding an event of interest, for classification or regression of experimental variables. These amplitude patterns can be extracted across space/electrodes (spatial decoding), time (temporal decoding), or both (spatiotemporal decoding). DDTBOX can also extract SVM feature weights, generate empirical chance distributions based on shuffled-labels decoding for group-level statistical testing, provide estimates of the prevalence of decodable information in the population, and perform a variety of corrections for multiple comparisons. It also includes plotting functions for single subject and group results. DDTBOX complements conventional analyses of ERP components, as subtle multivariate patterns can be detected that would be overlooked in standard analyses. It further allows for a more explorative search for information when no ERP component is known to be specifically linked to a cognitive process of interest. In summary, DDTBOX is an easy-to-use and open-source toolbox that allows for characterising the time-course of information related to various perceptual and cognitive processes. It can be applied to data from a large number of experimental paradigms and could therefore be a valuable tool for the neuroimaging community.

Human VMPFC encodes early signatures of confidence in perceptual decisions

Choice confidence, an individual’s internal estimate of judgment accuracy, plays a critical role in adaptive behaviour, yet its neural representations during decision formation remain underexplored. Here, we recorded simultaneous EEG-fMRI while participants performed a direction discrimination task and rated their confidence on each trial. Using multivariate single-trial discriminant analysis of the EEG, we identified a stimulus-independent component encoding confidence, which appeared prior to subjects’ explicit choice and confidence report, and was consistent with a confidence measure predicted by an accumulation-to-bound model of decision-making. Importantly, trial-to-trial variability in this electrophysiologically-derived confidence signal was uniquely associated with fMRI responses in the ventromedial prefrontal cortex (VMPFC), a region not typically associated with confidence for perceptual decisions. Furthermore, activity in the VMPFC was functionally coupled with regions of the frontal cortex linked to perceptual decision-making and metacognition. Our results suggest that the VMPFC holds an early confidence representation arising from decision dynamics, preceding and potentially informing metacognitive evaluation.

While waiting to cross the road on a foggy morning, you see a shape in the distance that appears to be an approaching car. How do you decide if it is safe to cross? We often have to make important decisions about the world based on imperfect information. What guides our subsequent actions in these situations is a sense of accuracy, or confidence, that we associate with our initial judgments. You would not step off the kerb if you were only 10% confident the car was a safe distance away. But how, when, and where in the brain does such confidence emerge?

Gherman and Philiastides examined how brain activity relates to confidence during the early stages of decision-making, that is, before people have explicitly committed to a particular choice. Healthy volunteers were asked to judge the direction in which dots were moving across a screen. They then had to rate how confident they were in their decision. Two techniques – EEG and fMRI – tracked their brain activity during the task. EEG uses scalp electrodes to reveal when and how electrical activity is changing inside the brain, while fMRI, a type of brain scan, shows where these changes in brain activity occur. Used together, the two techniques provide a greater understanding of brain activity than either used alone.

Activity in multiple regions of the brain correlated with confidence at different stages of the task. Certain brain networks showed confidence-related activity while the volunteers tried to judge the direction of movement, and others were engaged when volunteers made their confidence ratings. However, activity in only one area reliably indicated how confident the volunteers felt before they had made their choice. This area, the ventromedial prefrontal cortex, also helps process rewards. This suggests that feelings of confidence early in the decision-making process could guide our behaviour by virtue of being rewarding.

Many brain disorders – including depression, schizophrenia and Parkinson's disease – compromise decision-making. Patients show changes in accuracy, response times, and in their ability to accurately evaluate their decisions. The methods used in the current study could help reveal the neural changes that cause these impairments. This could lead to new methods to diagnose and predict cognitive deficits, and new ways to treat them at an earlier stage.

A multimodal encoding model applied to imaging decision-related neural cascades in the human brain

Perception and cognition in the brain are naturally characterized as spatiotemporal processes. Decision-making, for example, depends on coordinated patterns of neural activity cascading across the brain, running in time from stimulus to response and in space from primary sensory regions to the frontal lobe. Measuring this cascade is key to developing an understanding of brain function. Here we report on a novel methodology that employs multi-modal imaging for inferring this cascade in humans at unprecedented spatiotemporal resolution. Specifically, we develop an encoding model to link simultaneously measured electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) signals to infer high-resolution spatiotemporal brain dynamics during a perceptual decision. After demonstrating replication of results from the literature, we report previously unobserved sequential reactivation of a substantial fraction of the pre-response network whose magnitude correlates with a proxy for decision confidence. Our encoding model, which temporally tags BOLD activations using time localized EEG variability, identifies a coordinated and spatially distributed neural cascade that is associated with a perceptual decision. In general the methodology illuminates complex brain dynamics that would otherwise be unobservable using fMRI or EEG acquired separately.

Neural correlates of evidence accumulation during value-based decisions revealed via simultaneous EEG-fMRI

Current computational accounts posit that, in simple binary choices, humans accumulate evidence in favour of the different alternatives before committing to a decision. Neural correlates of this accumulating activity have been found during perceptual decisions in parietal and prefrontal cortex; however the source of such activity in value-based choices remains unknown. Here we use simultaneous EEG–fMRI and computational modelling to identify EEG signals reflecting an accumulation process and demonstrate that the within- and across-trial variability in these signals explains fMRI responses in posterior-medial frontal cortex. Consistent with its role in integrating the evidence prior to reaching a decision, this region also exhibits task-dependent coupling with the ventromedial prefrontal cortex and the striatum, brain areas known to encode the subjective value of the decision alternatives. These results further endorse the proposition of an evidence accumulation process during value-based decisions in humans and implicate the posterior-medial frontal cortex in this process.

Parietal and prefrontal cortices gather information to make perceptual decisions, but it is not known if the same is true for value-based choices. Here, authors use simultaneous EEG-fMRI and modelling to show that during value- and reward-based decisions this evidence is accumulated in the posterior medial frontal cortex.

Neural and neurochemical basis of reinforcement-guided decision making

Decision making is an adaptive behavior that takes into account several internal and external input variables and leads to the choice of a course of action over other available and often competing alternatives. While it has been studied in diverse fields ranging from mathematics, economics, ecology, and ethology to psychology and neuroscience, recent cross talk among perspectives from different fields has yielded novel descriptions of decision processes. Reinforcement-guided decision making models are based on economic and reinforcement learning theories, and their focus is on the maximization of acquired benefit over a defined period of time. Studies based on reinforcement-guided decision making have implicated a large network of neural circuits across the brain. This network includes a wide range of cortical (e.g., orbitofrontal cortex and anterior cingulate cortex) and subcortical (e.g., nucleus accumbens and subthalamic nucleus) brain areas and uses several neurotransmitter systems (e.g., dopaminergic and serotonergic systems) to communicate and process decision-related information. This review discusses distinct as well as overlapping contributions of these networks and neurotransmitter systems to the processing of decision making. We end the review by touching on neural circuitry and neuromodulatory regulation of exploratory decision making.

Space-by-time decomposition for single-trial decoding of M/EEG activity

We develop a novel methodology for the single-trial analysis of multichannel time-varying neuroimaging signals. We introduce the space-by-time M/EEG decomposition, based on Non-negative Matrix Factorization (NMF), which describes single-trial M/EEG signals using a set of non-negative spatial and temporal components that are linearly combined with signed scalar activation coefficients. We illustrate the effectiveness of the proposed approach on an EEG dataset recorded during the performance of a visual categorization task. Our method extracts three temporal and two spatial functional components achieving a compact yet full representation of the underlying structure, which validates and summarizes succinctly results from previous studies. Furthermore, we introduce a decoding analysis that allows determining the distinct functional role of each component and relating them to experimental conditions and task parameters. In particular, we demonstrate that the presented stimulus and the task difficulty of each trial can be reliably decoded using specific combinations of components from the identified space-by-time representation. When comparing with a sliding-window linear discriminant algorithm, we show that our approach yields more robust decoding performance across participants. Overall, our findings suggest that the proposed space-by-time decomposition is a meaningful low-dimensional representation that carries the relevant information of single-trial M/EEG signals.

•

Space-by-time decomposition for multichannel time-varying signal analysis.

•

Extraction of spatial and temporal components of single-trial M/EEG activity.

•

Full and succinct characterization of EEG data during a visual categorization task.

•

Single-trial decoding based on task-relevant features.

•

Robust and consistent decoding results across participants.

Two spatiotemporally distinct value systems shape reward-based learning in the human brain

Avoiding repeated mistakes and learning to reinforce rewarding decisions is critical for human survival and adaptive actions. Yet, the neural underpinnings of the value systems that encode different decision-outcomes remain elusive. Here coupling single-trial electroencephalography with simultaneously acquired functional magnetic resonance imaging, we uncover the spatiotemporal dynamics of two separate but interacting value systems encoding decision-outcomes. Consistent with a role in regulating alertness and switching behaviours, an early system is activated only by negative outcomes and engages arousal-related and motor-preparatory brain structures. Consistent with a role in reward-based learning, a later system differentially suppresses or activates regions of the human reward network in response to negative and positive outcomes, respectively. Following negative outcomes, the early system interacts and downregulates the late system, through a thalamic interaction with the ventral striatum. Critically, the strength of this coupling predicts participants' switching behaviour and avoidance learning, directly implicating the thalamostriatal pathway in reward-based learning.

Human scalp potentials reflect a mixture of decision-related signals during perceptual choices

Single-unit animal studies have consistently reported decision-related activity mirroring a process of temporal accumulation of sensory evidence to a fixed internal decision boundary. To date, our understanding of how response patterns seen in single-unit data manifest themselves at the macroscopic level of brain activity obtained from human neuroimaging data remains limited. Here, we use single-trial analysis of human electroencephalography data to show that population responses on the scalp can capture choice-predictive activity that builds up gradually over time with a rate proportional to the amount of sensory evidence, consistent with the properties of a drift-diffusion-like process as characterized by computational modeling. Interestingly, at time of choice, scalp potentials continue to appear parametrically modulated by the amount of sensory evidence rather than converging to a fixed decision boundary as predicted by our model. We show that trial-to-trial fluctuations in these response-locked signals exert independent leverage on behavior compared with the rate of evidence accumulation earlier in the trial. These results suggest that in addition to accumulator signals, population responses on the scalp reflect the influence of other decision-related signals that continue to covary with the amount of evidence at time of choice.

Experience does not equal expertise in recognizing infrequent incoming gunfire: neural markers for experience and task expertise at peak behavioral performance

For a soldier, decisions to use force can happen rapidly and sometimes lead to undesired consequences. In many of these situations, there is a rapid assessment by the shooter that recognizes a threat and responds to it with return fire. But the neural processes underlying these rapid decisions are largely unknown, especially amongst those with extensive weapons experience and expertise. In this paper, we investigate differences in weapons experts and non-experts during an incoming gunfire detection task. Specifically, we analyzed the electroencephalography (EEG) of eleven expert marksmen/soldiers and eleven non-experts while they listened to an audio scene consisting of a sequence of incoming and non-incoming gunfire events. Subjects were tasked with identifying each event as quickly as possible and committing their choice via a motor response. Contrary to our hypothesis, experts did not have significantly better behavioral performance or faster response time than novices. Rather, novices indicated trends of better behavioral performance than experts. These group differences were more dramatic in the EEG correlates of incoming gunfire detection. Using machine learning, we found condition-discriminating EEG activity among novices showing greater magnitude and covering longer periods than those found in experts. We also compared group-level source reconstruction on the maximum discriminating neural correlates and found that each group uses different neural structures to perform the task. From condition-discriminating EEG and source localization, we found that experts perceive more categorical overlap between incoming and non-incoming gunfire. Consequently, the experts did not perform as well behaviorally as the novices. We explain these unexpected group differences as a consequence of experience with gunfire not being equivalent to expertise in recognizing incoming gunfire.

Neural representations of confidence emerge from the process of decision formation during perceptual choices

Choice confidence represents the degree of belief that one's actions are likely to be correct or rewarding and plays a critical role in optimizing our decisions. Despite progress in understanding the neurobiology of human perceptual decision-making, little is known about the representation of confidence. Importantly, it remains unclear whether confidence forms an integral part of the decision process itself or represents a purely post-decisional signal. To address this issue we employed a paradigm whereby on some trials, prior to indicating their decision, participants could opt-out of the task for a small but certain reward. This manipulation captured participants' confidence on individual trials and allowed us to discriminate between electroencephalographic signals associated with certain-vs.-uncertain trials. Discrimination increased gradually and peaked well before participants indicated their choice. These signals exhibited a temporal profile consistent with a process of evidence accumulation, culminating at time of peak discrimination. Moreover, trial-by-trial fluctuations in the accumulation rate of nominally identical stimuli were predictive of participants' likelihood to opt-out of the task, suggesting that confidence emerges from the decision process itself and is computed continuously as the process unfolds. Correspondingly, source reconstruction placed these signals in regions previously implicated in decision making, within the prefrontal and parietal cortices. Crucially, control analyses ensured that these results could not be explained by stimulus difficulty, lapses in attention or decision accuracy.

Prestimulus alpha power predicts fidelity of sensory encoding in perceptual decision making

Pre-stimulus α power has been shown to correlate with the behavioral accuracy of perceptual decisions. In most cases, these correlations have been observed by comparing α power for different behavioral outcomes (e.g. correct vs incorrect trials). In this paper we investigate such covariation within the context of behaviorally-latent fluctuations in task-relevant post-stimulus neural activity. Specially we consider variations of pre-stimulus α power with post-stimulus EEG components in a two alternative forced choice visual discrimination task. EEG components, discriminative of stimulus class, are identified using a linear multivariate classifier and only the variability of the components for correct trials (regardless of stimulus class, and for nominally identical stimuli) are correlated with the corresponding pre-stimulus α power. We find a significant relationship between the mean and variance of the pre-stimulus α power and the variation of the trial-to-trial magnitude of an early post-stimulus EEG component. This relationship is not seen for a later EEG component that is also discriminative of stimulus class and which has been previously linked to the quality of evidence driving the decision process. Our results suggest that early perceptual representations, rather than temporally later neural correlates of the perceptual decision, are modulated by pre-stimulus state.

Soldiers and marksmen under fire: monitoring performance with neural correlates of small arms fire localization

Important decisions in the heat of battle occur rapidly and a key aptitude of a good combat soldier is the ability to determine whether he is under fire. This rapid decision requires the soldier to make a judgment in a fraction of a second, based on a barrage of multisensory cues coming from multiple modalities. The present study uses an oddball paradigm to examine listener ability to differentiate shooter locations from audio recordings of small arms fire. More importantly, we address the neural correlates involved in this rapid decision process by employing single-trial analysis of electroencephalography (EEG). In particular, we examine small arms expert listeners as they differentiate the sounds of small arms firing events recorded at different observer positions relative to a shooter. Using signal detection theory, we find clear neural signatures related to shooter firing angle by identifying the times of neural discrimination on a trial-to-trial basis. Similar to previous results in oddball experiments, we find common windows relative to the response and the stimulus when neural activity discriminates between target stimuli (forward fire: observer 0° to firing angle) vs. standards (off-axis fire: observer 90° to firing angle). We also find, using windows of maximum discrimination, that auditory target vs. standard discrimination yields neural sources in Brodmann Area 19 (BA 19), i.e., in the visual cortex. In summary, we show that single-trial analysis of EEG yields informative scalp distributions and source current localization of discriminating activity when the small arms experts discriminate between forward and off-axis fire observer positions. Furthermore, this perceptual decision implicates brain regions involved in visual processing, even though the task is purely auditory. Finally, we utilize these techniques to quantify the level of expertise in these subjects for the chosen task, having implications for human performance monitoring in combat.

Single-trial linear correlation analysis: application to characterization of stimulus modality effects

A key objective in systems and cognitive neuroscience is to establish associations between behavioral measures and concurrent neuronal activity. Single-trial analysis has been proposed as a novel method for characterizing such correlates by first extracting neural components that maximally discriminate trials on a categorical variable, (e.g., hard vs. easy, correct vs. incorrect etc.), and then correlate those components to a continues dependent variable of interest, e.g., reaction time, difficulty Index, etc. However, often times in experiment design it is difficult to either define meaningful categorical variables, or to record enough trials for the method to extract the discriminant components. Experiments designed for the study of the effects of stimulus presentation modality in working memory provide such a scenario, as will be exemplified. In this paper, we proposed a new approach to single-trial analysis in which we directly extract neural activity that maximally correlates to single-trial manual response times; eliminating the need to define an arbitrary categorical variable. We demonstrate our method on real electroencephalography (EEG) data recordings from the study of stimulus presentation modality effect (SPME).

EEG/fMRI fusion based on independent component analysis: integration of data-driven and model-driven methods

Simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) provide complementary noninvasive information of brain activity, and EEG/fMRI fusion can achieve higher spatiotemporal resolution than each modality separately. This focuses on independent component analysis (ICA)-based EEG/fMRI fusion. In order to appreciate the issues, we first describe the potential and limitations of the developed fusion approaches: fMRI-constrained EEG imaging, EEG-informed fMRI analysis, and symmetric fusion. We then outline some newly developed hybrid fusion techniques using ICA and the combination of data-/model-driven methods, with special mention of the spatiotemporal EEG/fMRI fusion (STEFF). Finally, we discuss the current trend in methodological development and the existing limitations for extrapolating neural dynamics.