Identifying object categories from event-related EEG: toward decoding of conceptual representations

Subword Representations Successfully Decode Brain Responses to Morphologically Complex Written Words

This study extends the idea of decoding word-evoked brain activations using a corpus-semantic vector space to multimorphemic words in the agglutinative Finnish language. The corpus-semantic models are trained on word segments, and decoding is carried out with word vectors that are composed of these segments. We tested several alternative vector-space models using different segmentations: no segmentation (whole word), linguistic morphemes, statistical morphemes, random segmentation, and character-level 1-, 2- and 3-grams, and paired them with recorded MEG responses to multimorphemic words in a visual word recognition task. For all variants, the decoding accuracy exceeded the standard word-label permutation-based significance thresholds at 350-500 ms after stimulus onset. However, the critical segment-label permutation test revealed that only those segmentations that were morphologically aware reached significance in the brain decoding task. The results suggest that both whole-word forms and morphemes are represented in the brain and show that neural decoding using corpus-semantic word representations derived from compositional subword segments is applicable also for multimorphemic word forms. This is especially relevant for languages with complex morphology, because a large proportion of word forms are rare and it can be difficult to find statistically reliable surface representations for them in any large corpus.

Decoding time-resolved neural representations of orientation ensemble perception

The visual system can compute summary statistics of several visual elements at a glance. Numerous studies have shown that an ensemble of different visual features can be perceived over 50-200 ms; however, the time point at which the visual system forms an accurate ensemble representation associated with an individual's perception remains unclear. This is mainly because most previous studies have not fully addressed time-resolved neural representations that occur during ensemble perception, particularly lacking quantification of the representational strength of ensembles and their correlation with behavior. Here, we conducted orientation ensemble discrimination tasks and electroencephalogram (EEG) recordings to decode orientation representations over time while human observers discriminated an average of multiple orientations. We modeled EEG signals as a linear sum of hypothetical orientation channel responses and inverted this model to quantify the representational strength of orientation ensemble. Our analysis using this inverted encoding model revealed stronger representations of the average orientation over 400-700 ms. We also correlated the orientation representation estimated from EEG signals with the perceived average orientation reported in the ensemble discrimination task with adjustment methods. We found that the estimated orientation at approximately 600-700 ms significantly correlated with the individual differences in perceived average orientation. These results suggest that although ensembles can be quickly and roughly computed, the visual system may gradually compute an orientation ensemble over several hundred milliseconds to achieve a more accurate ensemble representation.

Image classification and reconstruction from low-density EEG

Recent advances in visual decoding have enabled the classification and reconstruction of perceived images from the brain. However, previous approaches have predominantly relied on stationary, costly equipment like fMRI or high-density EEG, limiting the real-world availability and applicability of such projects. Additionally, several EEG-based paradigms have utilized artifactual, rather than stimulus-related information yielding flawed classification and reconstruction results. Our goal was to reduce the cost of the decoding paradigm, while increasing its flexibility. Therefore, we investigated whether the classification of an image category and the reconstruction of the image itself is possible from the visually evoked brain activity measured by a portable, 8-channel EEG. To compensate for the low electrode count and to avoid flawed predictions, we designed a theory-guided EEG setup and created a new experiment to obtain a dataset from 9 subjects. We compared five contemporary classification models with our setup reaching an average accuracy of 34.4% for 20 image classes on hold-out test recordings. For the reconstruction, the top-performing model was used as an EEG-encoder which was combined with a pretrained latent diffusion model via double-conditioning. After fine-tuning, we reconstructed images from the test set with a 1000 trial 50-class top-1 accuracy of 35.3%. While not reaching the same performance as MRI-based paradigms on unseen stimuli, our approach greatly improved the affordability and mobility of the visual decoding technology.

MEG Evidence That Modality-Independent Conceptual Representations Contain Semantic and Visual Features

The semantic knowledge stored in our brains can be accessed from different stimulus modalities. For example, a picture of a cat and the word "cat" both engage similar conceptual representations. While existing research has found evidence for modality-independent representations, their content remains unknown. Modality-independent representations could be semantic, or they might also contain perceptual features. We developed a novel approach combining word/picture cross-condition decoding with neural network classifiers that learned latent modality-independent representations from MEG data (25 human participants, 15 females, 10 males). We then compared these representations to models representing semantic, sensory, and orthographic features. Results show that modality-independent representations correlate both with semantic and visual representations. There was no evidence that these results were due to picture-specific visual features or orthographic features automatically activated by the stimuli presented in the experiment. These findings support the notion that modality-independent concepts contain both perceptual and semantic representations.

Spatiotemporal Properties of Common Semantic Categories for Words and Pictures

The timing of semantic processing during object recognition in the brain is a topic of ongoing discussion. One way of addressing this question is by applying multivariate pattern analysis to human electrophysiological responses to object images of different semantic categories. However, although multivariate pattern analysis can reveal whether neuronal activity patterns are distinct for different stimulus categories, concerns remain on whether low-level visual features also contribute to the classification results. To circumvent this issue, we applied a cross-decoding approach to magnetoencephalography data from stimuli from two different modalities: images and their corresponding written words. We employed items from three categories and presented them in a randomized order. We show that if the classifier is trained on words, pictures are classified between 150 and 430 msec after stimulus onset, and when training on pictures, words are classified between 225 and 430 msec. The topographical map, identified using a searchlight approach for cross-modal activation in both directions, showed left lateralization, confirming the involvement of linguistic representations. These results point to semantic activation of pictorial stimuli occurring at ∼150 msec, whereas for words, the semantic activation occurs at ∼230 msec.

Evaluating the Feasibility of Visual Imagery for an EEG-Based Brain-Computer Interface

Visual imagery, or the mental simulation of visual information from memory, could serve as an effective control paradigm for a brain-computer interface (BCI) due to its ability to directly convey the user's intention with many natural ways of envisioning an intended action. However, multiple initial investigations into using visual imagery as a BCI control strategies have been unable to fully evaluate the capabilities of true spontaneous visual mental imagery. One major limitation in these prior works is that the target image is typically displayed immediately preceding the imagery period. This paradigm does not capture spontaneous mental imagery as would be necessary in an actual BCI application but something more akin to short-term retention in visual working memory. Results from the present study show that short-term visual imagery following the presentation of a specific target image provides a stronger, more easily classifiable neural signature in EEG than spontaneous visual imagery from long-term memory following an auditory cue for the image. We also show that short-term visual imagery and visual perception share commonalities in the most predictive electrodes and spectral features. However, visual imagery received greater influence from frontal electrodes whereas perception was mostly confined to occipital electrodes. This suggests that visual perception is primarily driven by sensory information whereas visual imagery has greater contributions from areas associated with memory and attention. This work provides the first direct comparison of short-term and long-term visual imagery tasks and provides greater insight into the feasibility of using visual imagery as a BCI control strategy.

Impaired Cortical Tracking of Speech in Children with Developmental Language Disorder

In developmental language disorder (DLD), learning to comprehend and express oneself with spoken language is impaired, but the reason for this remains unknown. Using millisecond-scale magnetoencephalography recordings combined with machine learning models, we investigated whether the possible neural basis of this disruption lies in poor cortical tracking of speech. The stimuli were common spoken Finnish words (e.g., dog, car, hammer) and sounds with corresponding meanings (e.g., dog bark, car engine, hammering). In both children with DLD (10 boys and 7 girls) and typically developing (TD) control children (14 boys and 3 girls), aged 10-15 years, the cortical activation to spoken words was best modeled as time-locked to the unfolding speech input at ∼100 ms latency between sound and cortical activation. Amplitude envelope (amplitude changes) and spectrogram (detailed time-varying spectral content) of the spoken words, but not other sounds, were very successfully decoded based on time-locked brain responses in bilateral temporal areas; based on the cortical responses, the models could tell at ∼75-85% accuracy which of the two sounds had been presented to the participant. However, the cortical representation of the amplitude envelope information was poorer in children with DLD compared with TD children at longer latencies (at ∼200-300 ms lag). We interpret this effect as reflecting poorer retention of acoustic-phonetic information in short-term memory. This impaired tracking could potentially affect the processing and learning of words as well as continuous speech. The present results offer an explanation for the problems in language comprehension and acquisition in DLD.

Temporal differences and commonalities between hand and tool neural processing

Object recognition is a complex cognitive process that relies on how the brain organizes object-related information. While spatial principles have been extensively studied, less studied temporal dynamics may also offer valuable insights into this process, particularly when neural processing overlaps for different categories, as it is the case of the categories of hands and tools. Here we focus on the differences and/or similarities between the time-courses of hand and tool processing under electroencephalography (EEG). Using multivariate pattern analysis, we compared, for different time points, classification accuracy for images of hands or tools when compared to images of animals. We show that for particular time intervals (~ 136-156 ms and ~ 252-328 ms), classification accuracy for hands and for tools differs. Furthermore, we show that classifiers trained to differentiate between tools and animals generalize their learning to classification of hand stimuli between ~ 260-320 ms and ~ 376-500 ms after stimulus onset. Classifiers trained to distinguish between hands and animals, on the other hand, were able to extend their learning to the classification of tools at ~ 150 ms. These findings suggest variations in semantic features and domain-specific differences between the two categories, with later-stage similarities potentially related to shared action processing for hands and tools.

Cortical time-course of evidence accumulation during semantic processing

Our understanding of the surrounding world and communication with other people are tied to mental representations of concepts. In order for the brain to recognize an object, it must determine which concept to access based on information available from sensory inputs. In this study, we combine magnetoencephalography and machine learning to investigate how concepts are represented and accessed in the brain over time. Using brain responses from a silent picture naming task, we track the dynamics of visual and semantic information processing, and show that the brain gradually accumulates information on different levels before eventually reaching a plateau. The timing of this plateau point varies across individuals and feature models, indicating notable temporal variation in visual object recognition and semantic processing.

Semantic Categorization of Naming Responses Based on Prearticulatory Electrical Brain Activity

PURPOSE

Object naming requires visual decoding, conceptualization, semantic categorization, and phonological encoding, all within 400 to 600 ms of stimulus presentation and before a word is spoken. In this study, we sought to predict semantic categories of naming responses based on prearticulatory brain activity recorded with scalp EEG in healthy individuals.

METHODS

We assessed 19 healthy individuals who completed a naming task while undergoing EEG. The naming task consisted of 120 drawings of animate/inanimate objects or abstract drawings. We applied a one-dimensional, two-layer, neural network to predict the semantic categories of naming responses based on prearticulatory brain activity.

RESULTS

Classifications of animate, inanimate, and abstract responses had an average accuracy of 80%, sensitivity of 72%, and specificity of 87% across participants. Across participants, time points with the highest average weights were between 470 and 490 milliseconds after stimulus presentation, and electrodes with the highest weights were located over the left and right frontal brain areas.

CONCLUSIONS

Scalp EEG can be successfully used in predicting naming responses through prearticulatory brain activity. Interparticipant variability in feature weights suggests that individualized models are necessary for highest accuracy. Our findings may inform future applications of EEG in reconstructing speech for individuals with and without speech impairments.

The time course of cross-modal representations of conceptual categories

To what extent does language production activate cross-modal conceptual representations? In picture naming, we view specific exemplars of concepts and then name them with a label, like "dog". In overt reading, the written word does not express a specific exemplar. Here we used a decoding approach with magnetoencephalography (MEG) to address whether picture naming and overt word reading involve shared representations of superordinate categories (e.g., animal). This addresses a fundamental question about the modality-generality of conceptual representations and their temporal evolution. Crucially, we do this using a language production task that does not require explicit categorization judgment and that controls for word form properties across semantic categories. We trained our models to classify the animal/tool distinction using MEG data of one modality at each time point and then tested the generalization of those models on the other modality. We obtained evidence for the automatic activation of cross-modal semantic category representations for both pictures and words later than their respective modality-specific representations. Cross-modal representations were activated at 150 ms and lasted until around 450 ms. The time course of lexical activation was also assessed revealing that semantic category is represented before lexical access for pictures but after lexical access for words. Notably, this earlier activation of semantic category in pictures occurred simultaneously with visual representations. We thus show evidence for the spontaneous activation of cross-modal semantic categories in picture naming and word reading. These results serve to anchor a more comprehensive spatio-temporal delineation of the semantic feature space during production planning.

Parallel or sequential? Decoding conceptual and phonological/phonetic information from MEG signals during language production

Speaking requires the temporally coordinated planning of core linguistic information, from conceptual meaning to articulation. Recent neurophysiological results suggested that these operations involve a cascade of neural events with subsequent onset times, whilst competing evidence suggests early parallel neural activation. To test these hypotheses, we examined the sources of neuromagnetic activity recorded from 34 participants overtly naming 134 images from 4 object categories (animals, tools, foods and clothes). Within each category, word length and phonological neighbourhood density were co-varied to target phonological/phonetic processes. Multivariate pattern analyses (MVPA) searchlights in source space decoded object categories in occipitotemporal and middle temporal cortex, and phonological/phonetic variables in left inferior frontal (BA 44) and motor cortex early on. The findings suggest early activation of multiple variables due to intercorrelated properties and interactivity of processing, thus raising important questions about the representational properties of target words during the preparatory time enabling overt speaking.

EEG-based BCI Dataset of Semantic Concepts for Imagination and Perception Tasks

Electroencephalography (EEG) is a widely-used neuroimaging technique in Brain Computer Interfaces (BCIs) due to its non-invasive nature, accessibility and high temporal resolution. A range of input representations has been explored for BCIs. The same semantic meaning can be conveyed in different representations, such as visual (orthographic and pictorial) and auditory (spoken words). These stimuli representations can be either imagined or perceived by the BCI user. In particular, there is a scarcity of existing open source EEG datasets for imagined visual content, and to our knowledge there are no open source EEG datasets for semantics captured through multiple sensory modalities for both perceived and imagined content. Here we present an open source multisensory imagination and perception dataset, with twelve participants, acquired with a 124 EEG channel system. The aim is for the dataset to be open for purposes such as BCI related decoding and for better understanding the neural mechanisms behind perception, imagination and across the sensory modalities when the semantic category is held constant.

Spatiotemporal phase slip patterns for visual evoked potentials, covert object naming tasks, and insight moments extracted from 256 channel EEG recordings

Phase slips arise from state transitions of the coordinated activity of cortical neurons which can be extracted from the EEG data. The phase slip rates (PSRs) were studied from the high-density (256 channel) EEG data, sampled at 16.384 kHz, of five adult subjects during covert visual object naming tasks. Artifact-free data from 29 trials were averaged for each subject. The analysis was performed to look for phase slips in the theta (4-7 Hz), alpha (7-12 Hz), beta (12-30 Hz), and low gamma (30-49 Hz) bands. The phase was calculated with the Hilbert transform, then unwrapped and detrended to look for phase slip rates in a 1.0 ms wide stepping window with a step size of 0.06 ms. The spatiotemporal plots of the PSRs were made by using a montage layout of 256 equidistant electrode positions. The spatiotemporal profiles of EEG and PSRs during the stimulus and the first second of the post-stimulus period were examined in detail to study the visual evoked potentials and different stages of visual object recognition in the visual, language, and memory areas. It was found that the activity areas of PSRs were different as compared with EEG activity areas during the stimulus and post-stimulus periods. Different stages of the insight moments during the covert object naming tasks were examined from PSRs and it was found to be about 512 ± 21 ms for the 'Eureka' moment. Overall, these results indicate that information about the cortical phase transitions can be derived from the measured EEG data and can be used in a complementary fashion to study the cognitive behavior of the brain.

Attentional capture mediates the emergence and suppression of intrusive memories

Intrusive memories hijack consciousness and their control may lead to forgetting. However, the contribution of reflexive attention to qualifying a memory signal as interfering is unknown. We used machine learning to decode the brain's electrical activity and pinpoint the otherwise hidden emergence of intrusive memories reported during a memory suppression task. Importantly, the algorithm was trained on an independent attentional model of visual activity, mimicking either the abrupt and interfering appearance of visual scenes into conscious awareness or their deliberate exploration. Intrusion of memories into conscious awareness were decoded above chance. The decoding accuracy increased when the algorithm was trained using a model of reflexive attention. Conscious detection of intrusive activity decoded from the brain signal was central to the future silencing of suppressed memories and later forgetting. Unwanted memories require the reflexive orienting of attention and access to consciousness to be suppressed effectively by inhibitory control.

The Time Course of Language Production as Revealed by Pattern Classification of MEG Sensor Data

Language production involves a complex set of computations, from conceptualization to articulation, which are thought to engage cascading neural events in the language network. However, recent neuromagnetic evidence suggests simultaneous meaning-to-speech mapping in picture naming tasks, as indexed by early parallel activation of frontotemporal regions to lexical semantic, phonological, and articulatory information. Here we investigate the time course of word production, asking to what extent such "earliness" is a distinctive property of the associated spatiotemporal dynamics. Using MEG, we recorded the neural signals of 34 human subjects (26 males) overtly naming 134 images from four semantic object categories (animals, foods, tools, clothes). Within each category, we covaried word length, as quantified by the number of syllables contained in a word, and phonological neighborhood density to target lexical and post-lexical phonological/phonetic processes. Multivariate pattern analyses searchlights in sensor space distinguished the stimulus-locked spatiotemporal responses to object categories early on, from 150 to 250 ms after picture onset, whereas word length was decoded in left frontotemporal sensors at 250-350 ms, followed by the latency of phonological neighborhood density (350-450 ms). Our results suggest a progression of neural activity from posterior to anterior language regions for the semantic and phonological/phonetic computations preparing overt speech, thus supporting serial cascading models of word production.SIGNIFICANCE STATEMENT Current psycholinguistic models make divergent predictions on how a preverbal message is mapped onto articulatory output during the language planning. Serial models predict a cascading sequence of hierarchically organized neural computations from conceptualization to articulation. In contrast, parallel models posit early simultaneous activation of multiple conceptual, phonological, and articulatory information in the language system. Here we asked whether such earliness is a distinctive property of the neural dynamics of word production. The combination of the millisecond precision of MEG with multivariate pattern analyses revealed subsequent onset times for the neural events supporting semantic and phonological/phonetic operations, progressing from posterior occipitotemporal to frontal sensor areas. The findings bring new insights for refining current theories of language production.

Resolving the time course of visual and auditory object categorization

Humans can effortlessly categorize objects, both when they are conveyed through visual images and spoken words. To resolve the neural correlates of object categorization, studies have so far primarily focused on the visual modality. It is therefore still unclear how the brain extracts categorical information from auditory signals. In the current study, we used EEG (n = 48) and time-resolved multivariate pattern analysis to investigate 1) the time course with which object category information emerges in the auditory modality and 2) how the representational transition from individual object identification to category representation compares between the auditory modality and the visual modality. Our results show that 1) auditory object category representations can be reliably extracted from EEG signals and 2) a similar representational transition occurs in the visual and auditory modalities, where an initial representation at the individual-object level is followed by a subsequent representation of the objects’ category membership. Altogether, our results suggest an analogous hierarchy of information processing across sensory channels. However, there was no convergence toward conceptual modality-independent representations, thus providing no evidence for a shared supramodal code.

NEW & NOTEWORTHY Object categorization operates on inputs from different sensory modalities, such as vision and audition. This process was mainly studied in vision. Here, we explore auditory object categorization. We show that auditory object category representations can be reliably extracted from EEG signals and, similar to vision, auditory representations initially carry information about individual objects, which is followed by a subsequent representation of the objects’ category membership.

Neural decoding of semantic concepts: A systematic literature review

Objective Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Results Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

When the Whole Is Less Than the Sum of Its Parts: Maximum Object Category Information and Behavioral Prediction in Multiscale Activation Patterns

Neural codes are reflected in complex neural activation patterns. Conventional electroencephalography (EEG) decoding analyses summarize activations by averaging/down-sampling signals within the analysis window. This diminishes informative fine-grained patterns. While previous studies have proposed distinct statistical features capable of capturing variability-dependent neural codes, it has been suggested that the brain could use a combination of encoding protocols not reflected in any one mathematical feature alone. To check, we combined 30 features using state-of-the-art supervised and unsupervised feature selection procedures (n = 17). Across three datasets, we compared decoding of visual object category between these 17 sets of combined features, and between combined and individual features. Object category could be robustly decoded using the combined features from all of the 17 algorithms. However, the combination of features, which were equalized in dimension to the individual features, were outperformed across most of the time points by the multiscale feature of Wavelet coefficients. Moreover, the Wavelet coefficients also explained the behavioral performance more accurately than the combined features. These results suggest that a single but multiscale encoding protocol may capture the EEG neural codes better than any combination of protocols. Our findings put new constraints on the models of neural information encoding in EEG.

Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Semantic knowledge about individual entities (i.e., the referents of proper names such as Jacinta Ardern) is fine-grained, episodic, and strongly social in nature, when compared with knowledge about generic entities (the referents of common nouns such as politician). We investigate the semantic representations of individual entities in the brain; and for the first time we approach this question using both neural data, in the form of newly-acquired EEG data, and distributional models of word meaning, employing them to isolate semantic information regarding individual entities in the brain. We ran two sets of analyses. The first set of analyses is only concerned with the evoked responses to individual entities and their categories. We find that it is possible to classify them according to both their coarse and their fine-grained category at appropriate timepoints, but that it is hard to map representational information learned from individuals to their categories. In the second set of analyses, we learn to decode from evoked responses to distributional word vectors. These results indicate that such a mapping can be learnt successfully: this counts not only as a demonstration that representations of individuals can be discriminated in EEG responses, but also as a first brain-based validation of distributional semantic models as representations of individual entities. Finally, in-depth analyses of the decoder performance provide additional evidence that the referents of proper names and categories have little in common when it comes to their representation in the brain.

Time-resolved multivariate pattern analysis of infant EEG data: A practical tutorial

Time-resolved multivariate pattern analysis (MVPA), a popular technique for analyzing magneto- and electro-encephalography (M/EEG) neuroimaging data, quantifies the extent and time-course by which neural representations support the discrimination of relevant stimuli dimensions. As EEG is widely used for infant neuroimaging, time-resolved MVPA of infant EEG data is a particularly promising tool for infant cognitive neuroscience. MVPA has recently been applied to common infant imaging methods such as EEG and fNIRS. In this tutorial, we provide and describe code to implement time-resolved, within-subject MVPA with infant EEG data. An example implementation of time-resolved MVPA based on linear SVM classification is described, with accompanying code in Matlab and Python. Results from a test dataset indicated that in both infants and adults this method reliably produced above-chance accuracy for classifying stimuli images. Extensions of the classification analysis are presented including both geometric- and accuracy-based representational similarity analysis, implemented in Python. Common choices of implementation are presented and discussed. As the amount of artifact-free EEG data contributed by each participant is lower in studies of infants than in studies of children and adults, we also explore and discuss the impact of varying participant-level inclusion thresholds on resulting MVPA findings in these datasets.

Categorizing objects from MEG signals using EEGNet

Magnetoencephalography (MEG) signals have demonstrated their practical application to reading human minds. Current neural decoding studies have made great progress to build subject-wise decoding models to extract and discriminate the temporal/spatial features in neural signals. In this paper, we used a compact convolutional neural network-EEGNet-to build a common decoder across subjects, which deciphered the categories of objects (faces, tools, animals, and scenes) from MEG data. This study investigated the influence of the spatiotemporal structure of MEG on EEGNet's classification performance. Furthermore, the EEGNet replaced its convolution layers with two sets of parallel convolution structures to extract the spatial and temporal features simultaneously. Our results showed that the organization of MEG data fed into the EEGNet has an effect on EEGNet classification accuracy, and the parallel convolution structures in EEGNet are beneficial to extracting and fusing spatial and temporal MEG features. The classification accuracy demonstrated that the EEGNet succeeds in building the common decoder model across subjects, and outperforms several state-of-the-art feature fusing methods.

Decoding of semantic categories of imagined concepts of animals and tools in fNIRS

Objective.Semantic decoding refers to the identification of semantic concepts from recordings of an individual's brain activity. It has been previously reported in functional magnetic resonance imaging and electroencephalography. We investigate whether semantic decoding is possible with functional near-infrared spectroscopy (fNIRS). Specifically, we attempt to differentiate between the semantic categories of animals and tools. We also identify suitable mental tasks for potential brain-computer interface (BCI) applications.Approach.We explore the feasibility of a silent naming task, for the first time in fNIRS, and propose three novel intuitive mental tasks based on imagining concepts using three sensory modalities: visual, auditory, and tactile. Participants are asked to visualize an object in their minds, imagine the sounds made by the object, and imagine the feeling of touching the object. A general linear model is used to extract hemodynamic responses that are then classified via logistic regression in a univariate and multivariate manner.Main results.We successfully classify all tasks with mean accuracies of 76.2% for the silent naming task, 80.9% for the visual imagery task, 72.8% for the auditory imagery task, and 70.4% for the tactile imagery task. Furthermore, we show that consistent neural representations of semantic categories exist by applying classifiers across tasks.Significance.These findings show that semantic decoding is possible in fNIRS. The study is the first step toward the use of semantic decoding for intuitive BCI applications for communication.

No fruits without color: Cross-modal priming and EEG reveal different roles for different features across semantic categories

Category-specific impairments witnessed in patients with semantic deficits have broadly dissociated into natural and artificial kinds. However, how the category of food (more specifically, fruits and vegetables) fits into this distinction has been difficult to interpret, given a pattern of deficit that has inconsistently mapped onto either kind, despite its intuitive membership to the natural domain. The present study explores the effects of a manipulation of a visual sensory (i.e., color) or functional (i.e., orientation) feature on the consequential semantic processing of fruits and vegetables (and tools, by comparison), first at the behavioral and then at the neural level. The categorization of natural (i.e., fruits/vegetables) and artificial (i.e., utensils) entities was investigated via cross-modal priming. Reaction time analysis indicated a reduction in priming for color-modified natural entities and orientation-modified artificial entities. Standard event-related potentials (ERP) analysis was performed, in addition to linear classification. For natural entities, a N400 effect at central channel sites was observed for the color-modified condition compared relative to normal and orientation conditions, with this difference confirmed by classification analysis. Conversely, there was no significant difference between conditions for the artificial category in either analysis. These findings provide strong evidence that color is an integral property to the categorization of fruits/vegetables, thus substantiating the claim that feature-based processing guides as a function of semantic category.

Rapidly Decoding Image Categories From MEG Data Using a Multivariate Short-Time FC Pattern Analysis Approach

Recent advances in the development of multivariate analysis methods have led to the application of multivariate pattern analysis (MVPA) to investigate the interactions between brain regions using graph theory (functional connectivity, FC) and decode visual categories from functional magnetic resonance imaging (fMRI) data from a continuous multicategory paradigm. To estimate stable FC patterns from fMRI data, previous studies required long periods in the order of several minutes, in comparison to the human brain that categories visual stimuli within hundreds of milliseconds. Constructing short-time dynamic FC patterns in the order of milliseconds and decoding visual categories is a relatively novel concept. In this study, we developed a multivariate decoding algorithm based on FC patterns and applied it to magnetoencephalography (MEG) data. MEG data were recorded from participants presented with image stimuli in four categories (faces, scenes, animals and tools). MEG data from 17 participants demonstrate that short-time dynamic FC patterns yield brain activity patterns that can be used to decode visual categories with high accuracy. Our results show that FC patterns change over the time window, and FC patterns extracted in the time window of 0∼200 ms after the stimulus onset were most stable. Further, the categorizing accuracy peaked (the mean binary accuracy is above 78.6% at individual level) in the FC patterns estimated within the 0∼200 ms interval. These findings elucidate the underlying connectivity information during visual category processing on a relatively smaller time scale and demonstrate that the contribution of FC patterns to categorization fluctuates over time.

High-pass filtering artifacts in multivariate classification of neural time series data

BACKGROUND

Traditionally, EEG/MEG data are high-pass filtered and baseline-corrected to remove slow drifts. Minor deleterious effects of high-pass filtering in traditional time-series analysis have been well-documented, including temporal displacements. However, its effects on time-resolved multivariate pattern classification analyses (MVPA) are largely unknown.

NEW METHOD

To prevent potential displacement effects, we extend an alternative method of removing slow drift noise - robust detrending - with a procedure in which we mask out all cortical events from each trial. We refer to this method as trial-masked robust detrending.

RESULTS

In both real and simulated EEG data of a working memory experiment, we show that both high-pass filtering and standard robust detrending create artifacts that result in the displacement of multivariate patterns into activity silent periods, particularly apparent in temporal generalization analyses, and especially in combination with baseline correction. We show that trial-masked robust detrending is free from such displacements.

COMPARISON WITH EXISTING METHOD(S)

Temporal displacement may emerge even with modest filter cut-off settings such as 0.05 Hz, and even in regular robust detrending. However, trial-masked robust detrending results in artifact-free decoding without displacements. Baseline correction may unwittingly obfuscate spurious decoding effects and displace them to the rest of the trial.

CONCLUSIONS

Decoding analyses benefit from trial-masked robust detrending, without the unwanted side effects introduced by filtering or regular robust detrending. However, for sufficiently clean data sets and sufficiently strong signals, no filtering or detrending at all may work adequately. Implications for other types of data are discussed, followed by a number of recommendations.

The Perils and Pitfalls of Block Design for EEG Classification Experiments

A recent paper [31] claims to classify brain processing evoked in subjects watching ImageNet stimuli as measured with EEG and to employ a representation derived from this processing to construct a novel object classifier. That paper, together with a series of subsequent papers [11, 18, 20, 24, 25, 30, 34], claims to achieve successful results on a wide variety of computer-vision tasks, including object classification, transfer learning, and generation of images depicting human perception and thought using brain-derived representations measured through EEG. Our novel experiments and analyses demonstrate that their results crucially depend on the block design that they employ, where all stimuli of a given class are presented together, and fail with a rapid-event design, where stimuli of different classes are randomly intermixed. The block design leads to classification of arbitrary brain states based on block-level temporal correlations that are known to exist in all EEG data, rather than stimulus-related activity. Because every trial in their test sets comes from the same block as many trials in the corresponding training sets, their block design thus leads to classifying arbitrary temporal artifacts of the data instead of stimulus-related activity. This invalidates all subsequent analyses performed on this data in multiple published papers and calls into question all of the reported results. We further show that a novel object classifier constructed with a random codebook performs as well as or better than a novel object classifier constructed with the representation extracted from EEG data, suggesting that the performance of their classifier constructed with a representation extracted from EEG data does not benefit from the brain-derived representation. Together, our results illustrate the far-reaching implications of the temporal autocorrelations that exist in all neuroimaging data for classification experiments. Further, our results calibrate the underlying difficulty of the tasks involved and caution against overly optimistic, but incorrect, claims to the contrary.

Reaction times predict dynamic brain representations measured with MEG for only some object categorisation tasks

Behavioural categorisation reaction times (RTs) provide a useful way to link behaviour to brain representations measured with neuroimaging. In this framework, objects are assumed to be represented in a multidimensional activation space, with the distances between object representations indicating their degree of neural similarity. Faster RTs have been reported to correlate with greater distances from a classification decision boundary for animacy. Objects inherently belong to more than one category, yet it is not known whether the RT-distance relationship, and its evolution over the time-course of the neural response, is similar across different categories. Here we used magnetoencephalography (MEG) to address this question. Our stimuli included typically animate and inanimate objects, as well as more ambiguous examples (i.e., robots and toys). We conducted four semantic categorisation tasks on the same stimulus set assessing animacy, living, moving, and human-similarity concepts, and linked the categorisation RTs to MEG time-series decoding data. Our results show a sustained RT-distance relationship throughout the time course of object processing for not only animacy, but also categorisation according to human-similarity. Interestingly, this sustained RT-distance relationship was not observed for the living and moving category organisations, despite comparable classification accuracy of the MEG data across all four category organisations. Our findings show that behavioural RTs predict representational distance for an organisational principle other than animacy, however further research is needed to determine why this relationship is observed only for some category organisations and not others.

Electroencephalography decoding of Chinese characters in primary school children and its prediction for word reading performance and development

Research on what neural mechanisms facilitate word reading development in non-alphabetic scripts is relatively rare. The present study was among the first to adopt a multivariate pattern classification analysis to decode electroencephalographic signals recorded for primary school children (N = 236) while performing a Chinese character decision task. Chinese is an ideal script for studying the relationship between neural discriminability (i.e., decodability) of the orthography and behavioral word reading skills since the mapping from orthography to phonology is relatively arbitrary in Chinese. This was also among the first empirical attempts to examine the extent to which decoding performance can predict current and subsequent word reading skills using a longitudinal design. Results showed that neural activation patterns of real characters can be distinguished from activation patterns for pseudo-characters, non-characters, and random stroke combinations in both younger and older children. Topography of the transformed classifier weights revealed two distinct cognitive sub-processes underlying single character recognition, but temporal generalization analysis suggested common neural mechanisms between the distinct cognitive sub-processes. Suggestive evidence from correlational and hierarchical regression analyses showed that decoding performance, assessed on average 2 months before the year 2 behavioral testing, predicted both year 1 word reading performance and the development of word reading fluency over the year. Results demonstrate that decoding performance, one indicator of how the neural system is functionally organized in processing characters and character-like stimuli, can serve as a useful neural marker in predicting current word reading skills and the capacity to learn to read.

Dynamic Time-Locking Mechanism in the Cortical Representation of Spoken Words

Human speech has a unique capacity to carry and communicate rich meanings. However, it is not known how the highly dynamic and variable perceptual signal is mapped to existing linguistic and semantic representations. In this novel approach, we used the natural acoustic variability of sounds and mapped them to magnetoencephalography (MEG) data using physiologically-inspired machine-learning models. We aimed at determining how well the models, differing in their representation of temporal information, serve to decode and reconstruct spoken words from MEG recordings in 16 healthy volunteers. We discovered that dynamic time-locking of the cortical activation to the unfolding speech input is crucial for the encoding of the acoustic-phonetic features of speech. In contrast, time-locking was not highlighted in cortical processing of non-speech environmental sounds that conveyed the same meanings as the spoken words, including human-made sounds with temporal modulation content similar to speech. The amplitude envelope of the spoken words was particularly well reconstructed based on cortical evoked responses. Our results indicate that speech is encoded cortically with especially high temporal fidelity. This speech tracking by evoked responses may partly reflect the same underlying neural mechanism as the frequently reported entrainment of the cortical oscillations to the amplitude envelope of speech. Furthermore, the phoneme content was reflected in cortical evoked responses simultaneously with the spectrotemporal features, pointing to an instantaneous transformation of the unfolding acoustic features into linguistic representations during speech processing.

Group-Level Neural Responses to Service-to-Service Brand Extension

Brand extension is a marketing strategy leveraging well-established brand to promote new offerings provided as goods or service. The previous neurophysiological studies on goods-to-goods brand extension have proposed that categorization and semantic memory processes are involved in brand extension evaluation. However, it is unknown whether these same processes also underlie service-to-service brand extension. The present study, therefore, aims to investigate neural processes in consumers underlying their judgment of service-to-service brand extension. Specifically, we investigated human electroencephalographic responses to extended services that were commonly considered to fit well or badly with parent brand among consumers. For this purpose, we proposed a new stimulus grouping method to find commonly acceptable or unacceptable service extensions. In the experiment, participants reported the acceptability of 56 brand extension pairs, consisting of parent brand name (S1) and extended service name (S2). From individual acceptability responses, we assigned each pair to one of the three fit levels: high- (i.e., highly acceptable), low-, and mid-fit. Next, we selected stimuli that received high/low-fit evaluations from a majority of participants (i.e., >85%) and assigned them to a high/low population-fit group. A comparison of event-related potentials (ERPs) between population-fit groups through a paired t-test showed significant differences in the fronto-central N2 and fronto-parietal P300 amplitudes. We further evaluated inter-subject variability of these ERP components by a decoding analysis that classified N2 and/or P300 amplitudes into a high, or low population-fit class using a support vector machine. Leave-one-subject-out validation revealed classification accuracy of 60.35% with N2 amplitudes, 78.95% with P300, and 73.68% with both, indicating a relatively high inter-subject variability of N2 but low for P300. This validation showed that fronto-parietal P300 reflected neural processes more consistent across subjects in service-to-service brand extension. We further observed that the left frontal P300 amplitude was increased as fit-level increased across stimuli, indicating a semantic retrieval process to evaluate a semantic link between S1 and S2. Parietal P300 showed a higher amplitude in the high population-fit group, reflecting a similarity-based categorization process. In sum, our results suggest that service-to-service brand extension evaluation may share similar neural processes with goods-to-goods brand extension.

Single-Trial Decoding of Scalp EEG under Natural Conditions

There is significant current interest in decoding mental states from electroencephalography (EEG) recordings. EEG signals are subject-specific, are sensitive to disturbances, and have a low signal-to-noise ratio, which has been mitigated by the use of laboratory-grade EEG acquisition equipment under highly controlled conditions. In the present study, we investigate single-trial decoding of natural, complex stimuli based on scalp EEG acquired with a portable, 32 dry-electrode sensor system in a typical office setting. We probe generalizability by a leave-one-subject-out cross-validation approach. We demonstrate that support vector machine (SVM) classifiers trained on a relatively small set of denoised (averaged) pseudotrials perform on par with classifiers trained on a large set of noisy single-trial samples. We propose a novel method for computing sensitivity maps of EEG-based SVM classifiers for visualization of EEG signatures exploited by the SVM classifiers. Moreover, we apply an NPAIRS resampling framework for estimation of map uncertainty, and thus show that effect sizes of sensitivity maps for classifiers trained on small samples of denoised data and large samples of noisy data are similar. Finally, we demonstrate that the average pseudotrial classifier can successfully predict the class of single trials from withheld subjects, which allows for fast classifier training, parameter optimization, and unbiased performance evaluation in machine learning approaches for brain decoding.

Temporal dynamics of access to amodal representations of category-level conceptual information

Categories describe semantic divisions between classes of objects and category-based models are widely used for investigation of the conceptual system. One critical issue in this endeavour is the isolation of conceptual from perceptual contributions to category-differences. An unambiguous way to address this confound is combining multiple input-modalities. To this end, we showed participants person/place stimuli using name and picture modalities. Using multivariate methods, we searched for category-sensitive neural patterns shared across input-modalities and thus independent from perceptual properties. The millisecond temporal resolution of magnetoencephalography (MEG) allowed us to consider the precise timing of conceptual access and, by confronting latencies between the two modalities ("time generalization"), how latencies of processing depends on the input-modality. Our results identified category-sensitive conceptual representations common between modalities at three stages and that conceptual access for words was delayed by about 90 msec with respect to pictures. We also show that for pictures, the first conceptual pattern of activity (shared between both words and pictures) occurs as early as 110 msec. Collectively, our results indicated that conceptual access at the category-level is a multistage process and that different delays in access across these two input-modalities determine when these representations are activated.

Semantic Features Reveal Different Networks During Word Processing: An EEG Source Localization Study

The neural principles behind semantic category representation are still under debate. Dominant theories mostly focus on distinguishing concrete from abstract concepts but, in such theories, divisions into categories of concrete concepts are more developed than for their abstract counterparts. An encompassing theory on semantic category representation could be within reach when charting the semantic attributes that are capable of describing both concept types. A good candidate are the three semantic dimensions defined by Osgood (potency, valence, arousal). However, to show to what extent they affect semantic processing, specific neuroimaging tools are required. Electroencephalography (EEG) is on par with the temporal resolution of cognitive behavior and source reconstruction. Using high-density set-ups, it is able to yield a spatial resolution in the scale of millimeters, sufficient to identify anatomical brain parcellations that could differentially contribute to semantic category representation. Cognitive neuroscientists traditionally focus on scalp domain analysis and turn to source reconstruction when an effect in the scalp domain has been detected. Traditional methods will potentially miss out on the fine-grained effects of semantic features as they are possibly obscured by the mixing of source activity due to volume conduction. For this reason, we have developed a mass-univariate analysis in the source domain using a mixed linear effect model. Our analyses reveal distinct networks of sources for different semantic features that are active during different stages of lexico-semantic processing of single words. With our method we identified differences in the spatio-temporal activation patterns of abstract and concrete words, high and low potency words, high and low valence words, and high and low arousal words, and in this way shed light on how word categories are represented in the brain.

Words affect visual perception by activating object shape representations

Linguistic labels are known to facilitate object recognition, yet the mechanism of this facilitation is not well understood. Previous psychophysical studies have suggested that words guide visual perception by activating information about visual object shape. Here we aimed to test this hypothesis at the neural level, and to tease apart the visual and semantic contribution of words to visual object recognition. We created a set of object pictures from two semantic categories with varying shapes, and obtained subjective ratings of their shape and category similarity. We then conducted a word-picture matching experiment, while recording participants’ EEG, and tested if the shape or the category similarity between the word’s referent and target picture explained the spatiotemporal pattern of the picture-evoked responses. The results show that hearing a word activates representations of its referent’s shape, which interacts with the visual processing of a subsequent picture within 100 ms from its onset. Furthermore, non-visual categorical information, carried by the word, affects the visual processing at later stages. These findings advance our understanding of the interaction between language and visual perception and provide insights into how the meanings of words are represented in the brain.

Attending to Visual Stimuli versus Performing Visual Imagery as a Control Strategy for EEG-based Brain-Computer Interfaces

Currently the most common imagery task used in Brain-Computer Interfaces (BCIs) is motor imagery, asking a user to imagine moving a part of the body. This study investigates the possibility to build BCIs based on another kind of mental imagery, namely "visual imagery". We study to what extent can we distinguish alternative mental processes of observing visual stimuli and imagining it to obtain EEG-based BCIs. Per trial, we instructed each of 26 users who participated in the study to observe a visual cue of one of two predefined images (a flower or a hammer) and then imagine the same cue, followed by rest. We investigated if we can differentiate between the different subtrial types from the EEG alone, as well as detect which image was shown in the trial. We obtained the following classifier performances: (i) visual imagery vs. visual observation task (71% of classification accuracy), (ii) visual observation task towards different visual stimuli (classifying one observation cue versus another observation cue with an accuracy of 61%) and (iii) resting vs. observation/imagery (77% of accuracy between imagery task versus resting state, and the accuracy of 75% between observation task versus resting state). Our results show that the presence of visual imagery and specifically related alpha power changes are useful to broaden the range of BCI control strategies.

A study of decoding human brain activities from simultaneous data of EEG and fMRI using MVPA

Neuroscientists have investigated the functionality of the brain in detail and achieved remarkable results but this area still need further research. Functional magnetic resonance imaging (fMRI) is considered as the most reliable and accurate technique to decode the human brain activity, on the other hand electroencephalography (EEG) is a portable and low cost solution in brain research. The purpose of this study is to find whether EEG can be used to decode the brain activity patterns like fMRI. In fMRI, data from a very specific brain region is enough to decode the brain activity patterns due to the quality of data. On the other hand, EEG can measure the rapid changes in neuronal activity patterns due to its higher temporal resolution i.e., in msec. These rapid changes mostly occur in different brain regions. In this study, multivariate pattern analysis (MVPA) is used both for EEG and fMRI data analysis and the information is extracted from distributed activation patterns of the brain. The significant information among different classes is extracted using two sample t test in both data sets. Finally, the classification analysis is done using the support vector machine. A fair comparison of both data sets is done using the same analysis techniques, moreover simultaneously collected data of EEG and fMRI is used for this comparison. The final analysis is done with the data of eight participants; the average result of all conditions are found which is 65.7% for EEG data set and 64.1% for fMRI data set. It concludes that EEG is capable of doing brain decoding with the data from multiple brain regions. In other words, decoding accuracy with EEG MVPA is as good as fMRI MVPA and is above chance level.

Exploring the Organization of Semantic Memory through Unsupervised Analysis of Event-related Potentials

Modern multivariate methods have enabled the application of unsupervised techniques to analyze neurophysiological data without strict adherence to predefined experimental conditions. We demonstrate a multivariate method that leverages priming effects on the evoked potential to perform hierarchical clustering on a set of word stimuli. The current study focuses on the semantic relationships that play a key role in the organization of our mental lexicon of words and concepts. The N400 component of the event-related potential is considered a reliable neurophysiological response that is indicative of whether accessing one concept facilitates subsequent access to another (i.e., one "primes" the other). To further our understanding of the organization of the human mental lexicon, we propose to utilize the N400 component to drive a clustering algorithm that can uncover, given a set of words, which particular subsets of words show mutual priming. Such a scheme requires a reliable measurement of the amplitude of the N400 component without averaging across many trials, which was here achieved using a recently developed multivariate analysis method based on beamforming. We validated our method by demonstrating that it can reliably detect, without any prior information about the nature of the stimuli, a well-known feature of the organization of our semantic memory: the distinction between animate and inanimate concepts. These results motivate further application of our method to data-driven exploration of disputed or unknown relationships between stimuli.

Classifying the Perceptual Interpretations of a Bistable Image Using EEG and Artificial Neural Networks

In order to classify different human brain states related to visual perception of ambiguous images, we use an artificial neural network (ANN) to analyze multichannel EEG. The classifier built on the basis of a multilayer perceptron achieves up to 95% accuracy in classifying EEG patterns corresponding to two different interpretations of the Necker cube. The important feature of our classifier is that trained on one subject it can be used for the classification of EEG traces of other subjects. This result suggests the existence of common features in the EEG structure associated with distinct interpretations of bistable objects. We firmly believe that the significance of our results is not limited to visual perception of the Necker cube images; the proposed experimental approach and developed computational technique based on ANN can also be applied to study and classify different brain states using neurophysiological data recordings. This may give new directions for future research in the field of cognitive and pathological brain activity, and for the development of brain-computer interfaces.

EEG Differentiation Analysis and Stimulus Set Meaningfulness

A set of images can be considered as meaningfully different for an observer if they can be distinguished phenomenally from one another. Each phenomenal difference must be supported by some neurophysiological differences. Differentiation analysis aims to quantify neurophysiological differentiation evoked by a given set of stimuli to assess its meaningfulness to the individual observer. As a proof of concept using high-density EEG, we show increased neurophysiological differentiation for a set of natural, meaningfully different images in contrast to another set of artificially generated, meaninglessly different images in nine participants. Stimulus-evoked neurophysiological differentiation (over 257 channels, 800 ms) was systematically greater for meaningful vs. meaningless stimulus categories both at the group level and for individual subjects. Spatial breakdown showed a central-posterior peak of differentiation, consistent with the visual nature of the stimulus sets. Temporal breakdown revealed an early peak of differentiation around 110 ms, prominent in the central-posterior region; and a later, longer-lasting peak at 300-500 ms that was spatially more distributed. The early peak of differentiation was not accompanied by changes in mean ERP amplitude, whereas the later peak was associated with a higher amplitude ERP for meaningful images. An ERP component similar to visual-awareness-negativity occurred during the nadir of differentiation across all image types. Control stimulus sets and further analysis indicate that changes in neurophysiological differentiation between meaningful and meaningless stimulus sets could not be accounted for by spatial properties of the stimuli or by stimulus novelty and predictability.

Top-down modulation of alpha power and pattern similarity for threatening representations in visual short-term memory

Recent studies have shown that top-down attention biases task-relevant representations in visual short-term memory (VSTM). Accumulating evidence has also revealed the modulatory effects of emotional arousal on attentional processing. However, it remains unclear how top-down attention interacts with emotional memoranda in VSTM. In this study, we investigated the mechanisms of alpha oscillations and their spatiotemporal characteristics that underlie top-down attention to threatening representations during VSTM maintenance with electroencephalography. Participants were instructed to remember a threatening object and a neutral object in a cued variant delayed response task. Retrospective cues (retro-cues) were presented to direct attention to the hemifield of a threatening object (i.e., cue-to-threat trials) or a neutral object (i.e., cue-to-neutral trials) during a retention interval prior to the probe test. We found a significant retro-cue-related alpha lateralisation over posterior regions during VSTM maintenance. The novel finding was that the magnitude of alpha lateralisation was greater for cue-to-threat objects compared to cue-to-neutral ones. These results indicated that directing attention towards threatening representations compared to neutral representations could result in greater regulation of alpha activity contralateral to the cued hemifield. Importantly, we estimated the spatiotemporal pattern similarity in alpha activity and found significantly higher similarity indexes for the posterior regions relative to the anterior regions and for the cue-to-threat objects relative to cue-to-neutral objects over the posterior regions. Together, our findings provided the oscillatory evidence of greater top-down modulations of alpha lateralisation and spatiotemporal pattern similarity for attending to threatening representations in VSTM.

The neuro-cognitive representations of symbols: the case of concrete words

We live our lives surrounded by symbols (e.g., road signs, logos, but especially words and numbers), and throughout our life we use them to evoke, communicate and reflect upon ideas and things that are not currently present to our senses. Symbols are represented in our brains at different levels of complexity: at the first and most simple level, as physical entities, in the corresponding primary and secondary sensory cortices. The crucial property of symbols, however, is that, despite the simplicity of their surface forms, they have the power of evoking higher order multifaceted representations that are implemented in distributed neural networks spanning a large portion of the cortex. The rich internal states that reflect our knowledge of the meaning of symbols are what we call semantic representations. In this review paper, we summarize our current knowledge of both the cognitive and neural substrates of semantic representations, focusing on concrete words (i.e., nouns or verbs referring to concrete objects and actions), which, together with numbers, are the most-studied and well defined classes of symbols. Following a systematic descriptive approach, we will organize this literature review around two key questions: what is the content of semantic representations? And, how are semantic representations implemented in the brain, in terms of localization and dynamics? While highlighting the main current opposing perspectives on these topics, we propose that a fruitful way to make substantial progress in this domain would be to adopt a geometrical view of semantic representations as points in high dimensional space, and to operationally partition the space of concrete word meaning into motor-perceptual and conceptual dimensions. By giving concrete examples of the kinds of research that can be done within this perspective, we illustrate how we believe this framework will foster theoretical speculations as well as empirical research.

Electroencephalogram-based decoding cognitive states using convolutional neural network and likelihood ratio based score fusion

Electroencephalogram (EEG)-based decoding human brain activity is challenging, owing to the low spatial resolution of EEG. However, EEG is an important technique, especially for brain-computer interface applications. In this study, a novel algorithm is proposed to decode brain activity associated with different types of images. In this hybrid algorithm, convolutional neural network is modified for the extraction of features, a t-test is used for the selection of significant features and likelihood ratio-based score fusion is used for the prediction of brain activity. The proposed algorithm takes input data from multichannel EEG time-series, which is also known as multivariate pattern analysis. Comprehensive analysis was conducted using data from 30 participants. The results from the proposed method are compared with current recognized feature extraction and classification/prediction techniques. The wavelet transform-support vector machine method is the most popular currently used feature extraction and prediction method. This method showed an accuracy of 65.7%. However, the proposed method predicts the novel data with improved accuracy of 79.9%. In conclusion, the proposed algorithm outperformed the current feature extraction and prediction method.

Object Recognition in Mental Representations: Directions for Exploring Diagnostic Features through Visual Mental Imagery

One of the fundamental goals of object recognition research is to understand how a cognitive representation produced from the output of filtered and transformed sensory information facilitates efficient viewer behavior. Given that mental imagery strongly resembles perceptual processes in both cortical regions and subjective visual qualities, it is reasonable to question whether mental imagery facilitates cognition in a manner similar to that of perceptual viewing: via the detection and recognition of distinguishing features. Categorizing the feature content of mental imagery holds potential as a reverse pathway by which to identify the components of a visual stimulus which are most critical for the creation and retrieval of a visual representation. This review will examine the likelihood that the information represented in visual mental imagery reflects distinctive object features thought to facilitate efficient object categorization and recognition during perceptual viewing. If it is the case that these representational features resemble their sensory counterparts in both spatial and semantic qualities, they may well be accessible through mental imagery as evaluated through current investigative techniques. In this review, methods applied to mental imagery research and their findings are reviewed and evaluated for their efficiency in accessing internal representations, and implications for identifying diagnostic features are discussed. An argument is made for the benefits of combining mental imagery assessment methods with diagnostic feature research to advance the understanding of visual perceptive processes, with suggestions for avenues of future investigation.