Automated classification of fMRI data employing trial-based imagery tasks

fMRI volume classification using a 3D convolutional neural network robust to shifted and scaled neuronal activations

Deep-learning methods based on deep neural networks (DNNs) have recently been successfully utilized in the analysis of neuroimaging data. A convolutional neural network (CNN) is a type of DNN that employs a convolution kernel that covers a local area of the input sample and moves across the sample to provide a feature map for the subsequent layers. In our study, we hypothesized that a 3D-CNN model with down-sampling operations such as pooling and/or stride would have the ability to extract robust feature maps from the shifted and scaled neuronal activations in a single functional MRI (fMRI) volume for the classification of task information associated with that volume. Thus, the 3D-CNN model would be able to ameliorate the potential misalignment of neuronal activations and over-/under-activation in local brain regions caused by imperfections in spatial alignment algorithms, confounded by variability in blood-oxygenation-level-dependent (BOLD) responses across sessions and/or subjects. To this end, the fMRI volumes acquired from four sensorimotor tasks (left-hand clenching, right-hand clenching, auditory attention, and visual stimulation) were used as input for our 3D-CNN model to classify task information using a single fMRI volume. The classification performance of the 3D-CNN was systematically evaluated using fMRI volumes obtained from various minimal preprocessing scenarios applied to raw fMRI volumes that excluded spatial normalization to a template and those obtained from full preprocessing that included spatial normalization. Alternative classifier models such as the 1D fully connected DNN (1D-fcDNN) and support vector machine (SVM) were also used for comparison. The classification performance was also assessed for several k-fold cross-validation (CV) schemes, including leave-one-subject-out CV (LOOCV). Overall, the classification results of the 3D-CNN model were superior to that of the 1D-fcDNN and SVM models. When using the fully-processed fMRI volumes with LOOCV, the mean error rates (± the standard error of the mean) for the 3D-CNN, 1D-fcDNN, and SVM models were 2.1% (± 0.9), 3.1% (± 1.2), and 4.1% (± 1.5), respectively (p = 0.041 from a one-way ANOVA). The error rates for 3-fold CV were higher (2.4% ± 1.0, 4.2% ± 1.3, and 10.1% ± 2.0; p < 0.0003 from a one-way ANOVA). The mean error rates also increased considerably using the raw fMRI 3D volume data without preprocessing (26.2% for the 3D-CNN, 75.0% for the 1D-fcDNN, and 75.0% for the SVM). Furthermore, the ability of the pre-trained 3D-CNN model to handle shifted and scaled neuronal activations was demonstrated in an online scenario for five-class classification (i.e., four sensorimotor tasks and the resting state) using the real-time fMRI of three participants. The resulting classification accuracy was 78.5% (± 1.4), 26.7% (± 5.9), and 21.5% (± 3.1) for the 3D-CNN, 1D-fcDNN, and SVM models, respectively. The superior performance of the 3D-CNN compared to the 1D-fcDNN was verified by analyzing the resulting feature maps and convolution filters that handled the shifted and scaled neuronal activations and by utilizing an independent public dataset from the Human Connectome Project.

Exploration and Research of Human Identification Scheme Based on Inertial Data

The identification work based on inertial data is not limited by space, and has high flexibility and concealment. Previous research has shown that inertial data contains information related to behavior categories. This article discusses whether inertial data contains information related to human identity. The classification experiment, based on the neural network feature fitting function, achieves 98.17% accuracy on the test set, confirming that the inertial data can be used for human identification. The accuracy of the classification method without feature extraction on the test set is only 63.84%, which further indicates the need for extracting features related to human identity from the changes in inertial data. In addition, the research on classification accuracy based on statistical features discusses the effect of different feature extraction functions on the results. The article also discusses the dimensionality reduction processing and visualization results of the collected data and the extracted features, which helps to intuitively assess the existence of features and the quality of different feature extraction effects.

Real-time fMRI for brain-computer interfacing

Brain-computer interfaces (BCIs) based on functional magnetic resonance imaging (fMRI) provide an important complement to other noninvasive BCIs. While fMRI has several disadvantages (being nonportable, methodologically challenging, costly, and noisy), it is the only method providing high spatial resolution whole-brain coverage of brain activation. These properties allow relating mental activities to specific brain regions and networks providing a transparent scheme for BCI users to encode information and for real-time fMRI BCI systems to decode the intents of the user. Various mental activities have been used successfully in fMRI BCIs so far that can be classified into the four categories: (a) higher-order cognitive tasks (e.g., mental calculation), (b) covert language-related tasks (e.g., mental speech and mental singing), (c) imagery tasks (motor, visual, auditory, tactile, and emotion imagery), and (d) selective attention tasks (visual, auditory, and tactile attention). While the ultimate spatial and temporal resolution of fMRI BCIs is limited by the physiologic properties of the hemodynamic response, technical and analytical advances will likely lead to substantially improved fMRI BCIs in the future using, for example, decoding of imagined letter shapes at 7T as the basis for more "natural" communication BCIs.

Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: Evaluation using sensorimotor tasks

Feedforward deep neural networks (DNNs), artificial neural networks with multiple hidden layers, have recently demonstrated a record-breaking performance in multiple areas of applications in computer vision and speech processing. Following the success, DNNs have been applied to neuroimaging modalities including functional/structural magnetic resonance imaging (MRI) and positron-emission tomography data. However, no study has explicitly applied DNNs to 3D whole-brain fMRI volumes and thereby extracted hidden volumetric representations of fMRI that are discriminative for a task performed as the fMRI volume was acquired. Our study applied fully connected feedforward DNN to fMRI volumes collected in four sensorimotor tasks (i.e., left-hand clenching, right-hand clenching, auditory attention, and visual stimulus) undertaken by 12 healthy participants. Using a leave-one-subject-out cross-validation scheme, a restricted Boltzmann machine-based deep belief network was pretrained and used to initialize weights of the DNN. The pretrained DNN was fine-tuned while systematically controlling weight-sparsity levels across hidden layers. Optimal weight-sparsity levels were determined from a minimum validation error rate of fMRI volume classification. Minimum error rates (mean±standard deviation; %) of 6.9 (±3.8) were obtained from the three-layer DNN with the sparsest condition of weights across the three hidden layers. These error rates were even lower than the error rates from the single-layer network (9.4±4.6) and the two-layer network (7.4±4.1). The estimated DNN weights showed spatial patterns that are remarkably task-specific, particularly in the higher layers. The output values of the third hidden layer represented distinct patterns/codes of the 3D whole-brain fMRI volume and encoded the information of the tasks as evaluated from representational similarity analysis. Our reported findings show the ability of the DNN to classify a single fMRI volume based on the extraction of hidden representations of fMRI volumes associated with tasks across multiple hidden layers. Our study may be beneficial to the automatic classification/diagnosis of neuropsychiatric and neurological diseases and prediction of disease severity and recovery in (pre-) clinical settings using fMRI volumes without requiring an estimation of activation patterns or ad hoc statistical evaluation.

Improving hearing aid fitting using the speech-evoked auditory brainstem response

Measuring brain responses to speech may help improve the process of hearing aid fitting, especially in young children. Speech-evoked auditory brainstem responses (sABR) may be particularly useful because they provide a spectro-temporal representation of auditory neural activity in response to speech. However, use of the sABR in evaluating hearing aid performance has not been explored. This paper reviews recent work on measuring brain responses to speech, illustrates how sABR can provide insights into internal auditory processing, and proposes ways in which these responses may be used to improve hearing aid fitting.

Decision tree structure based classification of EEG signals recorded during two dimensional cursor movement imagery

BACKGROUND

Input signals of an EEG based brain computer interface (BCI) system are naturally non-stationary, have poor signal to noise ratio, depend on physical or mental tasks and are contaminated with various artifacts such as external electromagnetic waves, electromyogram and electrooculogram. All these disadvantages have motivated researchers to substantially improve speed and accuracy of all components of the communication system between brain and a BCI output device.

NEW METHOD

In this study, a fast and accurate decision tree structure based classification method was proposed for classifying EEG data to up/down/right/left computer cursor movement imagery EEG data. The data sets were acquired from three healthy human subjects in age group of between 24 and 29 years old in two sessions on different days.

RESULTS

The proposed decision tree structure based method was successfully applied to the present data sets and achieved 55.92%, 57.90% and 82.24% classification accuracy rate on the test data of three subjects.

COMPARISON WITH EXISTING METHOD(S)

The results indicated that the proposed method provided 12.25% improvement over the best results of the most closely related studies although the EEG signals were collected on two different sessions with about 1 week interval.

CONCLUSIONS

The proposed method required only a training set of the subject and automatically generated specific DTS for each new subject by determining the most appropriate feature set and classifier for each node. Additionally, with further developments of feature extraction and/or classification algorithms, any existing node can be easily replaced with new one without breaking the whole DTS. This attribute makes the proposed method flexible.

Enhanced activation of motor execution networks using action observation combined with imagination of lower limb movements

The combination of first-person observation and motor imagery, i.e. first-person observation of limbs with online motor imagination, is commonly used in interactive 3D computer gaming and in some movie scenes. These scenarios are designed to induce a cognitive process in which a subject imagines himself/herself acting as the agent in the displayed movement situation. Despite the ubiquity of this type of interaction and its therapeutic potential, its relationship to passive observation and imitation during observation has not been directly studied using an interactive paradigm. In the present study we show activation resulting from observation, coupled with online imagination and with online imitation of a goal-directed lower limb movement using functional MRI (fMRI) in a mixed block/event-related design. Healthy volunteers viewed a video (first-person perspective) of a foot kicking a ball. They were instructed to observe-only the action (O), observe and simultaneously imagine performing the action (O-MI), or imitate the action (O-IMIT). We found that when O-MI was compared to O, activation was enhanced in the ventralpremotor cortex bilaterally, left inferior parietal lobule and left insula. The O-MI and O-IMIT conditions shared many activation foci in motor relevant areas as confirmed by conjunction analysis. These results show that (i) combining observation with motor imagery (O-MI) enhances activation compared to observation-only (O) in the relevant foot motor network and in regions responsible for attention, for control of goal-directed movements and for the awareness of causing an action, and (ii) it is possible to extensively activate the motor execution network using O-MI, even in the absence of overt movement. Our results may have implications for the development of novel virtual reality interactions for neurorehabilitation interventions and other applications involving training of motor tasks.

The brain's silent messenger: using selective attention to decode human thought for brain-based communication

The interpretation of human thought from brain activity, without recourse to speech or action, is one of the most provoking and challenging frontiers of modern neuroscience. In particular, patients who are fully conscious and awake, yet, due to brain damage, are unable to show any behavioral responsivity, expose the limits of the neuromuscular system and the necessity for alternate forms of communication. Although it is well established that selective attention can significantly enhance the neural representation of attended sounds, it remains, thus far, untested as a response modality for brain-based communication. We asked whether its effect could be reliably used to decode answers to binary (yes/no) questions. Fifteen healthy volunteers answered questions (e.g., "Do you have brothers or sisters?") in the fMRI scanner, by selectively attending to the appropriate word ("yes" or "no"). Ninety percent of the answers were decoded correctly based on activity changes within the attention network. The majority of volunteers conveyed their answers with less than 3 min of scanning, suggesting that this technique is suited for communication in a reasonable amount of time. Formal comparison with the current best-established fMRI technique for binary communication revealed improved individual success rates and scanning times required to detect responses. This novel fMRI technique is intuitive, easy to use in untrained participants, and reliably robust within brief scanning times. Possible applications include communication with behaviorally nonresponsive patients.

Can a single brain region predict a disorder?

We perform prediction of diverse disorders (Cocaine Use, Schizophrenia and Alzheimers disease) in unseen subjects from brain fMRI. First, we show that for multi-subject prediction of simple cognitive states (e.g. motor vs. calculation and reading), voxels-as-features methods produce clusters that are similar for different leave-one-subject-out folds; while for group classification (e.g. cocaine addicted vs. control subjects), voxels are scattered and less stable. Therefore, we chose to use a single region per experimental condition and a majority vote classifier. Interestingly, our method outperforms state-of-the-art techniques. Our method can integrate multiple experimental conditions and successfully predict disorders in unseen subjects (leave-one-subjectout generalization accuracy: 89.3% and 90.9% for Cocaine Use, 96.4% for Schizophrenia and 81.5% for Alzheimers disease). Our experimental results not only span diverse disorders, but also different experimental designs (block design and event related tasks), facilities, magnetic fields (1.5Tesla, 3Tesla, 4Tesla) and speed of acquisition (interscan interval from 1600ms to 3500ms). We further argue that our method produces a meaningful low dimensional representation that retains discriminability.

Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study

Both embodied and symbolic accounts of conceptual organization would predict partial sharing and partial differentiation between the neural activations seen for concepts activated via different stimulus modalities. But cross-participant and cross-session variability in BOLD activity patterns makes analyses of such patterns with MVPA methods challenging. Here, we examine the effect of cross-modal and individual variation on the machine learning analysis of fMRI data recorded during a word property generation task. We present the same set of living and non-living concepts (land-mammals, or work tools) to a cohort of Japanese participants in two sessions: the first using auditory presentation of spoken words; the second using visual presentation of words written in Japanese characters. Classification accuracies confirmed that these semantic categories could be detected in single trials, with within-session predictive accuracies of 80–90%. However cross-session prediction (learning from auditory-task data to classify data from the written-word-task, or vice versa) suffered from a performance penalty, achieving 65–75% (still individually significant at p « 0.05). We carried out several follow-on analyses to investigate the reason for this shortfall, concluding that distributional differences in neither time nor space alone could account for it. Rather, combined spatio-temporal patterns of activity need to be identified for successful cross-session learning, and this suggests that feature selection strategies could be modified to take advantage of this.

Dreaming and the brain: from phenomenology to neurophysiology

Dreams are a remarkable experiment in psychology and neuroscience, conducted every night in every sleeping person. They show that the human brain, disconnected from the environment, can generate an entire world of conscious experiences by itself. Content analysis and developmental studies have promoted understanding of dream phenomenology. In parallel, brain lesion studies, functional imaging and neurophysiology have advanced current knowledge of the neural basis of dreaming. It is now possible to start integrating these two strands of research to address fundamental questions that dreams pose for cognitive neuroscience: how conscious experiences in sleep relate to underlying brain activity; why the dreamer is largely disconnected from the environment; and whether dreaming is more closely related to mental imagery or to perception.

The neural substrates for atypical planning and execution of word production in stuttering

Using an fMRI-based classification approach and the structural equation modeling (SEM) method, this study examined the neural bases of atypical planning and execution processes involved in stuttering. Twelve stuttering speakers and 12 controls were asked to name pictures under different conditions (single-syllable, multi-syllable, or repeated-syllable) in the scanner. The contrasts between conditions provided information about planning and execution processes. The classification analysis showed that, as compared to non-stuttering controls, stuttering speakers' atypical planning of speech was evident in their neural activities in the bilateral inferior frontal gyrus (IFG) and right putamen and their atypical execution of speech was evident in their activations in the right cerebellum and insula, left premotor area (PMA), and angular gyrus (AG). SEM results further revealed two parallel neural circuits-the basal ganglia-IFG/PMA circuit and the cerebellum-PMA circuit-that were involved in atypical planning and execution processes of stuttering, respectively. The AG appeared to be involved in the interface of atypical planning and execution in stuttering. These results are discussed in terms of their implications to the theories about stuttering and to clinical applications.

Another kind of 'BOLD Response': answering multiple-choice questions via online decoded single-trial brain signals

The term 'locked-in'syndrome (LIS) describes a medical condition in which persons concerned are severely paralyzed and at the same time fully conscious and awake. The resulting anarthria makes it impossible for these patients to naturally communicate, which results in diagnostic as well as serious practical and ethical problems. Therefore, developing alternative, muscle-independent communication means is of prime importance. Such communication means can be realized via brain-computer interfaces (BCIs) circumventing the muscular system by using brain signals associated with preserved cognitive, sensory, and emotional brain functions. Primarily, BCIs based on electrophysiological measures have been developed and applied with remarkable success. Recently, also blood flow-based neuroimaging methods, such as functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS), have been explored in this context. After reviewing recent literature on the development of especially hemodynamically based BCIs, we introduce a highly reliable and easy-to-apply communication procedure that enables untrained participants to motor-independently and relatively effortlessly answer multiple-choice questions based on intentionally generated single-trial fMRI signals that can be decoded online. Our technique takes advantage of the participants' capability to voluntarily influence certain spatio-temporal aspects of the blood oxygenation level-dependent (BOLD) signal: source location (by using different mental tasks), signal onset and offset. We show that healthy participants are capable of hemodynamically encoding at least four distinct information units on a single-trial level without extensive pretraining and with little effort. Moreover, real-time data analysis based on simple multi-filter correlations allows for automated answer decoding with a high accuracy (94.9%) demonstrating the robustness of the presented method. Following our 'proof of concept', the next step will involve clinical trials with LIS patients, undertaken in close collaboration with their relatives and caretakers in order to elaborate individually tailored communication protocols. As our procedure can be easily transferred to MRI-equipped clinical sites, it may constitute a simple and effective possibility for online detection of residual consciousness and for LIS patients to communicate basic thoughts and needs in case no other alternative communication means are available (yet)--especially in the acute phase of the LIS. Future research may focus on further increasing the efficiency and accuracy of fMRI-based BCIs by implementing sophisticated data analysis methods (e.g., multivariate and independent component analysis) and neurofeedback training techniques. Finally, the presented BCI approach could be transferred to portable fNIRS systems as only this would enable hemodynamically based communication in daily life situations.