F-score feature selection based Bayesian reconstruction of visual image from human brain activity

Reconstructing Rapid Natural Vision with fMRI-Conditional Video Generative Adversarial Network

Recent functional magnetic resonance imaging (fMRI) studies have made significant progress in reconstructing perceived visual content, which advanced our understanding of the visual mechanism. However, reconstructing dynamic natural vision remains a challenge because of the limitation of the temporal resolution of fMRI. Here, we developed a novel fMRI-conditional video generative adversarial network (f-CVGAN) to reconstruct rapid video stimuli from evoked fMRI responses. In this model, we employed a generator to produce spatiotemporal reconstructions and employed two separate discriminators (spatial and temporal discriminators) for the assessment. We trained and tested the f-CVGAN on two publicly available video-fMRI datasets, and the model produced pixel-level reconstructions of 8 perceived video frames from each fMRI volume. Experimental results showed that the reconstructed videos were fMRI-related and captured important spatial and temporal information of the original stimuli. Moreover, we visualized the cortical importance map and found that the visual cortex is extensively involved in the reconstruction, whereas the low-level visual areas (V1/V2/V3/V4) showed the largest contribution. Our work suggests that slow blood oxygen level-dependent signals describe neural representations of the fast perceptual process that can be decoded in practice.

A neural decoding algorithm that generates language from visual activity evoked by natural images

Transforming neural activities into language is revolutionary for human-computer interaction as well as functional restoration of aphasia. Present rapid development of artificial intelligence makes it feasible to decode the neural signals of human visual activities. In this paper, a novel Progressive Transfer Language Decoding Model (PT-LDM) is proposed to decode visual fMRI signals into phrases or sentences when natural images are being watched. The PT-LDM consists of an image-encoder, a fMRI encoder and a language-decoder. The results showed that phrases and sentences were successfully generated from visual activities. Similarity analysis showed that three often-used evaluation indexes BLEU, ROUGE and CIDEr reached 0.182, 0.197 and 0.680 averagely between the generated texts and the corresponding annotated texts in the testing set respectively, significantly higher than the baseline. Moreover, we found that higher visual areas usually had better performance than lower visual areas and the contribution curve of visual response patterns in language decoding varied at successively different time points. Our findings demonstrate that the neural representations elicited in visual cortices when scenes are being viewed have already contained semantic information that can be utilized to generate human language. Our study shows potential application of language-based brain-machine interfaces in the future, especially for assisting aphasics in communicating more efficiently with fMRI signals.

A dual-channel language decoding from brain activity with progressive transfer training

When we view a scene, the visual cortex extracts and processes visual information in the scene through various kinds of neural activities. Previous studies have decoded the neural activity into single/multiple semantic category tags which can caption the scene to some extent. However, these tags are isolated words with no grammatical structure, insufficiently conveying what the scene contains. It is well‐known that textual language (sentences/phrases) is superior to single word in disclosing the meaning of images as well as reflecting people's real understanding of the images. Here, based on artificial intelligence technologies, we attempted to build a dual‐channel language decoding model (DC‐LDM) to decode the neural activities evoked by images into language (phrases or short sentences). The DC‐LDM consisted of five modules, namely, Image‐Extractor, Image‐Encoder, Nerve‐Extractor, Nerve‐Encoder, and Language‐Decoder. In addition, we employed a strategy of progressive transfer to train the DC‐LDM for improving the performance of language decoding. The results showed that the texts decoded by DC‐LDM could describe natural image stimuli accurately and vividly. We adopted six indexes to quantitatively evaluate the difference between the decoded texts and the annotated texts of corresponding visual images, and found that Word2vec‐Cosine similarity (WCS) was the best indicator to reflect the similarity between the decoded and the annotated texts. In addition, among different visual cortices, we found that the text decoded by the higher visual cortex was more consistent with the description of the natural image than the lower one. Our decoding model may provide enlightenment in language‐based brain‐computer interface explorations.

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Brain decoding based on functional magnetic resonance imaging has recently enabled the identification of visual perception and mental states. However, due to the limitations of sample size and the lack of an effective reconstruction model, accurate reconstruction of natural images is still a major challenge. The current, rapid development of deep learning models provides the possibility of overcoming these obstacles. Here, we propose a deep learning-based framework that includes a latent feature extractor, a latent feature decoder, and a natural image generator, to achieve the accurate reconstruction of natural images from brain activity. The latent feature extractor is used to extract the latent features of natural images. The latent feature decoder predicts the latent features of natural images based on the response signals from the higher visual cortex. The natural image generator is applied to generate reconstructed images from the predicted latent features of natural images and the response signals from the visual cortex. Quantitative and qualitative evaluations were conducted with test images. The results showed that the reconstructed image achieved comparable, accurate reproduction of the presented image in both high-level semantic category information and low-level pixel information. The framework we propose shows promise for decoding the brain activity.

eHSCPr discriminating the cell identity involved in endothelial to hematopoietic transition

MOTIVATION

Hematopoietic stem cells (HSCs) give rise to all blood cells and play a vital role throughout the whole lifespan through their pluripotency and self-renewal properties. Accurately identifying the stages of early HSCs is extremely important, as it may open up new prospects for extracorporeal blood research. Existing experimental techniques for identifying the early stages of HSCs development are time-consuming and expensive. Machine learning has shown its excellence in massive single-cell data processing and it is desirable to develop related computational models as good complements to experimental techniques.

RESULTS

In this study, we presented a novel predictor called eHSCPr specifically for predicting the early stages of HSCs development. To reveal the distinct genes at each developmental stage of HSCs, we compared F-score with three state-of-art differential gene selection methods (limma, DESeq2, edgeR) and evaluated their performance. F-score captured the more critical surface markers of endothelial cells and hematopoietic cells, and the area under receiver operating characteristic curve (ROC) value was 0.987. Based on SVM, the 10-fold cross-validation accuracy of eHSCpr in the independent dataset and the training dataset reached 94.84% and 94.19%, respectively. Importantly, we performed transcription analysis on the F-score gene set, which indeed further enriched the signal markers of HSCs development stages. eHSCPr can be a powerful tool for predicting early stages of HSCs development, facilitating hypothesis-driven experimental design and providing crucial clues for the in vitro blood regeneration studies.

AVAILABILITY

http://bioinfor.imu.edu.cn/ehscpr.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Long short-term memory-based neural decoding of object categories evoked by natural images

Visual perceptual decoding is one of the important and challenging topics in cognitive neuroscience. Building a mapping model between visual response signals and visual contents is the key point of decoding. Most previous studies used peak response signals to decode object categories. However, brain activities measured by functional magnetic resonance imaging are a dynamic process with time dependence, so peak signals cannot fully represent the whole process, which may affect the performance of decoding. Here, we propose a decoding model based on long short‐term memory (LSTM) network to decode five object categories from multitime response signals evoked by natural images. Experimental results show that the average decoding accuracy using the multitime (2–6 s) response signals is 0.540 from the five subjects, which is significantly higher than that using the peak ones (6 s; accuracy: 0.492; p < .05). In addition, from the perspective of different durations, methods and visual areas, the decoding performances of the five object categories are deeply and comprehensively explored. The analysis of different durations and decoding methods reveals that the LSTM‐based decoding model with sequence simulation ability can fit the time dependence of the multitime visual response signals to achieve higher decoding performance. The comparative analysis of different visual areas demonstrates that the higher visual cortex (VC) contains more semantic category information needed for visual perceptual decoding than lower VC.

Integrating chemical similarity and bioequivalence: A pilot study on quality consistency evaluation of dispensing granule and traditional decoction of Scutellariae Radix by a totality-of-the-evidence approach

There is an increasing focus on the quality consistency evaluation of dispensing granule in traditional Chinese medicines (TCMs). According to the guideline from Chinese Pharmacopoeia Commission, the substantial equivalence of dispensing granule and traditional decoction should be determined, and the chromatographic fingerprint has been recommended as a comprehensive qualitative approach to assess the quality consistency between dispensing granule and traditional decoction. However, a high-degree chemical similarity does not equal a bioequivalence. Attempting to realize the quality evaluation by integrating chemical consistency and bioequivalence, we herein proposed a totality-of-the-evidence approach based on clustering analysis and equivalence evaluation taking the dispensing granule and traditional decoction of Scutellariae Radix (SR) as a typical case. Chemical fingerprints were developed by high performance liquid chromatography coupled with photodiode array detector and quadrupole time-of-flight mass spectrometry (HPLC-PDA/QTOF-MS). Subsequently, a feature selection strategy, integrated linear and nonlinear correlation analysis, was carried out to assess the correlation between chemical profiles and biological activities. Finally, quality consistency between the dispensing granule and the traditional decoction was determined by bioactive marker-guided hierarchical clustering analysis (HCA), k-means clustering method and bioequivalence evaluation. The available evidence suggested that not all the dispensing granule of SR were sufficiently similar to the traditional decoction. This study provides an applicable methodology for quality consistency evaluation of dispensing granule and traditional decoction in TCMs.