Artificial intelligence based multimodal language decoding from brain activity: A review

Decoding brain activity is conducive to the breakthrough of brain-computer interface (BCI) technology. The development of artificial intelligence (AI) continually promotes the progress of brain language decoding technology. Existent research has mainly focused on a single modality and paid insufficient attention to AI methods. Therefore, our objective is to provide an overview of relevant decoding research from the perspective of different modalities and methodologies. The modalities involve text, speech, image, and video, whereas the core method is using AI-built decoders to translate brain signals induced by multimodal stimuli into text or vocal language. The semantic information of brain activity can be successfully decoded into a language at various levels, ranging from words through sentences to discourses. However, the decoding effect is affected by various factors, such as the decoding model, vector representation model, and brain regions. Challenges and future directions are also discussed. The advances in brain language decoding and BCI technology will potentially assist patients with clinical aphasia in regaining the ability to communicate.

APT-Net: Adaptive encoding and parallel decoding transformer for medical image segmentation

There are limitations in the study of transformer-based medical image segmentation networks for token position encoding and decoding of images. The position encoding module cannot encode the position information adequately, and the serial decoder cannot utilize the contextual information efficiently. In this paper, we propose a new CNN-transformer hybrid structure for the medical image segmentation network APT-Net based on the encoder-decoder architecture. The network introduces an adaptive position encoding module for the fusion of position information of a multi-receptive field to provide more adequate position information for the token sequences in the transformer. In addition, the dual-path parallel decoder's basic and guide information paths simultaneously process multiscale feature maps to efficiently utilize contextual information. We conducted extensive experiments and reported a number of important metrics from multiple perspectives on seven datasets containing skin lesions, polyps, and glands. The IoU reached 0.783 and 0.851 on the ISIC2017 and Glas datasets, respectively. To the best of our knowledge, APT-Net achieves state-of-the-art performance on the Glas dataset and polyp segmentation tasks. Ablation experiments validate the effectiveness of the proposed adaptive position encoding module and the dual-path parallel decoder. Comparative experiments with state-of-the-art methods demonstrate the high accuracy and portability of APT-Net.

Adaptive cascade decoders for segmenting challenging regions in medical images

CNN-based techniques have achieved impressive outcomes in medical image segmentation but struggle to capture long-term dependencies between pixels. The Transformer, with its strong feature extraction and representation learning abilities, performs exceptionally well within the domain of medical image partitioning. However, there are still shortcomings in bridging local to global connections, resulting in occasional loss of positional information. To address this, we introduce a decoder based on dynamic convolution, called Adaptive Cascade Decoder (ACD). It can adaptively adjust the receptive field size based on medical images, adapting a set of parameters for each medical image individually. The ACD consists of an Adaptive Attention (ADA) and a Multi-Scale Convolution module (MSC). By enhancing feature extraction from local to global scales, it addresses issues of diminished contrast and fuzzy boundaries common in medical image segmentation. While increasing contextual connections, it also reduces certain parameters, thereby lowering memory consumption. Our model, T-ACD, uses the encoder backbone of TransUNet, which chunks feature maps from convolutional neural networks and feeds them as one-dimensional sequences into the Transformer. This leverages the Transformer's prowess in handling sequences, further refining the extracted features. In experiments involving heart and multi-organ segmentation, T-ACD excels in segmenting challenging areas. On the ACDC dataset, we achieve a DICE coefficient of 92.02 %. For the most challenging right ventricle, it is improved to 90.68 %, which is increased by 5.17 %. In the realm of medical segmentation, the core design of ACD can be generalized to other challenging organ segmentations.

A novel deep neural network-based technique for network embedding

In this paper, the graph segmentation (GSeg) method has been proposed. This solution is a novel graph neural network framework for network embedding that leverages the inherent characteristics of nodes and the underlying local network topology. The key innovation of GSeg lies in its encoder-decoder architecture, which is specifically designed to preserve the network's structural properties. The key contributions of GSeg are: (1) a novel graph neural network architecture that effectively captures local and global network structures, and (2) a robust node representation learning approach that achieves superior performance in various network analysis tasks. The methodology employed in our study involves the utilization of a graph neural network framework for the acquisition of node representations. The design leverages the inherent characteristics of nodes and the underlying local network topology. To enhance the architectural framework of encoder- decoder networks, the GSeg model is specifically devised to exhibit a structural resemblance to the SegNet model. The obtained empirical results on multiple benchmark datasets demonstrate that the GSeg outperforms existing state-of-the-art methods in terms of network structure preservation and prediction accuracy for downstream tasks. The proposed technique has potential utility across a range of practical applications in the real world.

ESIPT-based dual-emissive perimidine derivative as a rapid and sensitive sensor for Cu2+ and Al3+: Construction of memory device, 2-to-1 encoder and 1-to-2 decoder

An ESIPT based fluorescent perimidine derivative oPSDAN was developed and characterized by ¹H NMR, ¹³C NMR and mass spectroscopy. The study of the photo-physical properties of the sensor unveiled its selectivity and sensitivity towards Cu²⁺ and Al³⁺ ions. The sensing of ions was accompanied by colorimetric change (for Cu²⁺) as well as emission turn-off response. The binding stoichiometries of sensor oPSDAN with Cu²⁺ ion and Al³⁺ ions were determined to be 2:1 and 1:1, respectively. The binding constants and detection limits for Cu²⁺ and Al³⁺ were calculated from the UV-vis and fluorescence titration profiles as, 7.1 × 10⁴ M^-1, 1.9 × 10⁴ M^-1 and 9.89 nM, 1.5 × 10^-8 M, respectively. The mechanism was established by ¹H NMR as well as mass titrations and was supported by DFT and TD-DFT calculations. The UV-vis and fluorescence spectral results were further utilized for construction of memory device, encoder and decoder. Sensor-oPSDAN was also tested for determining Cu²⁺ ions in drinking water.

Multimodal Abstractive Summarization using bidirectional encoder representations from transformers with attention mechanism

In recent decades, abstractive text summarization using multimodal input has attracted many researchers due to the capability of gathering information from various sources to create a concise summary. However, the existing methodologies based on multimodal summarization provide only a summary for the short videos and poor results for the lengthy videos. To address the aforementioned issues, this research presented the Multimodal Abstractive Summarization using Bidirectional Encoder Representations from Transformers (MAS-BERT) with an attention mechanism. The purpose of the video summarization is to increase the speed of searching for a large collection of videos so that the users can quickly decide whether the video is relevant or not by reading the summary. Initially, the data is obtained from the publicly available How2 dataset and is encoded using the Bidirectional Gated Recurrent Unit (Bi-GRU) encoder and the Long Short Term Memory (LSTM) encoder. The textual data which is embedded in the embedding layer is encoded using a bidirectional GRU encoder and the features with audio and video data are encoded with LSTM encoder. After this, BERT based attention mechanism is used to combine the modalities and finally, the BI-GRU based decoder is used for summarizing the multimodalities. The results obtained through the experiments that show the proposed MAS-BERT has achieved a better result of 60.2 for Rouge-1 whereas, the existing Decoder-only Multimodal Transformer (D-MmT) and the Factorized Multimodal Transformer based Decoder Only Language model (FLORAL) has achieved 49.58 and 56.89 respectively. Our work facilitates users by providing better contextual information and user experience and would help video-sharing platforms for customer retention by allowing users to search for relevant videos by looking at its summary.