TAMNR: a network embedding learning algorithm using text attention mechanism

Because many existing algorithms are mainly trained based on the structural features of the networks, the results are more inclined to the structural commonality of the networks. These algorithms ignore the rich external information and node attributes (such as node text content, community and labels, etc.) that have important implications for network data analysis tasks. Existing network embedding algorithms considering text features usually regard the co-occurrence words in the node's text, or use an induced matrix completion algorithm to factorize the text feature matrix or the network structure feature matrix. Although this kind of algorithm can greatly improve the network embedding performance, they ignore the contribution rate of different co-occurrence words in the node's text. This article proposes a network embedding learning algorithm combining network structure and co-occurrence word features, also incorporating an attention mechanism to model the weight information of the co-occurrence words in the model. This mechanism filters out unimportant words and focuses on important words for learning and training tasks, fully considering the impact of the different co-occurrence words to the model. The proposed network representation algorithm is tested on three open datasets, and the experimental results demonstrate its strong advantages in node classification, visualization analysis, and case analysis tasks.

FIRE: knowledge-enhanced recommendation with feature interaction and intent-aware attention networks

To solve the information overload issue and enhance the user experience of various web applications, recommender systems aim to better model user interests and preferences. Knowledge Graphs (KGs), consisting of real-world objective facts and fruitful entities, play a vital role in recommender systems. Recently, a technological trend has been to develop end-to-end Graph Neural Networks (GNNs)-based knowledge-aware recommendation (a.k.a., Knowledge Graph Recommendation, KGR) models. Unfortunately, current GNNs-based KGR approaches focus on how to capture high-order feature information on KGs while neglecting the following two crucial limitations: 1) The explicitly high-order feature interaction and fusion mechanism and 2) The valid user intent modelling mechanism. As such, these issues lead to insufficient user/item representation learning capability and unsatisfactory KGR performance. In this work, we present a novel Knowledge-enhanced Re commendation with F eature I nteraction and Intent-aware Attention Networks (FIRE) to address the latent intent modelling and high-order feature interaction deficiencies ignored by existing KGR methods. Based on the prototype user/item representation learning leveraging the GNNs-based approach, our model offers the following major improvements: One is the innovative use of Convolutional Neural Networks (CNNs) that perform vertical convolutional (a.k.a., bit-level convolutional) and horizontal convolutional (a.k.a., vector-level convolutional) processes to model multi-granular high-order feature interactions to enhance item-side representation learning. Another is to model users' latent intent factors by utilizing a two-level attention mechanism (i.e., node- and intent-level attention mechanism) to enhance user-side representation learning. Extensive experiments on three KGs domain public datasets demonstrate that our method outperforms the existing state-of-the-art baseline. Last but not least, numerous ablation- and model studies demystify the working mechanism and elucidate the plausibility of the proposed model.

Cross-scale Attention Guided Multi-instance Learning for Crohn's Disease Diagnosis with Pathological Images

Multi-instance learning (MIL) is widely used in the computer-aided interpretation of pathological Whole Slide Images (WSIs) to solve the lack of pixel-wise or patch-wise annotations. Often, this approach directly applies "natural image driven" MIL algorithms which overlook the multi-scale (i.e. pyramidal) nature of WSIs. Off-the-shelf MIL algorithms are typically deployed on a single-scale of WSIs (e.g., 20× magnification), while human pathologists usually aggregate the global and local patterns in a multi-scale manner (e.g., by zooming in and out between different magnifications). In this study, we propose a novel cross-scale attention mechanism to explicitly aggregate inter-scale interactions into a single MIL network for Crohn's Disease (CD), which is a form of inflammatory bowel disease. The contribution of this paper is two-fold: (1) a cross-scale attention mechanism is proposed to aggregate features from different resolutions with multi-scale interaction; and (2) differential multi-scale attention visualizations are generated to localize explainable lesion patterns. By training ~250,000 H&E-stained Ascending Colon (AC) patches from 20 CD patient and 30 healthy control samples at different scales, our approach achieved a superior Area under the Curve (AUC) score of 0.8924 compared with baseline models. The official implementation is publicly available at https://github.com/hrlblab/CS-MIL.

Learning to predict in-hospital mortality risk in the intensive care unit with attention-based temporal convolution network

BACKGROUND

Dynamic prediction of patient mortality risk in the ICU with time series data is limited due to high dimensionality, uncertainty in sampling intervals, and other issues. A new deep learning method, temporal convolution network (TCN), makes it possible to deal with complex clinical time series data in ICU. We aimed to develop and validate it to predict mortality risk using time series data from MIMIC III dataset.

METHODS

A total of 21,139 records of ICU stays were analysed and 17 physiological variables from the MIMIC III dataset were used to predict mortality risk. Then we compared the model performance of the attention-based TCN with that of traditional artificial intelligence (AI) methods.

RESULTS

The area under receiver operating characteristic (AUCROC) and area under precision-recall curve (AUC-PR) of attention-based TCN for predicting the mortality risk 48 h after ICU admission were 0.837 (0.824 -0.850) and 0.454, respectively. The sensitivity and specificity of attention-based TCN were 67.1% and 82.6%, respectively, compared to the traditional AI method, which had a low sensitivity (< 50%).

CONCLUSIONS

The attention-based TCN model achieved better performance in the prediction of mortality risk with time series data than traditional AI methods and conventional score-based models. The attention-based TCN mortality risk model has the potential for helping decision-making for critical patients.

TRIAL REGISTRATION

Data used for the prediction of mortality risk were extracted from the freely accessible MIMIC III dataset. The project was approved by the Institutional Review Boards of Beth Israel Deaconess Medical Center (Boston, MA) and the Massachusetts Institute of Technology (Cambridge, MA). Requirement for individual patient consent was waived because the project did not impact clinical care and all protected health information was deidentified. The data were accessed via a data use agreement between PhysioNet, a National Institutes of Health-supported data repository (https://www.physionet.org/), and one of us (Yu-wen Chen, Certification Number: 28341490). All methods were carried out in accordance with the institutional guidelines and regulations.

Evaluation of cultivated land quality using attention mechanism-back propagation neural network

Cultivated land quality is related to the quality and safety of agricultural products and to ecological safety. Therefore, reasonably evaluating the quality of land, which is helpful in identifying its benefits, is crucial. However, most studies have used traditional methods to estimate cultivated land quality, and there is little research on using deep learning for this purpose. Using Ya'an cultivated land as the research object, this study constructs an evaluation system for cultivated land quality based on seven aspects, including soil organic matter and soil texture. An attention mechanism (AM) is introduced into a back propagation (BP) neural network model. Therefore, an AM-BP neural network that is suitable for Ya'an cultivated land is designed. The sample is divided into training and test sets by a ratio of 7:3. We can output the evaluation results of cultivated land quality through experiments. Furthermore, they can be visualized through a pie chart. The experimental results indicate that the model effect of the AM-BP neural network is better than that of the BP neural network. That is, the mean square error is reduced by approximately 0.0019 and the determination coefficient is increased by approximately 0.005. In addition, this study obtains better results via the ensemble model. The quality of cultivated land in Yucheng District is generally good, i.e.,mostly third and fourth grades. It conforms to the normal distribution. Lastly, the method has certain to evaluate cultivated land quality, providing a reference for future cultivated land quality evaluation.

Classification of ECG using ensemble of residual CNNs with or without attention mechanism

This paper introduces a winning solution (team ISIBrno-AIMT) to the official round of PhysioNet Challenge 2021. The main goal of the challenge was a classification of ECG recordings into 26 multi-label pathological classes with variable number of leads (e.g., 12,6,4,3,2). We introduced an ECG classification method based on the ResNet architecture with a multi-head attention mechanism for the official round of the challenge. However, empirical findings collected during model development suggested that the multi-head attention layer might not significantly impact the final classification performance. For this reason, during the follow-up round, we removed a multi-head attention layer to test the influence on model performance. Like the official round, the model is optimized using a mixture of loss functions, i.e., binary cross-entropy, custom challenge score loss function, and custom sparsity loss function. Probability thresholds for each classification class are estimated using the evolutionary optimization method. The final architecture consists of three submodels forming a majority voting classification ensemble. Our findings from the follow- up submission support the fact that the multi-head attention layer in the proposed architecture does not significantly affect the classification performance. The modified model without the multi-head attention layer increased the overall challenge score to 0.59 compared to the 0.58 from the official round.

TAHDNet: Time-Aware Hierarchical Dependency Network for Medication Recommendation

Medication recommendation is a hot topic in the research of applying neural networks to the healthcare area. Although extensive progressions have been made, current researches still face the following challenges: i). Existing methods are poor at efficiently capturing and leveraging local and global dependency information from patient visit records. ii). Current time-aware models based on irregularly interval medical records tend to ignore periodic variability in patient conditions, which limits the representational learning capability of these models. Therefore, we propose a Dynamic Time-aware Hierarchical Dependency Network (TAHDNet) for the medication recommendation task to address these challenges. Firstly, we use a Transformer-based model to learn the global information of the whole patient record through a self-supervised pre-training process. Secondly, a 1D-CNN model is used to learn the local dependencies on visitation level. Thirdly, we propose a dynamic time-aware module with a fused temporal decay function to assign different weights among different time intervals dynamically through a key-value attention mechanism. Experimental results on real-world datasets demonstrate the effectiveness of the model proposed in this paper.

Self-supervised recurrent depth estimation with attention mechanisms

Depth estimation has been an essential task for many computer vision applications, especially in autonomous driving, where safety is paramount. Depth can be estimated not only with traditional supervised learning but also via a self-supervised approach that relies on camera motion and does not require ground truth depth maps. Recently, major improvements have been introduced to make self-supervised depth prediction more precise. However, most existing approaches still focus on single-frame depth estimation, even in the self-supervised setting. Since most methods can operate with frame sequences, we believe that the quality of current models can be significantly improved with the help of information about previous frames. In this work, we study different ways of integrating recurrent blocks and attention mechanisms into a common self-supervised depth estimation pipeline. We propose a set of modifications that utilize temporal information from previous frames and provide new neural network architectures for monocular depth estimation in a self-supervised manner. Our experiments on the KITTI dataset show that proposed modifications can be an effective tool for exploiting temporal information in a depth prediction pipeline.

KGDAL: Knowledge Graph Guided Double Attention LSTM for Rolling Mortality Prediction for AKI-D Patients

With the rapid accumulation of electronic health record (EHR) data, deep learning (DL) models have exhibited promising performance on patient risk prediction. Recent advances have also demonstrated the effectiveness of knowledge graphs (KG) in providing valuable prior knowledge for further improving DL model performance. However, it is still unclear how KG can be utilized to encode high-order relations among clinical concepts and how DL models can make full use of the encoded concept relations to solve real-world healthcare problems and to interpret the outcomes. We propose a novel knowledge graph guided double attention LSTM model named KGDAL for rolling mortality prediction for critically ill patients with acute kidney injury requiring dialysis (AKI-D). KGDAL constructs a KG-based two-dimension attention in both time and feature spaces. In the experiment with two large healthcare datasets, we compared KGDAL with a variety of rolling mortality prediction models and conducted an ablation study to test the effectiveness, efficacy, and contribution of different attention mechanisms. The results showed that KGDAL clearly outperformed all the compared models. Also, KGDAL-derived patient risk trajectories may assist healthcare providers to make timely decisions and actions. The source code, sample data, and manual of KGDAL are available at https://github.com/lucasliu0928/KGDAL.

Mutual Correlation Attentive Factors in Dyadic Fusion Networks for Speech Emotion Recognition

Emotion recognition in dyadic communication is challenging because: 1. Extracting informative modality-specific representations requires disparate feature extractor designs due to the heterogenous input data formats. 2. How to effectively and efficiently fuse unimodal features and learn associations between dyadic utterances are critical to the model generalization in actual scenario. 3. Disagreeing annotations prevent previous approaches from precisely predicting emotions in context. To address the above issues, we propose an efficient dyadic fusion network that only relies on an attention mechanism to select representative vectors, fuse modality-specific features, and learn the sequence information. Our approach has three distinct characteristics: 1. Instead of using a recurrent neural network to extract temporal associations as in most previous research, we introduce multiple sub-view attention layers to compute the relevant dependencies among sequential utterances; this significantly improves model efficiency. 2. To improve fusion performance, we design a learnable mutual correlation factor inside each attention layer to compute associations across different modalities. 3. To overcome the label disagreement issue, we embed the labels from all annotators into a k-dimensional vector and transform the categorical problem into a regression problem; this method provides more accurate annotation information and fully uses the entire dataset. We evaluate the proposed model on two published multimodal emotion recognition datasets: IEMOCAP and MELD. Our model significantly outperforms previous state-of-the-art research by 3.8%-7.5% accuracy, using a more efficient model.

Machine Learning on Sequential Data Using a Recurrent Weighted Average

Recurrent Neural Networks (RNN) are a type of statistical model designed to handle sequential data. The model reads a sequence one symbol at a time. Each symbol is processed based on information collected from the previous symbols. With existing RNN architectures, each symbol is processed using only information from the previous processing step. To overcome this limitation, we propose a new kind of RNN model that computes a recurrent weighted average (RWA) over every past processing step. Because the RWA can be computed as a running average, the computational overhead scales like that of any other RNN architecture. The approach essentially reformulates the attention mechanism into a stand-alone model. The performance of the RWA model is assessed on the variable copy problem, the adding problem, classification of artificial grammar, classification of sequences by length, and classification of the MNIST images (where the pixels are read sequentially one at a time). On almost every task, the RWA model is found to fit the data significantly faster than a standard LSTM model.

Human Conversation Analysis Using Attentive Multimodal Networks with Hierarchical Encoder-Decoder

Human conversation analysis is challenging because the meaning can be expressed through words, intonation, or even body language and facial expression. We introduce a hierarchical encoder-decoder structure with attention mechanism for conversation analysis. The hierarchical encoder learns word-level features from video, audio, and text data that are then formulated into conversation-level features. The corresponding hierarchical decoder is able to predict different attributes at given time instances. To integrate multiple sensory inputs, we introduce a novel fusion strategy with modality attention. We evaluated our system on published emotion recognition, sentiment analysis, and speaker trait analysis datasets. Our system outperformed previous state-of-the-art approaches in both classification and regressions tasks on three datasets. We also outperformed previous approaches in generalization tests on two commonly used datasets. We achieved comparable performance in predicting co-existing labels using the proposed model instead of multiple individual models. In addition, the easily-visualized modality and temporal attention demonstrated that the proposed attention mechanism helps feature selection and improves model interpretability.