Causal Discovery with Attention-Based Convolutional Neural Networks

A synergistic future for AI and ecology

Research in both ecology and AI strives for predictive understanding of complex systems, where nonlinearities arise from multidimensional interactions and feedbacks across multiple scales. After a century of independent, asynchronous advances in computational and ecological research, we foresee a critical need for intentional synergy to meet current societal challenges against the backdrop of global change. These challenges include understanding the unpredictability of systems-level phenomena and resilience dynamics on a rapidly changing planet. Here, we spotlight both the promise and the urgency of a convergence research paradigm between ecology and AI. Ecological systems are a challenge to fully and holistically model, even using the most prominent AI technique today: deep neural networks. Moreover, ecological systems have emergent and resilient behaviors that may inspire new, robust AI architectures and methodologies. We share examples of how challenges in ecological systems modeling would benefit from advances in AI techniques that are themselves inspired by the systems they seek to model. Both fields have inspired each other, albeit indirectly, in an evolution toward this convergence. We emphasize the need for more purposeful synergy to accelerate the understanding of ecological resilience whilst building the resilience currently lacking in modern AI systems, which have been shown to fail at times because of poor generalization in different contexts. Persistent epistemic barriers would benefit from attention in both disciplines. The implications of a successful convergence go beyond advancing ecological disciplines or achieving an artificial general intelligence-they are critical for both persisting and thriving in an uncertain future.

scTIGER: A Deep-Learning Method for Inferring Gene Regulatory Networks from Case versus Control scRNA-seq Datasets

Inferring gene regulatory networks (GRNs) from single-cell RNA-seq (scRNA-seq) data is an important computational question to find regulatory mechanisms involved in fundamental cellular processes. Although many computational methods have been designed to predict GRNs from scRNA-seq data, they usually have high false positive rates and none infer GRNs by directly using the paired datasets of case-versus-control experiments. Here we present a novel deep-learning-based method, named scTIGER, for GRN detection by using the co-differential relationships of gene expression profiles in paired scRNA-seq datasets. scTIGER employs cell-type-based pseudotiming, an attention-based convolutional neural network method and permutation-based significance testing for inferring GRNs among gene modules. As state-of-the-art applications, we first applied scTIGER to scRNA-seq datasets of prostate cancer cells, and successfully identified the dynamic regulatory networks of AR, ERG, PTEN and ATF3 for same-cell type between prostatic cancerous and normal conditions, and two-cell types within the prostatic cancerous environment. We then applied scTIGER to scRNA-seq data from neurons with and without fear memory and detected specific regulatory networks for BDNF, CREB1 and MAPK4. Additionally, scTIGER demonstrates robustness against high levels of dropout noise in scRNA-seq data.

A Model for Types and Levels of Automation in Visual Analytics: A Survey, a Taxonomy, and Examples

The continuous growth in availability and access to data presents a major challenge to the human analyst. As the manual analysis of large and complex datasets is nowadays practically impossible, the need for assisting tools that can automate the analysis process while keeping the human analyst in the loop is imperative. A large and growing body of literature recognizes the crucial role of automation in Visual Analytics and suggests that automation is among the most important constituents for effective Visual Analytics systems. Today, however, there is no appropriate taxonomy nor terminology for assessing the extent of automation in a Visual Analytics system. In this article, we aim to address this gap by introducing a model of levels of automation tailored for the Visual Analytics domain. The consistent terminology of the proposed taxonomy could provide a ground for users/readers/reviewers to describe and compare automation in Visual Analytics systems. Our taxonomy is grounded on a combination of several existing and well-established taxonomies of levels of automation in the human-machine interaction domain and relevant models within the visual analytics field. To exemplify the proposed taxonomy, we selected a set of existing systems from the event-sequence analytics domain and mapped the automation of their visual analytics process stages against the automation levels in our taxonomy.

Time series causal relationships discovery through feature importance and ensemble models

Inferring causal relationships from observational data is a key challenge in understanding the interpretability of Machine Learning models. Given the ever-increasing amount of observational data available in many areas, Machine Learning algorithms used for forecasting have become more complex, leading to a less understandable path of how a decision is made by the model. To address this issue, we propose leveraging ensemble models, e.g., Random Forest, to assess which input features the trained model prioritizes when making a forecast and, in this way, establish causal relationships between the variables. The advantage of these algorithms lies in their ability to provide feature importance, which allows us to build the causal network. We present our methodology to estimate causality in time series from oil field production. As it is difficult to extract causal relations from a real field, we also included a synthetic oil production dataset and a weather dataset, which is also synthetic, to provide the ground truth. We aim to perform causal discovery, i.e., establish the existing connections between the variables in each dataset. Through an iterative process of improving the forecasting of a target's value, we evaluate whether the forecasting improves by adding information from a new potential driver; if so, we state that the driver causally affects the target. On the oil field-related datasets, our causal analysis results agree with the interwell connections already confirmed by tracer information; whenever the tracer data are available, we used it as our ground truth. This consistency between both estimated and confirmed connections provides us the confidence about the effectiveness of our proposed methodology. To our knowledge, this is the first time causal analysis using solely production data is employed to discover interwell connections in an oil field dataset.

Brain effective connectivity and functional connectivity as markers of lifespan vascular exposures in middle-aged adults: The Bogalusa Heart Study

Introduction

Effective connectivity (EC), the causal influence that functional activity in a source brain location exerts over functional activity in a target brain location, has the potential to provide different information about brain network dynamics than functional connectivity (FC), which quantifies activity synchrony between locations. However, head-to-head comparisons between EC and FC from either task-based or resting-state functional MRI (fMRI) data are rare, especially in terms of how they associate with salient aspects of brain health.

Methods

In this study, 100 cognitively-healthy participants in the Bogalusa Heart Study aged 54.2 ± 4.3years completed Stroop task-based fMRI, resting-state fMRI. EC and FC among 24 regions of interest (ROIs) previously identified as involved in Stroop task execution (EC-task and FC-task) and among 33 default mode network ROIs (EC-rest and FC-rest) were calculated from task-based and resting-state fMRI using deep stacking networks and Pearson correlation. The EC and FC measures were thresholded to generate directed and undirected graphs, from which standard graph metrics were calculated. Linear regression models related graph metrics to demographic, cardiometabolic risk factors, and cognitive function measures.

Results

Women and whites (compared to men and African Americans) had better EC-task metrics, and better EC-task metrics associated with lower blood pressure, white matter hyperintensity volume, and higher vocabulary score (maximum value of p = 0.043). Women had better FC-task metrics, and better FC-task metrics associated with APOE-ε4 3–3 genotype and better hemoglobin-A1c, white matter hyperintensity volume and digit span backwards score (maximum value of p = 0.047). Better EC rest metrics associated with lower age, non-drinker status, and better BMI, white matter hyperintensity volume, logical memory II total score, and word reading score (maximum value of p = 0.044). Women and non-drinkers had better FC-rest metrics (value of p = 0.004).

Discussion

In a diverse, cognitively healthy, middle-aged community sample, EC and FC based graph metrics from task-based fMRI data, and EC based graph metrics from resting-state fMRI data, were associated with recognized indicators of brain health in differing ways. Future studies of brain health should consider taking both task-based and resting-state fMRI scans and measuring both EC and FC analyses to get a more complete picture of functional networks relevant to brain health.

Granger causality using Jacobian in neural networks

Granger causality is a commonly used method for uncovering information flow and dependencies in a time series. Here, we introduce JGC (Jacobian Granger causality), a neural network-based approach to Granger causality using the Jacobian as a measure of variable importance, and propose a variable selection procedure for inferring Granger causal variables with this measure, using criteria of significance and consistency. The resulting approach performs consistently well compared to other approaches in identifying Granger causal variables, the associated time lags, as well as interaction signs. In addition, we also discuss the need for contemporaneous variables in Granger causal modeling as well as how these neural network-based approaches reduce the impact of nonseparability in dynamical systems, a problem where predictive information on a target variable is not unique to its causes, but also contained in the history of the target variable itself.

Entropy-Based Discovery of Summary Causal Graphs in Time Series

This study addresses the problem of learning a summary causal graph on time series with potentially different sampling rates. To do so, we first propose a new causal temporal mutual information measure for time series. We then show how this measure relates to an entropy reduction principle that can be seen as a special case of the probability raising principle. We finally combine these two ingredients in PC-like and FCI-like algorithms to construct the summary causal graph. There algorithm are evaluated on several datasets, which shows both their efficacy and efficiency.

Disentangling personalized treatment effects from “time-of-the-day” confounding in mobile health studies

Ideally, a patient’s response to medication can be monitored by measuring changes in performance of some activity. In observational studies, however, any detected association between treatment (“on-medication” vs “off-medication”) and the outcome (performance in the activity) might be due to confounders. In particular, causal inferences at the personalized level are especially vulnerable to confounding effects that arise in a cyclic fashion. For quick acting medications, effects can be confounded by circadian rhythms and daily routines. Using the time-of-the-day as a surrogate for these confounders and the performance measurements as captured on a smartphone, we propose a personalized statistical approach to disentangle putative treatment and “time-of-the-day” effects, that leverages conditional independence relations spanned by causal graphical models involving the treatment, time-of-the-day, and outcome variables. Our approach is based on conditional independence tests implemented via standard and temporal linear regression models. Using synthetic data, we investigate when and how residual autocorrelation can affect the standard tests, and how time series modeling (namely, ARIMA and robust regression via HAC covariance matrix estimators) can remedy these issues. In particular, our simulations illustrate that when patients perform their activities in a paired fashion, positive autocorrelation can lead to conservative results for the standard regression approach (i.e., lead to deflated true positive detection), whereas negative autocorrelation can lead to anticonservative behavior (i.e., lead to inflated false positive detection). The adoption of time series methods, on the other hand, leads to well controlled type I error rates. We illustrate the application of our methodology with data from a Parkinson’s disease mobile health study.

Going deep into schizophrenia with artificial intelligence

Despite years of research, the mechanisms governing the onset, relapse, symptomatology, and treatment of schizophrenia (SZ) remain elusive. The lack of appropriate analytic tools to deal with the heterogeneity and complexity of SZ may be one of the reasons behind this situation. Deep learning, a subfield of artificial intelligence (AI) inspired by the nervous system, has recently provided an accessible way of modeling and analyzing complex, high-dimensional, nonlinear systems. The unprecedented accuracy of deep learning algorithms in classification and prediction tasks has revolutionized a wide range of scientific fields and is rapidly permeating SZ research. Deep learning has the potential of becoming a valuable aid for clinicians in the prediction, diagnosis, and treatment of SZ, especially in combination with principles from Bayesian statistics. Furthermore, deep learning could become a powerful tool for uncovering the mechanisms underlying SZ thanks to a growing number of techniques designed for improving model interpretability and causal reasoning. The purpose of this article is to introduce SZ researchers to the field of deep learning and review its latest applications in SZ research. In general, existing studies have yielded impressive results in classification and outcome prediction tasks. However, methodological concerns related to the assessment of model performance in several studies, the widespread use of small training datasets, and the little clinical value of some models suggest that some of these results should be taken with caution.

Causal discovery of drivers of surface ozone variability in Antarctica using a deep learning algorithm

The discovery of causal structures behind a phenomenon under investigation has been at the heart of scientific inquiry since the beginning. Randomized control trials, the gold standard for causal analysis, may not always be feasible, such as in the domain of climate sciences. In the absence of interventional data, we are forced to depend only on observational data. This study demonstrates the application of one such causal discovery algorithm using a neural network for identifying the drivers of surface ozone variability in Antarctica. The analyses reveal the overarching influence of the stratosphere on the surface ozone variability in Antarctica, buttressed by the southern annular mode and tropospheric wave forcing in mid-latitudes. We find no significant and robust evidence for the influence of tropical teleconnection on the ground-level ozone in Antarctica. As the field of atmospheric science is now replete with a massive stock of observational data, both satellite and ground-based, this tool for automated causal structure discovery might prove to be invaluable for scientific investigation and flawless decision making.

Neural Networks for Directed Connectivity Estimation in Source-Reconstructed EEG Data

Directed connectivity between brain sources identified from scalp electroencephalography (EEG) can shed light on the brain’s information flows and provide a biomarker of neurological disorders. However, as volume conductance results in scalp activity being a mix of activities originating from multiple sources, the correct interpretation of their connectivity is a formidable challenge despite source localization being applied with some success. Traditional connectivity approaches rely on statistical assumptions that usually do not hold for EEG, calling for a model-free approach. We investigated several types of Artificial Neural Networks in estimating Directed Connectivity between Reconstructed EEG Sources and assessed their accuracy with respect to several ground truths. We show that a Long Short-Term Memory neural network with Non-Uniform Embedding yields the most promising results due to its relative robustness to differing dipole locations. We conclude that certain network architectures can compete with the already established methods for brain connectivity analysis.

Aristotle: stratified causal discovery for omics data

BACKGROUND

There has been a simultaneous increase in demand and accessibility across genomics, transcriptomics, proteomics and metabolomics data, known as omics data. This has encouraged widespread application of omics data in life sciences, from personalized medicine to the discovery of underlying pathophysiology of diseases. Causal analysis of omics data may provide important insight into the underlying biological mechanisms. Existing causal analysis methods yield promising results when identifying potential general causes of an observed outcome based on omics data. However, they may fail to discover the causes specific to a particular stratum of individuals and missing from others.

METHODS

To fill this gap, we introduce the problem of stratified causal discovery and propose a method, Aristotle, for solving it. Aristotle addresses the two challenges intrinsic to omics data: high dimensionality and hidden stratification. It employs existing biological knowledge and a state-of-the-art patient stratification method to tackle the above challenges and applies a quasi-experimental design method to each stratum to find stratum-specific potential causes.

RESULTS

Evaluation based on synthetic data shows better performance for Aristotle in discovering true causes under different conditions compared to existing causal discovery methods. Experiments on a real dataset on Anthracycline Cardiotoxicity indicate that Aristotle's predictions are consistent with the existing literature. Moreover, Aristotle makes additional predictions that suggest further investigations.

A fuzzy sensitivity analysis approach to estimate brain effective connectivity and its application to epileptic seizure detection

The brain is considered to be the most complicated organ in human body. Inferring and quantification of effective (causal) connectivity among regions of the brain is an important step in characterization of its complicated functions. The proposed method is comprised of modeling multivariate time series with Adaptive Neurofuzzy Inference System (ANFIS) and carrying out a sensitivity analysis using Fuzzy network parameters as a new approach to introduce a connectivity measure for detecting causal interactions between interactive input time series. The results of simulations indicate that this method is successful in detecting causal connectivity. After validating the performance of the proposed method on synthetic linear and nonlinear interconnected time series, it is applied to epileptic intracranial Electroencephalography (EEG) signals. The result of applying the proposed method on Freiburg epileptic intracranial EEG data recorded during seizure shows that the proposed method is capable of discriminating between the seizure and non-seizure states of the brain.

Echo state network models for nonlinear Granger causality

While Granger causality (GC) has been often employed in network neuroscience, most GC applications are based on linear multivariate autoregressive (MVAR) models. However, real-life systems like biological networks exhibit notable nonlinear behaviour, hence undermining the validity of MVAR-based GC (MVAR-GC). Most nonlinear GC estimators only cater for additive nonlinearities or, alternatively, are based on recurrent neural networks or long short-term memory networks, which present considerable training difficulties and tailoring needs. We reformulate the GC framework in terms of echo-state networks-based models for arbitrarily complex networks, and characterize its ability to capture nonlinear causal relations in a network of noisy Duffing oscillators, showing a net advantage of echo state GC (ES-GC) in detecting nonlinear, causal links. We then explore the structure of ES-GC networks in the human brain employing functional MRI data from 1003 healthy subjects drawn from the human connectome project, demonstrating the existence of previously unknown directed within-brain interactions. In addition, we examine joint brain-heart signals in 15 subjects where we explore directed interaction between brain networks and central vagal cardiac control in order to investigate the so-called central autonomic network in a causal manner. This article is part of the theme issue 'Advanced computation in cardiovascular physiology: new challenges and opportunities'.

KOMPOS: Connecting Causal Knots in Large Nonlinear Time Series with Non-Parametric Regression Splines

Recovering causality from copious time series data beyond mere correlations has been an important contributing factor in numerous scientific fields. Most existing works assume linearity in the data that may not comply with many real-world scenarios. Moreover, it is usually not sufficient to solely infer the causal relationships. Identifying the correct time delay of cause-effect is extremely vital for further insight and effective policies in inter-disciplinary domains. To bridge this gap, we propose KOMPOS, a novel algorithmic framework that combines a powerful concept from causal discovery of additive noise models with graphical ones. We primarily build our structural causal model from multivariate adaptive regression splines with inherent additive local nonlinearities, which render the underlying causal structure more easily identifiable. In contrast to other methods, our approach is not restricted to Gaussian or non-Gaussian noise due to the non-parametric attribute of the regression method. We conduct extensive experiments on both synthetic and real-world datasets, demonstrating the superiority of the proposed algorithm over existing causal discovery methods, especially for the challenging cases of autocorrelated and non-stationary time series.

Visual Causality Analysis of Event Sequence Data

Causality is crucial to understanding the mechanisms behind complex systems and making decisions that lead to intended outcomes. Event sequence data is widely collected from many real-world processes, such as electronic health records, web clickstreams, and financial transactions, which transmit a great deal of information reflecting the causal relations among event types. Unfortunately, recovering causalities from observational event sequences is challenging, as the heterogeneous and high-dimensional event variables are often connected to rather complex underlying event excitation mechanisms that are hard to infer from limited observations. Many existing automated causal analysis techniques suffer from poor explainability and fail to include an adequate amount of human knowledge. In this paper, we introduce a visual analytics method for recovering causalities in event sequence data. We extend the Granger causality analysis algorithm on Hawkes processes to incorporate user feedback into causal model refinement. The visualization system includes an interactive causal analysis framework that supports bottom-up causal exploration, iterative causal verification and refinement, and causal comparison through a set of novel visualizations and interactions. We report two forms of evaluation: a quantitative evaluation of the model improvements resulting from the user-feedback mechanism, and a qualitative evaluation through case studies in different application domains to demonstrate the usefulness of the system.

Benchmarking Attention-Based Interpretability of Deep Learning in Multivariate Time Series Predictions

The adaptation of deep learning models within safety-critical systems cannot rely only on good prediction performance but needs to provide interpretable and robust explanations for their decisions. When modeling complex sequences, attention mechanisms are regarded as the established approach to support deep neural networks with intrinsic interpretability. This paper focuses on the emerging trend of specifically designing diagnostic datasets for understanding the inner workings of attention mechanism based deep learning models for multivariate forecasting tasks. We design a novel benchmark of synthetically designed datasets with the transparent underlying generating process of multiple time series interactions with increasing complexity. The benchmark enables empirical evaluation of the performance of attention based deep neural networks in three different aspects: (i) prediction performance score, (ii) interpretability correctness, (iii) sensitivity analysis. Our analysis shows that although most models have satisfying and stable prediction performance results, they often fail to give correct interpretability. The only model with both a satisfying performance score and correct interpretability is IMV-LSTM, capturing both autocorrelations and crosscorrelations between multiple time series. Interestingly, while evaluating IMV-LSTM on simulated data from statistical and mechanistic models, the correctness of interpretability increases with more complex datasets.

Causal Discovery in Radiographic Markers of Knee Osteoarthritis and Prediction for Knee Osteoarthritis Severity With Attention-Long Short-Term Memory

The goal of this study is to build a prognostic model to predict the severity of radiographic knee osteoarthritis (KOA) and to identify long-term disease progression risk factors for early intervention and treatment. We designed a long short-term memory (LSTM) model with an attention mechanism to predict Kellgren/Lawrence (KL) grade for knee osteoarthritis patients. The attention scores reveal a time-associated impact of different variables on KL grades. We also employed a fast causal inference (FCI) algorithm to estimate the causal relation of key variables, which will aid in clinical interpretability. Based on the clinical information of current visits, we accurately predicted the KL grade of the patient's next visits with 90% accuracy. We found that joint space narrowing was a major contributor to KOA progression. Furthermore, our causal structure model indicated that knee alignments may lead to joint space narrowing, while symptoms (swelling, grinding, catching, and limited mobility) have little impact on KOA progression. This study evaluated a broad spectrum of potential risk factors from clinical data, questionnaires, and radiographic markers that are rarely considered in previous studies. Using our statistical model, providers are able to predict the risk of the future progression of KOA, which will provide a basis for selecting proper interventions, such as proceeding to joint arthroplasty for patients. Our causal model suggests that knee alignment should be considered in the primary treatment and KOA progression was independent of clinical symptoms.

Engineering the Microbiome to Prevent Adverse Events: Challenges and Opportunities

In the past decade of microbiome research, we have learned about numerous adverse interactions between the microbiome and medical interventions such as drugs, radiation, and surgery. What if we could alter our microbiomes to prevent these events? In this review, we discuss potential routes to mitigate microbiome adverse events, including applications from the emerging field of microbiome engineering. We highlight cases where the microbiome acts directly on a treatment, such as via differential drug metabolism, and cases where a treatment directly harms the microbiome, such as in radiation therapy. Understanding and preventing microbiome adverse events is a difficult challenge that will require a data-driven approach involving causal statistics, multiomics techniques, and a personalized means of mitigating adverse events. We propose research considerations to encourage productive work in preventing microbiome adverse events, and we highlight the many challenges and opportunities that await.

Attention-Based SeriesNet: An Attention-Based Hybrid Neural Network Model for Conditional Time Series Forecasting

Traditional time series forecasting techniques can not extract good enough sequence data features, and their accuracies are limited. The deep learning structure SeriesNet is an advanced method, which adopts hybrid neural networks, including dilated causal convolutional neural network (DC-CNN) and Long-short term memory recurrent neural network (LSTM-RNN), to learn multi-range and multi-level features from multi-conditional time series with higher accuracy. However, they didn’t consider the attention mechanisms to learn temporal features. Besides, the conditioning method for CNN and RNN is not specific, and the number of parameters in each layer is tremendous. This paper proposes the conditioning method for two types of neural networks, and respectively uses the gated recurrent unit network (GRU) and the dilated depthwise separable temporal convolutional networks (DDSTCNs) instead of LSTM and DC-CNN for reducing the parameters. Furthermore, this paper presents the lightweight RNN-based hidden state attention module (HSAM) combined with the proposed CNN-based convolutional block attention module (CBAM) for time series forecasting. Experimental results show our model is superior to other models from the viewpoint of forecasting accuracy and computation efficiency.

Multi-Step Short-Term Wind Speed Prediction Using a Residual Dilated Causal Convolutional Network with Nonlinear Attention

Wind energy is the most used renewable energy worldwide second only to hydropower. However, the stochastic nature of wind speed makes it harder for wind farms to manage the future power production and maintenance schedules efficiently. Many wind speed prediction models exist that focus on advance neural networks and/or preprocessing techniques to improve the accuracy. Since most of these models require a large amount of historic wind data and are validated using the data split method, the application to real-world scenarios cannot be determined. In this paper, we present a multi-step univariate prediction model for wind speed data inspired by the residual U-net architecture of the convolutional neural network (CNN). We propose a residual dilated causal convolutional neural network (Res-DCCNN) with nonlinear attention for multi-step-ahead wind speed forecasting. Our model can outperform long-term short-term memory networks (LSTM), gated recurrent units (GRU), and Res-DCCNN using sliding window validation techniques for 50-step-ahead wind speed prediction. We tested the performance of the proposed model on six real-world wind speed datasets with different probability distributions to confirm its effectiveness, and using several error metrics, we demonstrated that our proposed model was robust, precise, and applicable to real-world cases.

Approximate and Situated Causality in Deep Learning

Causality is the most important topic in the history of western science, and since the beginning of the statistical paradigm, its meaning has been reconceptualized many times. Causality entered into the realm of multi-causal and statistical scenarios some centuries ago. Despite widespread critics, today deep learning and machine learning advances are not weakening causality but are creating a new way of finding correlations between indirect factors. This process makes it possible for us to talk about approximate causality, as well as about a situated causality.