Explaining the black-box model: A survey of local interpretation methods for deep neural networks

An Intrinsically Explainable Method to Decode P300 Waveforms from EEG Signal Plots Based on Convolutional Neural Networks

This work proposes an intrinsically explainable, straightforward method to decode P300 waveforms from electroencephalography (EEG) signals, overcoming the black box nature of deep learning techniques. The proposed method allows convolutional neural networks to decode information from images, an area where they have achieved astonishing performance. By plotting the EEG signal as an image, it can be both visually interpreted by physicians and technicians and detected by the network, offering a straightforward way of explaining the decision. The identification of this pattern is used to implement a P300-based speller device, which can serve as an alternative communication channel for persons affected by amyotrophic lateral sclerosis (ALS). This method is validated by identifying this signal by performing a brain-computer interface simulation on a public dataset from ALS patients. Letter identification rates from the speller on the dataset show that this method can identify the P300 signature on the set of 8 patients. The proposed approach achieves similar performance to other state-of-the-art proposals while providing clinically relevant explainability (XAI).

Explainable Deep Learning-Based Feature Selection and Intrusion Detection Method on the Internet of Things

With the rapid advancement of the Internet of Things, network security has garnered increasing attention from researchers. Applying deep learning (DL) has significantly enhanced the performance of Network Intrusion Detection Systems (NIDSs). However, due to its complexity and "black box" problem, deploying DL-based NIDS models in practical scenarios poses several challenges, including model interpretability and being lightweight. Feature selection (FS) in DL models plays a crucial role in minimizing model parameters and decreasing computational overheads while enhancing NIDS performance. Hence, selecting effective features remains a pivotal concern for NIDSs. In light of this, this paper proposes an interpretable feature selection method for encrypted traffic intrusion detection based on SHAP and causality principles. This approach utilizes the results of model interpretation for feature selection to reduce feature count while ensuring model reliability. We evaluate and validate our proposed method on two public network traffic datasets, CICIDS2017 and NSL-KDD, employing both a CNN and a random forest (RF). Experimental results demonstrate superior performance achieved by our proposed method.

Explainable and generalizable AI-driven multiscale informatics for dynamic system modelling

Ultra-precision machining requires system modelling that both satisfies explainability and conforms to data fidelity. Existing modelling approaches, whether based on data-driven methods in present artificial intelligence (AI) or on first-principle knowledge, fall short of these qualities in high-demanding industrial applications. Therefore, this paper develops an explainable and generalizable 'grey-box' AI informatics method for real-world dynamic system modelling. Such a grey-box model serves as a multiscale 'world model' by integrating the first principles of the system in a white-box architecture with data-fitting black boxes for varying hyperparameters of the white box. The physical principles serve as an explainable global meta-structure of the real-world system driven by physical knowledge, while the black boxes enhance local fitting accuracy driven by training data. The grey-box model thus encapsulates implicit variables and relationships that a standalone white-box model or black-box model fails to capture. Case study on an industrial cleanroom high-precision temperature regulation system verifies that the grey-box method outperforms existing modelling methods and is suitable for varying operating conditions.

Development and Validation of an Explainable Deep Learning Model to Predict In-Hospital Mortality for Patients With Acute Myocardial Infarction: Algorithm Development and Validation Study

BACKGROUND

Acute myocardial infarction (AMI) is one of the most severe cardiovascular diseases and is associated with a high risk of in-hospital mortality. However, the current deep learning models for in-hospital mortality prediction lack interpretability.

OBJECTIVE

This study aims to establish an explainable deep learning model to provide individualized in-hospital mortality prediction and risk factor assessment for patients with AMI.

METHODS

In this retrospective multicenter study, we used data for consecutive patients hospitalized with AMI from the Chongqing University Central Hospital between July 2016 and December 2022 and the Electronic Intensive Care Unit Collaborative Research Database. These patients were randomly divided into training (7668/10,955, 70%) and internal test (3287/10,955, 30%) data sets. In addition, data of patients with AMI from the Medical Information Mart for Intensive Care database were used for external validation. Deep learning models were used to predict in-hospital mortality in patients with AMI, and they were compared with linear and tree-based models. The Shapley Additive Explanations method was used to explain the model with the highest area under the receiver operating characteristic curve in both the internal test and external validation data sets to quantify and visualize the features that drive predictions.

RESULTS

A total of 10,955 patients with AMI who were admitted to Chongqing University Central Hospital or included in the Electronic Intensive Care Unit Collaborative Research Database were randomly divided into a training data set of 7668 (70%) patients and an internal test data set of 3287 (30%) patients. A total of 9355 patients from the Medical Information Mart for Intensive Care database were included for independent external validation. In-hospital mortality occurred in 8.74% (670/7668), 8.73% (287/3287), and 9.12% (853/9355) of the patients in the training, internal test, and external validation cohorts, respectively. The Self-Attention and Intersample Attention Transformer model performed best in both the internal test data set and the external validation data set among the 9 prediction models, with the highest area under the receiver operating characteristic curve of 0.86 (95% CI 0.84-0.88) and 0.85 (95% CI 0.84-0.87), respectively. Older age, high heart rate, and low body temperature were the 3 most important predictors of increased mortality, according to the explanations of the Self-Attention and Intersample Attention Transformer model.

CONCLUSIONS

The explainable deep learning model that we developed could provide estimates of mortality and visual contribution of the features to the prediction for a patient with AMI. The explanations suggested that older age, unstable vital signs, and metabolic disorders may increase the risk of mortality in patients with AMI.

Predicting the distribution coefficient of cesium in solid phase groups using machine learning

The migration and retention of radioactive contaminants such as 137Cesium (137Cs) in various environmental media pose significant long-term storage challenges for nuclear waste. The distribution coefficient (Kd) is a critical parameter for assessing the mobility of radioactive contaminants and is influenced by various environmental conditions. This study presents machine-learning models based on the Japan Atomic Energy Agency Sorption Database (JAEA-SDB) to predict the Kd values for Cs in solid phase groups. We used three different machine learning models: random forest (RF), artificial neural network (ANN), and convolutional neural network (CNN). The models were trained on 14 input variables from the JAEA-SDB, including factors such as the Cs concentration, solid-phase properties, and solution conditions, which were preprocessed by normalization and log-transformation. The performances of the models were evaluated using the coefficient of determination (R2) and root mean squared error (RMSE). The RF, ANN, and CNN models achieved R2 values greater than 0.97, 0.86, and 0.88, respectively. We also analyzed the variable importance of RF using an out-of-bag (OOB) and a CNN with an attention module. Our results showed that the environmental media, initial radionuclide concentration, solid phase properties, and solution conditions were significant variables for Kd prediction. Our models accurately predict Kd values for different environmental conditions and can assess the environmental risk by analyzing the behavior of radionuclides in solid phase groups. The results of this study can improve safety analyses and long-term risk assessments related to waste disposal and prevent potential hazards and sources of contamination in the surrounding environment.

Probing with Each Step: How a Walking Crab-like Robot Classifies Buried Cylinders in Sand with Hall-Effect Sensors

Shallow underwater environments around the world are contaminated with unexploded ordnances (UXOs). Current state-of-the-art methods for UXO detection and localization use remote sensing systems. Furthermore, human divers are often tasked with confirming UXO existence and retrieval which poses health and safety hazards. In this paper, we describe the application of a crab robot with leg-embedded Hall effect-based sensors to detect and distinguish between UXOs and non-magnetic objects partially buried in sand. The sensors consist of Hall-effect magnetometers and permanent magnets embedded in load bearing compliant segments. The magnetometers are sensitive to magnetic objects in close proximity to the legs and their movement relative to embedded magnets, allowing for both proximity and force-related feedback in dynamically obtained measurements. A dataset of three-axis measurements is collected as the robot steps near and over different UXOs and UXO-like objects, and a convolutional neural network is trained on time domain inputs and evaluated by 5-fold cross validation. Additionally, we propose a novel method for interpreting the importance of measurements in the time domain for the trained classifier. The results demonstrate the potential for accurate and efficient UXO and non-UXO discrimination in the field.

Harnessing deep learning for population genetic inference

In population genetics, the emergence of large-scale genomic data for various species and populations has provided new opportunities to understand the evolutionary forces that drive genetic diversity using statistical inference. However, the era of population genomics presents new challenges in analysing the massive amounts of genomes and variants. Deep learning has demonstrated state-of-the-art performance for numerous applications involving large-scale data. Recently, deep learning approaches have gained popularity in population genetics; facilitated by the advent of massive genomic data sets, powerful computational hardware and complex deep learning architectures, they have been used to identify population structure, infer demographic history and investigate natural selection. Here, we introduce common deep learning architectures and provide comprehensive guidelines for implementing deep learning models for population genetic inference. We also discuss current challenges and future directions for applying deep learning in population genetics, focusing on efficiency, robustness and interpretability.

Spectrum Analysis for Fully Connected Neural Networks

This article studies the meaning of parameters of fully connected neural networks with single hidden layer from the perspective of spectrum. Under the constraints of numerical range, the corresponding relationship between parameters and the spectrum of network function can be established by the Fourier series coefficients of the activation function, which is truncated and periodically extended. This work is substantiated on the Mixed National Institute of Standards and Technology (MNIST) handwritten dataset and two illustrative examples with certain spectra. The simulations complete the conversion between spectrum and parameters with high precision and give the significance of hidden nodes to the spectrum of network function. Some algorithms derived from these properties, such as the parameter initialization method using spectrum and the pruning method by sorting amplification weights, are also presented to introduce how spectrum analysis affects neural network decision-making. Thus, spectrum analysis has great potential in network interpretation.

Understanding neural network through neuron level visualization

Neurons are the fundamental units of neural networks. In this paper, we propose a method for explaining neural networks by visualizing the learning process of neurons. For a trained neural network, the proposed method obtains the features learned by each neuron and displays the features in a human-understandable form. The features learned by different neurons are combined to analyze the working mechanism of different neural network models. The method is applicable to neural networks without requiring any changes to the architectures of the models. In this study, we apply the proposed method to both Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) trained using the backpropagation learning algorithm. We conduct experiments on models for image classification tasks to demonstrate the effectiveness of the method. Through these experiments, we gain insights into the working mechanisms of various neural network architectures and evaluate neural network interpretability from diverse perspectives.

A scoping review of artificial intelligence-based methods for diabetes risk prediction

The increasing prevalence of type 2 diabetes mellitus (T2DM) and its associated health complications highlight the need to develop predictive models for early diagnosis and intervention. While many artificial intelligence (AI) models for T2DM risk prediction have emerged, a comprehensive review of their advancements and challenges is currently lacking. This scoping review maps out the existing literature on AI-based models for T2DM prediction, adhering to the PRISMA extension for Scoping Reviews guidelines. A systematic search of longitudinal studies was conducted across four databases, including PubMed, Scopus, IEEE-Xplore, and Google Scholar. Forty studies that met our inclusion criteria were reviewed. Classical machine learning (ML) models dominated these studies, with electronic health records (EHR) being the predominant data modality, followed by multi-omics, while medical imaging was the least utilized. Most studies employed unimodal AI models, with only ten adopting multimodal approaches. Both unimodal and multimodal models showed promising results, with the latter being superior. Almost all studies performed internal validation, but only five conducted external validation. Most studies utilized the area under the curve (AUC) for discrimination measures. Notably, only five studies provided insights into the calibration of their models. Half of the studies used interpretability methods to identify key risk predictors revealed by their models. Although a minority highlighted novel risk predictors, the majority reported commonly known ones. Our review provides valuable insights into the current state and limitations of AI-based models for T2DM prediction and highlights the challenges associated with their development and clinical integration.

Modeling flood susceptibility zones using hybrid machine learning models of an agricultural dominant landscape of India

Flooding events are determining a significant amount of damages, in terms of economic loss and also casualties in Asia and Pacific areas. Due to complexity and ferocity of severe flooding, predicting flood-prone areas is a difficult task. Thus, creating flood susceptibility maps at local level is though challenging but an inevitable task. In order to implement a flood management plan for the Balrampur district, an agricultural dominant landscape of India, and strengthen its resilience, flood susceptibility modeling and mapping are carried out. In the present study, three hybrid machine learning (ML) models, namely, fuzzy-ANN (artificial neural network), fuzzy-RBF (radial basis function), and fuzzy-SVM (support vector machine) with 12 topographic, hydrological, and other flood influencing factors were used to determine flood-susceptible zones. To ascertain the relationship between the occurrences and flood influencing factors, correlation attribute evaluation (CAE) and multicollinearity diagnostic tests were used. The predictive power of these models was validated and compared using a variety of statistical techniques, including Wilcoxon signed-rank, t-paired tests and receiver operating characteristic (ROC) curves. Results show that fuzzy-RBF model outperformed other hybrid ML models for modeling flood susceptibility, followed by fuzzy-ANN and fuzzy-SVM. Overall, these models have shown promise in identifying flood-prone areas in the basin and other basins around the world. The outcomes of the work would benefit policymakers and government bodies to capture the flood-affected areas for necessary planning, action, and implementation.

CEModule: A Computation Efficient Module for Lightweight Convolutional Neural Networks

Lightweight convolutional neural networks (CNNs) rely heavily on the design of lightweight convolutional modules (LCMs). For an LCM, lightweight design based on repetitive feature maps (LoR) is currently one of the most effective approaches. An LoR mainly involves an extraction of feature maps from convolutional layers (CE) and feature map regeneration through cheap operations (RO). However, existing LoR approaches carry out lightweight improvements only from the aspect of RO but ignore the problems of poor generalization, low stability, and high computation workload incurred in the CE part. To alleviate these problems, this article introduces the concept of key features from a CNN model interpretation perspective. Subsequently, it presents a novel LCM, namely CEModule, focusing on the CE part. CEModule increases the number of key features to maintain a high level of accuracy in classification. In the meantime, CEModule employs a group convolution strategy to reduce floating-point operations (FLOPs) incurred in the training process. Finally, this article brings forth a dynamic adaptation algorithm ( α -DAM) to enhance the generalization of CEModule-enabled lightweight CNN models, including the developed CENet in dealing with datasets of different scales. Compared with the state-of-the-art results, CEModule reduces FLOPs by up to 54% on CIFAR-10 while maintaining a similar level of accuracy in classification. On ImageNet, CENet increases accuracy by 1.2% following the same FLOPs and training strategies.

Genetic algorithms based optimized hybrid deep learning model for explainable Alzheimer's prediction based on temporal multimodal cognitive data.. [DOI: 10.21203/rs.3.rs-3250006/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Alzheimer's Disease (AD) is an irreversible neurodegenerative disease. Its early detection is crucial to stop disease progression at an early stage. Most deep learning (DL) literature focused on neuroimage analysis. However, there is no noticed effect of these studies in the real environment. Model's robustness, cost, and interpretability are considered the main reasons for these limitations. The medical intuition of physicians is to evaluate the clinical biomarkers of patients then test their neuroimages. Cognitive scores provide an medically acceptable and cost-effective alternative for the neuroimages to predict AD progression. Each score is calculated from a collection of sub-scores which provide a deeper insight about patient conditions. No study in the literature have explored the role of these multimodal time series sub-scores to predict AD progression. We propose a hybrid CNN-LSTM DL model for predicting AD progression based on the fusion of four longitudinal cognitive sub-scores modalities. Bayesian optimizer has been used to select the best DL architecture. A genetic algorithms based feature selection optimization step has been added to the pipeline to select the best features from extracted deep representations of CNN-LSTM. The SoftMax classifier has been replaced by a robust and optimized random forest classifier. Extensive experiments using the ADNI dataset investigated the role of each optimization step, and the proposed model achieved the best results compared to other DL and classical machine learning models. The resulting model is robust, but it is a black box and it is difficult to understand the logic behind its decisions. Trustworthy AI models must be robust and explainable. We used SHAP and LIME to provide explainability features for the proposed model. The resulting trustworthy model has a great potential to be used to provide decision support in the real environments.

Modeling strength characteristics of basalt fiber reinforced concrete using multiple explainable machine learning with a graphical user interface

This study investigated the importance of applying explainable artificial intelligence (XAI) on different machine learning (ML) models developed to predict the strength characteristics of basalt-fiber reinforced concrete (BFRC). Even though ML is widely adopted in strength prediction in concrete, the black-box nature of predictions hinders the interpretation of results. Among several attempts to overcome this limitation by using explainable AI, researchers have employed only a single explanation method. In this study, we used three tree-based ML models (Decision tree, Gradient Boosting tree, and Light Gradient Boosting Machine) to predict the mechanical strength characteristics (compressive strength, flexural strength, and tensile strength) of basal fiber reinforced concrete (BFRC). For the first time, we employed two explanation methods (Shapley additive explanations (SHAP) and local interpretable model-agnostic explanations (LIME)) to provide explanations for all models. These explainable methods reveal the underlying decision-making criteria of complex machine learning models, improving the end user's trust. The comparison highlights that tree-based models obtained good accuracy in predicting strength characteristics yet, their explanations were different either by the magnitude of feature importance or the order of importance. This disagreement pushes towards complicated decision-making based on ML predictions which further stresses (1) extending XAI-based research in concrete strength predictions, and (2) involving domain experts to evaluate XAI results. The study concludes with the development of a "user-friendly computer application" which enables quick strength prediction of basalt fiber reinforced concrete (BFRC).

An Explainable Vision Transformer Model Based White Blood Cells Classification and Localization

White blood cells (WBCs) are crucial components of the immune system that play a vital role in defending the body against infections and diseases. The identification of WBCs subtypes is useful in the detection of various diseases, such as infections, leukemia, and other hematological malignancies. The manual screening of blood films is time-consuming and subjective, leading to inconsistencies and errors. Convolutional neural networks (CNN)-based models can automate such classification processes, but are incapable of capturing long-range dependencies and global context. This paper proposes an explainable Vision Transformer (ViT) model for automatic WBCs detection from blood films. The proposed model uses a self-attention mechanism to extract features from input images. Our proposed model was trained and validated on a public dataset of 16,633 samples containing five different types of WBCs. As a result of experiments on the classification of five different types of WBCs, our model achieved an accuracy of 99.40%. Moreover, the model's examination of misclassified test samples revealed a correlation between incorrect predictions and the presence or absence of granules in the cell samples. To validate this observation, we divided the dataset into two classes, Granulocytes and Agranulocytes, and conducted a secondary training process. The resulting ViT model, trained for binary classification, achieved impressive performance metrics during the test phase, including an accuracy of 99.70%, recall of 99.54%, precision of 99.32%, and F-1 score of 99.43%. To ensure the reliability of the ViT model's, we employed the Score-CAM algorithm to visualize the pixel areas on which the model focuses during its predictions. Our proposed method is suitable for clinical use due to its explainable structure as well as its superior performance compared to similar studies in the literature. The classification and localization of WBCs with this model can facilitate the detection and reporting process for the pathologist.

Integrating model explanations and hybrid priors into deep stacked networks for the "safe zone" prediction of acetabular cup

BACKGROUND

Existing state-of-the-art "safe zone" prediction methods are statistics-based methods, image-matching techniques, and machine learning methods. Yet, those methods bring a tension between accuracy and interpretability.

PURPOSE

To explore the model explanations and estimator consensus for "safe zone" prediction.

MATERIAL AND METHODS

We collected the pelvic datasets from Orthopaedic Hospital, and a novel acetabular cup detection method is proposed for automatic ROI segmentation. Hybrid priors comprising both specific priors from data and general priors from experts are constructed. Specifically, specific priors are constructed based on the fine-tuned ResNet-101 convolutional neural networks (CNN) model, and general priors are constructed based on expert knowledge. Our method considers the model explanations and dynamic consensus through appending a SHapley Additive exPlanations (SHAP) module and a dynamic estimator stacking.

RESULTS

The proposed method achieves an accuracy of 99.40% and an area under the curve of 0.9998. Experimental results show that our model achieves superior results to the state-of-the-art conventional ensemble classifiers and deep CNN models.

CONCLUSION

This new screening model provides a new option for the "safe zone" prediction of acetabular cup.

Creating meaningful work in the age of AI: explainable AI, explainability, and why it matters to organizational designers

In this paper, we contribute to research on enterprise artificial intelligence (AI), specifically to organizations improving the customer experiences and their internal processes through using the type of AI called machine learning (ML). Many organizations are struggling to get enough value from their AI efforts, and part of this is related to the area of explainability. The need for explainability is especially high in what is called black-box ML models, where decisions are made without anyone understanding how an AI reached a particular decision. This opaqueness creates a user need for explanations. Therefore, researchers and designers create different versions of so-called eXplainable AI (XAI). However, the demands for XAI can reduce the accuracy of the predictions the AI makes, which can reduce the perceived usefulness of the AI solution, which, in turn, reduces the interest in designing the organizational task structure to benefit from the AI solution. Therefore, it is important to ensure that the need for XAI is as low as possible. In this paper, we demonstrate how to achieve this by optimizing the task structure according to sociotechnical systems design principles. Our theoretical contribution is to the underexplored field of the intersection of AI design and organizational design. We find that explainability goals can be divided into two groups, pattern goals and experience goals, and that this division is helpful when defining the design process and the task structure that the AI solution will be used in. Our practical contribution is for AI designers who include organizational designers in their teams, and for organizational designers who answer that challenge.

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning

Recently, personality trait recognition, which aims to identify people's first impression behavior data and analyze people's psychological characteristics, has been an interesting and active topic in psychology, affective neuroscience and artificial intelligence. To effectively take advantage of spatio-temporal cues in audio-visual modalities, this paper proposes a new method of multimodal personality trait recognition integrating audio-visual modalities based on a hybrid deep learning framework, which is comprised of convolutional neural networks (CNN), bi-directional long short-term memory network (Bi-LSTM), and the Transformer network. In particular, a pre-trained deep audio CNN model is used to learn high-level segment-level audio features. A pre-trained deep face CNN model is leveraged to separately learn high-level frame-level global scene features and local face features from each frame in dynamic video sequences. Then, these extracted deep audio-visual features are fed into a Bi-LSTM and a Transformer network to individually capture long-term temporal dependency, thereby producing the final global audio and visual features for downstream tasks. Finally, a linear regression method is employed to conduct the single audio-based and visual-based personality trait recognition tasks, followed by a decision-level fusion strategy used for producing the final Big-Five personality scores and interview scores. Experimental results on the public ChaLearn First Impression-V2 personality dataset show the effectiveness of our method, outperforming other used methods.

Trichloroethylene remediation using zero-valent iron with kaolin clay, activated carbon and bacteria

Nanoscale particles of zero-valent iron were used to form a permeable reactive barrier whose performance in dechlorinating a solution of trichloroethylene was compared with that of a barrier formed from limestone. The iron was combined with kaolin by calcination. The test liquid contained sewage sludge, and also added NH₄Cl and KH₂PO₄. The average removal rates of trichloroethylene and phosphorus over 365 days both exceeded 94%. Chemical oxygen demand was reduced by 92% and ammonium nitrogen by 43.6%. All were significantly greater than the removals with the limestone barrier. The ceramsite barrier retained 85% of its effectiveness even after 365 days of use. Dechloromonas sp. was the main dechlorinating bacterium, but its removal ability is limited. The removal of trichloroethylene in such a barrier mainly depends on reduction by the zero-valent iron and biodegradation. The results show that the prepared ceramsite is stable and effective in removing trichloroethylene from water. It is a promising in-situ remediation material for groundwater.

Towards Interpretable Camera and LiDAR Data Fusion for Autonomous Ground Vehicles Localisation

Recent deep learning frameworks draw strong research interest in application of ego-motion estimation as they demonstrate a superior result compared to geometric approaches. However, due to the lack of multimodal datasets, most of these studies primarily focused on single-sensor-based estimation. To overcome this challenge, we collect a unique multimodal dataset named LboroAV2 using multiple sensors, including camera, light detecting and ranging (LiDAR), ultrasound, e-compass and rotary encoder. We also propose an end-to-end deep learning architecture for fusion of RGB images and LiDAR laser scan data for odometry application. The proposed method contains a convolutional encoder, a compressed representation and a recurrent neural network. Besides feature extraction and outlier rejection, the convolutional encoder produces a compressed representation, which is used to visualise the network's learning process and to pass useful sequential information. The recurrent neural network uses this compressed sequential data to learn the relationship between consecutive time steps. We use the Loughborough autonomous vehicle (LboroAV2) and the Karlsruhe Institute of Technology and Toyota Institute (KITTI) Visual Odometry (VO) datasets to experiment and evaluate our results. In addition to visualising the network's learning process, our approach provides superior results compared to other similar methods. The code for the proposed architecture is released in GitHub and accessible publicly.

Improving disease classification performance and explainability of deep learning models in radiology with heatmap generators

As deep learning is widely used in the radiology field, the explainability of Artificial Intelligence (AI) models is becoming increasingly essential to gain clinicians' trust when using the models for diagnosis. In this research, three experiment sets were conducted with a U-Net architecture to improve the disease classification performance while enhancing the heatmaps corresponding to the model's focus through incorporating heatmap generators during training. All experiments used the dataset that contained chest radiographs, associated labels from one of the three conditions ["normal", "congestive heart failure (CHF)", and "pneumonia"], and numerical information regarding a radiologist's eye-gaze coordinates on the images. The paper that introduced this dataset developed a U-Net model, which was treated as the baseline model for this research, to show how the eye-gaze data can be used in multi-modal training for explainability improvement and disease classification. To compare the classification performances among this research's three experiment sets and the baseline model, the 95% confidence intervals (CI) of the area under the receiver operating characteristic curve (AUC) were measured. The best method achieved an AUC of 0.913 with a 95% CI of [0.860, 0.966]. "Pneumonia" and "CHF" classes, which the baseline model struggled the most to classify, had the greatest improvements, resulting in AUCs of 0.859 with a 95% CI of [0.732, 0.957] and 0.962 with a 95% CI of [0.933, 0.989], respectively. The decoder of the U-Net for the best-performing proposed method generated heatmaps that highlight the determining image parts in model classifications. These predicted heatmaps, which can be used for the explainability of the model, also improved to align well with the radiologist's eye-gaze data. Hence, this work showed that incorporating heatmap generators and eye-gaze information into training can simultaneously improve disease classification and provide explainable visuals that align well with how the radiologist viewed the chest radiographs when making diagnosis.

Representation Learning Method with Semantic Propagation on Text-Augmented Knowledge Graphs

Knowledge graph representation learning aims to provide accurate entity and relation representations for tasks such as intelligent question answering and recommendation systems. Existing representation learning methods, which only consider triples, are not sufficiently accurate, so some methods use external auxiliary information such as text, type, and time to improve performance. However, they often encode this information independently, which makes it challenging to fully integrate this information with the knowledge graph at a semantic level. In this study, we propose a method called SP-TAG, which realizes the semantic propagation on text-augmented knowledge graphs. Specifically, SP-TAG constructs a text-augmented knowledge graph by extracting named entities from text descriptions and connecting them with the corresponding entities. Then, SP-TAG uses a graph convolutional network to propagate semantic information between the entities and new named entities so that the text and triple structure are fully integrated. The results of experiments on multiple benchmark datasets show that SP-TAG attains competitive performance. When the number of training samples is limited, SP-TAG maintains its high performance, verifying the importance of text augmentation and semantic propagation.

A Fair Performance Comparison between Complex-Valued and Real-Valued Neural Networks for Disease Detection

Our aim is to contribute to the classification of anomalous patterns in biosignals using this novel approach. We specifically focus on melanoma and heart murmurs. We use a comparative study of two convolution networks in the Complex and Real numerical domains. The idea is to obtain a powerful approach for building portable systems for early disease detection. Two similar algorithmic structures were chosen so that there is no bias determined by the number of parameters to train. Three clinical data sets, ISIC2017, PH2, and Pascal, were used to carry out the experiments. Mean comparison hypothesis tests were performed to ensure statistical objectivity in the conclusions. In all cases, complex-valued networks presented a superior performance for the Precision, Recall, F1 Score, Accuracy, and Specificity metrics in the detection of associated anomalies. The best complex number-based classifier obtained in the Receiving Operating Characteristic (ROC) space presents a Euclidean distance of 0.26127 with respect to the ideal classifier, as opposed to the best real number-based classifier, whose Euclidean distance to the ideal is 0.36022 for the same task of melanoma detection. The 27.46% superiority in this metric, as in the others reported in this work, suggests that complex-valued networks have a greater ability to extract features for more efficient discrimination in the dataset.

Privacy-Preserving and Explainable AI in Industrial Applications

The industrial environment has gone through the fourth revolution, also called “Industry 4.0”, where the main aspect is digitalization. Each device employed in an industrial process is connected to a network called the industrial Internet of things (IIOT). With IIOT manufacturers being capable of tracking every device, it has become easier to prevent or quickly solve failures. Specifically, the large amount of available data has allowed the use of artificial intelligence (AI) algorithms to improve industrial applications in many ways (e.g., failure detection, process optimization, and abnormality detection). Although data are abundant, their access has raised problems due to privacy concerns of manufacturers. Censoring sensitive information is not a desired approach because it negatively impacts the AI performance. To increase trust, there is also the need to understand how AI algorithms make choices, i.e., to no longer regard them as black boxes. This paper focuses on recent advancements related to the challenges mentioned above, discusses the industrial impact of proposed solutions, and identifies challenges for future research. It also presents examples related to privacy-preserving and explainable AI solutions, and comments on the interaction between the identified challenges in the conclusions.

Predicting Bulk Average Velocity with Rigid Vegetation in Open Channels Using Tree-Based Machine Learning: A Novel Approach Using Explainable Artificial Intelligence

Predicting the bulk-average velocity (U_B) in open channels with rigid vegetation is complicated due to the non-linear nature of the parameters. Despite their higher accuracy, existing regression models fail to highlight the feature importance or causality of the respective predictions. Therefore, we propose a method to predict U_B and the friction factor in the surface layer (f_S) using tree-based machine learning (ML) models (decision tree, extra tree, and XGBoost). Further, Shapley Additive exPlanation (SHAP) was used to interpret the ML predictions. The comparison emphasized that the XGBoost model is superior in predicting U_B (R = 0.984) and f_S (R = 0.92) relative to the existing regression models. SHAP revealed the underlying reasoning behind predictions, the dependence of predictions, and feature importance. Interestingly, SHAP adheres to what is generally observed in complex flow behavior, thus, improving trust in predictions.

Deep Personality Trait Recognition: A Survey

Automatic personality trait recognition has attracted increasing interest in psychology, neuropsychology, and computer science, etc. Motivated by the great success of deep learning methods in various tasks, a variety of deep neural networks have increasingly been employed to learn high-level feature representations for automatic personality trait recognition. This paper systematically presents a comprehensive survey on existing personality trait recognition methods from a computational perspective. Initially, we provide available personality trait data sets in the literature. Then, we review the principles and recent advances of typical deep learning techniques, including deep belief networks (DBNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs). Next, we describe the details of state-of-the-art personality trait recognition methods with specific focus on hand-crafted and deep learning-based feature extraction. These methods are analyzed and summarized in both single modality and multiple modalities, such as audio, visual, text, and physiological signals. Finally, we analyze the challenges and opportunities in this field and point out its future directions.

Interpretation of Machine-Learning-Based (Black-box) Wind Pressure Predictions for Low-Rise Gable-Roofed Buildings Using Shapley Additive Explanations (SHAP)

Conventional methods of estimating pressure coefficients of buildings retain time and cost constraints. Recently, machine learning (ML) has been successfully established to predict wind pressure coefficients. However, regardless of the accuracy, ML models are incompetent in providing end-users’ confidence as a result of the black-box nature of predictions. In this study, we employed tree-based regression models (Decision Tree, XGBoost, Extra-tree, LightGBM) to predict surface-averaged mean pressure coefficient (Cp,mean), fluctuation pressure coefficient (Cp, rms), and peak pressure coefficient (Cp,peak) of low-rise gable-roofed buildings. The accuracy of models was verified using Tokyo Polytechnic University (TPU) wind tunnel data. Subsequently, we used Shapley Additive Explanations (SHAP) to explain the black-box nature of the ML predictions. The comparison revealed that tree-based models are efficient and accurate in wind-predicting pressure coefficients. Interestingly, SHAP provided human-comprehensible explanations for the interaction of variables, the importance of features towards the outcome, and the underlying reasoning behind the predictions. Moreover, SHAP confirmed that tree-based predictions adhere to the flow physics of wind engineering, advancing the fidelity of ML-based predictions.

A multivariate multi-step LSTM forecasting model for tuberculosis incidence with model explanation in Liaoning Province, China

BACKGROUND

Tuberculosis (TB) is the respiratory infectious disease with the highest incidence in China. We aim to design a series of forecasting models and find the factors that affect the incidence of TB, thereby improving the accuracy of the incidence prediction.

RESULTS

In this paper, we developed a new interpretable prediction system based on the multivariate multi-step Long Short-Term Memory (LSTM) model and SHapley Additive exPlanation (SHAP) method. Four accuracy measures are introduced into the system: Root Mean Square Error, Mean Absolute Error, Mean Absolute Percentage Error, and symmetric Mean Absolute Percentage Error. The Autoregressive Integrated Moving Average (ARIMA) model and seasonal ARIMA model are established. The multi-step ARIMA-LSTM model is proposed for the first time to examine the performance of each model in the short, medium, and long term, respectively. Compared with the ARIMA model, each error of the multivariate 2-step LSTM model is reduced by 12.92%, 15.94%, 15.97%, and 14.81% in the short term. The 3-step ARIMA-LSTM model achieved excellent performance, with each error decreased to 15.19%, 33.14%, 36.79%, and 29.76% in the medium and long term. We provide the local and global explanation of the multivariate single-step LSTM model in the field of incidence prediction, pioneering.

CONCLUSIONS

The multivariate 2-step LSTM model is suitable for short-term prediction and obtained a similar performance as previous studies. The 3-step ARIMA-LSTM model is appropriate for medium-to-long-term prediction and outperforms these models. The SHAP results indicate that the five most crucial features are maximum temperature, average relative humidity, local financial budget, monthly sunshine percentage, and sunshine hours.

Machine Learning for Plant Stress Modeling: A Perspective towards Hormesis Management

Plant stress is one of the most significant factors affecting plant fitness and, consequently, food production. However, plant stress may also be profitable since it behaves hormetically; at low doses, it stimulates positive traits in crops, such as the synthesis of specialized metabolites and additional stress tolerance. The controlled exposure of crops to low doses of stressors is therefore called hormesis management, and it is a promising method to increase crop productivity and quality. Nevertheless, hormesis management has severe limitations derived from the complexity of plant physiological responses to stress. Many technological advances assist plant stress science in overcoming such limitations, which results in extensive datasets originating from the multiple layers of the plant defensive response. For that reason, artificial intelligence tools, particularly Machine Learning (ML) and Deep Learning (DL), have become crucial for processing and interpreting data to accurately model plant stress responses such as genomic variation, gene and protein expression, and metabolite biosynthesis. In this review, we discuss the most recent ML and DL applications in plant stress science, focusing on their potential for improving the development of hormesis management protocols.

Time-Dynamic Markov Random Fields for price outcome prediction in the presence of lobbying

This paper presents a mathematical/Artificial Intelligence (AI) model for the prediction of price outcomes in markets with the presence of lobbying, whose outputs are pricing trends that aggregate the opinions of lobbies on future prices. Our proposal succeeds in unraveling this complex real-world problem by reducing the solution to straightforward probability computations. We tested our method on real olive oil prices (Andalusia, Spain) with encouraging results in a challenging sector, where opacity in the entry of oil shipments which are stored while waiting for the price to rise, makes it very difficult to forecast the prices. Specifically, understanding by minimum price that the price level is at least reached, specific formulas for computing the likelihood of both the aggregate and the minimum market price are provided. These formulas are based on the price levels that lobbies expect which in turn, can be calculated using the probability that each lobby gives to market prices. An innovative quantitative study of the lobbies is also carried out by explicitly computing the weight of each lobby in the process thus solving a problem for which there were only qualitative references up until now. The structural model is based on Time Dynamic Markov random fields (TD-MRFs). This model requires significantly less information to produce an output and enjoys transparency during the process when compared with other approaches, such as neural networks (known as black boxes). Transparency also ensures that the internal structures can be fine tuned to fit to each context as well as possible.

Evaluating the capability of municipal solid waste separation in China based on AHP-EWM and BP neural network

With the increase in municipal solid waste (MSW), most cities face solid waste management issues. In this study, the analytic hierarchy process (AHP) and artificial neural network (ANN) models were improved to assess the MSW separation capability based on 18 selected indicators of solid waste separation in 15 cities in China. The entropy weight method (EWM) was used in AHP to optimize and determine the indicators and then evaluate their weights, which showed that the general public budget expenditure had the highest weight (0.5239). This implied that the MSW separation capability could be mainly influenced by government financial support. ANN based on scan optimization and machine learning methods were established (R² = 0.9992) to predict the missing indicators. The mapping relationship between MSW separation indicators and capabilities was also significantly improved from R² = 0.5317 to R² = 0.9993, thereby increasing the prediction accuracy of MSW separation capabilities to 95.15%. Thus, this research provides a new avenue for MSW separation and establishes a combined model to predict the separation capability in practical applications.

Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond

Explainable Artificial Intelligence (XAI) is an emerging research topic of machine learning aimed at unboxing how AI systems' black-box choices are made. This research field inspects the measures and models involved in decision-making and seeks solutions to explain them explicitly. Many of the machine learning algorithms cannot manifest how and why a decision has been cast. This is particularly true of the most popular deep neural network approaches currently in use. Consequently, our confidence in AI systems can be hindered by the lack of explainability in these black-box models. The XAI becomes more and more crucial for deep learning powered applications, especially for medical and healthcare studies, although in general these deep neural networks can return an arresting dividend in performance. The insufficient explainability and transparency in most existing AI systems can be one of the major reasons that successful implementation and integration of AI tools into routine clinical practice are uncommon. In this study, we first surveyed the current progress of XAI and in particular its advances in healthcare applications. We then introduced our solutions for XAI leveraging multi-modal and multi-centre data fusion, and subsequently validated in two showcases following real clinical scenarios. Comprehensive quantitative and qualitative analyses can prove the efficacy of our proposed XAI solutions, from which we can envisage successful applications in a broader range of clinical questions.

SurvNAM: The machine learning survival model explanation

An extension of the Neural Additive Model (NAM) called SurvNAM and its modifications are proposed to explain predictions of a black-box machine learning survival model. The method is based on applying the original NAM to solving the explanation problem in the framework of survival analysis. The basic idea behind SurvNAM is to train the network by means of a specific expected loss function which takes into account peculiarities of the survival model predictions. Moreover, the loss function approximates the black-box model by the extension of the Cox proportional hazards model, which uses the well-known Generalized Additive Model (GAM) in place of the simple linear relationship of covariates. The proposed method SurvNAM allows performing local and global explanations. The global explanation uses the whole training dataset. In contrast to the global explanation, a set of synthetic examples around the explained example are randomly generated for the local explanation. The proposed modifications of SurvNAM are based on using the Lasso-based regularization for functions from GAM and for a special representation of the GAM functions using their weighted linear and non-linear parts, which is implemented as a shortcut connection. Many numerical experiments illustrate efficiency of SurvNAM.

Towards a Reliable Evaluation of Local Interpretation Methods

The growing use of deep neural networks in critical applications is making interpretability urgently to be solved. Local interpretation methods are the most prevalent and accepted approach for understanding and interpreting deep neural networks. How to effectively evaluate the local interpretation methods is challenging. To address this question, a unified evaluation framework is proposed, which assesses local interpretation methods from three dimensions: accuracy, persuasibility and class discriminativeness. Specifically, in order to assess correctness, we designed an interactive user feature annotation tool to provide ground truth for local interpretation methods. To verify the usefulness of the interpretation method, we iteratively display part of the interpretation results, and then ask users whether they agree with the category information. At the same time, we designed and built a set of evaluation data sets with a rich hierarchical structure. Surprisingly, one finding is that the existing visual interpretation methods cannot satisfy all evaluation dimensions at the same time, and each has its own shortcomings.