A Systematic Review of Explainable Artificial Intelligence in Terms of Different Application Domains and Tasks

The (Im)perfect Automation Schema: Who Is Trusted More, Automated or Human Decision Support?

OBJECTIVE

This study's purpose was to better understand the dynamics of trust attitude and behavior in human-agent interaction.

BACKGROUND

Whereas past research provided evidence for a perfect automation schema, more recent research has provided contradictory evidence.

METHOD

To disentangle these conflicting findings, we conducted an online experiment using a simulated medical X-ray task. We manipulated the framing of support agents (i.e., artificial intelligence (AI) versus expert versus novice) between-subjects and failure experience (i.e., perfect support, imperfect support, back-to-perfect support) within subjects. Trust attitude and behavior as well as perceived reliability served as dependent variables.

RESULTS

Trust attitude and perceived reliability were higher for the human expert than for the AI than for the human novice. Moreover, the results showed the typical pattern of trust formation, dissolution, and restoration for trust attitude and behavior as well as perceived reliability. Forgiveness after failure experience did not differ between agents.

CONCLUSION

The results strongly imply the existence of an imperfect automation schema. This illustrates the need to consider agent expertise for human-agent interaction.

APPLICATION

When replacing human experts with AI as support agents, the challenge of lower trust attitude towards the novel agent might arise.

Explainable AI: Machine Learning Interpretation in Blackcurrant Powders

Recently, explainability in machine and deep learning has become an important area in the field of research as well as interest, both due to the increasing use of artificial intelligence (AI) methods and understanding of the decisions made by models. The explainability of artificial intelligence (XAI) is due to the increasing consciousness in, among other things, data mining, error elimination, and learning performance by various AI algorithms. Moreover, XAI will allow the decisions made by models in problems to be more transparent as well as effective. In this study, models from the 'glass box' group of Decision Tree, among others, and the 'black box' group of Random Forest, among others, were proposed to understand the identification of selected types of currant powders. The learning process of these models was carried out to determine accuracy indicators such as accuracy, precision, recall, and F1-score. It was visualized using Local Interpretable Model Agnostic Explanations (LIMEs) to predict the effectiveness of identifying specific types of blackcurrant powders based on texture descriptors such as entropy, contrast, correlation, dissimilarity, and homogeneity. Bagging (Bagging_100), Decision Tree (DT0), and Random Forest (RF7_gini) proved to be the most effective models in the framework of currant powder interpretability. The measures of classifier performance in terms of accuracy, precision, recall, and F1-score for Bagging_100, respectively, reached values of approximately 0.979. In comparison, DT0 reached values of 0.968, 0.972, 0.968, and 0.969, and RF7_gini reached values of 0.963, 0.964, 0.963, and 0.963. These models achieved classifier performance measures of greater than 96%. In the future, XAI using agnostic models can be an additional important tool to help analyze data, including food products, even online.

PERform: assessing model performance with predictivity and explainability readiness formula

In the rapidly evolving field of artificial intelligence (AI), explainability has been traditionally assessed in a post-modeling process and is often subjective. In contrary, many quantitative metrics have been routinely used to assess a model's performance. We proposed a unified formular named PERForm, by incorporating explainability as a weight into the existing statistical metrics to provide an integrated and quantitative measure of both predictivity and explainability to guide model selection, application, and evaluation. PERForm was designed as a generic formula and can be applied to any data types. We applied PERForm on a range of diverse datasets, including DILIst, Tox21, and three MAQC-II benchmark datasets, using various modeling algorithms to predict a total of 73 distinct endpoints. For example, AdaBoost algorithms exhibited superior performance (PERForm AUC for AdaBoost is 0.129 where Linear regression is 0) in DILIst prediction, where linear regression outperformed other models in the majority of Tox21 endpoints (PERForm AUC for linear regression is 0.301 where AdaBoost is 0.283 in average). This research marks a significant step toward comprehensively evaluating the utility of an AI model to advance transparency and interpretability, where the tradeoff between a model's performance and its interpretability can have profound implications.

Resilience-aware MLOps for AI-based medical diagnostic system

Background

The healthcare sector demands a higher degree of responsibility, trustworthiness, and accountability when implementing Artificial Intelligence (AI) systems. Machine learning operations (MLOps) for AI-based medical diagnostic systems are primarily focused on aspects such as data quality and confidentiality, bias reduction, model deployment, performance monitoring, and continuous improvement. However, so far, MLOps techniques do not take into account the need to provide resilience to disturbances such as adversarial attacks, including fault injections, and drift, including out-of-distribution. This article is concerned with the MLOps methodology that incorporates the steps necessary to increase the resilience of an AI-based medical diagnostic system against various kinds of disruptive influences.

Methods

Post-hoc resilience optimization, post-hoc predictive uncertainty calibration, uncertainty monitoring, and graceful degradation are incorporated as additional stages in MLOps. To optimize the resilience of the AI based medical diagnostic system, additional components in the form of adapters and meta-adapters are utilized. These components are fine-tuned during meta-training based on the results of adaptation to synthetic disturbances. Furthermore, an additional model is introduced for post-hoc calibration of predictive uncertainty. This model is trained using both in-distribution and out-of-distribution data to refine predictive confidence during the inference mode.

Results

The structure of resilience-aware MLOps for medical diagnostic systems has been proposed. Experimentally confirmed increase of robustness and speed of adaptation for medical image recognition system during several intervals of the system's life cycle due to the use of resilience optimization and uncertainty calibration stages. The experiments were performed on the DermaMNIST dataset, BloodMNIST and PathMNIST. ResNet-18 as a representative of convolutional networks and MedViT-T as a representative of visual transformers are considered. It is worth noting that transformers exhibited lower resilience than convolutional networks, although this observation may be attributed to potential imperfections in the architecture of adapters and meta-adapters.

Сonclusion

The main novelty of the suggested resilience-aware MLOps methodology and structure lie in the separating possibilities and activities on creating a basic model for normal operating conditions and ensuring its resilience and trustworthiness. This is significant for the medical applications as the developer of the basic model should devote more time to comprehending medical field and the diagnostic task at hand, rather than specializing in system resilience. Resilience optimization increases robustness to disturbances and speed of adaptation. Calibrated confidences ensure the recognition of a portion of unabsorbed disturbances to mitigate their impact, thereby enhancing trustworthiness.

Applying explainable artificial intelligence methods to models for diagnosing personal traits and cognitive abilities by social network data

This study utilizes advanced artificial intelligence techniques to analyze the social media behavior of 1358 users on VK, the largest Russian online social networking service. The analysis comprises 753,252 posts and reposts, combined with Big Five personality traits test results, as well as assessments of verbal and fluid intelligence. The objective of this research is to understand the manifestation of psychological attributes in social media users' behavior and determine their implications on user-interaction models. We employ the integrated gradients method to identify the most influential feature groups. The partial dependence plot technique aids in understanding how these features function across varying severity degrees of the predicted trait. To evaluate feature stability within the models, we cluster calculated Shapley values. Our findings suggest that the emotional tone (joy, surprise, anger, fear) of posts significantly influences the prediction of three personality traits: Extraversion, Agreeableness, and Openness to Experience. Additionally, user social engagement metrics (such as friend count, subscribers, likes, views, and comments) correlate directly with the predicted level of Logical thinking. We also observe a trend towards provocative and socially reprehensible content among users with high Neuroticism levels. The theme of religion demonstrates a multidirectional relationship with Consciousness and Agreeableness. Further findings, including an analysis of post frequency and key text characteristics, are also discussed, contributing to our understanding of the complex interplay between social media behavior and psychological traits. The study proposes a transition from the analysis of correlations between psychological (cognitive) traits to the analysis of indicators of behavior in a social network that are significant for diagnostic models of the corresponding traits.

Automated detection of Alzheimer's disease: a multi-modal approach with 3D MRI and amyloid PET

Recent advances in deep learning and imaging technologies have revolutionized automated medical image analysis, especially in diagnosing Alzheimer's disease through neuroimaging. Despite the availability of various imaging modalities for the same patient, the development of multi-modal models leveraging these modalities remains underexplored. This paper addresses this gap by proposing and evaluating classification models using 2D and 3D MRI images and amyloid PET scans in uni-modal and multi-modal frameworks. Our findings demonstrate that models using volumetric data learn more effective representations than those using only 2D images. Furthermore, integrating multiple modalities enhances model performance over single-modality approaches significantly. We achieved state-of-the-art performance on the OASIS-3 cohort. Additionally, explainability analyses with Grad-CAM indicate that our model focuses on crucial AD-related regions for its predictions, underscoring its potential to aid in understanding the disease's causes.

Explainable machine learning for the prediction and assessment of complex drought impacts

Drought is a common and costly natural disaster with broad social, economic, and environmental impacts. Machine learning (ML) has been widely applied in scientific research because of its outstanding performance on predictive tasks. However, for practical applications like disaster monitoring and assessment, the cost of the models failure, especially false negative predictions, might significantly affect society. Stakeholders are not satisfied with or do not "trust" the predictions from a so-called black box. The explainability of ML models becomes progressively crucial in studying drought and its impacts. In this work, we propose an explainable ML pipeline using the XGBoost model and SHAP model based on a comprehensive database of drought impacts in the U.S. The XGBoost models significantly outperformed the baseline models in predicting the occurrence of multi-dimensional drought impacts derived from the text-based Drought Impact Reporter, attaining an average F2 score of 0.883 at the national level and 0.942 at the state level. The interpretation of the models at the state scale indicates that the Standardized Precipitation Index (SPI) and Standardized Temperature Index (STI) contribute significantly to predicting multi-dimensional drought impacts. The time scalar, importance, and relationships of the SPI and STI vary depending on the types of drought impacts and locations. The patterns between the SPI variables and drought impacts indicated by the SHAP values reveal an expected relationship in which negative SPI values positively contribute to complex drought impacts. The explainability based on the SPI variables improves the trustworthiness of the XGBoost models. Overall, this study reveals promising results in accurately predicting complex drought impacts and rendering the relationships between the impacts and indicators more interpretable. This study also reveals the potential of utilizing explainable ML for the general social good to help stakeholders better understand the multi-dimensional drought impacts at the regional level and motivate appropriate responses.

A novel breast cancer detection architecture based on a CNN-CBR system for mammogram classification

This paper presents a novel framework for breast cancer detection using mammogram images. The proposed solution aims to output an explainable classification from a mammogram image. The classification approach uses a Case-Based Reasoning system (CBR). CBR accuracy strongly depends on the quality of the extracted features. To achieve relevant classification, we propose a pipeline that includes image enhancement and data augmentation to improve the quality of extracted features and provide a final diagnosis. An efficient segmentation method based on a U-Net architecture is used to extract Regions of interest (RoI) from mammograms. The purpose is to combine deep learning (DL) with CBR to improve classification accuracy. DL provides accurate mammogram segmentation, while CBR gives an explainable and accurate classification. The proposed approach was tested on the CBIS-DDSM dataset and achieved high performance with an accuracy (Acc) of 86.71 % and a recall of 91.34 %, outperforming some well-known machine learning (ML) and DL approaches.

Explainability and white box in drug discovery

Recently, artificial intelligence (AI) techniques have been increasingly used to overcome the challenges in drug discovery. Although traditional AI techniques generally have high accuracy rates, there may be difficulties in explaining the decision process and patterns. This can create difficulties in understanding and making sense of the outputs of algorithms used in drug discovery. Therefore, using explainable AI (XAI) techniques, the causes and consequences of the decision process are better understood. This can help further improve the drug discovery process and make the right decisions. To address this issue, Explainable Artificial Intelligence (XAI) emerged as a process and method that securely captures the results and outputs of machine learning (ML) and deep learning (DL) algorithms. Using techniques such as SHAP (SHApley Additive ExPlanations) and LIME (Locally Interpretable Model-Independent Explanations) has made the drug targeting phase clearer and more understandable. XAI methods are expected to reduce time and cost in future computational drug discovery studies. This review provides a comprehensive overview of XAI-based drug discovery and development prediction. XAI mechanisms to increase confidence in AI and modeling methods. The limitations and future directions of XAI in drug discovery are also discussed.

The role of explainable Artificial Intelligence in high-stakes decision-making systems: a systematic review

A high-stakes event is an extreme risk with a low probability of occurring, but severe consequences (e.g., life-threatening conditions or economic collapse). The accompanying lack of information is a source of high-stress pressure and anxiety for emergency medical services authorities. Deciding on the best proactive plan and action in this environment is a complicated process, which calls for intelligent agents to automatically produce knowledge in the manner of human-like intelligence. Research in high-stakes decision-making systems has increasingly focused on eXplainable Artificial Intelligence (XAI), but recent developments in prediction systems give little prominence to explanations based on human-like intelligence. This work investigates XAI based on cause-and-effect interpretations for supporting high-stakes decisions. We review recent applications in the first aid and medical emergency fields based on three perspectives: available data, desirable knowledge, and the use of intelligence. We identify the limitations of recent AI, and discuss the potential of XAI for dealing with such limitations. We propose an architecture for high-stakes decision-making driven by XAI, and highlight likely future trends and directions.

Machine learning algorithms distinguish discrete digital emotional fingerprints for web pages related to back pain

Back pain is the leading cause of disability worldwide. Its emergence relates not only to the musculoskeletal degeneration biological substrate but also to psychosocial factors; emotional components play a pivotal role. In modern society, people are significantly informed by the Internet; in turn, they contribute social validation to a "successful" digital information subset in a dynamic interplay. The Affective component of medical pages has not been previously investigated, a significant gap in knowledge since they represent a critical biopsychosocial feature. We tested the hypothesis that successful pages related to spine pathology embed a consistent emotional pattern, allowing discrimination from a control group. The pool of web pages related to spine or hip/knee pathology was automatically selected by relevance and popularity and submitted to automated sentiment analysis to generate emotional patterns. Machine Learning (ML) algorithms were trained to predict page original topics from patterns with binary classification. ML showed high discrimination accuracy; disgust emerged as a discriminating emotion. The findings suggest that the digital affective "successful content" (collective consciousness) integrates patients' biopsychosocial ecosystem, with potential implications for the emergence of chronic pain, and the endorsement of health-relevant specific behaviors. Awareness of such effects raises practical and ethical issues for health information providers.

A review of the application of artificial intelligence to nuclear reactors: Where we are and what's next

As a form of clean energy, nuclear energy has unique advantages compared to other energy sources in the present era, where low-carbon policies are being widely advocated. The exponential growth of artificial intelligence (AI) technology in recent decades has resulted in new opportunities and challenges in terms of improving the safety and economics of nuclear reactors. This study briefly introduces modern AI algorithms such as machine learning, deep learning, and evolutionary computing. Furthermore, several studies on the use of AI techniques for nuclear reactor design optimization as well as operation and maintenance (O&M) are reviewed and discussed. The existing obstacles that prevent the further fusion of AI and nuclear reactor technologies so that they can be scaled to real-world problems are classified into two categories: (1) data issues: insufficient experimental data increases the possibility of data distribution drift and data imbalance; (2) black-box dilemma: methods such as deep learning have poor interpretability. Finally, this study proposes two directions for the future fusion of AI and nuclear reactor technologies: (1) better integration of domain knowledge with data-driven approaches to reduce the high demand for data and improve the model performance and robustness; (2) promoting the use of explainable artificial intelligence (XAI) technologies to enhance the transparency and reliability of the model. In addition, causal learning warrants further attention owing to its inherent ability to solve out-of-distribution generalization (OODG) problems.

Explainable AI for Estimating Pathogenicity of Genetic Variants Using Large-Scale Knowledge Graphs

BACKGROUND

To treat diseases caused by genetic variants, it is necessary to identify disease-causing variants in patients. However, since there are a large number of disease-causing variants, the application of AI is required. We propose AI to solve this problem and report the results of its application in identifying disease-causing variants.

METHODS

To assist physicians in their task of identifying disease-causing variants, we propose an explainable AI (XAI) that combines high estimation accuracy with explainability using a knowledge graph. We integrated databases for genomic medicine and constructed a large knowledge graph that was used to achieve the XAI.

RESULTS

We compared our XAI with random forests and decision trees.

CONCLUSION

We propose an XAI that uses knowledge graphs for explanation. The proposed method achieves high estimation performance and explainability. This will support the promotion of genomic medicine.

Shapley values for cluster importance

This paper proposes a novel approach to explain the predictions made by data-driven methods. Since such predictions rely heavily on the data used for training, explanations that convey information about how the training data affects the predictions are useful. The paper proposes a novel approach to quantify how different data-clusters of the training data affect a prediction. The quantification is based on Shapley values, a concept which originates from coalitional game theory, developed to fairly distribute the payout among a set of cooperating players. A player’s Shapley value is a measure of that player’s contribution. Shapley values are often used to quantify feature importance, ie. how features affect a prediction. This paper extends this to cluster importance, letting clusters of the training data act as players in a game where the predictions are the payouts. The novel methodology proposed in this paper lets us explore and investigate how different clusters of the training data affect the predictions made by any black-box model, allowing new aspects of the reasoning and inner workings of a prediction model to be conveyed to the users. The methodology is fundamentally different from existing explanation methods, providing insight which would not be available otherwise, and should complement existing explanation methods, including explanations based on feature importance.

Modeling the Effect of Streetscape Environment on Crime Using Street View Images and Interpretable Machine-Learning Technique

Street crime is a common social problem that threatens the security of people's lives and property. Understanding the influencing mechanisms of street crime is an essential precondition for formulating crime prevention strategies. Widespread concern has contributed to the development of streetscape environment features as they can significantly affect the occurrence of street crime. Emerging street view images are a low-cost and highly accessible data source. On the other hand, machine-learning models such as XGBoost (eXtreme Gradient Boosting) usually have higher fitting accuracies than those of linear regression models. Therefore, they are popular for modeling the relationships between crime and related impact factors. However, due to the "black box" characteristic, researchers are unable to understand how each variable contributes to the occurrence of crime. Existing research mainly focuses on the independent impacts of streetscape environment features on street crime, but not on the interaction effects between these features and the community socioeconomic conditions and their local variations. In order to address the above limitations, this study first combines street view images, an objective detection network, and a semantic segmentation network to extract a systematic measurement of the streetscape environment. Then, controlling for socioeconomic factors, we adopted the XGBoost model to fit the relationships between streetscape environment features and street crime at the street segment level. Moreover, we used the SHAP (Shapley additive explanation) framework, a post-hoc machine-learning explainer, to explain the results of the XGBoost model. The results demonstrate that, from a global perspective, the number of people on the street, extracted from street view images, has the most significant impact on street property crime among all the street view variables. The local interpretability of the SHAP explainer demonstrates that a particular variable has different effects on street crime at different street segments. The nonlinear associations between streetscape environment features and street crime, as well as the interaction effects of different streetscape environment features are discussed. The positive effect of the number of pedestrians on street crime increases with the length of the street segment and the number of crime generators. The combination of street view images and interpretable machine-learning techniques is helpful in better accurately understanding the complex relationships between the streetscape environment and street crime. Furthermore, the readily comprehensible results can offer a reference for formulating crime prevention strategies.

Weight interpretation of artificial neural network model for analysis of rice (Oryza sativa L.) with near-infrared spectroscopy

•

ANN model was build based on NIR spectra and nutrient values of 110 rice samples.

•

Good correlation between ANN predicted and experimental nutrient values observed.

•

Scientific interpretation of weights agreed well with previously reported results.

•

Interpretation of weights was also in good agreement with conventional PLS analysis.

Prediction models for major nutrients of rice were built using near-infrared (NIR) spectral data based on the artificial neural network (ANN). Scientific interpretation of the weight values was proposed and performed to understand the wavenumbers contributing to the prediction of nutrients. NIR spectra were acquired from 110 rice samples. Carbohydrate and moisture contents were predicted with values for the determination coefficient, relative root mean square error, range error ratio, and residual prediction deviation of 0.98, 0.11 %, 44, and 7.3, and 0.97, 0.80 %, 27, and 5.8, respectively. The results agreed well with ones reported in the previous studies and acquired by the conventional partial least squares (PLS)-variable importance in projection method. This study demonstrates that the combination of NIR and ANN is a powerful and accurate tool to monitor nutrients of rice and scientific interpretation of weights can be performed to overcome black box nature of the ANN.

Design of Semiautomatic Digital Creation System for Electronic Music Based on Recurrent Neural Network

Semiautomated digital creation is increasingly important in the manipulation of electronic music. How to realize the learning of local effective features of audio data is a difficult point in the current research field. Based on recurrent neural network theory, this paper designs a semiautomatic digital creation system for electronic music for digital manipulation and genre classification. The recurrent neural network improves the transmission of electronic music information between the input and output of the network by adopting dense connections consistent with DenseNet and adopts an inception-like structure for the autonomous selection of effective recursive nuclear electronic music categories. In the simulation process, the prediction method based on semiautomatic digital audio clips is also adopted, which pays more attention to the learning of local effective features of audio data, which gives the model the ability to create audio samples of different lengths and improves the model's support for creative tasks in different scenarios. It includes the determination of the number of neurons, the selection of the function of neurons, the determination of the connection method, and the specific learning algorithm rules, and then the training samples are formed. The experimental results show that the recurrent neural network exhibits powerful feature extraction ability and classification ability of music information. The 10-fold cross-validation on GTZAN dataset and ISMIR2004 dataset has obtained 88.7% and 87.68%, surpassing similar ones. The model has reached a leading level. After further use of the MSD (Million Song Dataset) dataset for pre-semiautomatic training, the model effect has been further greatly improved. The accuracy rate on the dataset has been increased to 91.0% and 89.91%, respectively, which has improved the semiautomatic number and creative advancement.

Quality Models for Artificial Intelligence Systems: Characteristic-Based Approach, Development and Application

The factors complicating the specification of requirements for artificial intelligence systems (AIS) and their verification for the AIS creation and modernization are analyzed. The harmonization of definitions and building of a hierarchy of AIS characteristics for regulation of the development of techniques and tools for standardization, as well as evaluation and provision of requirements during the creation and implementation of AIS, is extremely important. The study aims to develop and demonstrate the use of quality models for artificial intelligence (AI), AI platform (AIP), and AIS based on the definition and ordering of characteristics. The principles of AI quality model development and its sequence are substantiated. Approaches to formulating definitions of AIS characteristics, methods of representation of dependencies, and hierarchies of characteristics are given. The definitions and harmonization options of hierarchical relations between 46 characteristics of AI and AIP are suggested. The quality models of AI, AIP, and AIS presented in analytical, tabular, and graph forms, are described. The so-called basic models with reduced sets of the most important characteristics are presented. Examples of AIS quality models for UAV video navigation systems and decision support systems for diagnosing diseases are described.