Explainable AI for clinical and remote health applications: a survey on tabular and time series data

A review of evaluation approaches for explainable AI with applications in cardiology

Explainable artificial intelligence (XAI) elucidates the decision-making process of complex AI models and is important in building trust in model predictions. XAI explanations themselves require evaluation as to accuracy and reasonableness and in the context of use of the underlying AI model. This review details the evaluation of XAI in cardiac AI applications and has found that, of the studies examined, 37% evaluated XAI quality using literature results, 11% used clinicians as domain-experts, 11% used proxies or statistical analysis, with the remaining 43% not assessing the XAI used at all. We aim to inspire additional studies within healthcare, urging researchers not only to apply XAI methods but to systematically assess the resulting explanations, as a step towards developing trustworthy and safe models.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10462-024-10852-w.

Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging

Explainable artificial intelligence (XAI) has experienced a vast increase in recognition over the last few years. While the technical developments are manifold, less focus has been placed on the clinical applicability and usability of systems. Moreover, not much attention has been given to XAI systems that can handle multimodal and longitudinal data, which we postulate are important features in many clinical workflows. In this study, we review, from a clinical perspective, the current state of XAI for multimodal and longitudinal datasets and highlight the challenges thereof. Additionally, we propose the XAI orchestrator, an instance that aims to help clinicians with the synopsis of multimodal and longitudinal data, the resulting AI predictions, and the corresponding explainability output. We propose several desirable properties of the XAI orchestrator, such as being adaptive, hierarchical, interactive, and uncertainty-aware.

Ethical Considerations in the Use of Artificial Intelligence and Machine Learning in Health Care: A Comprehensive Review

Artificial intelligence (AI) and machine learning (ML) technologies are revolutionizing health care by offering unprecedented opportunities to enhance patient care, optimize clinical workflows, and advance medical research. However, the integration of AI and ML into healthcare systems raises significant ethical considerations that must be carefully addressed to ensure responsible and equitable deployment. This comprehensive review explored the multifaceted ethical considerations surrounding the use of AI and ML in health care, including privacy and data security, algorithmic bias, transparency, clinical validation, and professional responsibility. By critically examining these ethical dimensions, stakeholders can navigate the ethical complexities of AI and ML integration in health care, while safeguarding patient welfare and upholding ethical principles. By embracing ethical best practices and fostering collaboration across interdisciplinary teams, the healthcare community can harness the full potential of AI and ML technologies to usher in a new era of personalized data-driven health care that prioritizes patient well-being and equity.

Explainable machine learning predicts survival of retroperitoneal liposarcoma: A study based on the SEER database and external validation in China

OBJECTIVE

We have developed explainable machine learning models to predict the overall survival (OS) of retroperitoneal liposarcoma (RLPS) patients. This approach aims to enhance the explainability and transparency of our modeling results.

METHODS

We collected clinicopathological information of RLPS patients from The Surveillance, Epidemiology, and End Results (SEER) database and allocated them into training and validation sets with a 7:3 ratio. Simultaneously, we obtained an external validation cohort from The First Affiliated Hospital of Naval Medical University (Shanghai, China). We performed LASSO regression and multivariate Cox proportional hazards analysis to identify relevant risk factors, which were then combined to develop six machine learning (ML) models: Cox proportional hazards model (Coxph), random survival forest (RSF), ranger, gradient boosting with component-wise linear models (GBM), decision trees, and boosting trees. The predictive performance of these ML models was evaluated using the concordance index (C-index), the integrated cumulative/dynamic area under the curve (AUC), and the integrated Brier score, as well as the Cox-Snell residual plot. We also used time-dependent variable importance, analysis of partial dependence survival plots, and the generation of aggregated survival SHapley Additive exPlanations (SurvSHAP) plots to provide a global explanation of the optimal model. Additionally, SurvSHAP (t) and survival local interpretable model-agnostic explanations (SurvLIME) plots were used to provide a local explanation of the optimal model.

RESULTS

The final ML models are consisted of six factors: patient's age, gender, marital status, surgical history, as well as tumor's histopathological classification, histological grade, and SEER stage. Our prognostic model exhibits significant discriminative ability, particularly with the ranger model performing optimally. In the training set, validation set, and external validation set, the AUC for 1, 3, and 5 year OS are all above 0.83, and the integrated Brier scores are consistently below 0.15. The explainability analysis of the ranger model also indicates that histological grade, histopathological classification, and age are the most influential factors in predicting OS.

CONCLUSIONS

The ranger ML prognostic model exhibits optimal performance and can be utilized to predict the OS of RLPS patients, offering valuable and crucial references for clinical physicians to make informed decisions in advance.

An interpretable data-driven prediction model to anticipate scoliosis in spinal muscular atrophy in the era of (gene-) therapies

5q-spinal muscular atrophy (SMA) is a neuromuscular disorder (NMD) that has become one of the first 5% treatable rare diseases. The efficacy of new SMA therapies is creating a dynamic SMA patient landscape, where disease progression and scoliosis development play a central role, however, remain difficult to anticipate. New approaches to anticipate disease progression and associated sequelae will be needed to continuously provide these patients the best standard of care. Here we developed an interpretable machine learning (ML) model that can function as an assistive tool in the anticipation of SMA-associated scoliosis based on disease progression markers. We collected longitudinal data from 86 genetically confirmed SMA patients. We selected six features routinely assessed over time to train a random forest classifier. The model achieved a mean accuracy of 0.77 (SD 0.2) and an average ROC AUC of 0.85 (SD 0.17). For class 1 'scoliosis' the average precision was 0.84 (SD 0.11), recall 0.89 (SD 0.22), F1-score of 0.85 (SD 0.17), respectively. Our trained model could predict scoliosis using selected disease progression markers and was consistent with the radiological measurements. During post validation, the model could predict scoliosis in patients who were unseen during training. We also demonstrate that rare disease data sets can be wrangled to build predictive ML models. Interpretable ML models can function as assistive tools in a changing disease landscape and have the potential to democratize expertise that is otherwise clustered at specialized centers.

Evaluating distributed-learning on real-world obstetrics data: comparing distributed, centralized and local models

This study focused on comparing distributed learning models with centralized and local models, assessing their efficacy in predicting specific delivery and patient-related outcomes in obstetrics using real-world data. The predictions focus on key moments in the obstetric care process, including discharge and various stages of hospitalization. Our analysis: using 6 different machine learning methods like Decision Trees, Bayesian methods, Stochastic Gradient Descent, K-nearest neighbors, AdaBoost, and Multi-layer Perceptron and 19 different variables with various distributions and types, revealed that distributed models were at least equal, and often superior, to centralized versions and local versions. We also describe thoroughly the preprocessing stage in order to help others implement this method in real-world scenarios. The preprocessing steps included cleaning and harmonizing missing values, handling missing data and encoding categorical variables with multisite logic. Even though the type of machine learning model and the distribution of the outcome variable can impact the result, we reached results of 66% being superior to the centralized and local counterpart and 77% being better than the centralized with AdaBoost. Our experiments also shed light in the preprocessing steps required to implement distributed models in a real-world scenario. Our results advocate for distributed learning as a promising tool for applying machine learning in clinical settings, particularly when privacy and data security are paramount, thus offering a robust solution for privacy-concerned clinical applications.

Applying explainable artificial intelligence methods to models for diagnosing personal traits and cognitive abilities by social network data

This study utilizes advanced artificial intelligence techniques to analyze the social media behavior of 1358 users on VK, the largest Russian online social networking service. The analysis comprises 753,252 posts and reposts, combined with Big Five personality traits test results, as well as assessments of verbal and fluid intelligence. The objective of this research is to understand the manifestation of psychological attributes in social media users' behavior and determine their implications on user-interaction models. We employ the integrated gradients method to identify the most influential feature groups. The partial dependence plot technique aids in understanding how these features function across varying severity degrees of the predicted trait. To evaluate feature stability within the models, we cluster calculated Shapley values. Our findings suggest that the emotional tone (joy, surprise, anger, fear) of posts significantly influences the prediction of three personality traits: Extraversion, Agreeableness, and Openness to Experience. Additionally, user social engagement metrics (such as friend count, subscribers, likes, views, and comments) correlate directly with the predicted level of Logical thinking. We also observe a trend towards provocative and socially reprehensible content among users with high Neuroticism levels. The theme of religion demonstrates a multidirectional relationship with Consciousness and Agreeableness. Further findings, including an analysis of post frequency and key text characteristics, are also discussed, contributing to our understanding of the complex interplay between social media behavior and psychological traits. The study proposes a transition from the analysis of correlations between psychological (cognitive) traits to the analysis of indicators of behavior in a social network that are significant for diagnostic models of the corresponding traits.

Explaining decisions of a light-weight deep neural network for real-time coronary artery disease classification in magnetic resonance imaging

In certain healthcare settings, such as emergency or critical care units, where quick and accurate real-time analysis and decision-making are required, the healthcare system can leverage the power of artificial intelligence (AI) models to support decision-making and prevent complications. This paper investigates the optimization of healthcare AI models based on time complexity, hyper-parameter tuning, and XAI for a classification task. The paper highlights the significance of a lightweight convolutional neural network (CNN) for analysing and classifying Magnetic Resonance Imaging (MRI) in real-time and is compared with CNN-RandomForest (CNN-RF). The role of hyper-parameter is also examined in finding optimal configurations that enhance the model's performance while efficiently utilizing the limited computational resources. Finally, the benefits of incorporating the XAI technique (e.g. GradCAM and Layer-wise Relevance Propagation) in providing transparency and interpretable explanations of AI model predictions, fostering trust, and error/bias detection are explored. Our inference time on a MacBook laptop for 323 test images of size 100x100 is only 2.6 sec, which is merely 8 milliseconds per image while providing comparable classification accuracy with the ensemble model of CNN-RF classifiers. Using the proposed model, clinicians/cardiologists can achieve accurate and reliable results while ensuring patients' safety and answering questions imposed by the General Data Protection Regulation (GDPR). The proposed investigative study will advance the understanding and acceptance of AI systems in connected healthcare settings.

[Use of AI-based applications by hospital staff: task profiles and qualification requirements]

BACKGROUND

Artificial intelligence (AI) is becoming increasingly important for the future development of hospitals. To unlock the large potential of AI, job profiles of hospital staff members need to be further developed in the direction of AI and digitization skills through targeted qualification measures. This affects both medical and non-medical processes along the entire value chain in hospitals. The aim of this paper is to provide an overview of the skills required to deal with smart technologies in a clinical context and to present measures for training employees.

METHODS

As part of the "SmartHospital.NRW" project in 2022, we conducted a literature review as well as interviews and workshops with experts. AI technologies and fields of application were identified.

RESULTS

Key findings include adapted and new task profiles, synergies and dependencies between individual task profiles, and the need for a comprehensive interdisciplinary and interprofessional exchange when using AI-based applications in hospitals.

DISCUSSION

Our article shows that hospitals need to promote digital health literacy skills for hospital staff members at an early stage and at the same time recruit technology- and AI-savvy staff. Interprofessional exchange formats and accompanying change management are essential for the use of AI in hospitals.

AI-Assisted Fatigue and Stamina Control for Performance Sports on IMU-Generated Multivariate Times Series Datasets

BACKGROUND

Optimal sports performance requires a balance between intensive training and adequate rest. IMUs provide objective, quantifiable data to analyze performance dynamics, despite the challenges in quantifying athlete training loads. The ability of AI to analyze complex datasets brings innovation to the monitoring and optimization of athlete training cycles. Traditional techniques rely on subjective assessments to prevent overtraining, which can lead to injury and underperformance. IMUs provide objective, quantitative data on athletes' physical status during action. AI and machine learning can turn these data into useful insights, enabling data-driven athlete performance management. With IMU-generated multivariate time series data, this paper uses AI to construct a robust model for predicting fatigue and stamina.

MATERIALS AND METHODS

IMUs linked to 19 athletes recorded triaxial acceleration, angular velocity, and magnetic orientation throughout repeated sessions. Standardized training included steady-pace runs and fatigue-inducing techniques. The raw time series data were used to train a supervised ML model based on frequency and time-domain characteristics. The performances of Random Forest, Gradient Boosting Machines, and LSTM networks were compared. A feedback loop adjusted the model in real time based on prediction error and bias estimation.

RESULTS

The AI model demonstrated high predictive accuracy for fatigue, showing significant correlations between predicted fatigue levels and observed declines in performance. Stamina predictions enabled individualized training adjustments that were in sync with athletes' physiological thresholds. Bias correction mechanisms proved effective in minimizing systematic prediction errors. Moreover, real-time adaptations of the model led to enhanced training periodization strategies, reducing the risk of overtraining and improving overall athletic performance.

CONCLUSIONS

In sports performance analytics, the AI-assisted model using IMU multivariate time series data is effective. Training can be tailored and constantly altered because the model accurately predicts fatigue and stamina. AI models can effectively forecast the beginning of weariness before any physical symptoms appear. This allows for timely interventions to prevent overtraining and potential accidents. The model shows an exceptional ability to customize training programs according to the physiological reactions of each athlete and enhance the overall training effectiveness. In addition, the study demonstrated the model's efficacy in real-time monitoring performance, improving the decision-making abilities of both coaches and athletes. The approach enables ongoing and thorough data analysis, supporting strategic planning for training and competition, resulting in optimized performance outcomes. These findings highlight the revolutionary capability of AI in sports science, offering a future where data-driven methods greatly enhance athlete training and performance management.