Cross-property deep transfer learning framework for enhanced predictive analytics on small materials data

XElemNet: towards explainable AI for deep neural networks in materials science

Recent progress in deep learning has significantly impacted materials science, leading to accelerated material discovery and innovation. ElemNet, a deep neural network model that predicts formation energy from elemental compositions, exemplifies the application of deep learning techniques in this field. However, the "black-box" nature of deep learning models often raises concerns about their interpretability and reliability. In this study, we propose XElemNet to explore the interpretability of ElemNet by applying a series of explainable artificial intelligence (XAI) techniques, focusing on post-hoc analysis and model transparency. The experiments with artificial binary datasets reveal ElemNet's effectiveness in predicting convex hulls of element-pair systems across periodic table groups, indicating its capability to effectively discern elemental interactions in most cases. Additionally, feature importance analysis within ElemNet highlights alignment with chemical properties of elements such as reactivity and electronegativity. XElemNet provides insights into the strengths and limitations of ElemNet and offers a potential pathway for explaining other deep learning models in materials science.

Machine Learning-Enabled Image Classification for Automated Electron Microscopy

Traditionally, materials discovery has been driven more by evidence and intuition than by systematic design. However, the advent of "big data" and an exponential increase in computational power have reshaped the landscape. Today, we use simulations, artificial intelligence (AI), and machine learning (ML) to predict materials characteristics, which dramatically accelerates the discovery of novel materials. For instance, combinatorial megalibraries, where millions of distinct nanoparticles are created on a single chip, have spurred the need for automated characterization tools. This paper presents an ML model specifically developed to perform real-time binary classification of grayscale high-angle annular dark-field images of nanoparticles sourced from these megalibraries. Given the high costs associated with downstream processing errors, a primary requirement for our model was to minimize false positives while maintaining efficacy on unseen images. We elaborate on the computational challenges and our solutions, including managing memory constraints, optimizing training time, and utilizing Neural Architecture Search tools. The final model outperformed our expectations, achieving over 95% precision and a weighted F-score of more than 90% on our test data set. This paper discusses the development, challenges, and successful outcomes of this significant advancement in the application of AI and ML to materials discovery.

From bulk effective mass to 2D carrier mobility accurate prediction via adversarial transfer learning

Data scarcity is one of the critical bottlenecks to utilizing machine learning in material discovery. Transfer learning can use existing big data to assist property prediction on small data sets, but the premise is that there must be a strong correlation between large and small data sets. To extend its applicability in scenarios with different properties and materials, here we develop a hybrid framework combining adversarial transfer learning and expert knowledge, which enables the direct prediction of carrier mobility of two-dimensional (2D) materials using the knowledge learned from bulk effective mass. Specifically, adversarial training ensures that only common knowledge between bulk and 2D materials is extracted while expert knowledge is incorporated to further improve the prediction accuracy and generalizability. Successfully, 2D carrier mobilities are predicted with the accuracy over 90% from only crystal structure, and 21 2D semiconductors with carrier mobilities far exceeding silicon and suitable bandgap are successfully screened out. This work enables transfer learning in simultaneous cross-property and cross-material scenarios, providing an effective tool to predict intricate material properties with limited data.

Machine Learning to Promote Efficient Screening of Low-Contact Electrode for 2D Semiconductor Transistor Under Limited Data

Low-barrier and high-injection electrodes are crucial for high-performance (HP) 2D semiconductor devices. Conventional trial-and-error methodologies for electrode material screening are impractical because of their low efficiency and arbitrary specificity. Although machine learning has emerged as a promising alternative to tackle this problem, its practical application in semiconductor devices is hindered by its substantial data requirements. In this paper, a comprehensive scheme combining an autoencoding regularized adversarial neural network and a feature-adaptive variational active learning algorithm for screening low-contact electrode materials for 2D semiconductor transistors with limited data is proposed. The proposed scheme exhibits exceptional performance by training with only 15% of the total data points, where the mean square errors are 0.17 and 0.27 eV for the vertical and lateral Schottky barrier, respectively, and 2.88% for tunneling probability. Further, it exhibits an optimal predictive performance for 100 randomly sampled training datasets, reveals the underlying physical insight based on the identified features, and realizes continual improvement by employing detailed density-of-states descriptors. Finally, the empirical evaluations of the transport characteristics are conducted and verified by constructing MOSFET devices. These findings demonstrate the considerable potential of machine-learning techniques for screening high-efficiency electrode materials and constructing HP 2D semiconductor devices.

Coarse-Grained Crystal Graph Neural Networks for Reticular Materials Design

Reticular materials, including metal-organic frameworks and covalent organic frameworks, combine the relative ease of synthesis and an impressive range of applications in various fields from gas storage to biomedicine. Diverse properties arise from the variation of building units─metal centers and organic linkers─in almost infinite chemical space. Such variation substantially complicates the experimental design and promotes the use of computational methods. In particular, the most successful artificial intelligence algorithms for predicting the properties of reticular materials are atomic-level graph neural networks, which optionally incorporate domain knowledge. Nonetheless, the data-driven inverse design involving these models suffers from the incorporation of irrelevant and redundant features such as a full atomistic graph and network topology. In this study, we propose a new way of representing materials, aiming to overcome the limitations of existing methods; the message passing is performed on a coarse-grained crystal graph that comprises molecular building units. To highlight the merits of our approach, we assessed the predictive performance and energy efficiency of neural networks built on different materials representations, including composition-based and crystal-structure-aware models. Coarse-grained crystal graph neural networks showed decent accuracy at low computational costs, making them a valuable alternative to omnipresent atomic-level algorithms. Moreover, the presented models can be successfully integrated into an inverse materials design pipeline as estimators of the objective function. Overall, the coarse-grained crystal graph framework is aimed at challenging the prevailing atom-centric perspective on reticular materials design.

A Machine Learning Study on High Thermal Conductivity Assisted to Discover Chalcogenides with Balanced Infrared Nonlinear Optical Performance

Exploration of novel nonlinear optical (NLO) chalcogenides with high laser-induced damage thresholds (LIDT) is critical for mid-infrared (mid-IR) solid-state laser applications. High lattice thermal conductivity (κL ) is crucial to increasing LIDT yet often neglected in the search for NLO crystals due to lack of accurate κL data. A machine learning (ML) approach to predict κL for over 6000 chalcogenides is hereby proposed. Combining ML-generated κL data and first-principles calculation, a high-throughput screening route is initiated, and ten new potential mid-IR NLO chalcogenides with optimal bandgap, NLO coefficients, and thermal conductivity are discovered, in which Li2 SiS3 and AlZnGaS4 are highlighted. Big-data analysis on structural chemistry proves that the chalcogenides having dense and simple lattice structures with low anisotropy, light atoms, and strong covalent bonds are likely to possess higher κL . The four-coordinated motifs in which central cations show the bond valence sum of +2 to +3 and are from IIIA, IVA, VA, and IIB groups, such as those in diamond-like defect-chalcopyrite chalcogenides, are preferred to fulfill the desired structural chemistry conditions for balanced NLO and thermal properties. This work provides not only an efficient strategy but also interpretable research directions in the search for NLO crystals with high thermal conductivity.

MatGPT: A Vane of Materials Informatics from Past, Present, to Future

Combining materials science, artificial intelligence (AI), physical chemistry, and other disciplines, materials informatics is continuously accelerating the vigorous development of new materials. The emergence of "GPT (Generative Pre-trained Transformer) AI" shows that the scientific research field has entered the era of intelligent civilization with "data" as the basic factor and "algorithm + computing power" as the core productivity. The continuous innovation of AI will impact the cognitive laws and scientific methods, and reconstruct the knowledge and wisdom system. This leads to think more about materials informatics. Here, a comprehensive discussion of AI models and materials infrastructures is provided, and the advances in the discovery and design of new materials are reviewed. With the rise of new research paradigms triggered by "AI for Science", the vane of materials informatics: "MatGPT", is proposed and the technical path planning from the aspects of data, descriptors, generative models, pretraining models, directed design models, collaborative training, experimental robots, as well as the efforts and preparations needed to develop a new generation of materials informatics, is carried out. Finally, the challenges and constraints faced by materials informatics are discussed, in order to achieve a more digital, intelligent, and automated construction of materials informatics with the joint efforts of more interdisciplinary scientists.

A Multiple-Fidelity Method for Accurate Simulation of MoS2 Properties Using JAX-ReaxFF and Neural Network Potentials

Reactive force field (ReaxFF) is a commonly used force field for modeling chemical reactions at the atomic level. Recently, JAX-ReaxFF, combined with automatic differentiation, has been used to efficiently parametrize ReaxFF. However, its analytical formula may lead to inaccurate predictions. While neural network-based potentials (NNPs) trained on density functional theory-labeled data offer a more accurate method, it requires a large amount of training data to be trained from scratch. To overcome these issues, we present a multiple-fidelity method that combines JAX-ReaxFF and NNP and apply the method on MoS2, a promising two-dimensional semiconductor for flexible electronics. By incorporating implicit prior physical information, ReaxFF can serve as a cost-effective way to generate pretraining data, facilitating more accurate simulations of MoS2. Moreover, in the Mo-S-H system, the pretraining strategy can reduce root-mean-square errors of energy by 20%. This approach can be extended to a wide variety of material systems, accelerating their computational research.

Examining graph neural networks for crystal structures: Limitations and opportunities for capturing periodicity

Graph neural networks (GNNs) have recently been used to learn the representations of crystal structures through an end-to-end data-driven approach. However, a systematic top-down approach to evaluate and understand the limitations of GNNs in accurately capturing crystal structures has yet to be established. In this study, we introduce an approach using human-designed descriptors as a compendium of human knowledge to investigate the extent to which GNNs can comprehend crystal structures. Our findings reveal that current state-of-the-art GNNs fall short in accurately capturing the periodicity of crystal structures. We analyze this failure by exploring three aspects: local expressive power, long-range information processing, and readout function. To address these identified limitations, we propose a straightforward and general solution: the hybridization of descriptors with GNNs, which directly supplements the missing information to GNNs. The hybridization enhances the predictive accuracy of GNNs for specific material properties, most notably phonon internal energy and heat capacity, which heavily rely on the periodicity of materials.

SEMPro: A Data-Driven Pipeline To Learn Structure-Property Insights from Scanning Electron Microscopy Images

Analyzing hydrogel microstructure through scanning electron microscopy (SEM) images is crucial in understanding hydrogel properties. However, the analysis of SEM images in hydrogel research heavily relies on the intuition of individual researchers and is constrained by the limited size of the dataset. To address this, we propose SEMPro, a data-driven solution using web-scraping and deep learning (DL) to compile and analyze the structure-property relationships of hydrogels through SEM images. It accurately predicts the elastic modulus from SEM images within the same order of magnitude and displays a learned extraction of modulus-relevant features in SEM images as seen through the nontrivial activation mapping and transfer learning. By employing Explainable AI through activation map exposure, SEMPro validates the model predictions. SEMPro represents a closed-loop data collection and analysis pipeline, providing critical insights into hydrogels and soft materials. This innovative approach has the potential to revolutionize hydrogel research, offering high-dimensional insights for further advancements.

Orchestrating information across tissues via a novel multitask GAT framework to improve quantitative gene regulation relation modeling for survival analysis

Survival analysis is critical to cancer prognosis estimation. High-throughput technologies facilitate the increase in the dimension of genic features, but the number of clinical samples in cohorts is relatively small due to various reasons, including difficulties in participant recruitment and high data-generation costs. Transcriptome is one of the most abundantly available OMIC (referring to the high-throughput data, including genomic, transcriptomic, proteomic and epigenomic) data types. This study introduced a multitask graph attention network (GAT) framework DQSurv for the survival analysis task. We first used a large dataset of healthy tissue samples to pretrain the GAT-based HealthModel for the quantitative measurement of the gene regulatory relations. The multitask survival analysis framework DQSurv used the idea of transfer learning to initiate the GAT model with the pretrained HealthModel and further fine-tuned this model using two tasks i.e. the main task of survival analysis and the auxiliary task of gene expression prediction. This refined GAT was denoted as DiseaseModel. We fused the original transcriptomic features with the difference vector between the latent features encoded by the HealthModel and DiseaseModel for the final task of survival analysis. The proposed DQSurv model stably outperformed the existing models for the survival analysis of 10 benchmark cancer types and an independent dataset. The ablation study also supported the necessity of the main modules. We released the codes and the pretrained HealthModel to facilitate the feature encodings and survival analysis of transcriptome-based future studies, especially on small datasets. The model and the code are available at http://www.healthinformaticslab.org/supp/.

Data quantity governance for machine learning in materials science

Data-driven machine learning (ML) is widely employed in the analysis of materials structure-activity relationships, performance optimization and materials design due to its superior ability to reveal latent data patterns and make accurate prediction. However, because of the laborious process of materials data acquisition, ML models encounter the issue of the mismatch between a high dimension of feature space and a small sample size (for traditional ML models) or the mismatch between model parameters and sample size (for deep-learning models), usually resulting in terrible performance. Here, we review the efforts for tackling this issue via feature reduction, sample augmentation and specific ML approaches, and show that the balance between the number of samples and features or model parameters should attract great attention during data quantity governance. Following this, we propose a synergistic data quantity governance flow with the incorporation of materials domain knowledge. After summarizing the approaches to incorporating materials domain knowledge into the process of ML, we provide examples of incorporating domain knowledge into governance schemes to demonstrate the advantages of the approach and applications. The work paves the way for obtaining the required high-quality data to accelerate materials design and discovery based on ML.

Improving deep learning model performance under parametric constraints for materials informatics applications

Modern machine learning (ML) and deep learning (DL) techniques using high-dimensional data representations have helped accelerate the materials discovery process by efficiently detecting hidden patterns in existing datasets and linking input representations to output properties for a better understanding of the scientific phenomenon. While a deep neural network comprised of fully connected layers has been widely used for materials property prediction, simply creating a deeper model with a large number of layers often faces with vanishing gradient problem, causing a degradation in the performance, thereby limiting usage. In this paper, we study and propose architectural principles to address the question of improving the performance of model training and inference under fixed parametric constraints. Here, we present a general deep-learning framework based on branched residual learning (BRNet) with fully connected layers that can work with any numerical vector-based representation as input to build accurate models to predict materials properties. We perform model training for materials properties using numerical vectors representing different composition-based attributes of the respective materials and compare the performance of the proposed models against traditional ML and existing DL architectures. We find that the proposed models are significantly more accurate than the ML/DL models for all data sizes by using different composition-based attributes as input. Further, branched learning requires fewer parameters and results in faster model training due to better convergence during the training phase than existing neural networks, thereby efficiently building accurate models for predicting materials properties.

Toward the Golden Age of Materials Informatics: Perspective and Opportunities

Materials informatics is reaching the transition point and is evolving from its early stages of adoption and development and moving toward its golden age. Here, the transformation of the early stage of materials informatics toward the next level of materials informatics is explored. In particular, it has become crucial to be able to manipulate materials synthesis data, materials properties data, and materials characterization data. Through the use of ontology, material design and understanding can be carried out simultaneously in a whitebox manner. Here, a perspective on the ultimate goal of materials informatics along with potential key components is discussed.

MPpredictor: An Artificial Intelligence-Driven Web Tool for Composition-Based Material Property Prediction

The applications of artificial intelligence, machine learning, and deep learning techniques in the field of materials science are becoming increasingly common due to their promising abilities to extract and utilize data-driven information from available data and accelerate materials discovery and design for future applications. In an attempt to assist with this process, we deploy predictive models for multiple material properties, given the composition of the material. The deep learning models described here are built using a cross-property deep transfer learning technique, which leverages source models trained on large data sets to build target models on small data sets with different properties. We deploy these models in an online software tool that takes a number of material compositions as input, performs preprocessing to generate composition-based attributes for each material, and feeds them into the predictive models to obtain up to 41 different material property values. The material property predictor is available online at http://ai.eecs.northwestern.edu/MPpredictor.

Which molecular properties determine the impact sensitivity of an explosive? A machine learning quantitative investigation of nitroaromatic explosives

We decomposed density functional theory charge densities of 53 nitroaromatic molecules into atom-centered electric multipoles using the distributed multipole analysis that provides a detailed picture of the molecular electronic structure. Three electric multipoles, (the charge of the nitro groups), (the total dipole, i.e., polarization, of the nitro groups), (the total electron delocalization of the C ring atoms), and the number of explosophore groups (#NO₂) were selected as features for a comprehensive machine learning (ML) investigation. The target property was the impact sensitivity h₅₀ (cm) values quantified by drop-weight measurements, with a large h₅₀ (e.g., 150 cm) indicating that an explosive is insensitive and vice versa. After a preliminary screening of 42 ML algorithms, four were selected based on the lowest root mean square errors: Extra Trees, Random Forests, Gradient Boosting, and AdaBoost. Compared to experimental data, the predicted h₅₀ values of molecules having very different sensitivities for the four algorithms have differences in the range 19-28%. The most important properties for predicting h₅₀ are the electron delocalization in the ring atoms and the polarization of the nitro groups with averaged weights of 39% and 35%, followed by the charge (16%) and number (10%) of nitro groups. A significant result is how the contribution of these properties to h₅₀ depends on their actual sensitivities: for the most sensitive explosives (h₅₀ up to ∼50 cm), the four properties contribute to reducing h₅₀, and for intermediate ones (∼50 cm ≲ h₅₀ ≲ 100 cm) #NO₂ and contribute to increasing it and the other two properties to reducing it. For highly insensitive explosives (h₅₀ ≳ 200 cm), all four properties essentially contribute to increasing it. These results furnish a consistent molecular basis of the sensitivities of known explosives that also can be used for developing safer new ones.

A primer on artificial intelligence in plant digital phenomics: embarking on the data to insights journey

Artificial intelligence (AI) has emerged as a fundamental component of global agricultural research that is poised to impact on many aspects of plant science. In digital phenomics, AI is capable of learning intricate structure and patterns in large datasets. We provide a perspective and primer on AI applications to phenome research. We propose a novel human-centric explainable AI (X-AI) system architecture consisting of data architecture, technology infrastructure, and AI architecture design. We clarify the difference between post hoc models and 'interpretable by design' models. We include guidance for effectively using an interpretable by design model in phenomic analysis. We also provide directions to sources of tools and resources for making data analytics increasingly accessible. This primer is accompanied by an interactive online tutorial.

Building Chemical Property Models for Energetic Materials from Small Datasets Using a Transfer Learning Approach

For many experimentally measured chemical properties that cannot be directly computed from first-principles, the existing physics-based models do not extrapolate well to out-of-sample molecules, and experimental datasets themselves are too small for traditional machine learning (ML) approaches. To overcome these limitations, we apply a transfer learning approach, whereby we simultaneously train a multi-target regression model on a small number of molecules with experimentally measured values and a large number of molecules with related computed properties. We demonstrate this methodology on predicting the experimentally measured impact sensitivity of energetic crystals, finding that both characteristics of the computed dataset and model architecture are important to prediction accuracy of the small experimental dataset. Our directed-message passing neural network (D-MPNN) ML model using transfer learning outperforms direct-ML and physics-based models on a diverse test set, and the new methods described here are widely applicable to modeling many other structure-property relationships.

Deep Transfer Learning Enables Robust Prediction of Antimicrobial Resistance for Novel Antibiotics

Antimicrobial resistance (AMR) has become one of the serious global health problems, threatening the effective treatment of a growing number of infections. Machine learning and deep learning show great potential in rapid and accurate AMR predictions. However, a large number of samples for the training of these models is essential. In particular, for novel antibiotics, limited training samples and data imbalance hinder the models' generalization performance and overall accuracy. We propose a deep transfer learning model that can improve model performance for AMR prediction on small, imbalanced datasets. As our approach relies on transfer learning and secondary mutations, it is also applicable to novel antibiotics and emerging resistances in the future and enables quick diagnostics and personalized treatments.

Generative Adversarial Networks and Mixture Density Networks-Based Inverse Modeling for Microstructural Materials Design

There are two broad modeling paradigms in scientific applications: forward and inverse. While forward modeling estimates the observations based on known causes, inverse modeling attempts to infer the causes given the observations. Inverse problems are usually more critical as well as difficult in scientific applications as they seek to explore the causes that cannot be directly observed. Inverse problems are used extensively in various scientific fields, such as geophysics, health care and materials science. Exploring the relationships from properties to microstructures is one of the inverse problems in material science. It is challenging to solve the microstructure discovery inverse problem, because it usually needs to learn a one-to-many nonlinear mapping. Given a target property, there are multiple different microstructures that exhibit the target property, and their discovery also requires significant computing time. Further, microstructure discovery becomes even more difficult because the dimension of properties (input) is much lower than that of microstructures (output). In this work, we propose a framework consisting of generative adversarial networks and mixture density networks for inverse modeling of structure-property linkages in materials, i.e., microstructure discovery for a given property. The results demonstrate that compared to baseline methods, the proposed framework can overcome the above-mentioned challenges and discover multiple promising solutions in an efficient manner.

Prior Knowledge for Predictive Modeling: The Case of Acute Aquatic Toxicity

Early assessment of the potential impact of chemicals on health and the environment requires toxicological properties of the molecules. Predictive modeling is often used to estimate the property values in silico from pre-existing experimental data, which is often scarce and uncertain. One of the ways to advance the predictive modeling procedure might be the use of knowledge existing in the field. Scientific publications contain a vast amount of knowledge. However, the amount of manual work required to process the enormous volumes of information gathered in scientific articles might hinder its utilization. This work explores the opportunity of semiautomated knowledge extraction from scientific papers and investigates a few potential ways of its use for predictive modeling. The knowledge extraction and predictive modeling are applied to the field of acute aquatic toxicity. Acute aquatic toxicity is an important parameter of the safety assessment of chemicals. The extensive amount of diverse information existing in the field makes acute aquatic toxicity an attractive area for investigation of knowledge use for predictive modeling. The work demonstrates that the knowledge collection and classification procedure could be useful in hybrid modeling studies concerning the model and predictor selection, addressing data gaps, and evaluation of models' performance.

Moving closer to experimental level materials property prediction using AI

While experiments and DFT-computations have been the primary means for understanding the chemical and physical properties of crystalline materials, experiments are expensive and DFT-computations are time-consuming and have significant discrepancies against experiments. Currently, predictive modeling based on DFT-computations have provided a rapid screening method for materials candidates for further DFT-computations and experiments; however, such models inherit the large discrepancies from the DFT-based training data. Here, we demonstrate how AI can be leveraged together with DFT to compute materials properties more accurately than DFT itself by focusing on the critical materials science task of predicting “formation energy of a material given its structure and composition”. On an experimental hold-out test set containing 137 entries, AI can predict formation energy from materials structure and composition with a mean absolute error (MAE) of 0.064 eV/atom; comparing this against DFT-computations, we find that AI can significantly outperform DFT computations for the same task (discrepancies of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$>0.076$$\end{document}>0.076 eV/atom) for the first time.

Transfer learning using attentions across atomic systems with graph neural networks (TAAG)

Recent advances in Graph Neural Networks (GNNs) have transformed the space of molecular and catalyst discovery. Despite the fact that the underlying physics across these domains remain the same, most prior work has focused on building domain-specific models either in small molecules or in materials. However, building large datasets across all domains is computationally expensive; therefore, the use of transfer learning (TL) to generalize to different domains is a promising but under-explored approach to this problem. To evaluate this hypothesis, we use a model that is pretrained on the Open Catalyst Dataset (OC20), and we study the model's behavior when fine-tuned for a set of different datasets and tasks. This includes MD17, the *CO adsorbate dataset, and OC20 across different tasks. Through extensive TL experiments, we demonstrate that the initial layers of GNNs learn a more basic representation that is consistent across domains, whereas the final layers learn more task-specific features. Moreover, these well-known strategies show significant improvement over the non-pretrained models for in-domain tasks with improvements of 53% and 17% for the *CO dataset and across the Open Catalyst Project (OCP) task, respectively. TL approaches result in up to 4× speedup in model training depending on the target data and task. However, these do not perform well for the MD17 dataset, resulting in worse performance than the non-pretrained model for few molecules. Based on these observations, we propose transfer learning using attentions across atomic systems with graph Neural Networks (TAAG), an attention-based approach that adapts to prioritize and transfer important features from the interaction layers of GNNs. The proposed method outperforms the best TL approach for out-of-domain datasets, such as MD17, and gives a mean improvement of 6% over a model trained from scratch.