Scalable deeper graph neural networks for high-performance materials property prediction

Scalable crystal structure relaxation using an iteration-free deep generative model with uncertainty quantification

In computational molecular and materials science, determining equilibrium structures is the crucial first step for accurate subsequent property calculations. However, the recent discovery of millions of new crystals and super large twisted structures has challenged traditional computational methods, both ab initio and machine-learning-based, due to their computationally intensive iterative processes. To address these scalability issues, here we introduce DeepRelax, a deep generative model capable of performing geometric crystal structure relaxation rapidly and without iterations. DeepRelax learns the equilibrium structural distribution, enabling it to predict relaxed structures directly from their unrelaxed ones. The ability to perform structural relaxation at the millisecond level per structure, combined with the scalability of parallel processing, makes DeepRelax particularly useful for large-scale virtual screening. We demonstrate DeepRelax's reliability and robustness by applying it to five diverse databases, including oxides, Materials Project, two-dimensional materials, van der Waals crystals, and crystals with point defects. DeepRelax consistently shows high accuracy and efficiency, validated by density functional theory calculations. Finally, we enhance its trustworthiness by integrating uncertainty quantification. This work significantly accelerates computational workflows, offering a robust and trustworthy machine-learning method for material discovery and advancing the application of AI for science.

PointGAT: A Quantum Chemical Property Prediction Model Integrating Graph Attention and 3D Geometry

Predicting quantum chemical properties is a fundamental challenge for computational chemistry. While the development of graph neural networks has advanced molecular representation learning and property prediction, their performance could be further enhanced by incorporating three-dimensional (3D) structural geometry into two-dimensional (2D) molecular graph representation. In this study, we introduce the PointGAT model for quantum molecular property prediction, which integrates 3D molecular coordinates with graph-attention modeling. Comparison with other current models in molecular prediction tasks showed that PointGAT could provide higher predictive accuracy in various benchmark data sets from MoleculeNet, including ESOL, FreeSolv, Lipop, HIV, and 6 out of 12 tasks of the QM9 data set. To further examine PointGAT prediction of quantum mechanical (QM) energies, we constructed a C10 data set comprising 11,841 charged and chiral carbocation intermediates with QM energies calculated at the DM21/6-31G*//B3LYP/6-31G* levels. Notably, PointGAT achieved an R2 value of 0.950 and an MAE of 1.616 kcal/mol, outperforming even the best-performing graph neural network model with a reduction of 0.216 kcal/mol in MAE and an improvement of 0.050 in R2. Additional ablation studies indicated that incorporating molecular geometry into the model resulted in markedly higher predictive accuracy, reducing the MAE value from 1.802 to 1.616 kcal/mol. Moreover, visualization of PointGAT atomic attention weights suggested its predictions were interpretable. Findings in this study support the application of PointGAT as a powerful and versatile tool for quantum chemical property prediction that can facilitate high-accuracy modeling for fundamental exploration of chemical space as well as drug design and molecular engineering.

The carbon footprint of predicting CO2 storage capacity in metal-organic frameworks within neural networks

While artificial intelligence drives remarkable progress in natural sciences, its broader societal implications are mostly disregarded. In this study, we evaluate environmental impacts of deep learning in materials science through extensive benchmarking. In particular, a set of diverse neural networks is trained for a given supervised learning task to assess greenhouse gas (GHG) emissions during training and inference phases. A chronological perspective showed diminishing returns, manifesting themselves as a 28% decrease in mean absolute error and nearly a 15,000% increase in the carbon footprint of model training in 2016-2022. By means of up-to-date graphics processing units, it is possible to partially offset the immense growth of GHG emissions. Nonetheless, the practice of employing energy-efficient hardware is overlooked by the materials informatics community, as follows from a literature analysis in the field. On the basis of our findings, we encourage researchers to report GHG emissions together with standard performance metrics.

Accelerating material property prediction using generically complete isometry invariants

Periodic material or crystal property prediction using machine learning has grown popular in recent years as it provides a computationally efficient replacement for classical simulation methods. A crucial first step for any of these algorithms is the representation used for a periodic crystal. While similar objects like molecules and proteins have a finite number of atoms and their representation can be built based upon a finite point cloud interpretation, periodic crystals are unbounded in size, making their representation more challenging. In the present work, we adapt the Pointwise Distance Distribution (PDD), a continuous and generically complete isometry invariant for periodic point sets, as a representation for our learning algorithm. The PDD distinguished all (more than 660 thousand) periodic crystals in the Cambridge Structural Database as purely periodic sets of points without atomic types. We develop a transformer model with a modified self-attention mechanism that combines PDD with compositional information via a spatial encoding method. This model is tested on the crystals of the Materials Project and Jarvis-DFT databases and shown to produce accuracy on par with state-of-the-art methods while being several times faster in both training and prediction time.

Polymer-Unit Graph: Advancing Interpretability in Graph Neural Network Machine Learning for Organic Polymer Semiconductor Materials

The graph representation of complex materials plays a crucial role in the field of inorganic and organic materials investigations for developing data-centric materials science, such as those using graph neural networks (GNNs). However, the currently prevalent GNN models are primarily employed for investigating periodic crystals and organic small molecule data, yet they still encounter challenges in terms of interpretability and computational efficiency when applied to polymer monomers and organic macromolecules data. There is still a lack of graph representation of organic polymers and macromolecules specifically tailored for GNN models to explore the structural characteristics. The Polymer-unit Graph, a novel coarse-grained graph representation method introduced in study, is dedicated to expressing and analyzing polymers and macromolecules. By incorporating the Polymer-unit Graph into the GNN models and analyzing the organic semiconductor (OSC) materials database, it becomes possible to uncover intricate structure-property relationships involving branched-chain engineering, fluoridation substitution, and donor-acceptor combination effects on the elementary structure of OSC polymers. Furthermore, the Polymer-unit Graph enables visualizing the relationship between target properties and polymer units while reducing training time by an impressive 98% and minimizing molecular graph representation models. In conclusion, the Polymer-unit Graph successfully integrates the concept of Polymer-unit into the field of GNNs, enabling more accurate analysis and understanding of organic polymers and macromolecules.

Coarse-Grained Crystal Graph Neural Networks for Reticular Materials Design

Reticular materials, including metal-organic frameworks and covalent organic frameworks, combine the relative ease of synthesis and an impressive range of applications in various fields from gas storage to biomedicine. Diverse properties arise from the variation of building units─metal centers and organic linkers─in almost infinite chemical space. Such variation substantially complicates the experimental design and promotes the use of computational methods. In particular, the most successful artificial intelligence algorithms for predicting the properties of reticular materials are atomic-level graph neural networks, which optionally incorporate domain knowledge. Nonetheless, the data-driven inverse design involving these models suffers from the incorporation of irrelevant and redundant features such as a full atomistic graph and network topology. In this study, we propose a new way of representing materials, aiming to overcome the limitations of existing methods; the message passing is performed on a coarse-grained crystal graph that comprises molecular building units. To highlight the merits of our approach, we assessed the predictive performance and energy efficiency of neural networks built on different materials representations, including composition-based and crystal-structure-aware models. Coarse-grained crystal graph neural networks showed decent accuracy at low computational costs, making them a valuable alternative to omnipresent atomic-level algorithms. Moreover, the presented models can be successfully integrated into an inverse materials design pipeline as estimators of the objective function. Overall, the coarse-grained crystal graph framework is aimed at challenging the prevailing atom-centric perspective on reticular materials design.

Physics-Guided Dual Self-Supervised Learning for Structure-Based Material Property Prediction

Deep learning models have been widely used for high-performance material property prediction. However, training such models usually requires a large amount of labeled data, which are usually unavailable. Self-supervised learning (SSL) methods have been proposed to address this data scarcity issue. Herein, we present DSSL, a physics-guided dual SSL framework, for graph neural network-based material property prediction, which combines node masking-based generative SSL with atomic coordinate perturbation-based contrastive SSL strategies to capture local and global information about input crystals. Moreover, we achieve physics-guided pretraining by using the macroproperty (e.g., elasticity)-related microproperty prediction of atomic stiffness as an additional pretext task. We pretrain our DSSL model on the Materials Project database and fine-tune it with 10 material property data sets. The experimental results demonstrate that teaching neural networks some physics using the SSL strategy can afford ≤26.89% performance improvement compared to that of the baseline models.

Prediction and Interpretability of Glass Transition Temperature of Homopolymers by Data-Augmented Graph Convolutional Neural Networks

Establishing the structure-property relationship by machine learning (ML) models is extremely valuable for accelerating the molecular design of polymers. However, existing ML models for the polymers are subject to scarcity issues of training data and fewer variations of graph structures of molecules. In addition, limited works have explored the interpretability of ML models to infer the latent knowledge in the field of polymer science that could inspire ML-assisted molecular design. In this contribution, we integrate graph convolutional neural networks (GCNs) with data augmentation strategy to predict the glass transition temperature Tg of polymers. It is demonstrated that the data-augmented GCN model outperforms the conventional models and achieves a higher accuracy for the prediction of Tg despite a small amount of training data. Furthermore, taking advantage of molecular graph representations, the data-augmented GCN model has the capability to infer the importance of atoms or substructures from the understanding of Tg, which generally agrees with the experimental findings in the field of polymer science. The inferred knowledge of the GCN model is used to advise on the design of functional polymers with specific Tg. The data-augmented GCN model possesses prominent superiorities in the establishment of structure-property relationship and also provides an efficient way for accelerating the rational design of polymer molecules.

Examining graph neural networks for crystal structures: Limitations and opportunities for capturing periodicity

Graph neural networks (GNNs) have recently been used to learn the representations of crystal structures through an end-to-end data-driven approach. However, a systematic top-down approach to evaluate and understand the limitations of GNNs in accurately capturing crystal structures has yet to be established. In this study, we introduce an approach using human-designed descriptors as a compendium of human knowledge to investigate the extent to which GNNs can comprehend crystal structures. Our findings reveal that current state-of-the-art GNNs fall short in accurately capturing the periodicity of crystal structures. We analyze this failure by exploring three aspects: local expressive power, long-range information processing, and readout function. To address these identified limitations, we propose a straightforward and general solution: the hybridization of descriptors with GNNs, which directly supplements the missing information to GNNs. The hybridization enhances the predictive accuracy of GNNs for specific material properties, most notably phonon internal energy and heat capacity, which heavily rely on the periodicity of materials.

Accurate, interpretable predictions of materials properties within transformer language models

Property prediction accuracy has long been a key parameter of machine learning in materials informatics. Accordingly, advanced models showing state-of-the-art performance turn into highly parameterized black boxes missing interpretability. Here, we present an elegant way to make their reasoning transparent. Human-readable text-based descriptions automatically generated within a suite of open-source tools are proposed as materials representation. Transformer language models pretrained on 2 million peer-reviewed articles take as input well-known terms such as chemical composition, crystal symmetry, and site geometry. Our approach outperforms crystal graph networks by classifying four out of five analyzed properties if one considers all available reference data. Moreover, fine-tuned text-based models show high accuracy in the ultra-small data limit. Explanations of their internal machinery are produced using local interpretability techniques and are faithful and consistent with domain expert rationales. This language-centric framework makes accurate property predictions accessible to people without artificial-intelligence expertise.

BNM-CDGNN: Batch Normalization Multilayer Perceptron Crystal Distance Graph Neural Network for Excellent-Performance Crystal Property Prediction

Recently, in the field of crystal property prediction, the graph neural network (GNN) model has made rapid progress. The GNN model can effectively capture high-dimensional crystal features from the crystal structure, thereby achieving optimal performance in property prediction. However, the existing GNN model faces limitations in handling the hidden layer after the pooling layer, which restricts the training performance of the model. In the present research, we propose a novel GNN model called the batch normalization multilayer perceptron crystal distance graph neural network (BNM-CDGNN). BNM-CDGNN encodes the crystal's geometry structure only based on the distance vector between atoms. The graph convolutional layer utilizes the radial basis function as the attention mask, ensuring the crystal's rotation invariance and adding the geometric information on the crystal. Subsequently, the average pooling layer is connected after the convolutional layer to enhance the model's ability to learn precise information. BNM-CDGNN connects multiple hidden layers after the average pooling layers, and these layers are processed by the batch normalization layer. Finally, the fully connected layer maps the results to the target property. BNM-CDGNN significantly enhances the accuracy of crystal property prediction compared with previous baseline models such as SchNet, MPNN, CGCNN, MEGNet, and GATGNN.

Estimation of Particle Location in Granular Materials Based on Graph Neural Networks

Particle locations determine the whole structure of a granular system, which is crucial to understanding various anomalous behaviors in glasses and amorphous solids. How to accurately determine the coordinates of each particle in such materials within a short time has always been a challenge. In this paper, we use an improved graph convolutional neural network to estimate the particle locations in two-dimensional photoelastic granular materials purely from the knowledge of the distances for each particle, which can be estimated in advance via a distance estimation algorithm. The robustness and effectiveness of our model are verified by testing other granular systems with different disorder degrees, as well as systems with different configurations. In this study, we attempt to provide a new route to the structural information of granular systems irrelevant to dimensionality, compositions, or other material properties.

Machine Learning Accelerated Discovery of Promising Thermal Energy Storage Materials with High Heat Capacity

Thermal energy storage offers numerous benefits by reducing energy consumption and promoting the use of renewable energy sources. Thermal energy storage materials have been investigated for many decades with the aim of improving the overall efficiency of energy systems. However, finding solid materials that meet the requirement of high heat capacity has been a grand challenge for material scientists. Herewith, by training various machine learning models on 3377 high-quality data from full density functional theory (DFT) calculations, we efficiently search for potential materials with high heat capacity. We build four traditional machine learning models and two graph neural network models. Cross-comparison of the prediction performance and model accuracy was conducted among different models. The deeperGATGNN model exhibits high prediction accuracy and is used for predicting the heat capacity of 32,026 structures screened from the open quantum material database. We gain deep insight into the correlation between heat capacity and structure descriptors such as space group, prototype, lattice volume, atomic weight, etc. Twenty-two structures were predicted to possess high heat capacity, and the results were further validated with DFT calculations. We also identified one special structure, namely, MnIn₂Se₄, with space group no. 227 (Fd3̅m), that exhibits extremely high heat capacity, even higher than that of the Dulong-Petit limit at room temperature. This study paves the way for accelerating the discovery of novel thermal energy storage materials by combining machine learning with minimal DFT inquiry.

Predicting Lattice Vibrational Frequencies Using Deep Graph Neural Networks

Lattice vibrational frequencies are related to many important materials properties such as thermal and electrical conductivity as well as superconductivity. However, computational calculation of vibrational frequencies using density functional theory methods is computationally too demanding for large number of samples in materials screening. Here we propose a deep graph neural network based algorithm for predicting crystal vibrational frequencies from crystal structures. Our algorithm addresses the variable dimension of vibrational frequency spectrum using the zero padding scheme. Benchmark studies on two data sets with 15,000 mixed-structure and 35,552 rhombohedra samples show that the aggregated R ² scores of the prediction reach 0.554 and 0.724. We also evaluate the structural transferability by predicting the vibration frequencies for 239 individual cubic target structures. The R ² scores for more than 40% of the targets are greater than 0.8 and can reach as high as 0.98 for the model trained with mixed samples, while the average mean absolute error is 43.69 Thz showing low transferability across structure types. Our work demonstrates the capability of deep graph neural networks to learn to predict lattice vibration frequency when sufficient number of training samples are available.

Formula Graph Self-Attention Network for Representation-Domain Independent Materials Discovery

The success of machine learning (ML) in materials property prediction depends heavily on how the materials are represented for learning. Two dominant families of material descriptors exist, one that encodes crystal structure in the representation and the other that only uses stoichiometric information with the hope of discovering new materials. Graph neural networks (GNNs) in particular have excelled in predicting material properties within chemical accuracy. However, current GNNs are limited to only one of the above two avenues owing to the little overlap between respective material representations. Here, a new concept of formula graph which unifies stoichiometry-only and structure-based material descriptors is introduced. A self-attention integrated GNN that assimilates a formula graph is further developed and it is found that the proposed architecture produces material embeddings transferable between the two domains. The proposed model can outperform some previously reported structure-agnostic models and their structure-based counterparts while exhibiting better sample efficiency and faster convergence. Finally, the model is applied in a challenging exemplar to predict the complex dielectric function of materials and nominate new substances that potentially exhibit epsilon-near-zero phenomena.