End-to-end prediction of multimaterial stress fields and fracture patterns using cycle-consistent adversarial and transformer neural networks

ProtAgents: protein discovery via large language model multi-agent collaborations combining physics and machine learning

Designing de novo proteins beyond those found in nature holds significant promise for advancements in both scientific and engineering applications. Current methodologies for protein design often rely on AI-based models, such as surrogate models that address end-to-end problems by linking protein structure to material properties or vice versa. However, these models frequently focus on specific material objectives or structural properties, limiting their flexibility when incorporating out-of-domain knowledge into the design process or comprehensive data analysis is required. In this study, we introduce ProtAgents, a platform for de novo protein design based on Large Language Models (LLMs), where multiple AI agents with distinct capabilities collaboratively address complex tasks within a dynamic environment. The versatility in agent development allows for expertise in diverse domains, including knowledge retrieval, protein structure analysis, physics-based simulations, and results analysis. The dynamic collaboration between agents, empowered by LLMs, provides a versatile approach to tackling protein design and analysis problems, as demonstrated through diverse examples in this study. The problems of interest encompass designing new proteins, analyzing protein structures and obtaining new first-principles data - natural vibrational frequencies - via physics simulations. The concerted effort of the system allows for powerful automated and synergistic design of de novo proteins with targeted mechanical properties. The flexibility in designing the agents, on one hand, and their capacity in autonomous collaboration through the dynamic LLM-based multi-agent environment on the other hand, unleashes great potentials of LLMs in addressing multi-objective materials problems and opens up new avenues for autonomous materials discovery and design.

Generative Retrieval-Augmented Ontologic Graph and Multiagent Strategies for Interpretive Large Language Model-Based Materials Design

Transformer neural networks show promising capabilities, in particular for uses in materials analysis, design, and manufacturing, including their capacity to work effectively with human language, symbols, code, and numerical data. Here, we explore the use of large language models (LLMs) as a tool that can support engineering analysis of materials, applied to retrieving key information about subject areas, developing research hypotheses, discovery of mechanistic relationships across disparate areas of knowledge, and writing and executing simulation codes for active knowledge generation based on physical ground truths. Moreover, when used as sets of AI agents with specific features, capabilities, and instructions, LLMs can provide powerful problem-solution strategies for applications in analysis and design problems. Our experiments focus on using a fine-tuned model, MechGPT, developed based on training data in the mechanics of materials domain. We first affirm how fine-tuning endows LLMs with a reasonable understanding of subject area knowledge. However, when queried outside the context of learned matter, LLMs can have difficulty recalling correct information and may hallucinate. We show how this can be addressed using retrieval-augmented Ontological Knowledge Graph strategies. The graph-based strategy helps us not only to discern how the model understands what concepts are important but also how they are related, which significantly improves generative performance and also naturally allows for injection of new and augmented data sources into generative AI algorithms. We find that the additional feature of relatedness provides advantages over regular retrieval augmentation approaches and not only improves LLM performance but also provides mechanistic insights for exploration of a material design process. Illustrated for a use case of relating distinct areas of knowledge, here, music and proteins, such strategies can also provide an interpretable graph structure with rich information at the node, edge, and subgraph level that provides specific insights into mechanisms and relationships. We discuss other approaches to improve generative qualities, including nonlinear sampling strategies and agent-based modeling that offer enhancements over single-shot generations, whereby LLMs are used to both generate content and assess content against an objective target. Examples provided include complex question answering, code generation, and execution in the context of automated force-field development from actively learned density functional theory (DFT) modeling and data analysis.

The effect of increasing temperature on simulated nanocomposites reinforced with SWBNNs and its effect on characteristics related to mechanics and the physical attributes using the MDs approach

This study examines the effect of increasing temperature (300, 350, 400, 450 and 500 K) on simulated nanocomposites reinforced with exploration of the impact of single-walled boron nitride nanotubes (SWBNNTs) on both the mechanical properties (including Young's modulus, Poisson's ratio, shear modulus, and bulk modulus) and the physical property of density, achieved through molecular dynamics (MDs) simulations. MDs utilized to simulate nanocomposite models consisting of five case studies of SWBNNs with different chiralities (5, 0), (10, 0), (15, 0), (20, 0), and (25, 0) as the reinforcement and using thermoplastic polyurethane (TPU) as the common matrix. The results reveal that with increasing temperature and chiralities of SWBNNTs, the density and Poisson's ratio increase dramatically, and Young's, shear, and bulk moduli decrease continuously. At a consistent temperature, there is a noteworthy trend in the mechanical properties of SWBNNTs with various chiralities. This includes the increase in Young's modulus, Poisson's ratio, shear modulus, and bulk modulus in the simulated nanocomposite, ranging from SWBNNTs (5, 0) to (25, 0). Similarly, the physical property of density exhibits an increasing trend from SWBNNTs (5, 0) to (20, 0) and then decreases at SWBNNTs (25, 0). To validate the accuracy of these findings, a Radial Distribution Function (RDF) diagram is generated using Materials Studio software.

Unsupervised cross-domain translation via deep learning and adversarial attention neural networks and application to music-inspired protein designs

Taking inspiration from nature about how to design materials has been a fruitful approach, used by humans for millennia. In this paper we report a method that allows us to discover how patterns in disparate domains can be reversibly related using a computationally rigorous approach, the AttentionCrossTranslation model. The algorithm discovers cycle- and self-consistent relationships and offers a bidirectional translation of information across disparate knowledge domains. The approach is validated with a set of known translation problems, and then used to discover a mapping between musical data-based on the corpus of note sequences in J.S. Bach's Goldberg Variations created in 1741-and protein sequence data-information sampled more recently. Using protein folding algorithms, 3D structures of the predicted protein sequences are generated, and their stability is validated using explicit solvent molecular dynamics. Musical scores generated from protein sequences are sonified and rendered into audible sound.

Computational Design and Manufacturing of Sustainable Materials through First-Principles and Materiomics

Engineered materials are ubiquitous throughout society and are critical to the development of modern technology, yet many current material systems are inexorably tied to widespread deterioration of ecological processes. Next-generation material systems can address goals of environmental sustainability by providing alternatives to fossil fuel-based materials and by reducing destructive extraction processes, energy costs, and accumulation of solid waste. However, development of sustainable materials faces several key challenges including investigation, processing, and architecting of new feedstocks that are often relatively mechanically weak, complex, and difficult to characterize or standardize. In this review paper, we outline a framework for examining sustainability in material systems and discuss how recent developments in modeling, machine learning, and other computational tools can aid the discovery of novel sustainable materials. We consider these through the lens of materiomics, an approach that considers material systems holistically by incorporating perspectives of all relevant scales, beginning with first-principles approaches and extending through the macroscale to consider sustainable material design from the bottom-up. We follow with an examination of how computational methods are currently applied to select examples of sustainable material development, with particular emphasis on bioinspired and biobased materials, and conclude with perspectives on opportunities and open challenges.

Multiscale Modeling at the Interface of Molecular Mechanics and Natural Language through Attention Neural Networks

Humans are continually bombarded with massive amounts of data. To deal with this influx of information, we use the concept of attention in order to perceive the most relevant input from vision, hearing, touch, and others. Thereby, the complex ensemble of signals is used to generate output by querying the processed data in appropriate ways. Attention is also the hallmark of the development of scientific theories, where we elucidate which parts of a problem are critical, often expressed through differential equations. In this Account we review the emergence of attention-based neural networks as a class of approaches that offer many opportunities to describe materials across scales and modalities, including how universal building blocks interact to yield a set of material properties. In fact, the self-assembly of hierarchical, structurally complex, and multifunctional biomaterials remains a grand challenge in modeling, theory, and experiment. Expanding from the process by which material building blocks physically interact to form a type of material, in this Account we view self-assembly as both the functional emergence of properties from interacting building blocks as well as the physical process by which elementary building blocks interact and yield structure and, thereby, functions. This perspective, integrated through the theory of materiomics, allows us to solve multiscale problems with a first-principles-based computational approach based on attention-based neural networks that transform information to feature to property while providing a flexible modeling approach that can integrate theory, simulation, and experiment. Since these models are based on a natural language framework, they offer various benefits including incorporation of general domain knowledge via general-purpose pretraining, which can be accomplished without labeled data or large amounts of lower-quality data. Pretrained models then offer a general-purpose platform that can be fine-tuned to adapt these models to make specific predictions, often with relatively little labeled data. The transferrable power of the language-based modeling approach realizes a neural olog description, where mathematical categorization is learned by multiheaded attention, without domain knowledge in its formulation. It can hence be applied to a range of complex modeling tasks─such as physical field predictions, molecular properties, or structure predictions, all using an identical formulation. This offers a complementary modeling approach that is already finding numerous applications, with great potential to solve complex assembly problems, enabling us to learn, build, and utilize functional categorization of how building blocks yield a range of material functions. In this Account, we demonstrate the approach in various application areas, including protein secondary structure prediction and prediction of normal-mode frequencies as well as predicting mechanical fields near cracks. Unifying these diverse problem areas is the building block approach, where the models are based on a universally applicable platform that offers benefits ranging from transferability, interpretability, and cross-domain pollination of knowledge as exemplified through a transformer model applied to predict how musical compositions infer de novo protein structures. We discuss future potentialities of this approach for a variety of material phenomena across scales, including the use in multiparadigm modeling schemes.

Modeling Atomistic Dynamic Fracture Mechanisms Using a Progressive Transformer Diffusion Model

Dynamic fracture is an important area of materials analysis, assessing the atomic-level mechanisms by which materials fail over time. Here, we focus on brittle materials failure and show that an atomistically derived progressive transformer diffusion machine learning model can effectively describe the dynamics of fracture, capturing important aspects such as crack dynamics, instabilities, and initiation mechanisms. Trained on a small dataset of atomistic simulations, the model generalizes well and offers a rapid assessment of dynamic fracture mechanisms for complex geometries, expanding well beyond the original set of atomistic simulation results. Various validation cases, progressively more distinct from the data used for training, are presented and analyzed. The validation cases feature distinct geometric details, including microstructures generated by a generative neural network used here to identify novel bio-inspired material designs for mechanical performance. For all cases, the model performs well and captures key aspects of material failure.

CollagenTransformer: End-to-End Transformer Model to Predict Thermal Stability of Collagen Triple Helices Using an NLP Approach

Collagen is one of the most important structural proteins in biology, and its structural hierarchy plays a crucial role in many mechanically important biomaterials. Here, we demonstrate how transformer models can be used to predict, directly from the primary amino acid sequence, the thermal stability of collagen triple helices, measured via the melting temperature T_m. We report two distinct transformer architectures to compare performance. First, we train a small transformer model from scratch, using our collagen data set featuring only 633 sequence-to-T_m pairings. Second, we use a large pretrained transformer model, ProtBERT, and fine-tune it for a particular downstream task by utilizing sequence-to-T_m pairings, using a deep convolutional network to translate natural language processing BERT embeddings into required features. Both the small transformer model and the fine-tuned ProtBERT model have similar R² values of test data (R² = 0.84 vs 0.79, respectively), but the ProtBERT is a much larger pretrained model that may not always be applicable for other biological or biomaterials questions. Specifically, we show that the small transformer model requires only 0.026% of the number of parameters compared to the much larger model but reaches almost the same accuracy for the test set. We compare the performance of both models against 71 newly published sequences for which T_m has been obtained as a validation set and find reasonable agreement, with ProtBERT outperforming the small transformer model. The results presented here are, to our best knowledge, the first demonstration of the use of transformer models for relatively small data sets and for the prediction of specific biophysical properties of interest. We anticipate that the work presented here serves as a starting point for transformer models to be applied to other biophysical problems.