Massively Parallel Quantum Chemistry: A high-performance research platform for electronic structure

Forte: A suite of advanced multireference quantum chemistry methods

Forte is an open-source library specialized in multireference electronic structure theories for molecular systems and the rapid prototyping of new methods. This paper gives an overview of the capabilities of Forte, its software architecture, and examples of applications enabled by the methods it implements.

Revisiting Artifacts of Kohn-Sham Density Functionals for Biosimulation

We revisit the problem of unphysical charge density delocalization/fractionalization induced by the self-interaction error of common approximate Kohn-Sham (KS) density functional theory functionals on simulation of small to medium-sized proteins in a vacuum. Aside from producing unphysical electron densities and total energies, the vanishing of the HOMO-LUMO gap associated with the unphysical charge delocalization leads to an unphysical low-energy spectrum and catastrophic failure of most popular solvers for the KS self-consistent field (SCF) problem. We apply a robust quasi-Newton SCF solver [ Phys. Chem. Chem. Phys. 2024, 26, 6557] to obtain solutions for some of these difficult cases. The anatomy of the charge delocalization is revealed by the natural deformation orbitals obtained from the density matrix difference between the Hartree-Fock and KS solutions; the charge delocalization not only can occur between charged fragments (such as in zwitterionic polypeptides) but also involves neutral fragments. The vanishing-gap phenomenon and troublesome SCF convergence are both attributed to the unphysical KS Fock operator eigenspectra of molecular fragments (e.g., amino acids or their side chains). Analysis of amino acid pairs suggests that the unphysical charge delocalization can be partially ameliorated by the use of some range-separated hybrid functionals but not by semilocal or standard hybrid functionals. Last, we demonstrate that solutions without the unphysical charge delocalization can be located even for semilocal KS functionals highly prone to such defects, but such solutions have non-Aufbau character and are unstable with respect to mixing of the non-overlapping "frontier" orbitals. Caution should be exercised when unexpectedly small (or vanishing) HOMO-LUMO gaps and atypical SCF convergence patterns (e.g., oscillatory) are observed in KS DFT simulations in any context (bio or otherwise).

Economical quasi-Newton unitary optimization of electronic orbitals

We present an efficient quasi-Newton orbital solver optimized to reduce the number of gradient evaluations and other computational steps of comparable cost. The solver optimizes orthogonal orbitals by sequences of unitary rotations generated by the (preconditioned) limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm equipped with trust-region step restriction. The low-rank structure of the L-BFGS inverse Hessian is exploited when solving the trust-region problem. The efficiency of the proposed "Quasi-Newton Unitary Optimization with Trust-Region" (QUOTR) solver is compared to that of the standard Roothaan-Hall approach accelerated by the Direct Inversion of Iterative Subspace (DIIS), and other exact and approximate Newton solvers for mean-field (Hartree-Fock and Kohn-Sham) problems.

Direct Determination of Optimal Real-Space Orbitals for Correlated Electronic Structure of Molecules

We demonstrate how to determine numerically nearly exact orthonormal orbitals that are optimal for the evaluation of the energy of arbitrary (correlated) states of atoms and molecules by minimization of the energy Lagrangian. Orbitals are expressed in real space using a multiresolution spectral element basis that is refined adaptively to achieve the user-specified target precision while avoiding the ill-conditioning issues that plague AO basis set expansions traditionally used for correlated models of molecular electronic structure. For light atoms, the orbital solver, in conjunction with a variational electronic structure model [selected Configuration Interaction (CI)] provides energies of comparable precision to a state-of-the-art atomic CI solver. The computed electronic energies of atoms and molecules are significantly more accurate than the counterparts obtained with the orbital sets of the same rank expanded in Gaussian AO bases, and can be determined even when linear dependence issues preclude the use of the AO bases. It is feasible to optimize more than 100 fully correlated numerical orbitals on a single computer node, and significant room exists for additional improvement. These findings suggest that real-space orbital representations might be the preferred alternative to AO representations for high-end models of correlated electronic states of molecules and materials.

A Perspective on Sustainable Computational Chemistry Software Development and Integration

The power of quantum chemistry to predict the ground and excited state properties of complex chemical systems has driven the development of computational quantum chemistry software, integrating advances in theory, applied mathematics, and computer science. The emergence of new computational paradigms associated with exascale technologies also poses significant challenges that require a flexible forward strategy to take full advantage of existing and forthcoming computational resources. In this context, the sustainability and interoperability of computational chemistry software development are among the most pressing issues. In this perspective, we discuss software infrastructure needs and investments with an eye to fully utilize exascale resources and provide unique computational tools for next-generation science problems and scientific discoveries.

Toward an extreme-scale electronic structure system

Electronic structure calculations have the potential to predict key matter transformations for applications of strategic technological importance, from drug discovery to material science and catalysis. However, a predictive physicochemical characterization of these processes often requires accurate quantum chemical modeling of complex molecular systems with hundreds to thousands of atoms. Due to the computationally demanding nature of electronic structure calculations and the complexity of modern high-performance computing hardware, quantum chemistry software has historically failed to operate at such large molecular scales with accuracy and speed that are useful in practice. In this paper, novel algorithms and software are presented that enable extreme-scale quantum chemistry capabilities with particular emphasis on exascale calculations. This includes the development and application of the multi-Graphics Processing Unit (GPU) library LibCChem 2.0 as part of the General Atomic and Molecular Electronic Structure System package and of the standalone Extreme-scale Electronic Structure System (EXESS), designed from the ground up for scaling on thousands of GPUs to perform high-performance accurate quantum chemistry calculations at unprecedented speed and molecular scales. Among various results, we report that the EXESS implementation enables Hartree-Fock/cc-pVDZ plus RI-MP2/cc-pVDZ/cc-pVDZ-RIFIT calculations on an ionic liquid system with 623 016 electrons and 146 592 atoms in less than 45 min using 27 600 GPUs on the Summit supercomputer with a 94.6% parallel efficiency.

Distributed memory, GPU accelerated Fock construction for hybrid, Gaussian basis density functional theory

With the growing reliance of modern supercomputers on accelerator-based architecture such a graphics processing units (GPUs), the development and optimization of electronic structure methods to exploit these massively parallel resources has become a recent priority. While significant strides have been made in the development GPU accelerated, distributed memory algorithms for many modern electronic structure methods, the primary focus of GPU development for Gaussian basis atomic orbital methods has been for shared memory systems with only a handful of examples pursing massive parallelism. In the present work, we present a set of distributed memory algorithms for the evaluation of the Coulomb and exact exchange matrices for hybrid Kohn-Sham DFT with Gaussian basis sets via direct density-fitted (DF-J-Engine) and seminumerical (sn-K) methods, respectively. The absolute performance and strong scalability of the developed methods are demonstrated on systems ranging from a few hundred to over one thousand atoms using up to 128 NVIDIA A100 GPUs on the Perlmutter supercomputer.

Tensor Hypercontraction Form of the Perturbative Triples Energy in Coupled-Cluster Theory

We present the working equations for a reduced-scaling method of evaluating the perturbative triples (T) energy in coupled-cluster theory, through the tensor hypercontraction (THC) of the triples amplitudes (t_ijk^abc). Through our method, we can reduce the scaling of the (T) energy from the traditional O(N7) to a more modest O(N5). We also discuss implementation details to aid future research, development, and software realization of this method. Additionally, we show that this method yields submillihartree (mEh) differences from CCSD(T) when evaluating absolute energies and sub-0.1 kcal/mol energy differences when evaluating relative energies. Finally, we demonstrate that this method converges to the true CCSD(T) energy through the systematic increasing of the rank or eigenvalue tolerance of the orthogonal projector, as well as exhibiting sublinear to linear error growth with respect to system size.

Memory-Efficient Recursive Evaluation of 3-Center Gaussian Integrals

To improve the efficiency of Gaussian integral evaluation on modern accelerated architectures, FLOP-efficient Obara-Saika-based recursive evaluation schemes are optimized for the memory footprint. For the 3-center 2-particle integrals that are key for the evaluation of Coulomb and other 2-particle interactions in the density-fitting approximation, the use of multiquantal recurrences (in which multiple quanta are created or transferred at once) is shown to produce significant memory savings. Other innovations include leveraging register memory for reduced memory footprint and direct compile-time generation of optimized kernels (instead of custom code generation) with compile-time features of modern C++/CUDA. Performance of conventional and CUDA-based implementations of the proposed schemes is illustrated for both the individual batches of integrals involving up to Gaussians with low and high angular momenta (up to L = 6) and contraction degrees, as well as for the density-fitting-based evaluation of the Coulomb potential. The computer implementation is available in the open-source LibintX library.

Quantum Mechanics/Molecular Mechanics Simulations on NVIDIA and AMD Graphics Processing Units

We have ported and optimized the graphics processing unit (GPU)-accelerated QUICK and AMBER-based ab initio quantum mechanics/molecular mechanics (QM/MM) implementation on AMD GPUs. This encompasses the entire Fock matrix build and force calculation in QUICK including one-electron integrals, two-electron repulsion integrals, exchange-correlation quadrature, and linear algebra operations. General performance improvements to the QUICK GPU code are also presented. Benchmarks carried out on NVIDIA V100 and AMD MI100 cards display similar performance on both hardware for standalone HF/DFT calculations with QUICK and QM/MM molecular dynamics simulations with QUICK/AMBER. Furthermore, with respect to the QUICK/AMBER release version 21, significant speedups are observed for QM/MM molecular dynamics simulations. This significantly increases the range of scientific problems that can be addressed with open-source QM/MM software on state-of-the-art computer hardware.

Multinode Multi-GPU Two-Electron Integrals: Code Generation Using the Regent Language

The computation of two-electron repulsion integrals (ERIs) is often the most expensive step of integral-direct self-consistent field methods. Formally it scales as O(N⁴), where N is the number of Gaussian basis functions used to represent the molecular wave function. In practice, this scaling can be reduced to O(N²) or less by neglecting small integrals with screening methods. The contributions of the ERIs to the Fock matrix are of Coulomb (J) and exchange (K) type and require separate algorithms to compute matrix elements efficiently. We previously implemented highly efficient GPU-accelerated J-matrix and K-matrix algorithms in the electronic structure code TeraChem. Although these implementations supported the use of multiple GPUs on a node, they did not support the use of multiple nodes. This presents a key bottleneck to cutting-edge ab initio simulations of large systems, e.g., excited state dynamics of photoactive proteins. We present our implementation of multinode multi-GPU J- and K-matrix algorithms in TeraChem using the Regent programming language. Regent directly supports distributed computation in a task-based model and can generate code for a variety of architectures, including NVIDIA GPUs. We demonstrate multinode scaling up to 45 GPUs (3 nodes) and benchmark against hand-coded TeraChem integral code. We also outline our metaprogrammed Regent implementation, which enables flexible code generation for integrals of different angular momenta.

Robust Pipek-Mezey Orbital Localization in Periodic Solids

We describe a robust method for determining Pipek-Mezey (PM) Wannier functions (WF), recently introduced by Jónsson et al. (J. Chem. Theor. Chem. 2017, 13, 460), which provide some formal advantages over the more common Boys (also known as maximally-localized) Wannier functions. The Broyden-Fletcher-Goldfarb-Shanno-based PMWF solver is demonstrated to yield dramatically faster convergence compared to the alternatives (steepest ascent and conjugate gradient) in a variety of one-, two-, and three-dimensional solids (including some with vanishing gaps) and can be used to obtain Wannier functions robustly in supercells with thousands of atoms. Evaluation of the PM functional and its gradient in periodic linear combination of atomic orbital representation used a particularly simple definition of atomic charges obtained by Moore-Penrose pseudoinverse projection onto the minimal atomic orbital basis. An automated "canonicalize phase then randomize" method for generating the initial guess for WFs contributes significantly to the robustness of the solver.

Knowledge Structure of the Application of High-Performance Computing: A Co-Word Analysis

As high-performance computing (HPC) plays a key role in the Fourth Industrial Revolution, the application of HPC in various industries is becoming increasingly important. Several studies have reviewed the research trends of HPC but considered only the functional aspects, causing limitations when discussing the application. Thus, this study aims to identify the knowledge structure of the application of HPC, enabling practical and policy support in various industrial fields. Co-word analysis is mainly used to establish the knowledge structure. We first collected 28,941 published papers related to HPC applications and built a co-word network that used author keywords. We performed centrality analysis and cluster analysis of the co-word network; as a result, we derived the major keywords and 18 areas of HPC applications. To validate the knowledge structure, we conducted a case study to find opportunities for HPC research plans in the research community. As a result, we discovered 17 new research topics and presented their research priorities by conducting expert interviews and Analytic Hierarchy Process. The findings of this study contribute to an understanding of the application of HPC, to exploring promising research fields for technological and social development, and to supporting research plans for successful technology commercialization.

GQCP: The Ghent Quantum Chemistry Package

The Ghent Quantum Chemistry Package (GQCP) is an open-source electronic structure software package that aims to provide an intuitive and expressive software framework for electronic structure software development. Its high-level interfaces (accessible through C++ and Python) have been specifically designed to correspond to theoretical concepts, while retaining access to lower-level intermediates and allowing structural run-time modifications of quantum chemical solvers. GQCP focuses on providing quantum chemical method developers with the computational "building blocks" that allow them to flexibly develop proof of principle implementations for new methods and applications up to the level of two-component spinor bases.

Harnessing the Power of Multi-GPU Acceleration into the Quantum Interaction Computational Kernel Program

We report a new multi-GPU capable ab initio Hartree-Fock/density functional theory implementation integrated into the open source QUantum Interaction Computational Kernel (QUICK) program. Details on the load balancing algorithms for electron repulsion integrals and exchange correlation quadrature across multiple GPUs are described. Benchmarking studies carried out on up to four GPU nodes, each containing four NVIDIA V100-SXM2 type GPUs demonstrate that our implementation is capable of achieving excellent load balancing and high parallel efficiency. For representative medium to large size protein/organic molecular systems, the observed parallel efficiencies remained above 82% for the Kohn-Sham matrix formation and above 90% for nuclear gradient calculations. The accelerations on NVIDIA A100, P100, and K80 platforms also have realized parallel efficiencies higher than 68% in all tested cases, paving the way for large-scale ab initio electronic structure calculations with QUICK.

Robust Approximation of Tensor Networks: Application to Grid-Free Tensor Factorization of the Coulomb Interaction

Approximation of a tensor network by approximating (e.g., factorizing) one or more of its constituent tensors can be improved by canceling the leading-order error due to the constituents' approximation. The utility of such robust approximation is demonstrated for robust canonical polyadic (CP) approximation of a (density-fitting) factorized two-particle Coulomb interaction tensor. The resulting algebraic (grid-free) approximation for the Coulomb tensor, closely related to the factorization appearing in pseudospectral and tensor hypercontraction approaches, is efficient and accurate, with significantly reduced rank compared to the naive (nonrobust) approximation. Application of the robust approximation to the particle-particle ladder term in the coupled-cluster singles and doubles reduces the size complexity from O (N⁶) to O (N⁵) with robustness ensuring negligible errors in chemically relevant energy differences using CP ranks approximately equal to the size of the density-fitting basis.

From NWChem to NWChemEx: Evolving with the Computational Chemistry Landscape

Since the advent of the first computers, chemists have been at the forefront of using computers to understand and solve complex chemical problems. As the hardware and software have evolved, so have the theoretical and computational chemistry methods and algorithms. Parallel computers clearly changed the common computing paradigm in the late 1970s and 80s, and the field has again seen a paradigm shift with the advent of graphical processing units. This review explores the challenges and some of the solutions in transforming software from the terascale to the petascale and now to the upcoming exascale computers. While discussing the field in general, NWChem and its redesign, NWChemEx, will be highlighted as one of the early codesign projects to take advantage of massively parallel computers and emerging software standards to enable large scientific challenges to be tackled.