TeraChem: Accelerating electronic structure and ab initio molecular dynamics with graphical processing units

Developed over the past decade, TeraChem is an electronic structure and ab initio molecular dynamics software package designed from the ground up to leverage graphics processing units (GPUs) to perform large-scale ground and excited state quantum chemistry calculations in the gas and the condensed phase. TeraChem's speed stems from the reformulation of conventional electronic structure theories in terms of a set of individually optimized high-performance electronic structure operations (e.g., Coulomb and exchange matrix builds, one- and two-particle density matrix builds) and rank-reduction techniques (e.g., tensor hypercontraction). Recent efforts have encapsulated these core operations and provided language-agnostic interfaces. This greatly increases the accessibility and flexibility of TeraChem as a platform to develop new electronic structure methods on GPUs and provides clear optimization targets for emerging parallel computing architectures.

Solving the structure of Lgl2, a difficult blind test of unsupervised structure determination

In the companion paper by Ufimtsev and Levitt [Ufimtsev IS, Levitt M (2019) Proc Natl Acad Sci USA, 10.1073/pnas.1821512116], we presented a method for unsupervised solution of protein crystal structures and demonstrated its utility by solving several test cases of known structure in the 2.9- to 3.45-Å resolution range. Here we apply this method to solve the crystal structure of a 966-amino acid construct of human lethal giant larvae protein (Lgl2) that resisted years of structure determination efforts, at 3.2-Å resolution. The structure was determined starting with a molecular replacement (MR) model identified by unsupervised refinement of a pool of 50 candidate MR models. This initial model had 2.8-Å RMSD from the solution. The solved structure was validated by comparison with a model subsequently derived from an alternative crystal form diffracting to higher resolution. This model could phase an anomalous difference Fourier map from an Hg derivative, and a single-wavelength anomalous dispersion phased density map made from these sites aligned with the refined structure.

Quantum Chemistry on Graphical Processing Units. 1. Strategies for Two-Electron Integral Evaluation

Modern videogames place increasing demands on the computational and graphical hardware, leading to novel architectures that have great potential in the context of high performance computing and molecular simulation. We demonstrate that Graphical Processing Units (GPUs) can be used very efficiently to calculate two-electron repulsion integrals over Gaussian basis functions [Formula: see text] the first step in most quantum chemistry calculations. A benchmark test performed for the evaluation of approximately 10(6) (ss|ss) integrals over contracted s-orbitals showed that a naïve algorithm implemented on the GPU achieves up to 130-fold speedup over a traditional CPU implementation on an AMD Opteron. Subsequent calculations of the Coulomb operator for a 256-atom DNA strand show that the GPU advantage is maintained for basis sets including higher angular momentum functions.

Generating Efficient Quantum Chemistry Codes for Novel Architectures

We describe an extension of our graphics processing unit (GPU) electronic structure program TeraChem to include atom-centered Gaussian basis sets with d angular momentum functions. This was made possible by a "meta-programming" strategy that leverages computer algebra systems for the derivation of equations and their transformation to correct code. We generate a multitude of code fragments that are formally mathematically equivalent, but differ in their memory and floating-point operation footprints. We then select between different code fragments using empirical testing to find the highest performing code variant. This leads to an optimal balance of floating-point operations and memory bandwidth for a given target architecture without laborious manual tuning. We show that this approach is capable of similar performance compared to our hand-tuned GPU kernels for basis sets with s and p angular momenta. We also demonstrate that mixed precision schemes (using both single and double precision) remain stable and accurate for molecules with d functions. We provide benchmarks of the execution time of entire self-consistent field (SCF) calculations using our GPU code and compare to mature CPU based codes, showing the benefits of the GPU architecture for electronic structure theory with appropriately redesigned algorithms. We suggest that the meta-programming and empirical performance optimization approach may be important in future computational chemistry applications, especially in the face of quickly evolving computer architectures.

Excited-State Electronic Structure with Configuration Interaction Singles and Tamm-Dancoff Time-Dependent Density Functional Theory on Graphical Processing Units

Excited-state calculations are implemented in a development version of the GPU-based TeraChem software package using the configuration interaction singles (CIS) and adiabatic linear response Tamm–Dancoff time-dependent density functional theory (TDA-TDDFT) methods. The speedup of the CIS and TDDFT methods using GPU-based electron repulsion integrals and density functional quadrature integration allows full ab initio excited-state calculations on molecules of unprecedented size. CIS/6-31G and TD-BLYP/6-31G benchmark timings are presented for a range of systems, including four generations of oligothiophene dendrimers, photoactive yellow protein (PYP), and the PYP chromophore solvated with 900 quantum mechanical water molecules. The effects of double and single precision integration are discussed, and mixed precision GPU integration is shown to give extremely good numerical accuracy for both CIS and TDDFT excitation energies (excitation energies within 0.0005 eV of extended double precision CPU results).

GPU-accelerated molecular modeling coming of age

Graphics processing units (GPUs) have traditionally been used in molecular modeling solely for visualization of molecular structures and animation of trajectories resulting from molecular dynamics simulations. Modern GPUs have evolved into fully programmable, massively parallel co-processors that can now be exploited to accelerate many scientific computations, typically providing about one order of magnitude speedup over CPU code and in special cases providing speedups of two orders of magnitude. This paper surveys the development of molecular modeling algorithms that leverage GPU computing, the advances already made and remaining issues to be resolved, and the continuing evolution of GPU technology that promises to become even more useful to molecular modeling. Hardware acceleration with commodity GPUs is expected to benefit the overall computational biology community by bringing teraflops performance to desktop workstations and in some cases potentially changing what were formerly batch-mode computational jobs into interactive tasks.

Observation of a Zundel-like transition state during proton transfer in aqueous hydroxide solutions

It is generally accepted that the anomalous diffusion of the aqueous hydroxide ion results from its ability to accept a proton from a neighboring water molecule; yet, many questions exist concerning the mechanism for this process. What is the solvation structure of the hydroxide ion? In what way do water hydrogen bond dynamics influence the transfer of a proton to the ion? We present the results of femtosecond pump-probe and 2D infrared experiments that probe the O-H stretching vibration of a solution of dilute HOD dissolved in NaOD/D(2)O. Upon the addition of NaOD, measured pump-probe transients and 2D IR spectra show a new feature that decays with a 110-fs time scale. The calculation of 2D IR spectra from an empirical valence bond molecular dynamics simulation of a single NaOH molecule in a bath of H(2)O indicates that this fast feature is due to an overtone transition of Zundel-like H(3)O(2)(-) states, wherein a proton is significantly shared between a water molecule and the hydroxide ion. Given the frequency of vibration of shared protons, the observations indicate the shared proton state persists for 2-3 vibrational periods before the proton localizes on a hydroxide. Calculations based on the EVB-MD model argue that the collective electric field in the proton transfer direction is the appropriate coordinate to describe the creation and relaxation of these Zundel-like transition states.

A multistate empirical valence bond model for solvation and transport simulations of OH- in aqueous solutions

We describe a new multistate empirical valence bond (MS-EVB) model of OH(-) in aqueous solutions. This model is based on the recently proposed "charged ring" parameterization for the intermolecular interaction of hydroxyl ion with water [Ufimtsev, et al., Chem. Phys. Lett., 2007, 442, 128] and is suitable for classical molecular simulations of OH(-) solvation and transport. The model reproduces the hydration structure of OH(-)(aq) in good agreement with experimental data and the results of ab initio molecular dynamics simulations. It also accurately captures the major structural, energetic, and dynamic aspects of the proton transfer processes involving OH(-) (aq). The model predicts an approximately two-fold increase of the OH(-) mobility due to proton exchange reactions.

Quantum Chemistry on Graphical Processing Units. 3. Analytical Energy Gradients, Geometry Optimization, and First Principles Molecular Dynamics

We demonstrate that a video gaming machine containing two consumer graphical cards can outpace a state-of-the-art quad-core processor workstation by a factor of more than 180× in Hartree-Fock energy + gradient calculations. Such performance makes it possible to run large scale Hartree-Fock and Density Functional Theory calculations, which typically require hundreds of traditional processor cores, on a single workstation. Benchmark Born-Oppenheimer molecular dynamics simulations are performed on two molecular systems using the 3-21G basis set - a hydronium ion solvated by 30 waters (94 atoms, 405 basis functions) and an aspartic acid molecule solvated by 147 waters (457 atoms, 2014 basis functions). Our GPU implementation can perform 27 ps/day and 0.7 ps/day of ab initio molecular dynamics simulation on a single desktop computer for these systems.