Affordable and accurate large-scale hybrid-functional calculations on GPU-accelerated supercomputers

Efficient all-electron hybrid density functionals for atomistic simulations beyond 10 000 atoms

Hybrid density functional approximations (DFAs) offer compelling accuracy for ab initio electronic-structure simulations of molecules, nanosystems, and bulk materials, addressing some deficiencies of computationally cheaper, frequently used semilocal DFAs. However, the computational bottleneck of hybrid DFAs is the evaluation of the non-local exact exchange contribution, which is the limiting factor for the application of the method for large-scale simulations. In this work, we present a drastically optimized resolution-of-identity-based real-space implementation of the exact exchange evaluation for both non-periodic and periodic boundary conditions in the all-electron code FHI-aims, targeting high-performance central processing unit (CPU) compute clusters. The introduction of several new refined message passing interface (MPI) parallelization layers and shared memory arrays according to the MPI-3 standard were the key components of the optimization. We demonstrate significant improvements of memory and performance efficiency, scalability, and workload distribution, extending the reach of hybrid DFAs to simulation sizes beyond ten thousand atoms. In addition, we also compare the runtime performance of the PBE, HSE06, and PBE0 functionals. As a necessary byproduct of this work, other code parts in FHI-aims have been optimized as well, e.g., the computation of the Hartree potential and the evaluation of the force and stress components. We benchmark the performance and scaling of the hybrid DFA-based simulations for a broad range of chemical systems, including hybrid organic-inorganic perovskites, organic crystals, and ice crystals with up to 30 576 atoms (101 920 electrons described by 244 608 basis functions).

Speeding-up Hybrid Functional-Based Ab Initio Molecular Dynamics Using Multiple Time-stepping and Resonance-Free Thermostat

Ab initio molecular dynamics (AIMD) based on density functional theory (DFT) has become a workhorse for studying the structure, dynamics, and reactions in condensed matter systems. Currently, AIMD simulations are primarily carried out at the level of generalized gradient approximation (GGA), which is at the second rung of DFT functionals in terms of accuracy. Hybrid DFT functionals, which form the fourth rung in the accuracy ladder, are not commonly used in AIMD simulations as the computational cost involved is 100 times or higher. To facilitate AIMD simulations with hybrid functionals, we propose here an approach using multiple time stepping with adaptively compressed exchange operator and resonance-free thermostat, that could speed up the calculations by ∼30 times or more for systems with a few hundred of atoms. We demonstrate that by achieving this significant speed up and making the compute time of hybrid functional-based AIMD simulations at par with that of GGA functionals, we are able to study several complex condensed matter systems and model chemical reactions in solution with hybrid functionals that were earlier unthinkable to be performed.

GPU Acceleration of Large-Scale Full-Frequency GW Calculations

Many-body perturbation theory is a powerful method to simulate electronic excitations in molecules and materials starting from the output of density functional theory calculations. By implementing the theory efficiently so as to run at scale on the latest leadership high-performance computing systems it is possible to extend the scope of GW calculations. We present a GPU acceleration study of the full-frequency GW method as implemented in the WEST code. Excellent performance is achieved through the use of (i) optimized GPU libraries, e.g., cuFFT and cuBLAS, (ii) a hierarchical parallelization strategy that minimizes CPU-CPU, CPU-GPU, and GPU-GPU data transfer operations, (iii) nonblocking MPI communications that overlap with GPU computations, and (iv) mixed precision in selected portions of the code. A series of performance benchmarks has been carried out on leadership high-performance computing systems, showing a substantial speedup of the GPU-accelerated version of WEST with respect to its CPU version. Good strong and weak scaling is demonstrated using up to 25 920 GPUs. Finally, we showcase the capability of the GPU version of WEST for large-scale, full-frequency GW calculations of realistic systems, e.g., a nanostructure, an interface, and a defect, comprising up to 10 368 valence electrons.

Improving the scaling and performance of multiple time stepping-based molecular dynamics with hybrid density functionals

Density functionals at the level of the generalized gradient approximation (GGA) and a plane-wave basis set are widely used today to perform ab initio molecular dynamics (AIMD) simulations. Going up in the ladder of accuracy of density functionals from GGA (second rung) to hybrid density functionals (fourth rung) is much desired pertaining to the accuracy of the latter in describing structure, dynamics, and energetics of molecular and condensed matter systems. On the other hand, hybrid density functional based AIMD simulations are about two orders of magnitude slower than GGA based AIMD for systems containing ~100 atoms using ~100 compute cores. Two methods, namely MTACE and s-MTACE, based on a multiple time step integrator and adaptively compressed exchange operator formalism are able to provide a speed-up of about 7-9 in performing hybrid density functional based AIMD. In this work, we report an implementation of these methods using a task-group based parallelization within the CPMD program package, with the intention to take advantage of the large number of compute cores available on modern high-performance computing platforms. We present here the boost in performance achieved through this algorithm. This work also identifies the computational bottleneck in the s-MTACE method and proposes a way to overcome it.

Exploring metastable states in UO2using hybrid functionals and dynamical mean field theory

A detailed exploration of thef-atomic orbital occupancy space for UO₂is performed using a first principles approach based on density functional theory (DFT), employing a full hybrid functional within a systematic basis set. Specifically, the PBE0 functional is combined with an occupancy biasing scheme implemented in a wavelet-based algorithm which is adapted to large supercells. The results are compared with previous DFT +Ucalculations reported in the literature, while dynamical mean field theory is also performed to provide a further base for comparison. This work shows that the computational complexity of the energy landscape of a correlatedf-electron oxide is much richer than has previously been demonstrated. The resulting calculations provide evidence of the existence of multiple previously unexplored metastable electronic states of UO₂, including those with energies which are lower than previously reported ground states.

Achieving an Order of Magnitude Speedup in Hybrid-Functional- and Plane-Wave-Based Ab Initio Molecular Dynamics: Applications to Proton-Transfer Reactions in Enzymes and in Solution

Ab initio molecular dynamics (MD) with hybrid density functionals and a plane wave basis is computationally expensive due to the high computational cost of exact exchange energy evaluation. Recently, we proposed a strategy to combine adaptively compressed exchange (ACE) operator formulation and a multiple time step integration scheme to reduce the computational cost significantly [J. Chem. Phys. 2019, 151, 151102 ]. However, it was found that the construction of the ACE operator, which has to be done at least once in every MD time step, is computationally expensive. In the present work, systematic improvements are introduced to further speed up by employing localized orbitals for the construction of the ACE operator. By this, we could achieve a computational speedup of an order of magnitude for a periodic system containing 32 water molecules. Benchmark calculations were carried out to show the accuracy and efficiency of the method in predicting the structural and dynamical properties of bulk water. To demonstrate the applicability, computationally intensive free-energy computations at the level of hybrid density functional theory were performed to investigate (a) methyl formate hydrolysis reaction in neutral aqueous media and (b) proton-transfer reaction within the active-site residues of the class C β-lactamase enzyme.

Fingerprint-Based Detection of Non-Local Effects in the Electronic Structure of a Simple Single Component Covalent System

Using fingerprints used mainly in machine learning schemes of the potential energy surface, we detect in a fully algorithmic way long range effects on local physical properties in a simple covalent system of carbon atoms. The fact that these long range effects exist for many configurations implies that atomistic simulation methods, such as force fields or modern machine learning schemes, that are based on locality assumptions, are limited in accuracy. We show that the basic driving mechanism for the long range effects is charge transfer. If the charge transfer is known, locality can be recovered for certain quantities such as the band structure energy.

The CECAM electronic structure library and the modular software development paradigm

First-principles electronic structure calculations are now accessible to a very large community of users across many disciplines, thanks to many successful software packages, some of which are described in this special issue. The traditional coding paradigm for such packages is monolithic, i.e., regardless of how modular its internal structure may be, the code is built independently from others, essentially from the compiler up, possibly with the exception of linear-algebra and message-passing libraries. This model has endured and been quite successful for decades. The successful evolution of the electronic structure methodology itself, however, has resulted in an increasing complexity and an ever longer list of features expected within all software packages, which implies a growing amount of replication between different packages, not only in the initial coding but, more importantly, every time a code needs to be re-engineered to adapt to the evolution of computer hardware architecture. The Electronic Structure Library (ESL) was initiated by CECAM (the European Centre for Atomic and Molecular Calculations) to catalyze a paradigm shift away from the monolithic model and promote modularization, with the ambition to extract common tasks from electronic structure codes and redesign them as open-source libraries available to everybody. Such libraries include "heavy-duty" ones that have the potential for a high degree of parallelization and adaptation to novel hardware within them, thereby separating the sophisticated computer science aspects of performance optimization and re-engineering from the computational science done by, e.g., physicists and chemists when implementing new ideas. We envisage that this modular paradigm will improve overall coding efficiency and enable specialists (whether they be computer scientists or computational scientists) to use their skills more effectively and will lead to a more dynamic evolution of software in the community as well as lower barriers to entry for new developers. The model comes with new challenges, though. The building and compilation of a code based on many interdependent libraries (and their versions) is a much more complex task than that of a code delivered in a single self-contained package. Here, we describe the state of the ESL, the different libraries it now contains, the short- and mid-term plans for further libraries, and the way the new challenges are faced. The ESL is a community initiative into which several pre-existing codes and their developers have contributed with their software and efforts, from which several codes are already benefiting, and which remains open to the community.

Electro-thermal transport in disordered nanostructures: a modeling perspective

Following the emergence of novel classes of atomic systems with amorphous active regions, device simulations had to rapidly evolve to devise strategies to account for the influence of disordered phases, defects, and interfaces into its core physical models. We review here how molecular dynamics and quantum transport can be combined to shed light on the performance of, for example, conductive bridging random access memories (CBRAM), a type of non-volatile memory. In particular, we show that electro-thermal effects play a critical role in such devices and therefore present a method based on density functional theory and the non-equilibrium Green's function formalism to accurately describe them. Three CBRAM configurations are investigated to illustrate the functionality of the proposed modeling approach.