1
|
Di Felice R, Mayes ML, Richard RM, Williams-Young DB, Chan GKL, de Jong WA, Govind N, Head-Gordon M, Hermes MR, Kowalski K, Li X, Lischka H, Mueller KT, Mutlu E, Niklasson AMN, Pederson MR, Peng B, Shepard R, Valeev EF, van Schilfgaarde M, Vlaisavljevich B, Windus TL, Xantheas SS, Zhang X, Zimmerman PM. A Perspective on Sustainable Computational Chemistry Software Development and Integration. J Chem Theory Comput 2023; 19:7056-7076. [PMID: 37769271 PMCID: PMC10601486 DOI: 10.1021/acs.jctc.3c00419] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Indexed: 09/30/2023]
Abstract
The power of quantum chemistry to predict the ground and excited state properties of complex chemical systems has driven the development of computational quantum chemistry software, integrating advances in theory, applied mathematics, and computer science. The emergence of new computational paradigms associated with exascale technologies also poses significant challenges that require a flexible forward strategy to take full advantage of existing and forthcoming computational resources. In this context, the sustainability and interoperability of computational chemistry software development are among the most pressing issues. In this perspective, we discuss software infrastructure needs and investments with an eye to fully utilize exascale resources and provide unique computational tools for next-generation science problems and scientific discoveries.
Collapse
Affiliation(s)
- Rosa Di Felice
- Departments
of Physics and Astronomy and Quantitative and Computational Biology, University of Southern California, Los Angeles, California 90089, United States
- CNR-NANO
Modena, Modena 41125, Italy
| | - Maricris L. Mayes
- Department
of Chemistry and Biochemistry, University
of Massachusetts Dartmouth, North Dartmouth, Massachusetts 02747, United States
| | | | | | - Garnet Kin-Lic Chan
- Division
of Chemistry and Chemical Engineering, California
Institute of Technology, Pasadena, California 91125, United States
| | - Wibe A. de Jong
- Lawrence
Berkeley National Laboratory, Berkeley, California 94720, United States
| | - Niranjan Govind
- Physical
Sciences Division, Pacific Northwest National
Laboratory, Richland, Washington 99354, United States
| | - Martin Head-Gordon
- Pitzer Center
for Theoretical Chemistry, Department of Chemistry, University of California, Berkeley, California 94720, United States
| | - Matthew R. Hermes
- Department
of Chemistry, Chicago Center for Theoretical Chemistry, University of Chicago, Chicago, Illinois 60637, United States
| | - Karol Kowalski
- Physical
Sciences Division, Pacific Northwest National
Laboratory, Richland, Washington 99354, United States
| | - Xiaosong Li
- Department
of Chemistry, University of Washington, Seattle, Washington 98195, United States
| | - Hans Lischka
- Department
of Chemistry and Biochemistry, Texas Tech
University, Lubbock, Texas 79409, United States
| | - Karl T. Mueller
- Physical
and Computational Sciences Directorate, Pacific Northwest National Laboratory, Richland, Washington 99354, United States
| | - Erdal Mutlu
- Advanced
Computing, Mathematics, and Data Division, Pacific Northwest National Laboratory, Richland, Washington 99354, United States
| | - Anders M. N. Niklasson
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, United States
| | - Mark R. Pederson
- Department
of Physics, The University of Texas at El
Paso, El Paso, Texas 79968, United States
| | - Bo Peng
- Physical
Sciences Division, Pacific Northwest National
Laboratory, Richland, Washington 99354, United States
| | - Ron Shepard
- Chemical
Sciences and Engineering Division, Argonne
National Laboratory, Lemont, Illinois 60439, United States
| | - Edward F. Valeev
- Department
of Chemistry, Virginia Tech, Blacksburg, Virginia 24061, United States
| | | | - Bess Vlaisavljevich
- Department
of Chemistry, University of South Dakota, Vermillion, South Dakota 57069, United States
| | - Theresa L. Windus
- Department
of Chemistry, Iowa State University and
Ames Laboratory, Ames, Iowa 50011, United States
| | - Sotiris S. Xantheas
- Department
of Chemistry, University of Washington, Seattle, Washington 98195, United States
- Advanced
Computing, Mathematics and Data Division, Pacific Northwest National Laboratory, Richland, Washington 99354, United States
| | - Xing Zhang
- Division
of Chemistry and Chemical Engineering, California
Institute of Technology, Pasadena, California 91125, United States
| | - Paul M. Zimmerman
- Department
of Chemistry, University of Michigan, Ann Arbor, Michigan 48109, United States
| |
Collapse
|
2
|
Finkelstein J, Negre CFA, Fattebert JL. A fast, dense Chebyshev solver for electronic structure on GPUs. J Chem Phys 2023; 159:101101. [PMID: 37694745 DOI: 10.1063/5.0164255] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 08/14/2023] [Indexed: 09/12/2023] Open
Abstract
Matrix diagonalization is almost always involved in computing the density matrix needed in quantum chemistry calculations. In the case of modest matrix sizes (≲4000), performance of traditional dense diagonalization algorithms on modern GPUs is underwhelming compared to the peak performance of these devices. This motivates the exploration of alternative algorithms better suited to these types of architectures. We newly derive, and present in detail, an existing Chebyshev expansion algorithm [Liang et al., J. Chem. Phys. 119, 4117-4125 (2003)] whose number of required matrix multiplications scales with the square root of the number of terms in the expansion. Focusing on dense matrices of modest size, our implementation on GPUs results in large speed ups when compared to diagonalization. Additionally, we improve upon this existing method by capitalizing on the inherent task parallelism and concurrency in the algorithm. This improvement is implemented on GPUs by using CUDA and HIP streams via the MAGMA library and leads to a significant speed up over the serial-only approach for smaller (≲1000) matrix sizes. Finally, we apply our technique to a model system with a high density of states around the Fermi level, which typically presents significant challenges.
Collapse
Affiliation(s)
- Joshua Finkelstein
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
| | - Christian F A Negre
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
| | - Jean-Luc Fattebert
- Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37830, USA
| |
Collapse
|
3
|
Khadatkar S, Motamarri P. Subspace recursive Fermi-operator expansion strategies for large-scale DFT eigenvalue problems on HPC architectures. J Chem Phys 2023; 159:031102. [PMID: 37470423 DOI: 10.1063/5.0150287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Accepted: 06/19/2023] [Indexed: 07/21/2023] Open
Abstract
Quantum mechanical calculations for material modeling using Kohn-Sham density functional theory (DFT) involve the solution of a nonlinear eigenvalue problem for N smallest eigenvector-eigenvalue pairs, with N proportional to the number of electrons in the material system. These calculations are computationally demanding and have asymptotic cubic scaling complexity with the number of electrons. Large-scale matrix eigenvalue problems arising from the discretization of the Kohn-Sham DFT equations employing a systematically convergent basis traditionally rely on iterative orthogonal projection methods, which are shown to be computationally efficient and scalable on massively parallel computing architectures. However, as the size of the material system increases, these methods are known to incur dominant computational costs through the Rayleigh-Ritz projection step of the discretized Kohn-Sham Hamiltonian matrix and the subsequent subspace diagonalization of the projected matrix. This work explores the potential of polynomial expansion approaches based on recursive Fermi-operator expansion as an alternative to the subspace diagonalization of the projected Hamiltonian matrix to reduce the computational cost. Subsequently, we perform a detailed comparison of various recursive polynomial expansion approaches to the traditional approach of explicit diagonalization on both multi-node central processing unit and graphics processing unit architectures and assess their relative performance in terms of accuracy, computational efficiency, scaling behavior, and energy efficiency.
Collapse
Affiliation(s)
- Sameer Khadatkar
- Department of Computational and Data Sciences, Indian Institute of Science, Bengaluru 560012, India
| | - Phani Motamarri
- Department of Computational and Data Sciences, Indian Institute of Science, Bengaluru 560012, India
| |
Collapse
|
4
|
Woo J, Kim S, Kim WY. Dynamic Precision Approach for Accelerating Large-Scale Eigenvalue Solvers in Electronic Structure Calculations on Graphics Processing Units. J Chem Theory Comput 2023; 19:1457-1465. [PMID: 36812094 DOI: 10.1021/acs.jctc.2c00983] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Abstract
Single precision (SP) arithmetic can be greatly accelerated as compared to double precision (DP) arithmetic on graphics processing units (GPUs). However, the use of SP in the whole process of electronic structure calculations is inappropriate for the required accuracy. We propose a 3-fold dynamic precision approach for accelerated calculations but still with the accuracy of DP. Here, SP, DP, and mixed precision are dynamically switched during an iterative diagonalization process. We applied this approach to the locally optimal block preconditioned conjugate gradient method to accelerate a large-scale eigenvalue solver for the Kohn-Sham equation. We determined a proper threshold for switching each precision scheme by examining the convergence pattern on the eigenvalue solver only with the kinetic energy operator of the Kohn-Sham Hamiltonian. As a result, we achieved up to 8.53× and 6.60× speedups for band structure and self-consistent field calculations, respectively, for test systems under various boundary conditions on NVIDIA GPUs.
Collapse
Affiliation(s)
- Jeheon Woo
- Department of Chemistry, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Republic of Korea
| | - Seonghwan Kim
- Department of Chemistry, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Republic of Korea
| | - Woo Youn Kim
- Department of Chemistry, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Republic of Korea
| |
Collapse
|
5
|
Negre CFA, Wall ME, Niklasson AMN. Graph-based quantum response theory and shadow Born-Oppenheimer molecular dynamics. J Chem Phys 2023; 158:074108. [PMID: 36813723 DOI: 10.1063/5.0137119] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2023] Open
Abstract
Graph-based linear scaling electronic structure theory for quantum-mechanical molecular dynamics simulations [A. M. N. Niklasson et al., J. Chem. Phys. 144, 234101 (2016)] is adapted to the most recent shadow potential formulations of extended Lagrangian Born-Oppenheimer molecular dynamics, including fractional molecular-orbital occupation numbers [A. M. N. Niklasson, J. Chem. Phys. 152, 104103 (2020) and A. M. N. Niklasson, Eur. Phys. J. B 94, 164 (2021)], which enables stable simulations of sensitive complex chemical systems with unsteady charge solutions. The proposed formulation includes a preconditioned Krylov subspace approximation for the integration of the extended electronic degrees of freedom, which requires quantum response calculations for electronic states with fractional occupation numbers. For the response calculations, we introduce a graph-based canonical quantum perturbation theory that can be performed with the same natural parallelism and linear scaling complexity as the graph-based electronic structure calculations for the unperturbed ground state. The proposed techniques are particularly well-suited for semi-empirical electronic structure theory, and the methods are demonstrated using self-consistent charge density-functional tight-binding theory both for the acceleration of self-consistent field calculations and for quantum-mechanical molecular dynamics simulations. Graph-based techniques combined with the semi-empirical theory enable stable simulations of large, complex chemical systems, including tens-of-thousands of atoms.
Collapse
Affiliation(s)
- Christian F A Negre
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
| | - Michael E Wall
- Computer, Computational, and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
| | - Anders M N Niklasson
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
| |
Collapse
|
6
|
Finkelstein J, Rubensson EH, Mniszewski SM, Negre CFA, Niklasson AMN. Quantum Perturbation Theory Using Tensor Cores and a Deep Neural Network. J Chem Theory Comput 2022; 18:4255-4268. [PMID: 35670603 DOI: 10.1021/acs.jctc.2c00274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
Time-independent quantum response calculations are performed using Tensor cores. This is achieved by mapping density matrix perturbation theory onto the computational structure of a deep neural network. The main computational cost of each deep layer is dominated by tensor contractions, i.e., dense matrix-matrix multiplications, in mixed-precision arithmetics, which achieves close to peak performance. Quantum response calculations are demonstrated and analyzed using self-consistent charge density-functional tight-binding theory as well as coupled-perturbed Hartree-Fock theory. For linear response calculations, a novel parameter-free convergence criterion is presented that is well-suited for numerically noisy low-precision floating point operations and we demonstrate a peak performance of almost 200 Tflops using the Tensor cores of two Nvidia A100 GPUs.
Collapse
Affiliation(s)
- Joshua Finkelstein
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545 United States
| | - Emanuel H Rubensson
- Division of Scientific Computing, Department of Information Technology, Uppsala University, Box 337, Uppsala SE-751 05, Sweden
| | - Susan M Mniszewski
- Computer, Computational, and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, United States
| | - Christian F A Negre
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545 United States
| | - Anders M N Niklasson
- Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545 United States
| |
Collapse
|
7
|
Negre CFA, Lopez-Bezanilla A, Zhang Y, Akrobotu PD, Mniszewski SM, Tretiak S, Dub PA. Toward a QUBO-Based Density Matrix Electronic Structure Method. J Chem Theory Comput 2022; 18:4177-4185. [PMID: 35658437 DOI: 10.1021/acs.jctc.2c00090] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
Density matrix electronic structure theory is used in many quantum chemistry methods to "alleviate" the computational cost that arises from directly using wave functions. Although density matrix based methods are computationally more efficient than wave function based methods, significant computational effort is involved. Because the Schrödinger equation needs to be solved as an eigenvalue problem, the time-to-solution scales cubically with the system size in mean-field type approaches such as Hartree-Fock and density functional theory and is solved as many times in order to reach charge or field self-consistency. We hereby propose and study a method to compute the density matrix by using a quadratic unconstrained binary optimization (QUBO) solver. This method could be useful to solve the problem with quantum computers and, more specifically, quantum annealers. Our proposed approach is based on a direct construction of the density matrix using a QUBO eigensolver. We explore the main parameters of the algorithm focusing on precision and efficiency. We show that, while direct construction of the density matrix using a QUBO formulation is possible, the efficiency and precision have room for improvement. Moreover, calculations performed with quantum annealing on D-Wave's new Advantage quantum computer are compared with results obtained with classical simulated annealing, further highlighting some problems of the proposed method. We also suggest alternative methods that could lead to a more efficient QUBO-based density matrix construction.
Collapse
Affiliation(s)
| | | | | | - Prosper D Akrobotu
- Department of Mathematical Sciences, University of Texas at Dallas, Richardson, Texas 75080, United States
| | | | | | | |
Collapse
|