Measuring Density-Driven Errors Using Kohn-Sham Inversion

Correcting Dispersion Corrections with Density-Corrected DFT

Almost all empirical parametrizations of dispersion corrections in DFT use only energy errors, thereby mixing functional and density-driven errors. We introduce density and dispersion-corrected DFT (D2C-DFT), a dual-calibration approach that accounts for density delocalization errors when parametrizing dispersion interactions. We simply exclude density-sensitive reactions from the training data. We find a significant reduction in both errors and variation among several semilocal functionals and their global hybrids when tailored dispersion corrections are employed with Hartree-Fock densities.

Targeting spectroscopic accuracy for dispersion bound systems from ab initio techniques: Translational eigenstates of Ne@C70

We investigate the endofullerene system Ne@C70 by constructing a three-dimensional Potential Energy Surface (PES) describing the translational motion of the Ne atom. This is constructed from electronic structure calculations from a plethora of methods, including MP2, SCS-MP2, SOS-MP2, RPA@PBE, and C(HF)-RPA, which were previously used for He@C60 in Panchagnula et al. [J. Chem. Phys. 160, 104303 (2024)], alongside B86bPBE-25X-XDM and B86bPBE-50X-XDM. The reduction in symmetry moving from C60 to C70 introduces a double well potential along the anisotropic direction, which forms a test of the sensitivity and effectiveness of the electronic structure methods. The nuclear Hamiltonian is diagonalized using a symmetrized double minimum basis set outlined in Panchagnula and Thom [J. Chem. Phys. 159, 164308 (2023)], with translational energies having error bars ±1 and ±2 cm-1. We find no consistency between electronic structure methods as they find a range of barrier heights and minima positions of the double well and different translational eigenspectra, which also differ from the Lennard-Jones (LJ) PES given in Mandziuk and Bačić [J. Chem. Phys. 101, 2126-2140 (1994)]. We find that generating effective LJ parameters for each electronic structure method cannot reproduce the full PES nor recreate the eigenstates, and this suggests that the LJ form of the PES, while simple, may not be best suited to describe these systems. Even though MP2 and RPA@PBE performed best for He@C60, due to the lack of concordance between all electronic structure methods, we require more experimental data in order to properly validate the choice.

Unconventional Error Cancellation Explains the Success of Hartree-Fock Density Functional Theory for Barrier Heights

Energy barriers, which control the rates of chemical reactions, are seriously underestimated by computationally efficient semilocal approximations for the exchange-correlation energy. The accuracy of a semilocal density functional approximation is strongly boosted for reaction barrier heights by evaluating that approximation non-self-consistently on Hartree-Fock electron densities, which has been known for ∼30 years. The conventional explanation is that the Hartree-Fock theory yields the more accurate density. This work presents a benchmark Kohn-Sham inversion of accurate coupled-cluster densities for the reaction H2 + F → HHF → H + HF and finds a strong, understandable cancellation between positive (excessively overcorrected) density-driven and large negative functional-driven errors (expected from stretched radical bonds in the transition state) within this Hartree-Fock density functional theory. This confirms earlier conclusions (Kaplan, A. D., et al. J. Chem. Theory Comput. 2023, 19, 532-543) based on 76 barrier heights and three less reliable, but less expensive, fully nonlocal density functional proxies for the exact density.

Consistent density functional theory-based description of ion hydration through density-corrected many-body representations

Delocalization error constrains the accuracy of density functional theory in describing molecular interactions in ion-water systems. Using Na+ and Cl- in water as model systems, we calculate the effects of delocalization error in the SCAN functional for describing ion-water and water-water interactions in hydrated ions, and demonstrate that density-corrected SCAN (DC-SCAN) predicts n-body and interaction energies with an accuracy approaching coupled cluster theory. The performance of DC-SCAN is size-consistent, maintaining an accurate description of molecular interactions well beyond the first solvation shell. Molecular dynamics simulations at ambient conditions with many-body MB-SCAN(DC) potentials, derived from the many-body expansion, predict the solvation structure of Na+ and Cl- in quantitative agreement with reference data, while simultaneously reproducing the structure of liquid water. Beyond rationalizing the accuracy of density-corrected models of ion hydration, our findings suggest that our unified density-corrected MB formalism holds great promise for efficient DFT-based simulations of condensed-phase systems with chemical accuracy.

Exact and Model Exchange-Correlation Potentials for Open-Shell Systems

The conventional approaches to the inverse density functional theory problem typically assume nondegeneracy of the Kohn-Sham (KS) eigenvalues, greatly hindering their use in open-shell systems. We present a generalization of the inverse density functional theory problem that can seamlessly admit degenerate KS eigenvalues. Additionally, we allow for fractional occupancy of the Kohn-Sham orbitals to also handle noninteracting ensemble-v-representable densities, as opposed to just noninteracting pure-v-representable densities. We present the exact exchange-correlation (XC) potentials for six open-shell systems─four atoms (Li, C, N, and O) and two molecules (CN and CH2)─using accurate ground-state densities from configuration interaction calculations. We compare these exact XC potentials with model XC potentials obtained using nonlocal (B3LYP, SCAN0) and local/semilocal (SCAN, PBE, PW92) XC functionals. Although the relative errors in the densities obtained from these DFT functionals are of O (10-3 to 10-2), the relative errors in the model XC potentials remain substantially large─O (10-1 to 100).

Corrected density functional theory and the random phase approximation: Improved accuracy at little extra cost

We recently introduced an efficient methodology to perform density-corrected Hartree-Fock density functional theory [DC(HF)-DFT] calculations and an extension to it we called "corrected" HF DFT [C(HF)-DFT] [Graf and Thom, J. Chem. Theory Comput. 19 5427-5438 (2023)]. In this work, we take a further step and combine C(HF)-DFT, augmented with a straightforward orbital energy correction, with the random phase approximation (RPA). We refer to the resulting methodology as corrected HF RPA [C(HF)-RPA]. We evaluate the proposed methodology across various RPA methods: direct RPA (dRPA), RPA with an approximate exchange kernel, and RPA with second-order screened exchange. C(HF)-dRPA demonstrates very promising performance; for RPA with exchange methods, on the other hand, we often find over-corrections.

Density-Corrected Density Functional Theory for Open Shells: How to Deal with Spin Contamination

Density functional theory (DFT) is usually used self-consistently to predict chemical properties, but the use of the Hartree-Fock (HF) density improves energetics in certain, well-characterized cases. Density-corrected (DC) DFT provides the theory behind this, but unrestricted Hartree-Fock (UHF) densities yield poor energetics in cases of strong spin contamination. Here we compare with restricted open-shell HF (ROHF) across 13 different functionals and two DC-DFT methods. For significant spin contamination, ROHF densities outperform UHF densities by as much as a factor of 3, depending on the energy functional, and ROHF-DFT improves over self-consistent DFT for most of the tested functionals. We refine the DC(HF)-DFT algorithm to use ROHF densities in cases of severe spin contamination.

Simple and Efficient Route toward Improved Energetics within the Framework of Density-Corrected Density Functional Theory

The crucial step in density-corrected Hartree-Fock density functional theory (DC(HF)-DFT) is to decide whether the density produced by the density functional for a specific calculation is erroneous and, hence, should be replaced by, in this case, the HF density. We introduce an indicator, based on the difference in noninteracting kinetic energies between DFT and HF calculations, to determine when the HF density is the better option. Our kinetic energy indicator directly compares the self-consistent density of the analyzed functional with the HF density, is size-intensive, reliable, and most importantly highly efficient. Moreover, we present a procedure that makes best use of the computed quantities necessary for DC(HF)-DFT by additionally evaluating a related hybrid functional and, in that way, not only "corrects" the density but also the functional itself; we call that procedure corrected Hartree-Fock density functional theory (C(HF)-DFT).

Toward routine Kohn-Sham inversion using the "Lieb-response" approach

Kohn-Sham (KS) inversion, in which the effective KS mean-field potential is found for a given density, provides insights into the nature of exact density functional theory (DFT) that can be exploited for the development of density functional approximations. Unfortunately, despite significant and sustained progress in both theory and software libraries, KS inversion remains rather difficult in practice, especially in finite basis sets. The present work presents a KS inversion method, dubbed the "Lieb-response" approach, that naturally works with existing Fock-matrix DFT infrastructure in finite basis sets, is numerically efficient, and directly provides meaningful matrix and energy quantities for pure-state and ensemble systems. Some additional work yields potential. It thus enables the routine inversion of even difficult KS systems, as illustrated in a variety of problems within this work, and provides outputs that can be used for embedding schemes or machine learning of density functional approximations. The effect of finite basis sets on KS inversion is also analyzed and investigated.

Extending density functional theory with near chemical accuracy beyond pure water

Density functional simulations of condensed phase water are typically inaccurate, due to the inaccuracies of approximate functionals. A recent breakthrough showed that the SCAN approximation can yield chemical accuracy for pure water in all its phases, but only when its density is corrected. This is a crucial step toward first-principles biosimulations. However, weak dispersion forces are ubiquitous and play a key role in noncovalent interactions among biomolecules, but are not included in the new approach. Moreover, naïve inclusion of dispersion in HF-SCAN ruins its high accuracy for pure water. Here we show that systematic application of the principles of density-corrected DFT yields a functional (HF-r²SCAN-DC4) which recovers and not only improves over HF-SCAN for pure water, but also captures vital noncovalent interactions in biomolecules, making it suitable for simulations of solutions.

Understanding Density-Driven Errors for Reaction Barrier Heights

Delocalization errors, such as charge-transfer and some self-interaction errors, plague computationally efficient and otherwise accurate density functional approximations (DFAs). Evaluating a semilocal DFA non-self-consistently on the Hartree-Fock (HF) density is often recommended as a computationally inexpensive remedy for delocalization errors. For sophisticated meta-GGAs like SCAN, this approach can achieve remarkable accuracy. This HF-DFT (also known as DFA@HF) is often presumed to work, when it significantly improves over the DFA, because the HF density is more accurate than the self-consistent DFA density in those cases. By applying the metrics of density-corrected density functional theory (DFT), we show that HF-DFT works for barrier heights by making a localizing charge-transfer error or density overcorrection, thereby producing a somewhat reliable cancellation of density- and functional-driven errors for the energy. A quantitative analysis of the charge-transfer errors in a few randomly selected transition states confirms this trend. We do not have the exact functional and electron densities that would be needed to evaluate the exact density- and functional-driven errors for the large BH76 database of barrier heights. Instead, we have identified and employed three fully nonlocal proxy functionals (SCAN 50% global hybrid, range-separated hybrid LC-ωPBE, and SCAN-FLOSIC) and their self-consistent proxy densities. These functionals are chosen because they yield reasonably accurate self-consistent barrier heights and because their self-consistent total energies are nearly piecewise linear in fractional electron number─two important points of similarity to the exact functional. We argue that density-driven errors of the energy in a self-consistent density functional calculation are second order in the density error and that large density-driven errors arise primarily from incorrect electron transfers over length scales larger than the diameter of an atom.

Reconstruction of Exchange-Correlation Potentials from Their Matrix Representations

Within a basis set of one-electron functions that form linearly independent products (LIPs), it is always possible to construct a unique local (multiplicative) real-space potential that is precisely equivalent to an arbitrary given operator. Although standard basis sets of quantum chemistry rarely form LIPs in a numerical sense, occupied and low-lying virtual canonical Kohn-Sham orbitals often do so, at least for small atoms and molecules. Using these principles, we construct atomic and molecular exchange-correlation potentials from their matrix representations in LIP basis sets of occupied canonical Kohn-Sham orbitals. The reconstructions are found to imitate the original potentials in a consistent but exaggerated way. Since the original and reconstructed potentials produce the same ground-state electron density and energy within the associated LIP basis set, the procedure may be regarded as a rigorous solution to the Kohn-Sham inversion problem within the subspace spanned by the occupied Kohn-Sham orbitals.

How Good Is the Density-Corrected SCAN Functional for Neutral and Ionic Aqueous Systems, and What Is So Right about the Hartree-Fock Density?

Density functional theory (DFT) is the most widely used electronic structure method, due to its simplicity and cost effectiveness. The accuracy of a DFT calculation depends not only on the choice of the density functional approximation (DFA) adopted but also on the electron density produced by the DFA. SCAN is a modern functional that satisfies all known constraints for meta-GGA functionals. The density-driven errors, defined as energy errors arising from errors of the self-consistent DFA electron density, can hinder SCAN from achieving chemical accuracy in some systems, including water. Density-corrected DFT (DC-DFT) can alleviate this shortcoming by adopting a more accurate electron density which, in most applications, is the electron density obtained at the Hartree-Fock level of theory due to its relatively low computational cost. In this work, we present extensive calculations aimed at determining the accuracy of the DC-SCAN functional for various aqueous systems. DC-SCAN (SCAN@HF) shows remarkable consistency in reproducing reference data obtained at the coupled cluster level of theory, with minimal loss of accuracy. Density-driven errors in the description of ionic aqueous clusters are thoroughly investigated. By comparison with the orbital-optimized CCD density in the water dimer, we find that the self-consistent SCAN density transfers a spurious fraction of an electron across the hydrogen bond to the hydrogen atom (H^*, covalently bound to the donor oxygen atom) from the acceptor (O_A) and donor (O_D) oxygen atoms, while HF makes a much smaller spurious transfer in the opposite direction, consistent with DC-SCAN (SCAN@HF) reduction of SCAN overbinding due to delocalization error. While LDA seems to be the conventional extreme of density delocalization error, and HF the conventional extreme of (usually much smaller) density localization error, these two densities do not quite yield the conventional range of density-driven error in energy differences. Finally, comparisons of the DC-SCAN results with those obtained with the Fermi-Löwdin orbital self-interaction correction (FLOSIC) method show that DC-SCAN represents a more accurate approach to reducing density-driven errors in SCAN calculations of ionic aqueous clusters. While the HF density is superior to that of SCAN for noncompact water clusters, the opposite is true for the compact water molecule with exactly 10 electrons.

Core-hole delocalization for modeling X-ray spectroscopies: A cautionary tale

The influence of core-hole delocalization for X-ray photoelectron, X-ray absorption, and X-ray emission spectrum calculations is investigated in detail, using approaches including response theory, transition-potential methods, and ground state schemes. The question of a localized/delocalized vacancy is relevant for systems with symmetrically equivalent atoms, as well as near-degeneracies which can distribute the core-orbitals over several atoms. We show that issues relating to core-hole delocalization are present for calculations considering explicit core-hole states, e.g. when using a core-excited or core-ionized reference state, or for fractional occupation numbers. Including electron correlation eventually alleviates the issues, but even using CCSD(T) there is a noticable discrepancy between core-ionization energies obtained with a localized and delocalized core-hole (0.5 eV for the carbon K-edge). Within density functional theory, the discrepancy correlates to the exchange interaction involving the core orbitals of the same spin symmetry as the delocalized core-hole. The use of a localized core-hole allows for a reasonably good inclusion of relaxation at lower level of theory, whereas the proper symmetry solution involving a delocalized core-hole requires higher levels of theory to account for the correlation effects involved in orbital relaxation. For linear response methods, we further show that if X-ray absorption spectra are modelled by considering symmetry-unique sets of atoms, care has to be taken such that there are no delocalizations of the core orbitals, which would otherwise introduce shifts in absolute energies and relative features.

Assessing the Interplay between Functional-Driven and Density-Driven Errors in DFT Models of Water

We investigate the interplay between functional-driven and density-driven errors in different density functional approximations within density functional theory (DFT) and the implications of these errors for simulations of water with DFT-based data-driven potentials. Specifically, we quantify density-driven errors in two widely used dispersion-corrected functionals derived within the generalized gradient approximation (GGA), namely BLYP-D3 and revPBE-D3, and two modern meta-GGA functionals, namely strongly constrained and appropriately normed (SCAN) and B97M-rV. The effects of functional-driven and density-driven errors on the interaction energies are first assessed for the water clusters of the BEGDB dataset. Further insights into the nature of functional-driven errors are gained from applying the absolutely localized molecular orbital energy decomposition analysis (ALMO-EDA) to the interaction energies, which demonstrates that functional-driven errors are strongly correlated with the nature of the interactions. We discuss cases where density-corrected DFT (DC-DFT) models display higher accuracy than the original DFT models and cases where reducing the density-driven errors leads to larger deviations from the reference energies due to the presence of large functional-driven errors. Finally, molecular dynamics simulations are performed with data-driven many-body potentials derived from DFT and DC-DFT data to determine the effect that minimizing density-driven errors has on the description of liquid water. Besides rationalizing the performance of widely used DFT models of water, we believe that our findings unveil fundamental relations between the shortcomings of some common DFT approximations and the requirements for accurate descriptions of molecular interactions, which will aid the development of a consistent, DFT-based framework for the development of data-driven and machine-learned potentials for simulations of condensed-phase systems.

Improving Results by Improving Densities: Density-Corrected Density Functional Theory

Density functional theory (DFT) calculations have become widespread in both chemistry and materials, because they usually provide useful accuracy at much lower computational cost than wavefunction-based methods. All practical DFT calculations require an approximation to the unknown exchange-correlation energy, which is then used self-consistently in the Kohn-Sham scheme to produce an approximate energy from an approximate density. Density-corrected DFT is simply the study of the relative contributions to the total energy error. In the vast majority of DFT calculations, the error due to the approximate density is negligible. But with certain classes of functionals applied to certain classes of problems, the density error is sufficiently large as to contribute to the energy noticeably, and its removal leads to much better results. These problems include reaction barriers, torsional barriers involving π-conjugation, halogen bonds, radicals and anions, most stretched bonds, etc. In all such cases, use of a more accurate density significantly improves performance, and often the simple expedient of using the Hartree-Fock density is enough. This Perspective explains what DC-DFT is, where it is likely to improve results, and how DC-DFT can produce more accurate functionals. We also outline challenges and prospects for the field.

Elevating density functional theory to chemical accuracy for water simulations through a density-corrected many-body formalism

Density functional theory (DFT) has been extensively used to model the properties of water. Albeit maintaining a good balance between accuracy and efficiency, no density functional has so far achieved the degree of accuracy necessary to correctly predict the properties of water across the entire phase diagram. Here, we present density-corrected SCAN (DC-SCAN) calculations for water which, minimizing density-driven errors, elevate the accuracy of the SCAN functional to that of "gold standard" coupled-cluster theory. Building upon the accuracy of DC-SCAN within a many-body formalism, we introduce a data-driven many-body potential energy function, MB-SCAN(DC), that quantitatively reproduces coupled cluster reference values for interaction, binding, and individual many-body energies of water clusters. Importantly, molecular dynamics simulations carried out with MB-SCAN(DC) also reproduce the properties of liquid water, which thus demonstrates that MB-SCAN(DC) is effectively the first DFT-based model that correctly describes water from the gas to the liquid phase.

Inverse Kohn-Sham Density Functional Theory: Progress and Challenges

Inverse Kohn-Sham (iKS) methods are needed to fully understand the one-to-one mapping between densities and potentials on which density functional theory is based. They can contribute to the construction of empirical exchange-correlation functionals and to the development of techniques for density-based embedding. Unlike the forward Kohn-Sham problems, numerical iKS problems are ill-posed and can be unstable. We discuss some of the fundamental and practical difficulties of iKS problems with constrained-optimization methods on finite basis sets. Various factors that affect the performance are systematically compared and discussed, both analytically and numerically, with a focus on two of the most practical methods: the Wu-Yang method (WY) and the partial differential equation constrained optimization (PDE-CO). Our analysis of the WY and PDE-CO highlights the limitation of finite basis sets. We introduce new ideas to make iKS problems more tractable, provide an overall strategy for performing numerical density-to-potential inversions, and discuss challenges and future directions.

Improved Spin-State Energy Differences of Fe(II) Molecular and Crystalline Complexes via the Hubbard U-Corrected Density

We recently showed that the DFT+U approach with a linear-response U yields adiabatic energy differences biased toward high spin [Mariano et al. J. Chem. Theory Comput. 2020, 16, 6755-6762]. Such bias is removed here by employing a density-corrected DFT approach where the PBE functional is evaluated on the Hubbard U-corrected density. The adiabatic energy differences of six Fe(II) molecular complexes computed using this approach, named PBE[U] here, are in excellent agreement with coupled cluster-corrected CASPT2 values for both weak- and strong-field ligands resulting in a mean absolute error (MAE) of 0.44 eV, smaller than that of the recently proposed Hartree-Fock density-corrected DFT (1.22 eV) and any other tested functional, including the best performer TPSSh (0.49 eV). We take advantage of the computational efficiency of this approach and compute the adiabatic energy differences of five molecular crystals using PBE[U] with periodic boundary conditions. The results show, again, an excellent agreement (MAE = 0.07 eV) with experimentally extracted values and a superior performance compared with the best performers M06-L (MAE = 0.08 eV) and TPSSh (MAE = 0.31 eV) computed on molecular fragments.

Explaining and Fixing DFT Failures for Torsional Barriers

Most torsional barriers are predicted with high accuracies (about 1 kJ/mol) by standard semilocal functionals, but a small subset was found to have much larger errors. We created a database of almost 300 carbon-carbon torsional barriers, including 12 poorly behaved barriers, that stem from the Y═C-X group, where Y is O or S and X is a halide. Functionals with enhanced exchange mixing (about 50%) worked well for all barriers. We found that poor actors have delocalization errors caused by hyperconjugation. These problematic calculations are density-sensitive (i.e., DFT predictions change noticeably with the density), and using HF densities (HF-DFT) fixes these issues. For example, conventional B3LYP performs as accurately as exchange-enhanced functionals if the HF density is used. For long-chain conjugated molecules, HF-DFT can be much better than exchange-enhanced functionals. We suggest that HF-PBE0 has the best overall performance.

Density Sensitivity of Empirical Functionals

Empirical fitting of parameters in approximate density functionals is common. Such fits conflate errors in the self-consistent density with errors in the energy functional, but density-corrected DFT (DC-DFT) separates these two. We illustrate with catastrophic failures of a toy functional applied to H₂⁺ at varying bond lengths, where the standard fitting procedure misses the exact functional; Grimme's D3 fit to noncovalent interactions, which can be contaminated by large density errors such as in the WATER27 and B30 data sets; and double-hybrids trained on self-consistent densities, which can perform poorly on systems with density-driven errors. In these cases, more accurate results are found at no additional cost by using Hartree-Fock (HF) densities instead of self-consistent densities. For binding energies of small water clusters, errors are greatly reduced. Range-separated hybrids with 100% HF at large distances suffer much less from this effect.