Enhancing Adaptive Physics Refinement Simulations Through the Addition of Realistic Red Blood Cell Counts

Simulations of cancer cell transport require accurately modeling mm-scale and longer trajectories through a circulatory system containing trillions of deformable red blood cells, whose intercellular interactions require submicron fidelity. Using a hybrid CPU-GPU approach, we extend the advanced physics refinement (APR) method to couple a finely-resolved region of explicitly-modeled red blood cells to a coarsely-resolved bulk fluid domain. We further develop algorithms that: capture the dynamics at the interface of differing viscosities, maintain hematocrit within the cell-filled volume, and move the finely-resolved region and encapsulated cells while tracking an individual cancer cell. Comparison to a fully-resolved fluid-structure interaction model is presented for verification. Finally, we use the advanced APR method to simulate cancer cell transport over a mm-scale distance while maintaining a local region of RBCs, using a fraction of the computational power required to run a fully-resolved model.

High Performance Adaptive Physics Refinement to Enable Large-Scale Tracking of Cancer Cell Trajectory

The ability to track simulated cancer cells through the circulatory system, important for developing a mechanistic understanding of metastatic spread, pushes the limits of today's supercomputers by requiring the simulation of large fluid volumes at cellular-scale resolution. To overcome this challenge, we introduce a new adaptive physics refinement (APR) method that captures cellular-scale interaction across large domains and leverages a hybrid CPU-GPU approach to maximize performance. Through algorithmic advances that integrate multi-physics and multi-resolution models, we establish a finely resolved window with explicitly modeled cells coupled to a coarsely resolved bulk fluid domain. In this work we present multiple validations of the APR framework by comparing against fully resolved fluid-structure interaction methods and employ techniques, such as latency hiding and maximizing memory bandwidth, to effectively utilize heterogeneous node architectures. Collectively, these computational developments and performance optimizations provide a robust and scalable framework to enable system-level simulations of cancer cell transport.

Propagation pattern for moment representation of the lattice Boltzmann method

A propagation pattern for the moment representation of the regularized lattice Boltzmann method (LBM) in three dimensions is presented. Using effectively lossless compression, the simulation state is stored as a set of moments of the lattice Boltzmann distribution function, instead of the distribution function itself. An efficient cache-aware propagation pattern for this moment representation has the effect of substantially reducing both the storage and memory bandwidth required for LBM simulations. This paper extends recent work with the moment representation by expanding the performance analysis on central processing unit (CPU) architectures, considering how boundary conditions are implemented, and demonstrating the effectiveness of the moment representation on a graphics processing unit (GPU) architecture.

Multi-GPU Immersed Boundary Method Hemodynamics Simulations

Large-scale simulations of blood flow that resolve the 3D deformation of each comprising cell are increasingly popular owing to algorithmic developments in conjunction with advances in compute capability. Among different approaches for modeling cell-resolved hemodynamics, fluid structure interaction (FSI) algorithms based on the immersed boundary method are frequently employed for coupling separate solvers for the background fluid and the cells within one framework. GPUs can accelerate these simulations; however, both current pre-exascale and future exascale CPU-GPU heterogeneous systems face communication challenges critical to performance and scalability. We describe, to our knowledge, the largest distributed GPU-accelerated FSI simulations of high hematocrit cell-resolved flows with over 17 million red blood cells. We compare scaling on a fat node system with six GPUs per node and on a system with a single GPU per node. Through comparison between the CPU- and GPU-based implementations, we identify the costs of data movement in multiscale multi-grid FSI simulations on heterogeneous systems and show it to be the greatest performance bottleneck on the GPU.

Numerical simulation of a compound capsule in a constricted microchannel

Simulations of the passage of eukaryotic cells through a constricted channel aid in studying the properties of cancer cells and their transport in the bloodstream. Compound capsules, which explicitly model the outer cell membrane and nuclear lamina, have the potential to improve computational model fidelity. However, general simulations of compound capsules transiting a constricted microchannel have not been conducted and the influence of the compound capsule model on computational performance is not well known. In this study, we extend a parallel hemodynamics application to simulate the fluid-structure interaction between compound capsules and fluid. With this framework, we compare the deformation of simple and compound capsules in constricted microchannels, and explore how deformation depends on the capillary number and on the volume fraction of the inner membrane. The computational framework's parallel performance in this setting is evaluated and future development lessons are discussed.

Electron Elevator: Excitations across the Band Gap via a Dynamical Gap State

We use time-dependent density functional theory to study self-irradiated Si. We calculate the electronic stopping power of Si in Si by evaluating the energy transferred to the electrons per unit path length by an ion of kinetic energy from 1 eV to 100 keV moving through the host. Electronic stopping is found to be significant below the threshold velocity normally identified with transitions across the band gap. A structured crossover at low velocity exists in place of a hard threshold. An analysis of the time dependence of the transition rates using coupled linear rate equations enables one of the excitation mechanisms to be clearly identified: a defect state induced in the gap by the moving ion acts like an elevator and carries electrons across the band gap.

Massively parallel simulations of hemodynamics in the primary large arteries of the human vasculature

We present a computational model of three-dimensional and unsteady hemodynamics within the primary large arteries in the human on 1,572,864 cores of the IBM Blue Gene/Q. Models of large regions of the circulatory system are needed to study the impact of local factors on global hemodynamics and to inform next generation drug delivery mechanisms. The HARVEY code successfully addresses key challenges that can hinder effective solution of image-based hemodynamics on contemporary supercomputers, such as limited memory capacity and bandwidth, flexible load balancing, and scalability. This work is the first demonstration of large fluid dynamics simulations of the aortofemoral region of the circulatory system at resolutions as small as 10 μm.

Lithium ion solvation and diffusion in bulk organic electrolytes from first-principles and classical reactive molecular dynamics

Lithium-ion battery performance is strongly influenced by the ionic conductivity of the electrolyte, which depends on the speed at which Li ions migrate across the cell and relates to their solvation structure. The choice of solvent can greatly impact both the solvation and diffusivity of Li ions. In this work, we used first-principles molecular dynamics to examine the solvation and diffusion of Li ions in the bulk organic solvents ethylene carbonate (EC), ethyl methyl carbonate (EMC), and a mixture of EC and EMC. We found that Li ions are solvated by either carbonyl or ether oxygen atoms of the solvents and sometimes by the PF6(-) anion. Li(+) prefers a tetrahedrally coordinated first solvation shell regardless of which species are involved, with the specific preferred solvation structure dependent on the organic solvent. In addition, we calculated Li diffusion coefficients in each electrolyte, finding slightly larger diffusivities in the linear carbonate EMC compared to the cyclic carbonate EC. The magnitude of the diffusion coefficient correlates with the strength of Li(+) solvation. Corresponding analysis for the PF6(-) anion shows greater diffusivity associated with a weakly bound, poorly defined first solvation shell. These results can be used to aid in the design of new electrolytes to improve Li-ion battery performance.

Towards real-time simulation of cardiac electrophysiology in a human heart at high resolution

We have developed the capability to rapidly simulate cardiac electrophysiological phenomena in a human heart discretised at a resolution comparable with the length of a cardiac myocyte. Previous scientific investigation has generally invoked simplified geometries or coarse-resolution hearts, with simulation duration limited to 10s of heartbeats. Using state-of-the-art high-performance computing techniques coupled with one of the most powerful computers available (the 20 PFlop/s IBM BlueGene/Q at Lawrence Livermore National Laboratory), high-resolution simulation of the human heart can now be carried out over 1200 times faster compared with published results in the field. We demonstrate the utility of this capability by simulating, for the first time, the formation of transmural re-entrant waves in a 3D human heart. Such wave patterns are thought to underlie Torsades de Pointes, an arrhythmia that indicates a high risk of sudden cardiac death. Our new simulation capability has the potential to impact a multitude of applications in medicine, pharmaceuticals and implantable devices.

Optical properties of passivated silicon nanoclusters: The role of synthesis

The effect of preparation conditions on the structural and optical properties of silicon nanoparticles is investigated. Nanoscale reconstructions, unique to curved nanosurfaces, are presented for silicon nanocrystals and shown to have lower energy and larger optical gaps than bulk-derived structures. We find that high-temperature synthesis processes can produce metastable noncrystalline nanostructures with different core structures than bulk-derived crystalline clusters. The type of core structure that forms from a given synthesis process may depend on the passivation mechanism and time scale. The effect of oxygen on the optical of different types of silicon structures is calculated. In contrast to the behavior of bulklike nanostructures, for noncrystalline and reconstructed crystalline structures surface oxygen atoms do not decrease the gap. In some cases, the presence of oxygen atoms at the nanocluster surface can significantly increase the optical absorption gap, due to decreased angular distortion of the silicon bonds. The relationship between strain and the optical gap in silicon nanoclusters is discussed.

Towards an assessment of the accuracy of density functional theory for first principles simulations of water

A series of Car-Parrinello (CP) molecular dynamics simulations of water are presented, aimed at assessing the accuracy of density functional theory in describing the structural and dynamical properties of water at ambient conditions. We found negligible differences in structural properties obtained using the Perdew-Burke-Ernzerhof or the Becke-Lee-Yang-Parr exchange and correlation energy functionals; we also found that size effects, although not fully negligible when using 32 molecule cells, are rather small. In addition, we identified a wide range of values of the fictitious electronic mass (micro) entering the CP Lagrangian for which the electronic ground state is accurately described, yielding trajectories and average properties that are independent of the value chosen. However, care must be exercised not to carry out simulations outside this range, where structural properties may artificially depend on micro. In the case of an accurate description of the electronic ground state, and in the absence of proton quantum effects, we obtained an oxygen-oxygen correlation function that is overstructured compared to experiment, and a diffusion coefficient which is approximately ten times smaller.

Influence of synthesis conditions on the structural and optical properties of passivated silicon nanoclusters

First-principles molecular dynamics and quantum Monte Carlo techniques are employed to gain insight into the effect of preparation conditions on the structural and optical properties of silicon nanoparticles. Our results demonstrate that (i) kinetically limited nanostructures form different core structures than bulk-derived crystalline clusters, (ii) the type of core structure that forms depends on how the cluster is passivated during synthesis, and (iii) good agreement with measured optical gaps can be obtained for nanoparticles with core structures different from those derived from the bulk.

Superfluidity in a doped helium droplet

Path-integral Monte Carlo calculations of the superfluid density throughout 4He droplets doped with linear impurities are presented. After deriving a local estimator for the superfluid density distribution, we find a decreased superfluid response in the cylindrically symmetric region of the first solvation layer. The helium in this region has a superfluid transition temperature similar to that of a two-dimensional helium system and may be responsible for previously unexplained experimental Q-branch measurements.

Bose-Einstein condensation at a helium surface

A path integral Monte Carlo method was used to calculate the Bose-Einstein condensate fraction at the surface of a helium film at T = 0.77 K, as a function of density. Moving from the center of the slab to the surface, the condensate fraction was found to initially increase with decreasing density to a maximum value of 0.9 before decreasing. Long wavelength density correlations were observed in the static structure factor at the surface of the slab. Finally, a surface dispersion relation was calculated from imaginary-time density-density correlations.