A high performance computing framework for Bayesian uncertainty quantification of complex models
We present Π4U, an extensible framework, for non-intrusive Bayesian Uncertainty Quantification and Propagation (UQ+P) of complex and computationally demanding physical models, that can exploit massively parallel computer architectures. The framework incorporates Laplace asymptotic approximations as well as stochastic algorithms, along with distributed numerical differentiation and task-based parallelism for heterogeneous clusters. Sampling is based on the Transitional Markov Chain Monte Carlo (TMCMC) algorithm and its variants. The optimization tasks associated with the asymptotic approximations are treated via the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). A modified subset simulation method is used for posterior reliability measurements of rare events. The framework accommodates scheduling of multiple physical model evaluations based on an adaptive load balancing library and shows excellent scalability.
A general overview of the computational framework is shown in the figure below. The task-parallel runtime environment orches-trates task submission and scheduling. It exports a user-friendly programming interface for design and development of scalable UQ and Optimization in HPC platforms ranging from multi-core systems to hybrid CPU/GPU clusters. Sequential or parallel system solvers can be coupled with the framework.
The TORC tasking library
In order to meet the above requirements, we based the parallel implementation of our tools on TORC (Tasking library for Clusters). TORC is a novel task-parallel library that provides a programming and runtime environment where parallel programs can be executed unaltered on both shared and distributed memory platforms. TORC supports arbitrary nesting of tasks while all data transfers in the library are performed with explicit, transparent to the user, messaging. Due to the task stealing mechanism, the programmer has only to decide about the task distribution scheme. A task function can either include source code supplied by the user or invoke an external simulation program. The injected user code can embrace intra-node parallelism expressed with OpenMP directives or TORC tasks that are tied to the current node. We do not pose any restrictions on the external software, which can be sequential or parallel running on multicores and clusters (e.g. using OpenMP and MPI) or GPUs.
Available algorithms
- TMCMC (for exact Bayesian inference)
- ABC-SubSim (for approximate Bayesian inference)
- CMA-ES (for optimisation)
- Subset Simulation (for rare events sampling)
- A-PNDL (for adaptive parallel numerical differentiation)
Applications
Molecular Dynamics
We perform UQ+P in the most widely used MD model, that of water. We use a 5-site water model, TIP5P and we choose to resolve long range electrostatics using the Particle–Mesh–Ewald method. We perform UQ+P in the most widely used MD model, that of water. We use a 5-site water model, TIP5P and we choose to resolve long range electrostatics using the Particle–Mesh–Ewald method. The computational workflow includes equilibration using steepest descent of the system for at least 20,000 steps, subsequent equilibration for a further 1 ns using a time step of t=0.5 fs in the NPT ensemble, and finally a 1nsproduction run in the NPT ensemble for each run. We then extract the predicted Oxygen–Oxygen radial distribution function (g(r)) averaged over the production part of the trajectory.
Structural dynamics
We perform Bayesian UQ+P in a high fidelity finite element model of the Metsovo bridge. A detailed model of the bridge is created using 3-dimensional tetrahedron quadratic Lagrange finite elements. A coarse mesh is chosen to predict the lowest 20 modal frequencies and mode shapes of the bridge. The model has 97,636 finite elements and 562,101 DOFs. The model is parameterized using five parameters associated with the modulus of elasticity of one or more structural components. We performed our experiments for the bridge Bayesian inference problem on a multicore server with four 12-core AMD Opteron 6174 CPUs (2.2 GHz, 512 KB private L2 cache/core) and 32 GB of RAM. As the server is comprised of 8 6-core NUMA nodes, the TORC-based applications were launched with 8 MPI processes and 6 workers each, utilizing 48 workers in total. The inference results along with an algorithmic comparison are summarized below.
Discrete element simulations of granular materials
Predictions in the behavior of granular materials using Discrete Element Methods (DEM) hinge on the employed interaction potentials. Here we introduce a data driven, Bayesian framework to quantify DEM predictions. Our approach relies on experimentally measured coefficients of restitution for single steel particle–wall collisions. The calibration data entail both tangential and normal coefficients of restitution, for varying impact angles and speeds of the bouncing particle. The parametric uncertainty in multiple Force–Displacement models is estimated using an enhanced Transitional Markov Chain Monte Carlo implemented efficiently on parallel computer architectures. In turn, the parametric model uncertainties are propagated to predict Quantities of Interest (QoI) for two testbed applications: silo discharge and vibration induced mass-segregation. This work demonstrates that the classical way of calibrating DEM potentials, through parameter optimization, is insufficient and it fails to provide robust predictions. The present Bayesian framework provides robust predictions for the behavior of granular materials using DEM simulations. Most importantly the results demonstrate the importance of including parametric and modeling uncertainties in the potentials employed in Discrete Element Methods.
>> Get the code from here <<
Unknown Publication Date
- P. E. Hadjidoukas, P. Angelikopoulos, C. Papadimitriou, and P. Koumoutsakos, “Π4U: a high performance computing framework for Bayesian uncertainty quantification of complex models,” J. Comput. Phys., vol. 284, p. 1–21, 2015.
[BibTeX] [Abstract] [PDF] [DOI]
We present {Π}4U, an extensible framework, for non-intrusive Bayesian Uncertainty Quantification and Propagation (UQ+P) of complex and computationally demanding physical models, that can exploit massively parallel computer architectures. The framework incorporates Laplace asymptotic approximations as well as stochastic algorithms, along with distributed numerical differentiation and task-based parallelism for heterogeneous clusters. Sampling is based on the Transitional Markov Chain Monte Carlo (TMCMC) algorithm and its variants. The optimization tasks associated with the asymptotic approximations are treated via the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). A modified subset simulation method is used for posterior reliability measurements of rare events. The framework accommodates scheduling of multiple physical model evaluations based on an adaptive load balancing library and shows excellent scalability. In addition to the software framework, we also provide guidelines as to the applicability and efficiency of Bayesian tools when applied to computationally demanding physical models. Theoretical and computational developments are demonstrated with applications drawn from molecular dynamics, structural dynamics and granular flow.
@article{hadjidoukas2015b, author = {Hadjidoukas, Panagiotis E and Angelikopoulos, Panagiotis and Papadimitriou, Costas and Koumoutsakos, Petros}, doi = {https://doi.org/10.1016/j.jcp.2014.12.006}, journal = {{J. Comput. Phys.}}, pages = {1--21}, publisher = {Elsevier}, title = {
{4U}: A high performance computing framework for {B}ayesian uncertainty quantification of complex models},
url = {http://www.cse-lab.ethz.ch/wp-content/papercite-data/pdf/hadjidoukas2015b.pdf},
volume = {284},
year = {2015}
}