1. SOLVCON: Software-
Engineering Simulations of
Conservation Laws
Yung-Yu Chen!
SOLVCON Project!
@ PyConAPAC 2014 May 17

2. Introduction
❖ I am: an engineer using computers to solve problems in mechanics.!
❖ I want to solve conservation laws. I want to solve for various physical
processes. I should build a framework.!
❖ Examples of the simulations:
Aerodynamics Benchmarks
Stress Waves in!
Anisotropic Solids
Supersonic Jet in Cross Flow

3. Goals
❖ We want multiple solvers for physical processes governed by the equations:!
!
❖ We use the space-time Conservation Element and Solution Element (CESE)
method.!
❖ We want an extensible system, modularized for various physical processes.!
❖ Then plan for interoperation.!
❖ We want parallel code (scale from 1,000 cores to more). We want hybrid
parallelism for heterogeneous architecture.!
❖ We want the code to be organized. We don’t want spaghetti code patched
from paper to paper. We want to code systematically.

4. It’s an Engineering Problem
❖ “Develop a software framework for the CESE method.”!
❖ Balance the following points:!
❖ Extensibility and modularity.!
❖ Parallel computing.!
❖ Maintainability.!
❖ All goals are achieved, with rough ends.

5. Python
❖ Python is slow, but provides a good infrastructure for working with
low-level number-crunching code.!
❖ ndarray (N-dimensional array) of NumPy (http://
www.numpy.org/).!
❖ Cython (http://cython.org/) for interfacing between Python and C.!
❖ The versatile scientific Python ecosystem simplifies coding supportive
functionalities.!
❖ Practices emerged from other fields flow into sciences:!
❖ Version control, unit tests, documenting tools, etc.

6. Scripting Is a Must
❖ There are too many parameters and analyses required.!
❖ No interface is sufficient for the simulations: SOLVCON
needs to be a library.!
❖ Controlling a library by using low-level languages like
C and C++ is too painful to be realistic.

7. Design for Productivity
❖ The outstanding capability to interface to C makes Python an
ideal manager for low-level, high-performance C code.!
❖ C++ works as well but SOLVCON uses only C to avoid the
complexity.!
❖ The hybrid approach allows us to architect the system with the
high-level Python and still have the high performance of C.!
❖ Ideas can be quickly prototyped with Python. After results
get validated, the fast C version can then developed
iteratively.

8. Discretize Space
❖ The conservation laws we are
solving assume continuum
model.!
❖ In computer memory nothing
is continuous.!
❖ There must be a way to
discretize the space for the
equations.!
❖ “Meshing”.

9. Unstructured Meshes
❖ We chose to employ the unstructured meshes of mixed
elements to model complex geometry.!
❖ In computer memory they consist of several look-up
tables.
quadrilateral triangle
hexahedron tetrahedron prism pyramid

10. Look-Up Tables for Meshes
❖ SOLVCON categorize the look-up tables into three: (i)
coordinate, (ii) meta-data, and (iii) connectivity.!
❖ They are either 1-D or 2-D arrays.!
❖ These arrays contain “ghost cells” for boundary-
condition treatments.!
❖ The ghost cells are outside the computing space.!
❖ The CESE method needs only one layer of ghost cells.

11. Domain Decomposition
❖ Millions or billions of
unstructured mesh elements
can be automatically
decomposed into multiple
blocks for parallel computing.!
❖ A graph is built for
representing the connectivity
among all the mesh elements.!
❖ The partitioned graphs are
used to decompose the
original mesh into thousands
of blocks.

12. Two-Level Nested Loops
❖ The time-marching nature of
the CESE method dictates a
structure of two-level nested
loops.!
❖ The outer loop is for time-
marching.!
❖ Multiple inner loops
“sweep” over the discretized
space.!
❖ All facilities can built around
this execution flow.

13. Module Structure

14. Case-Solver Relation
❖ solvcon.case.MeshCase: the outer temporal loop.!
❖ solvcon.solver.MeshSolver: house of the inner spatial loops.

15. In-Situ Visualization/Analysis
❖ What’s in-situ visualization/analysis (ISV)?!
❖ Perform result visualization/analysis while time-marching.!
❖ Why in-situ?!
❖ [50 million-element mesh] * [10,000 time steps] * [6 single-precision solution
fields (CFD)] mean 12 TB of data. It is one simulation case.!
❖ Segregating result analysis from simulation code is difficult.!
❖ Architecture of most research codes isn’t good enough for the segregation.!
❖ Most codes simply produce the data and use another software to post-
process.!
❖ A common practice is to reduce the resolution.!
❖ Outputting every 100 time steps reduces the output dat to 120 GB.!
❖ But it defeats the purpose of high-resolution simulations.

16. SOLVCON’s Segregation
❖ SOLVCON provides
companions of MeshCase and
MeshSolver: MeshHook and
MeshAnchor, respectively.!
❖ MeshCase and MeshSolver
are for equation solutions.!
❖ MeshHook and MeshAnchor
are for solution analysis.!
❖ And there’re still a lot to do.

17. Documenting
❖ SOLVCON documents should contain what’s in
conventional research papers.!
❖ What we want is more than conventional papers.!
❖ Bi-directional linking between the code and the
documents.!
❖ Some part of the documents should be runnable. It’s
related to validation.

18. Sphinx
❖ Sphinx (http://sphinx-doc.org/) is the default tool for
Python documentation.!
❖ ReadTheDocs (https://readthedocs.org/) is a service to
host Sphinx documents.!
❖ SOLVCON authors and organizes its documents with
Sphinx.!
❖ The documents are published at http://
www.solvcon.net/ (ReadTheDocs is behind the scenes).

20. Authoring, Editing, and Reviewing
❖ We use Mercurial for version control and BitBucket
(https://bitbucket.org/solvcon/solvcon/) to host the
project and its source code.!
❖ Reviewing the documents goes through the pull-request
flow of the code review.!
❖ We treat the reStructuredText source files of the
documents like the program source code.

21. Integrated Testing
❖ Code without verification can’t be taken as working code,
no matter how carefully we wrote it.!
❖ With the testing system, refactoring and quick prototyping
become possible.!
❖ Many research codes don’t have a regression testing
system, so while coding the developers can’t be sure
how the changes impact the system.!
❖ Two kinds of tests: unit tests and answer tests. Both need
automation.

22. Unit Tests
❖ Unit tests are short and quick, and check for simple
output of elementary API behaviors.!
❖ SOLVCON uses standard Python unittest module
(http://docs.python.org/2/library/unittest.html).!
❖ nose (https://nose.readthedocs.org/) is used as the
automatic test runner.

24. Answer Tests
!
❖ Answer tests check for the correctness of simulations.
Usually an answer test is a meaningful simulation, and
can serve as an example to how to use SOLVCON to
simulate other problems. !
❖ Jenkins (http://jenkins-ci.org/) is installed at http://
ci.solvcon.net/ to automatically run the answer tests for
every commit of SOLVCON in a testing farm.

26. Challenges (Conclusions)
❖ The simulations were done with older version of
SOLVCON; it takes a lot of efforts to upgrade (ongoing).!
❖ Unfamiliarity of the collaborators to the authoring
system (Sphinx): http://www.solvcon.net/en/latest/bulk/theory.html!
❖ Unification of code and document needs extra efforts.!
❖ Feeding back to the main framework.!
❖ Server farm for continuous integration.

27. Current/Future Works
❖ New physical processes:!
❖ solvcon.parcel.bulk: acoustic wave solver based on
bulk modulus.!
❖ solvcon.parcel.vewave: visco-elastic wave solver.!
❖ Document the CESE method.!
❖ Documenting with iPython in addition to Sphinx?!
❖ Get more people to play together!