1. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Big Data Analytics at the MPCDF:
GPU Crystallography with Python
Giuseppe Di Bernardo
Max Planck Computing and Data Facility (MPCDF)
Garching bei München, Germany
giuseppe.di-bernardo@mpcdf.mpg.de
@jose_dibernardo
July 12, 2017
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 1 / 48
2. Basic Research at cutting-edge;
strictly curiosity-driven and quality oriented;
Flexible, dynamic, interdisciplinary MPIs;
Autonomy → scientists decide upon science;
Long-term trust systems;
Quality assurance by peers.
3. Sites of Max Planck Research Institutes
and Associated Institutes
MAX PLANCK INSTITUTES ABROAD
ITALY
Bibliotheca Hertziana, Rome
Art History Institute, Florence
THE NETHERLANDS
Max Planck Institute for Psycholinguistics, Nijmegen
USA, FLORIDA
Max Planck Florida, Jupiter
LUXEMBOURG
Max Planck Institute Luxembourg for International,
European and Regulatory Procedural Law,
Luxembourg
4. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Outline
1 X-ray Crystallography: a short overview
2 Big Data Analytics at MPCDF
3 Visualization of Crystal Nano-Structure
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 4 / 48
5. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Outline
1 X-ray Crystallography: a short overview
Atom Probe Tomography
Crystallography: implementation on a GPU
2 Big Data Analytics at MPCDF
PyCUDA
PyNX: A Python-based approach to GPU computing
3 Visualization of Crystal Nano-Structure
Python Plotting for Exploratory Data Analysis
Getting started with ParaView
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 5 / 48
6. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Atom Probe Tomography (APT)
Baptiste Gault & Shyam Katnagallu BDDMS - APT
Retrieve high-quality/crystalline data
Iterativily use the Fourier analysis for
reconstruction of APT data;
Need of the state-of-the art data mining and
visualization
Baptiste Gault & Shyam Katnagallu BDDMS - APT
1
by courtesy of B. Gault & S. Katnagallu (MPIE), and M. Rampp (MPCDF)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 6 / 48
A new playground for Big Data analysis1
Fourier Transforms on billion points data!!
7. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Scattering computing from an atomistic model
Theory: X-ray and neutron scattering
Scattering vector
Scattering density, at position r
A(S) =
V
ρ(r) exp(2iπ S · r)dV = FT[ρ(r)]
Scattered amplitude
divide et impera
FFT is very fast: O(N log N)
coordinates in reciprocal space
(hkl) are imposed by the
electron density grid
Our approach: Direct calculation
A(s) =
N−1
j=0
fj (s) exp(2iπS · rj)
O(N2
)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 7 / 48
8. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
GPU-accelerated analysis of large X-ray data
1 Direct calculation in 3D reciprocal space (hkl): atomic
positions (ri) & scattering factors (fi ):
A(s) =
N−1
j=0
fj (s)e2iπS·rj
computing any point in (hkl) from any structural model!
2 In crystallography usually we have:
many atoms, e.g. Natoms O(≥1 × 108
)
many reflections (Nhkl) in hkl, e.g. Nkhl O(128 × 128 × 128)
3 It requires massive parallelism
Nflop ≈ 10 × Natoms × Nhkl : O(≥1 × 1015
) flop
modern GPU: 1013
Flop/s, CPU server: 1012
Flop/s
algorithm well suited for GPU computations, scalable across
multiple GPUs → O(minutes)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 8 / 48
9. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
GPU-accelerated analysis of large X-ray data
1 Direct calculation in 3D reciprocal space (hkl): atomic
positions (ri) & scattering factors (fi ):
A(s) =
N−1
j=0
fj (s)e2iπS·rj
computing any point in (hkl) from any structural model!
2 In crystallography usually we have:
many atoms, e.g. Natoms O(≥1 × 108
)
many reflections (Nhkl) in hkl, e.g. Nkhl O(128 × 128 × 128)
3 It requires massive parallelism
Nflop ≈ 10 × Natoms × Nhkl : O(≥1 × 1015
) flop
modern GPU: 1013
Flop/s, CPU server: 1012
Flop/s
algorithm well suited for GPU computations, scalable across
multiple GPUs → O(minutes)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 8 / 48
10. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
GPU-accelerated analysis of large X-ray data
1 Direct calculation in 3D reciprocal space (hkl): atomic
positions (ri) & scattering factors (fi ):
A(s) =
N−1
j=0
fj (s)e2iπS·rj
computing any point in (hkl) from any structural model!
2 In crystallography usually we have:
many atoms, e.g. Natoms O(≥1 × 108
)
many reflections (Nhkl) in hkl, e.g. Nkhl O(128 × 128 × 128)
3 It requires massive parallelism
Nflop ≈ 10 × Natoms × Nhkl : O(≥1 × 1015
) flop
modern GPU: 1013
Flop/s, CPU server: 1012
Flop/s
algorithm well suited for GPU computations, scalable across
multiple GPUs → O(minutes)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 8 / 48
11. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Outline
1 X-ray Crystallography: a short overview
Atom Probe Tomography
Crystallography: implementation on a GPU
2 Big Data Analytics at MPCDF
PyCUDA
PyNX: A Python-based approach to GPU computing
3 Visualization of Crystal Nano-Structure
Python Plotting for Exploratory Data Analysis
Getting started with ParaView
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 9 / 48
13. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Why do Scripting for GPUs?
GPUs are everything that high-level
scripting languages are not.
Highly parallel
Very architecture-sensitive
Built for maximum FP/memory
throughput
→ complement each other
Scripting + GPU : A good combination
⇓
Python + CUDA2
= PyCUDA
Python + OpenCL = PyOpenCL
GPU Scripting PyOpenCL News RTCG Showcase Overview Being Productive
Why do Scripting for GPUs?
GPUs are everything that scripting
languages are not.
Highly parallel
Very architecture-sensitive
Built for maximum FP/memory
throughput
! complement each other
CPU: largely restricted to control
tasks (⇠1000/sec)
Scripting fast enough
Python + CUDA = PyCUDA
Python + OpenCL = PyOpenCL
Andreas Kl¨ockner PyCUDA: Even Simpler GPU Programming with Python
Figure: courtesy by
A. Klöckner, Nvidia
GTC 2010
2
CUDA is a general purpose parallel computing architecture that leverages the
parallel compute engine in NVIDIA graphics processing units (GPUs)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 11 / 48
14. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
But before PyCUDA: why Python at all?
general purpose
simple to learn and use
extensible and embeddable: Python C API
science oriented too (NumPy, SciPy, mpi4py)
great visualization tools (Bokeh, Matplotlib)
very well documented
NumPy: package for large, N-dimensional array object
Vectors, Matrices, ... array computing is fast
A+B, sin(A), dot(A,B)
la.solve(A,b), la.eig(A)
cube[;, ;, n-k:n+k], cube+5
FFT’s, tools for integrating C/C++ and Fortran code
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 12 / 48
15. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Whetting your appetite: SourceModule
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np
from pycuda.compiler import SourceModule
a = np.random.randn(4,4).astype(np.float32)
a_gpu = cuda.mem_alloc(a.nbytes)
# host-to-device
cuda.memcpy_htod(a_gpu, a)
mod = SourceModule("""
__global__ void multiply_by_two(float *a)
{
int idx = threadIdx.x + threadIdx.y*4;
a[idx] *= 2;
} Compute Kernel
""")
func = mod.get_function("multiply_by_two")
func(a_gpu, bloc=(4,4,1))
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 13 / 48
16. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Whetting your appetite: GPUArrays
import numpy as np
import pycuda.autoinit
import pycuda.gpuarray as gpuarray
a_gpu = gpuarray.to_gpu(np.random.randn(4,4).astype(np.float32))
a_doubled = (2*a_gpu).get()
print a_doubled
print a_gpu
GPUArrays: computational linear algebra
element-wise algebraic operations (+, -, *, /, sin, cos, exp)
tight integration with numpy
gpuarray.to_gpu(numpy_array)
numpy_array = gpuarray.get()
mixed data types (int32 + float32 = float64)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 14 / 48
17. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Workflow: "Edit-Run-Repeat"
A two-fold aim:
1 usage of existing CUDA C
2 on top of the first layer, PyCUDA ⇒ abstractions
GPU Scripting PyOpenCL News RTCG Showcase Overview Being Productive
PyCUDA: Workflow
Edit
PyCUDA
Run
SourceModule("...")
Cache?
nvcc
no
.cubin
Upload to GPU
Run on GPU
Figure: A. Klöckner et al. 2013, https://arxiv.org/abs/1304.5553
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 15 / 48
18. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python.
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
19. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
20. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
Robustness: automatic management of
object lifetimes and error checking
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
21. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
Robustness: automatic management of
object lifetimes and error checking
Completeness: full power of CUDA’s
driver API at your disposal
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
22. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
Robustness: automatic management of
object lifetimes and error checking
Completeness: full power of CUDA’s
driver API at your disposal
Convenience: provide abstractions
(comes with ready-made on-GPU linear
algebra, reduction, scan. Add-on packages
for FFT and LAPACK available.)
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
23. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
Robustness: automatic management of
object lifetimes and error checking
Completeness: full power of CUDA’s
driver API at your disposal
Convenience: provide abstractions
(comes with ready-made on-GPU linear
algebra, reduction, scan. Add-on packages
for FFT and LAPACK available.)
Integrate tightly with NumPy
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
24. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
Robustness: automatic management of
object lifetimes and error checking
Completeness: full power of CUDA’s
driver API at your disposal
Convenience: provide abstractions
(comes with ready-made on-GPU linear
algebra, reduction, scan. Add-on packages
for FFT and LAPACK available.)
Integrate tightly with NumPy
Speed: PyCUDA’s base layer is written in
C++ (near-zero wrapping overhead)
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
25. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA Philosophy: maps all CUDA into Python
PyCUDA lets you completely access to Nvidia’s CUDA parallel
computation API from Python. Key Features:
Robustness: automatic management of
object lifetimes and error checking
Completeness: full power of CUDA’s
driver API at your disposal
Convenience: provide abstractions
(comes with ready-made on-GPU linear
algebra, reduction, scan. Add-on packages
for FFT and LAPACK available.)
Integrate tightly with NumPy
Speed: PyCUDA’s base layer is written in
C++ (near-zero wrapping overhead)
Complete, helpful documentation
GPU Scripting PyOpenCL News RTCG Showcase
PyCUDA Philosophy
Andreas Kl¨ockner
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 16 / 48
26. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyCUDA: Vital Information
Availability: Freely down-loadable
from this location
Open source
MIT Licensed
requires NumPy, Python 2.4+
(Win/OS X/Linux)
Support via mailing list
For further information see:
Main PyCUDA page
PyCUDA Wiki
PyCUDA FAQ
http://mathema.tician.de/
software/pycuda
Complete documentation
MIT License
(no warranty, free for all use)
Requires: numpy, Python 2.4+
(Win/OS X/Linux)
Support via mailing list
Figure: courtesy by
A. Klöckner, Nvidia
GTC 2010
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 17 / 48
27. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Tools for Nano-structures X-tallograhy
It is an open-source library created by Vincent Favre-Nicolin at
ESRF3with the following main modules:
3
European Synchrotron Radiation Facility; favre@esrf.fr
4
New: in PyNX 3.1.0, is called pynx.scattering
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 18 / 48
28. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Tools for Nano-structures X-tallograhy
It is an open-source library created by Vincent Favre-Nicolin at
ESRF3with the following main modules:
1 pynx.gpu4: X-ray scattering computing using graphical
processing units, allowing up to 4 × 1011
reflections/atoms/seconds (e.g., 2x nVidia GeForce GTX 980);
3
European Synchrotron Radiation Facility; favre@esrf.fr
4
New: in PyNX 3.1.0, is called pynx.scattering
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 18 / 48
29. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Tools for Nano-structures X-tallograhy
It is an open-source library created by Vincent Favre-Nicolin at
ESRF3with the following main modules:
1 pynx.gpu4: X-ray scattering computing using graphical
processing units, allowing up to 4 × 1011
reflections/atoms/seconds (e.g., 2x nVidia GeForce GTX 980);
2 pynx.ptycho: simulation and analysis of experiments using
the ptychography technique;
3
European Synchrotron Radiation Facility; favre@esrf.fr
4
New: in PyNX 3.1.0, is called pynx.scattering
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 18 / 48
30. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Tools for Nano-structures X-tallograhy
It is an open-source library created by Vincent Favre-Nicolin at
ESRF3with the following main modules:
1 pynx.gpu4: X-ray scattering computing using graphical
processing units, allowing up to 4 × 1011
reflections/atoms/seconds (e.g., 2x nVidia GeForce GTX 980);
2 pynx.ptycho: simulation and analysis of experiments using
the ptychography technique;
3 pynx.wavefront: X-ray wavefront propagation in the near,
far field, or continuous;
3
European Synchrotron Radiation Facility; favre@esrf.fr
4
New: in PyNX 3.1.0, is called pynx.scattering
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 18 / 48
31. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Tools for Nano-structures X-tallograhy
It is an open-source library created by Vincent Favre-Nicolin at
ESRF3with the following main modules:
1 pynx.gpu4: X-ray scattering computing using graphical
processing units, allowing up to 4 × 1011
reflections/atoms/seconds (e.g., 2x nVidia GeForce GTX 980);
2 pynx.ptycho: simulation and analysis of experiments using
the ptychography technique;
3 pynx.wavefront: X-ray wavefront propagation in the near,
far field, or continuous;
4 pynx.cdi: Coherent Diffraction Imaging reconstruction
algorithms using GPU.
3
European Synchrotron Radiation Facility; favre@esrf.fr
4
New: in PyNX 3.1.0, is called pynx.scattering
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 18 / 48
32. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Hardware requirements
PyNX aims to help computing scattering (X-ray or neutrons) maps
for atomic structures, especially if they are distorted or disordered.
High-performance computing with GPUs
nVidia’s CUDA toolkit5
and the pyCUDA library
OpenCL6
language, along with pyOpenCL library
GPU-accelerated applications
PyNX provides fast parallel computation of scattering from
large assemblies of atoms ( 108
atoms) and 1D, 2D or
3D coordinates ( 106
grid points) in reciprocal lattice
space.
GPU Scripting PyOpenCL News RTCG Showcase Overview Being Productive
Why do Scripting for GPUs?
GPUs are everything that scripting
languages are not.
Highly parallel
Very architecture-sensitive
Built for maximum FP/memory
throughput
! complement each other
CPU: largely restricted to control
tasks (⇠1000/sec)
Scripting fast enough
Python + CUDA = PyCUDA
Python + OpenCL = PyOpenCL
Andreas Kl¨ockner PyCUDA: Even Simpler GPU Programming with Python
5
https://developer.nvidia.com/cuda-toolkit
6
https://www.khronos.org/opencl/
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 19 / 48
33. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Software requirements
PyNX supports Python version 2.7, 3.4 and above (Anaconda
distribution is recommended) .
Python interface ⇒ No need to learn CUDA
Install PyNX
PyNX is available from:
http://ftp.esrf.fr/pub/scisoft/PyNX/
http://gitlab.esrf.fr/favre/PyNX
PyPI:
$ pip install pynx
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 20 / 48
Python is also a great
glue language!
34. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Software requirements
PyNX supports Python version 2.7, 3.4 and above (Anaconda
distribution is recommended) .
Python interface ⇒ No need to learn CUDA
Install PyNX
PyNX is available from:
http://ftp.esrf.fr/pub/scisoft/PyNX/
http://gitlab.esrf.fr/favre/PyNX
PyPI:
$ pip install pynx
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 20 / 48
Python is also a great
glue language!
35. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Software requirements
PyNX supports Python version 2.7, 3.4 and above (Anaconda
distribution is recommended) .
Python interface ⇒ No need to learn CUDA
Install PyNX
PyNX is available from:
http://ftp.esrf.fr/pub/scisoft/PyNX/
http://gitlab.esrf.fr/favre/PyNX
PyPI:
$ pip install pynx
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 20 / 48
Python is also a great
glue language!
39. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX kernel: __global__ void CUDA_fhkl()
1 const unsigned long ix=threadIdx.x+blockDim.x*blockIdx.x; // Thread idx
2 const float h=twopi*vh[ix];
3 const float k=twopi*vk[ix];
4 const float l=twopi*vl[ix];
5 float fr=0,fi=0;
6 __shared__ float x[BLOCKSIZE]; // Shared memory between BLOCKSIZE %d
7 __shared__ float y[BLOCKSIZE]; // parallel threads
8 __shared__ float z[BLOCKSIZE];
9 long at=0;
10 for (;at<=(natoms-BLOCKSIZE);at+=BLOCKSIZE) {
11 __syncthreads();
12 x[threadIdx.x]=vx[at+threadIdx.x]; // "Coalesced" transfer
13 y[threadIdx.x]=vy[at+threadIdx.x]; // to shared memory
14 z[threadIdx.x]=vz[at+threadIdx.x];
15 __syncthreads();
16 for(unsigned int i=0;i<BLOCKSIZE;i++) {// Each thread computes
17 float s,c; // a single reflection
18 __sincosf(h*x[i] + k*y[i] + l*z[i] , &s,&c);
19 fr +=c; // fast, intrensic trigonometric function
20 fi +=s;
21 }__syncthreads();}
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 22 / 48
40. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX kernel: __global__ void CUDA_fhkl()
1 __syncthreads()
2 ;}
3
4 /* Take care of remaining atoms */
5 if(threadIdx.x<(natoms-at)) {
6 x[threadIdx.x]=vx[at+threadIdx.x];
7 y[threadIdx.x]=vy[at+threadIdx.x];
8 z[threadIdx.x]=vz[at+threadIdx.x];
9 }
10 __syncthreads();
11 for(long i=0;i<(natoms-at);i++) {
12 float s,c;
13 __sincosf(h*x[i] + k*y[i] + l*z[i] , &s,&c);
14 fr +=c;
15 fi +=s;
16 }
17 fhkl_real[ix]+=fr;
18 fhkl_imag[ix]+=fi;
19 }
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 23 / 48
41. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX Python interface: loading the data
1 import numpy as np
2 import time
3
4 @timeit
5 def do_work(*args, **kwargs):
6 def read_pos(file_name):
7 """Loads an APT.pos file
8 Columns: x, y, z, Da
9 """
10 f = open(file_name, mode='rb') # Load orthonormal coord.
11 dt_type = np.dtype({'names':['x', 'y', 'z', 'Da'],
12 'formats':['>f4', '>f4', '>f4', '>f4']})
13 pos = np.fromfile(f, dt_type, -1)
14 f.close()
15 print("the data contain: {0:.5e} atoms".format(pos.size))
16 return pos
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 24 / 48
42. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX Python interface: parsing the data
16 if args[1] is not None:
17 fname = args[1]
18
19 df = readpos(fname)
20 xEl = df['x'] # slicing columns by labels
21 yEl = df['y']
22 zEl = df['z']
23 lattice_parameter = 0.4045
24 xEl /= lattice_parameter # Convert to fractional coordinates
25 yEl /= lattice_parameter
26 zEl /= lattice_parameter
27 N = 128
28 h = np.linspace(-1.1, 1.1, num=N, endpoint=True) # HKL as 3D
29 k = np.linspace(-1.1, 1.1, num=N, endpoint=True)[:, np.
newaxis]
30 l = np.linspace(-1.1, 1.1, num=N, endpoint=True)[:, np.
newaxis, np.newaxis]
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 25 / 48
43. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX Python interface: The Fourier Transform
31 from pynx import gpu # main module for scattering computation
32 gpu_name = args[0] # identify GPU, e.g. GeForce GTX980
33
34 if args[0] is not None:
35 fhkl, dt = gpu.Fhkl_thread(h, k, l, xEl, yEl, zEl,
gpu_name=gpu_name,verbose=True,language="CUDA",
cl_platform="nvidia")
36 print("The FT computed in dt = {0:7.5f}:".format(dt))
37 print("{0:d} fourier number points".format(fhkl.size))
38
39 return fhkl # a numpy complex64 array
gpu.Fhkl_thread
computes F(hkl) = i exp[2jπ(xi × h + yi × k + zi × l)];
distributes the scattering maps on several GPU.
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 26 / 48
44. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Examples
PyNX to compute e.g., ...
1 2D scattering monoatomic cubic structure 100 × 100 × 100
2 complex refraction index of a crystilline material (grazing incidence)
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 27 / 48
45. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: Performance
k of atoms, and u(r) the displacement field inside the crys-
the composition of the blocks of atoms vary (e.g. due to
diffusion), it is also possible to include a variation of the
age scattering density in the FT (Takagi, 1969):
A(S) ≈ F(S) FT [ρ(r) exp(2iπH · u(r))] (6)
here ρ(r) is the relative scattering density in the crystal.
oth equation (5) and (6) allow the use of a fast Fourier trans-
, but are only valid as long as:1
| (S − H) · u(r) |≪ 1 (7)
oreover, use of equations (5) and (6) with a fast Fourier
form restricts the computation of scattering on a triperiodic
in reciprocal space - this is a limitation since modern data
ction often use 2D detectors, and the measured points in
rocal space are located on a curved surface (the projection
e detector on Ewald’s sphere). Furthermore, as the resolu-
in reciprocal space is inversely proportional to the size in
space, analysis of high-resolution data using a FFT calcu-
n demands a large model - even if the extent in reciprocal
e is very limited.
herefore, even if the speed of the FFT is optimal for large
alline structures - for N points in real space, N points
ciprocal space are calculated with a cost proportional to
og(N) instead of N2
- it is still interesting to consider a
t computation using equation (2) or (4) because it allows
putation for:
Equation (2) fulfills all requirements, assuming that bo
number of atoms and the number of points in reciprocal
are large (≫1000).
Figure 1
Computing speed depending on the number of atoms and reflections.
tests were run on a single nVidia GTX295 graphics card, using in p
the two multiprocessors available on the card. Nrefl indicates the num
reflections for the GPU calculations (black lines). The CPU (Central P
ing Unit) curves (red lines) correspond to a computing using a vectorized
optimized) C++ code running on a single core of an Intel Core2 Quad
running at 2.83 GHz, for Nrefl = 102, 103, 104 (the curves for 103 and
almost identical).
assesment of GPU-accelerated python software package PyNX
Effective (single precision) throughput per GPU:
up to 3.5 × 1011
reflections · atoms · s−1
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 28 / 48
cf. Favre-Nicolin et al.,
J. Appl. Cryst. 44 (2011) 635
46. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: benchmarks at the MPCDF
Logarithmic scale on the vertical axis!
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 29 / 48
47. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: benchmarks at the MPCDF
Logarithmic scale on the vertical axis!
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 29 / 48
48. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
PyNX: benchmarks at the MPCDF
Logarithmic scale on the vertical axis!
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 29 / 48
49. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Deploying Data Science Projects - Clusters
#!/bin/bash -l
#SBATCH -J pynx_run
# Queue:
#SBATCH –partition=gpu
# Node feature:
#SBATCH –constraint="gpu"
# N. of nodes and MPI tasks per node:
#SBATCH –nodes=1
#SBATCH –ntasks-per-node=32
# wall clock limit:
#SBATCH –time=02:00:00
echo "starting the job ..."
python crystAl.py -g="GeForce GTX 980"
echo "...done"
SLURM worload manager: basic usage
qsub: submit your job;
qstat: query queue/job status;
qdel: delete your job
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 30 / 48
Supercomputing Drives
Science through Simulations.
The MPG SuperComputer Hydra
at the MPCDF
In total there are ∼ 83.000 cores
with a main memory of 280 TB and
a peak performance of about 1.7 PetaFlop
50. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Outline
1 X-ray Crystallography: a short overview
Atom Probe Tomography
Crystallography: implementation on a GPU
2 Big Data Analytics at MPCDF
PyCUDA
PyNX: A Python-based approach to GPU computing
3 Visualization of Crystal Nano-Structure
Python Plotting for Exploratory Data Analysis
Getting started with ParaView
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 31 / 48
51. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Image Processing in Python: Toolbox for SciPy
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d.art3d import
Poly3DCollection
from mpl_toolkits.mplot3d import Axes3D
from skimage import measure
pow_spec = abs(np.log10(abs(fhkl)**2))
mean_iso = (pow_spec.max() + pow_spec.min())/2.
verts, faces =
measure.marching_cubes(pow_spec, mean_iso)
fig = plt.figure()
ax = plt.axes(projection="3d")
mesh = Poly3DCollection(verts[faces])
ax.add_collection3d(mesh)
plt.show()
Marching Cubes
1 MC is an algorithm to extract a 2D
surface mesh from a 3D volume;
2 ‘verts[faces]‘ to generate a collection
of triangles;
3 Display triangular mesh using Matplotlib.
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 32 / 48
52. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Converting your NumPy arrays to VTK files
The Visualization Toolkit (VTK) is an open-source software for 3D
computer graphics, image processing, volume rendering
At its core, VTK is implemented as a C++ toolkit;
supports automated wrapping of the C++ core into Python, Java, and Tcl
1 from pyevtk.hl import gridToVTK
2 h=k=l = np.linspace(-1.1, 1.1, num=N, endpoint=True, dtype='
float64')
3 gridToVTK("/viz/vtk/" + filename, h, k, l, pointData = {"pow_spec
" : pow_spec})
PyEVTK: a great little package by a Paulo Herrera
1 $ pip install pyevtk
2 save NumPy arrays straight to different types of VTK XML-based
3 visualize and process your NumPy arrays with any of the flagship
VTK applications such as ParaView, VisIt, Mayavi, main workhorses
for 3D analysis
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 33 / 48
53. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
What is ParaView?
ParaView is an open-source application and architecture
display and analysis of scientific data sets;
ParaView has been demonstrated to process billions of
unstructured cells and to process over a trillion structured
cells;
ParaView’s parallel framework has run on over 100,000
processing cores (from notebooks to world’s largest supercomputers);
ParaView’s key features are:
An open-source scalable, multi-platform for visualizing 2D/3D
data (excels at traditional scientific vis qualitative 3D rendering);
An extensible, modular architecture based on open standards
e.g. Custom apps, plugins, Python scripting, ParaViewWeb, Catalyst
Support for distributed computation models to process
large data sets;
An open, extensible, and intuitive user interface.
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 34 / 48
54. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: The dataset types (sampling structures)
ParaView was designed primarily to handle data with spatial
representation. Thus the data types used in ParaView are meshes
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 35 / 48
55. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: a standard de-facto
ParaView is primarily developed and published by Kitware
Inc.9
A flexible BSD 3-clause license;
Commercial maintenance and support;
ParaView is used by many academic, government, and
commercial institutions all over the world;
ParaView is downloaded roughly 100,000 times every year;
ParaView also won the HPCwire Readers’ Choice Award and
HPCwire Editors’ Choice Award for Best HPC Visualization
Product or Technology.
9
MPCDF has no self-interests nor any conflict of interest
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 36 / 48
56. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: The architecture
The application most people associate with ParaView is really just
a small client application built on top of a tall stack of libraries
that provide ParaView with its functionality.
a tall stack of libraries that provide ParaView with its functionality.
comes with a pvpython application that allows you to automate the visualization and
essing with Python scripting.
w Server library provides the abstraction layer necessary for running parallel, interactive
on. It relieves the client application from most of the issues concerning if and how
is running in parallel.
lization Toolkit (VTK) provides the basic visualization and rendering algorithms.
openGL MPI IceT
VTK
ParaView Server
UI (Qt Widgets, Python Wrappings)
ParaView Client pvpython Custom app
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 37 / 48
57. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
58. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
59. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
60. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
61. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
62. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
63. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
ParaView: Loading, filtering and rendering
GUI elements
Menu
Toolbars
Pipeline
Inspector
Help
Views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 38 / 48
64. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Python in Paraview ... believe or not
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 39 / 48
65. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
myparaview.py: change parameters
1 #### import the simple module from the paraview
2 from paraview.simple import *
3 #### disable automatic camera reset on 'Show'
4 paraview.simple._DisableFirstRenderCameraReset()
5
6 # create a new 'XML Rectilinear Grid Reader'
7 fccBigData_isosurf128vtr = XMLRectilinearGridReader(FileName=['/
Users/giuseppe/Projects/MPI-E_dev/viz/vtk/
fccBigData_isosurf128.vtr'])
8 fccBigData_isosurf128vtr.PointArrayStatus = ['iso_surf']
9 # get active view
10 renderView1 = GetActiveViewOrCreate('RenderView')
11 # uncomment following to set a specific view size
12 # renderView1.ViewSize = [951, 573]
13 # show data in view
14 fccBigData_isosurf128vtrDisplay = Show(fccBigData_isosurf128vtr,
renderView1)
15 # trace defaults for the display properties.
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 40 / 48
66. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
myparaview.py: use filters
18 fccBigData_isosurf128vtrDisplay.Representation = 'Outline'
19 fccBigData_isosurf128vtrDisplay.ColorArrayName = [None, '']
20 fccBigData_isosurf128vtrDisplay.OSPRayScaleArray = 'iso_surf'
21 fccBigData_isosurf128vtrDisplay.OSPRayScaleFunction = '
PiecewiseFunction'
22 fccBigData_isosurf128vtrDisplay.SelectOrientationVectors = 'None'
23 fccBigData_isosurf128vtrDisplay.ScaleFactor = 0.22000000000000003
24 fccBigData_isosurf128vtrDisplay.SelectScaleArray = 'None'
25 fccBigData_isosurf128vtrDisplay.GlyphType = 'Arrow'
26 fccBigData_isosurf128vtrDisplay.PolarAxes = '
PolarAxesRepresentation'
27 fccBigData_isosurf128vtrDisplay.GaussianRadius =
0.11000000000000001
28 fccBigData_isosurf128vtrDisplay.SetScaleArray = ['POINTS', '
iso_surf']
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 41 / 48
67. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
myparaview.py: create a visualization
31 fccBigData_isosurf128vtrDisplay.ScaleTransferFunction = '
PiecewiseFunction'
32 # hide data in view
33 Hide(fccBigData_isosurf128vtr, renderView1)
34
35 #### saving camera placements for all active views
36
37 # current camera placement for renderView1
38 renderView1.CameraPosition = [4.572332349968719,
-5.768623046054573, -0.07839074501841727]
39 renderView1.CameraViewUp = [0.3363494710978258,
0.2542755863640535, 0.9067595929856104]
40 renderView1.CameraParallelScale = 1.9052559296210743
41
42 #### uncomment the following to render all views
43 # RenderAllViews()
44 # alternatively, if you want to write images, you can use
SaveScreenshot(...).
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 42 / 48
68. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Filters Menu
...some of them
1 Calculator: evaluates a
user-defined expression on a
per-point or per-cell basis
2 Contour: extracts the
points, curves, or surfaces
where a scalar field is equal
to a user-defined value. This
surface is often also called
iso-surface
3 Threshold: extracts cells
that lie within a specified
range of a scalar-field24
• Filters Menu
– Recent
– Common
– Data Analysis
– Statistical
– Temporal
– Alphabetical
• Quick Launch
– PC/Linux
CTRL-Space
– Mac
ALT-Space
• Apply Undo/Redo
Filters
Calculator
Contour
Clip
Slice
Threshold
Extract
Subset
Glyph
Stream Tracer
Warp By Vector
Group Datasets
Extract Group
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 43 / 48
69. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Paraview: Multiple views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 44 / 48
70. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Paraview: Multiple views
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 44 / 48
71. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Query Data by Attribute Values
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 45 / 48
72. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Query Data by Attribute Values
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 45 / 48
73. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Query Data Spatially - Selection
...by clicking and dragging the mouse in the 3D view
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 46 / 48
74. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Next: extract the tuple of radius and angles
x
y
z
φ
θ
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 47 / 48
75. X-ray Crystallography: a short overview Big Data Analytics at MPCDF Visualization of Crystal Nano-Structure
Thank you for the attention!
Giuseppe Di Bernardo (MPCDF) GPU Crystallography with Python July 12, 2017 48 / 48