By Balazs Gerofi, RIKEN Advanced Institute For Computational Science RIKEN Advanced Institute for Computation Science is in charge of leading the development of Japan's next generation flagship supercomputer, the successor of the K. Part of this effort is to design and develop a system software stack that suits the needs of future extreme scale computing. In this talk, we focus on operating system (OS) requirements for HPC and discuss IHK/McKernel, a multi-kernel based operating system framework. IHK/McKernel runs Linux with a light-weight kernel (LWK) side-by-side on compute nodes with the primary motivation of providing scalable, consistent performance for large scale HPC simulations, but at the same time to retain a fully Linux compatible execution environment. We provide an overview of the project and discuss the status of its support for ARM architecture. Balazs Gerofi Bio Research Scientist at RIKEN Advanced Institute For Computational Science. Email bgerofi@riken.jp For more info on The Linaro High Performance Computing (HPC) visit https://www.linaro.org/sig/hpc/

1. An Evaluation of EasyBuild for
Open Source Software Deployment
Arm Environment
14:00 - 14:25
12th December 2017 - Arm HPC Workshop, Akihabara
System Software Development Team
RIKEN Advanced Institute for Computational Science
Takahiro OGURA
1

2. Agenda
2
1. Problems you might encountered when
building/executing OSS
2. How EasyBuild helps solving these problems
3. Structure of EasyBuild
4. Testing EasyBuild on Arm architecture
5. Creating recipes for EasyBuild
6. Conclusions

3. Problems you might encounter when building/executing OSS
3
 If the OSS (= HPC software stack not in Linux distro., such as PETSc)
you want use is not installed on the system you have to build it by your
own.
Problems you might encountered:
 It is time-consuming to obtain all the OSSs needed to build the target OSS
(ex. PETSc requires 40 OSSs to build)
 It is confusing and time-consuming to set/switch the environmental
variables
 Environment dependent build settings
 Lack of architecture support other than x86 (e.g., SPARC and Arm) often requires
manual patching
 Similarly for vendor-specific compilers and/or cross compilers

4. How to solve OSS building/executing problem
4
 Similar problems can also occur on post K computer
 Linaro’s effort solves many of them, but won’t cover all
 Operators should install OSSs instead of individual users
 Operator should share know-hows
 CEA in France is using EasyBuild to manage OSSs on x86 Xeon clusters to ease
the OSS building/execution
 EasyBuild: Framework for building and managing OSS commonly used in HPC
 Step 1. Understand EasyBuild’s framework and how it works
 Step 2. Test EasyBuild on Arm architecture machine
 Step 3. Test EasyBuild on cross compiling environment with vender-provided compiler
 Currently at Step 2.

5. How EasyBuild works with PETSc
5
 40 OSSs needed to build PETSc 3.7.3
 Some of these provided by RPMs
 Some of these provided by source (tar.gz) from the web sites
 Some of these provided by github repository
Autoconf-2.69
Automake-1.15
Autotools-20150215
Autotools-20150215-foss-2016a
binutils-2.25
Bison-3.0.4
Boost-1.61.0
bzip2-1.0.6
CMake-3.4.3
CMake-3.5.2
FFTW-3.3.4
FIAT-1.6.0
flex-2.5.39
GCC-4.9.3-2.25
GCCcore-4.9.3
GMP-6.1.0
gompi
hwloc-1.11.2-GCC-4.9.3-2.25
Hypre-2.11.1
libreadline-6.3
libtool-2.4.6
M4-1.4.17
METIS-5.1.0
mpmath-0.19
ncurses-6.0
numactl-2.0.11
numpy-1.8.2
OpenBLAS-0.2.15
OpenMPI-1.10.2
ParMETIS-4.0.3
Python-2.7.11
ScaLAPACK-2.0.2
ScientificPython-2.9.4
SCOTCH-6.0.4
SQLite-3.9.2
SuiteSparse-4.5.3
sympy-1.0
Tcl-8.6.4
Tk-8.6.4
zlib-1.2.8

6. How EasyBuild collects dependences
6
 Execute the following command:
$ eb PETSc-3.7.3-foss-2016a-Python-2.7.11.eb
 Then download, git clone the source and build by following
EasyBuild recipes
 However, EasyBuild does not support RPMs...
 You have to build and install OSSs even RPMs are provided
officially

7. Setting environmental variables
7
 You need to set the environmental variable such as
INCLUDE_PATH, LD_LIBRARY_PATH, PATH and etc. to build
the OSS
 You need to set the environmental variable such as PATH to
execute the OSS
 EasyBuild creates environment modules automatically
 Environment modules allows users set and unset the
environmental variables by a single command:
$ module load xxxx

8. How EasyBuild works
8
1. execute eb command. Specify Easyconfig file.
$ eb PETSc-3.7.3-foss-2016a-Python-2.7.11.eb
PETSc-3.7.3-foss-2016a-Python-2.7.11.eb
Easyconfig
(text file)
2. fetch sources
3. unpack sources
4. apply patches
5. configure
8. sanity check
7. install
6. build/make
Easyblock
(Python2 Script)
Internet
Download
git clone
Environment
Modulefiles
Include
headers
Executable
Binary files
Executable
Binary files
Executable
Binary files
Library
Binary files
Include
headers
Functions from
EasyBuild Framework

9. 9
 EasyBuild framework
 Python2 API set class library to use EasyBuild
 Python3 in not supported yet
 Easyblock
 Script written in Python2 using EasyBuild framework
 Need Easyblock for each OSS
 Easyconfig
 Text setting file which Easyblock reads
 Modify when changing building settings such as configure options
 Written in Key = Value format
Structure of EasyBuild
For more info: https://easybuild.readthedocs.io/en/latest/Concepts_and_Terminology.html

10. 10
 A part of classes of EasyBuild framework
 easybuild.framework.easyblock.EasyBlock
 Basic class for Easyblock
building and install software
load Enviroment Module
 easybuild.framework.easyconfig.easyconfig.EasyConfig
 Class to read Easyconfig
 easybuild.framework.easyconfig.easyconfig.copy_easyconfigs
 Class to copy Easyconfig file to designated path
 easybuild.framework.easyconfig.easyconfig.copy_patch_files
 Class to copy patch file to designated path
 easybuild.framework.easyconfig.default.get_easyconfig_parameter
_default
 Class to get Easyconfig values.
EasyBuild framework
For more info: https://easybuild.readthedocs.io/en/latest/api/easybuild.framework.html

11. 11
Major settings in Easyconfig
Setting Item Name(Key) Examples of value Description
Easyblock EB_PIP Name of Easyblock file.
name PIP Name of software.
version 0.1 Version of OSS.
homepage http://www.riken.jp/ Web site URL of OSS (Top page.)
description PiP is a library to … Description of OSS.
toolchain {'name': 'GCC', 'version': '4.9.3-2.25'} Dependence OSS name.
builddependencies ('PIP-glibc', '2.17') Dependence OSS name when building target OSS.
osdependencies ('nss-devel', '') Name of package which must installed in OS when
building target OSS.
sources ['%(name)s.tar.gz'] Name of source file.
source_urls ['http://www.python.org/ftp/%(namelo
wer)s/%(version)s/']
URL of source code tar ball.
patches ['PETSc_ranlib-fix.patch'] Name of patch file if needed.
configopts/buildopts/inst
allopts
'CC=gcc CXX=g++ --enable-add-ons' options for configure/make/make install
runtest 'check ' setting for test after make.
sanity_check_paths {'files': "bin/hello" } Check file/directory presence.
For more info: https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_parameters.html

12. 12
Step2. Test EasyBuild on Arm architecture machine
Testing environment
Hardware
• SoftIron OverDrive 3000
• CPU: AMD Opteron A1170
(2GHz 8 cores)
• memory: 16GB
• Network：10Gbase-T * 2
• Disk: 1TB * 1
Software
• OS: CentOS 7.3.1611
• Kernel: 4.5.0-29.el7.aarch64
• Python: 2.7.5
• Lua: 5.1.4
• Lmod: 7.6
• GCC: 4.8.5
• EasyBuild: 3.4.0, 3.4.1
Requirement for EasyBuild
• Linux OS
• Python 2.6 or higher
• Lmod or Tcl
※ Lmod is suggested
• Lua
• lua-filesystem
• lua-posix
• lua-devel
• libreadline
• ncurses
• C/C++ Compiler (GNU gcc etc.)
For more info：https://easybuild.readthedocs.io/en/latest/Installation.html

13. 13
 Installing Lmod
 No RPM for Lmod is available for Aarch64
So, we built it from the source code
Installing EasyBuild(1/2)
For more info：https://easybuild.readthedocs.io/en/latest/Installing-Lmod-without-root-permissions.html#installing-lmod
$ curl -LO http://sourceforge.net/projects/lmod/files/Lmod-7.6.tar.bz2
$ tar xfvj Lmod-7.6.tar.bz2 && cd Lmod-7.6
$ ./configure --prefix="<install path>" && make install
• Add PATH settings to .bashrc
• Test Lmod
$ lmod --version
Modules based on Lua: Version 7.6 (7.6-3-gbbb0e9b7) 2017-07-24 14:07 -05:00
by Robert McLay mclay@tacc.utexas.edu
export PATH=$HOME/lmod/7.6/libexec:$PATH

14. 14
 Installing EasyBuild
 Set environmental variables
 Download bootstrap script
 execute install script
 Set Environment Modules PATH and test eb command
Installing EasyBuild(2/2)
For more info：https://easyBuild.readthedocs.io/en/latest/Installation.html
For more info ：https://easybuild.readthedocs.io/en/latest/Installation_Alternative.html
$ export EASYBUILD_PREFIX=$HOME/.local/easybuild
$ curl -O https://raw.githubusercontent.com/easybuilders/easybuild-framework/develop/easybuil
d/scripts/bootstrap_eb.py
$ python bootstrap_eb.py $EASYBUILD_PREFIX
$ module use $/EASYBUILD_PREFIX/modules/all
$ module load EasyBuild
$ eb --version
This is EasyBuild 3.4.1 (framework: 3.4.1, easyblocks: 3.4.1) on host ***.aics.riken.jp.

15. 15
 PIP is a software being developed by our team
 PIP requires glibc to build and execute
 Make 2 sets (for glibc and PIP) of Easyblock and Easyconfig
Building PIP on Arm/Creating recipes for PIP
Easyconfig
PIP-0.1.eb
Easyblock
pip.py
Easyblock
pip_glibc.py
Easyconfig
PIP-glibc-2.
17.eb

16. 16
 pip_glibc.py = 100 lines, pip.py = 10 lines
 Rules for Easyblock classes name to add
 Name of a new Python class for each EasyBlock must start
from “EB_” and the target software name.
 Simply execute encode_class_name() to name new Python class
 Rules for Easyblock file name.
 To name a new Easyblock file simply execute
get_module_path().
Making Easyblock for PIP
For more info: https://easybuild.readthedocs.io/en/latest/Implementing-easyblocks.html
>>> from easybuild.tools.filetools import encode_class_name
>>> encode_class_name('PIP-glibc')
'EB_PIP_minus_glibc’ ←New class name
>>> from from easybuild.framework.easyconfig.easyconfig import get_module_path
>>> get_module_path('PIP-glibc')
'easybuild.easyblocks.pip_glibc’ ←New Easyblock file name

17. 17
pip.py (Easyblock for PIP)
class EB_PIP(ConfigureMake): # Class definition (Inherit ConfigureMake class)
def configure_step(self): # Function definition
# statically link to PIP-glib if it is a (build) dependency
pipglibc = get_software_root('PIP-glibc') # Get PIP-glibc path
if pipglibc:
glibc_libs = os.path.join(pipglibc, "lib") # Get path of lib directory for PIP-glibc
# Set configure options
cmd = './configure --prefix="%s" --with-glibc-libdir="%s"' % (self.installdir, glibc_libs)
# run configure
run_cmd(cmd, log_all=True, simple=True) # Execute configure and make install

18. 18
pip_glibc.py (Easyblock for PIP-glibc)(1/2)
＜snip＞
import easybuild.tools.environment as env
from easybuild.easyblocks.generic.configuremake import ConfigureMake # import configuremake module
from easybuild.tools.build_log import EasyBuildError # import build_log module
from easybuild.tools.run import run_cmd # import run module
class EB_PIP_minus_glibc(ConfigureMake): # Inherit ConfugureMake and define EB_PIP_minus_gbibc
def create_dir(self, dirname): # function to create new dir
"""
Create a dir to build in.
"""
dirpath = os.path.join(self.cfg['start_dir'], dirname)
try:
os.mkdir(dirpath)
os.chdir(dirpath)
self.log.debug("Created dir at %s" % dirpath)
return dirpath
except OSError, err:
raise EasyBuildError("Can't use dir %s to build in: %s", dirpath, err)
(go to next page)

19. 19
(continue from previous page)
def run_configure_cmd(self, cmd):
# Get a value “preconfigopts” from Easyconfig
(out, ec) = run_cmd("%s %s" % (self.cfg['preconfigopts'], cmd), log_all=True, simple=False)
if ec != 0:
raise EasyBuildError("Command '%s' exited with exit code != 0 (%s)", cmd, ec)
unknown_re = re.compile("WARNING: unrecognized options")
unknown_options = unknown_re.findall(out)
if unknown_options:
raise EasyBuildError("Unrecognized options found during configure: %s", unknown_options)
def configure_step(self): # overwrite configure_step function
configopts = ''
# configopts is only used in first configure
self.configopts = self.cfg['configopts'] # Get a value of “configopts” from Easyconfig
self.create_dir(self, "glibcobj")
# actual configure, but not on default path
# Use value from Easyconfig as a configure option
cmd = '../configure --prefix="%s" %s %s' % (self.installdir, self.configopts, configopts)
self.run_configure_cmd(cmd)
＜snip＞
pip_glibc.py(Easyblock for PIP-glibc)(2/2)

20. 20
Easyconfig for PIP/PIP-0.1.eb(1/2)
For more info: https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_parameters.html
KEY=VALUE Description
Easyblock = 'EB_PIP' Use EB_PIP as an Easyblock
name = 'PIP' Environment Module name
version = '0.1' Environment Module version
homepage = 'http://www.riken.jp/' Web site of OSS
description = "PiP is a library to create processes
in process virtual address space implemented at
user-level."
Description for the software
Used as Environment Module description
toolchain = {'name': 'dummy', 'version': ‘’} Tool-chain to use
builddependencies = [
('PIP-glibc', '2.17'),
]
Define dependency to build PIP
sources = ['%(name)s.tar.gz’] Path to the source code
To Download the source code define “source_urls”
（Go to next page）

21. 21
Easyconfig for PIP/PIP-0.1.eb(2/2)
KEY = VALUE Description
sanity_check_paths = {
'files': [
"bin/hello",
＜snip＞
],
'dirs': [
"bin/",
＜snip＞
], }
Check presence of directories and files
runtest = 'check ' Before install run make check
sanity_check_commands =
[('%(installdir)s/bin/hello', ‘’), ]
How to check the presence of directories and files
postinstallcmds =
[('%(installdir)s/bin/piplnlibs’), ]
Specify command after install_step(make install)
moduleclass = 'lib' Define software type
Can check by “$ eb –moduleclasses” command
For more info: https://easybuild.readthedocs.io/en/latest/version-specific/easyconfig_parameters.html

22. 22
Run PIP-0.1_Aarch64.eb
$ eb PIP-0.1_Aarch64.eb
== temporary log file in case of crash /tmp/eb-AviEmJ/easybuild-_KRKyK.log
== processing EasyBuild Easyconfig /home/user/PIP-0.1_Aarch64.eb
== building and installing PIP/0.1...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully
== Results of the build can be found in the log file(s) /home/user/easybuild/software/PIP/0.1/easybuild/easybuild-PIP-0.1-20171030.16383
9.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-AviEmJ/easybuild-_KRKyK.log* have been removed.
== Temporary directory /tmp/eb-AviEmJ has been removed.
$ module load PIP/0.1
$ hello
Hello (PID=3038)!!

23. Conclusions
23
 EasyBuild has potential for solving typical OSS
building/executing problems
 However, to learn/understand EasyBuild framework it takes
almost a month (for Python beginners) and the EasyBuild
framework needs more functions such as support for RPM,
cross-compiling features, etc.
 Once we understood the EasyBuild framework, it took only
a week to create Easyblock and Easyconfig for our software
 We will further evaluate whether EasyBuild can adopt cross-
compiling environments with vendor-provided compilers

24. 24
Thank you for your attention!
Any questions or comments?