YouTube: https://www.youtube.com/watch?v=MsXhC-Kb7SI&index=47&list=PLnKL6-WWWE_VNp6tUznu7Ca8hBF8yjKj2

1. www.icm.edu.pl
InfiniCortex
A Supercomputer
that encompasses the world
Marek Michalewicz
Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw, Poland
Institute for Advanced Computational Science, Stony Brook University, USA
EmTech Asia, Singapore, 14 February 2017

2. www.icm.edu.pl
CSIRO,
Australia
A*STAR,
Singapore
Nano-
technology
BREAK
My life - in (computing scale) numbers
1,000x >100x
>100x
Giga
Tera
Peta
1,000,000,000x memory and 100,000,000x computing speed increase!

3. www.icm.edu.pl
0
250
500
750
1000
1250
1500
2009 2010 2011 2012 2013 2014 2015 2016
SGI KNL
NSCC Fujitsu
IBM BlueGene Q Cumulus
IBM Power7 Cirrus
SGI Aurora
Fujitsu Fuji
HP Axle
ACRC old systems -10 computers!
TFLOPS
Growth of computational power
in Singapore

4. 4
Level 17 at Fusionopolis
A*CRC Datacenter 1

5. 5
A*CRC Datacenter 2
Matrix Building at Biopolis

6. Mellanox Metro-X testing since early 2013
goal: to connect HPC resources at Fusionopolis
with storage and genomics pipeline at Biopolis
A*CRC Metro-X testing team:
Stephen Wong
Tay Teck Wee
Steven Chew

8. www.icm.edu.pl
NOT GRID!
NOT CLOUD!
NOT “Internet”!
InfiniCortex is …

9. www.icm.edu.pl
The InfiniCortex uses a metaphor of a human brain’s outer layer, the Cortex, consisting
of highly connected and dense network of neurons enabling thinking ….
to deliver concurrent supercomputing across the globe utilising trans-continental
InfiniBand and Galaxy of Supercomputers
InfiniCortex is like a living global brain

10. www.icm.edu.pl
InfiniCortex Components
1. Galaxy of Supercomputers
• Supercomputer interconnect topology work
by Y. Deng, M. Michalewicz and L. Orlowski
• Obsidian Strategics Crossbow InfiniBand router
2. ACA 100 & ACE 10
• Asia Connects America 100 Gbps, by November 2014
• Asia Connects Europe 10Gbps, established February 2015
• SingAREN-TEIN-CC and SingAREN-Internet2 connections
3. InfiniBand over trans-continental distances
• Using Obsidian Strategics Longbow range extenders
• round the globe InfiniBand connectivity established by 2015
4. Application layer
• from simplest file transfer: dsync+
• to complex workflows: ADIOS, multi-scale models

11. www.icm.edu.pl
TITECH
(Tokyo)
A*STAR
(Singapore)
NCI
(Canberra)
Seattle
SC14
(New Orleans)
GA Tech
(Atlanta)
10Gbps InfiniBand
100Gbps InfiniBand
Enabling geographically dispersed HPC facilities to collaborate and function as ONE concurrent supercomputer,
bringing the capability to address and solve grand challenges to the next level of efficiency and scale.
InfiniCortex 2014 (phase 1)

12. www.icm.edu.pl

13. www.icm.edu.plIn collaboration with NCI ANU Andrew Howard and Jakub Chrzeszczyk

14. www.icm.edu.pl
10Gbps InfiniBand
100Gbps InfiniBand
100Gbps InfiniBand East-ward link: Singapore-trans-Pacific-USA-trans-Atlantic-Europe
10Gbps InfiniBand West-ward link: Singapore-Europe (via TEIN*CC)
InfiniCortex 2015 InfiniBand ring-around the
World
TITECH
(Tokyo)
NCI
(Canberra)
Seattle
GA Tech
(Atlanta)SC15
(Austin,TX)
PSNC
(Poznan)
A*STAR
(Singapore)
URCA
(Reims)

15. www.icm.edu.pl
largest spanning
InfiniBand network:
ring-around-the-world
with up-to100Gbps
sections Singapore-
USA
InfiniBand routing
across 7 subnets over
4 continents

16. www.icm.edu.pl
InfiniCloud 2015
True HPC Cloud around the Globe

17. www.icm.edu.pl
Running the pipeline on the virtual InfiniCloud cluster
Virtual
globally
distributed
cluster
CPU
Network
Cancer mutation calling pipeline
Output of mutations
geopipeline-compute001 | australia |
geopipeline-compute002 | australia |
geopipeline-compute003 | singapore |
geopipeline-compute004 | europe |
geopipeline-frontend001 | nova |

18. www.icm.edu.pl
7 InfiniBand sub-nets
7 countries: Singapore, USA, Australia, Japan,
Poland, France, Canada
100Gbps Singapore-Austin
10-30Gbps rest of network
~15 Universities and Research
entities
~40 partners and growing
HPC InfiniCloud over 4
continents

19. InfiniCortex demo, SC16, Salt Lake City,
UT, USA

20. www.icm.edu.pl

21. www.icm.edu.pl
No. Description
Quantit
y
Unit Price
Unit Price
S$
Total Price
S$
A*CRC
Spending
s
NSCC
Spending
s
Remarks
1 100G link from SG-LA 1 USD 105,000 149 100 149 100 0 74 550
50/50 Co-sharing between NSCC and Internet2-
Sponsored
2 100G link from LA to Austin 1 USD 100,000 142 000 142 000 0 0 Sponsored by Centurylink/SCINET
3 100G Cards for SLIX 4 66 250 SGD 66 250 265 000 0 0 Sponsored by Ciena/3D
4
Cross-Connect at Los
Angeles
1 USD, 10 000 14 200 14 200 0 0 Sponsored by SCINET
5 10G SG-Europe Link 1 EURO 16,000 24 800 24 800 0 12 400
50-50 Co-sharing between NSCC and TEIN*CC-
Sponsored
6 30G Poznan to Austin 1 USD 100,000 142 000 142 000 0 0 Sponsored by GEANT/Internet2/Pioneer/Poznan/Scinet
7
Connection from SCINET to
ACRC Booth
USD 7000 9 940 9 940 9 940 0
8 100G Network Switches 2 300 000 SGD 300 000 600 000 0 0 Sponsored by Arista
9
Long Haul InfiniBand
Switches and Routers
38 USD 135,000 191 700 7 284 600 0 0
34 InfiniBand switches and 1x InfiniBand Router was
sponsored by Obsidian
4 x InfiniBand switches and 1x InfiniBand Router are
A*CRC existing equipments using FY14 funds.
10
Long Haul InfiniBand
Routers
2 USD 12,500 17 750 35 500 0 0
1 x InfiniBand Router was A*CRC existing equipments
using FY14 funds
1 x InfiniBand Router was sponsored by Obsidian
11 40G Infiniband Switches 2 USD 22,500 31 950 63 900 0 0
Sponsored by Obsidian Strategics for inter-booth
connection
12 Servers 10 USD 15,000 21 300 213 000 0 0 Sponsored by HPE/Mellanox/Intel/NVIDIA
13 Storages 1 USD 52,260 74 210,00 74 210 0 0 Sponsored by DDN

22. www.icm.edu.pl
NGSP Sequencers at B2 (Illumina + PacBio)
NSCC Gateway
GIS
NSCC
STEP 2: Automated pipeline analysis once sequencing
completes. Processed data resides in NSCC
500Gbps Primary
link
Data Manager
STEP 3: Data manager indexes and
annotates processed data. Replicate
metadata to GIS. Allowing data to be
search ed and retrieved
Data
Manager
Compute
Tiered Storage
POLARIS, Genotyping and other
Platforms in L4~L8
Tiered Storage
STEP 1: Sequencers stream directly to NSCC
Storage (NO footprint in GIS)
Compute
1 Gbps per sequencer
10 Gbps
1 Gbps per machine
100 Gbps
10 Gbps
Genomic Institute of Singapore - National Supercomputing Centr
(GIS-NSCC) Integration

23. Science, Technology and Research Network (STAR-N) connects all
National Supercomputing Centre stakeholders: A*STAR, NUS, NTU and
Industrial users with 100Gbps + InfiniBand links.
NUS
NTU
A*STAR
Fusionopolis
SingAREN
Global Switch
A*STAR
Biopolis
WOODLANDS
100Gbps InfiniBand
500Gbps Infinera Cloud Express
10/40/100Gbps InfiniBand/IP
SELETAR
CHANGI
NOVENA
OUTRAM
ONE-NORTH
JURONG
Singapore InfiniBand connectivity
• A high bandwidth network
to connect the distributed
login nodes
• Provide high speed access
to users (both public and
private) anywhere
• Support transfer of large
data-sets (both locally and
internationally)
• Builds local and
international network
connectivity (Internet 2,
TEIN*CC)
• ASEAN, USA Europe,
Australia, Japan,
Middle East

24. www.icm.edu.pl

25. www.icm.edu.pl
Interdisciplinary Centre for Mathematical and
Computational Modelling, University of Warsaw
People at ICM:
• researchers,
• software developers,
• IT administration,
• HPC experts
Hardware
• 5 HPC installations, 2.1 PetaFlops peak
performance (CPU’s and GPU’s).
• dedicated installation for DataScience
Premises
• Supercomputer and Data center at Białołęka
• Backup installation at Kampus Ochota
A base:

26. www.icm.edu.pl
A selection of ICM’s services:
Computational grants allocation
• 350 active computational grants
• over 280 scientific publications in 2016
• 180 millions cpu hours in 2016
Meteo - numerical weather forecast
• 4 independent models, first service in production in Poland
• over 150 millions of visits a year
Virtural Library
• Access to the scientific publications for all academics in Poland
• In collection: Springer, Elsevier, Wiley, ACS, and others
• An access to single article within countrywide licence ~ 1€,
instead of ~ 30€

27. www.icm.edu.pl
• VisNow
• Generic open-source platform for data analysis and visualization, developed from
scratch at ICM
• Multiplatform, extensible, Java software
• Supporting large-scale datasets
• Image-based analysis of tissue regeneration process
• Classic approach to image-based features in MRI imaging
• Deep Learning techniques for machine learning classification
• Novel T2 mapping based imaging
• Quantitative automated assessment of tissue pathology
for Achilles tendon
Strength of ICM: projects

28. Strength of ICM: projects
Timothy
Timothy is a novel large scale parallel
computational model allowing 3-D simulations
of cell colonies growing and interacting with
variable environment in previously unavailable
tissue scale.
Vessel with concentration gradient
of oxygen in the surrounding tissue 3-D solic tumor development in
oxygen environment (necrotic core)

29. CERMINE
• Problem: Automatic extraction of information from PDFs with
scientific papers
• Learns automatically from data using modern machine learning
methods (CRF, SVM, k-means, …)
• Award winning ESWC 2015 SemPub Best Performing Approach
• Web service http://cermine.icm.edu.pl
• Used by:
Strength of ICM: projects

30. www.icm.edu.pl
Cosmology
• Evolution of
the Universe
Epidemiology
• Agent Based Modelling of the Influenza
Disease spread across Poland
• Census data based realistic and
georeferential society model
• Model includes many details:
- daily commuting and long-therm mobility
• - immunological distance
Computational and algorithmic support
for scientific projects:

31. www.icm.edu.pl
PCJ is a library for Java language that helps to
perform parallel and distributed calculations.
• Actively developed
• Local and remote memory access
• Resilience
• Program deployment
• Able to work on the multicore systems connected
with the typical interconnect such as ethernet or
infiniband providing users with the uniform view
across nodes.
• HPC Challenge Class 2 Best Productivity Award
• http://pcj.icm.edu.pl

32. PSNC: 1
Cyfronet: 2.4
WCNS: 1
ICM: 1.3
TASK: 1
PIONIER Academic Network
Consortium coordinated by
PSNC, Poznan
7,500 km own fiber
Five Academic Supercomputing
Centers, combined ~6.7 PFLOPS
Polish Academic Supercomputing and Networking Landscape, 2016

33. www.icm.edu.pl
Interdisciplinary Centre
for Mathematical and Computational Modelling
University of Warsaw, Poland

34. Poznan Supercomputing and
Networking Centre
SC16: booth
501
CYFRONET, Krakow

35. InfiniCortex Team
With help from:
SingAREN
A/Prof Francis Lee
Prof Lawrence
Wong
NTU
Stanley Goh
A*CRC
Tan Geok Lian (Networking)
Lim Seng (Networking)
Dr Jonathan Low (H/W, S/W,
Applications)
Dr Dominic Chien (S/W, Applications)
Dr Liou Sing-Wu (S/W, Applications)
Dr Gabriel Noaje (S/W, Applications)
Lukasz Orlowski (S/W, Applications)
A/Prof Tan Tin Wee (PI) Yves Poppe,
International
connectivity
Prof Yuefan DengDr Marek Michalewicz (PI) Dr David Southwell

36. (most) Project Partners 2014-2016
Huawei, HPE, Fujitsu, Aspera, Bright Cluster, Altair,
ByteScale, AristaFermiLab, George Washington University
Team Europe
GEANT, TEIN,
France: University of Reims,
Poland: PSNC, ICM

37. www.icm.edu.pl
Bell's law of computer classes
Roughly every decade a new, lower priced computer class forms based on a new
programming platform, network, and interface resulting in new usage and the
establishment of a new industry.
Computer classes that conform to the law
• mainframes (1960s)
• minicomputers (1970s)[3] These are essentially replaced by clusters of PCs for a twenty-year (1965-1985)
lifespan. [4]
• personal computers and workstations evolving into a network enabled by Local Area Networking or
Ethernet (1980s)
• web browser client-server structures enabled by the Internet (1990s)
• cloud computing, e.g. Amazon Web Services,{2006} or Microsoft Azure (2012)
• hand held devices from media players and cell phones to tablets, e.g.Creative, iPods, BlackBerrys, iPhones,
Smartphones, Kindles, iPads (c. 2000-2010)
• Wireless Sensor Networks that enable sensor [5] and actuator interconnection, enabling the evolving
Internet of Things. (c. >2005)
Can we consider InfiniCortex to be a new class of computers in Bell’s
Law sense?