EUDAT Porto Conference 2018

1. DEEP-Hybrid-DataCloud has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 777435.
DEEP general presentation
Brief project overview
Jesús Marco de Lucas, Álvaro López García
{marco,aloga}@ifca.unican.es
Spanish National Research Council
DEEP & XDC Kick of Meeting
Bologna, Italy
January 24, 2018

2. https://deep-hybrid-datacloud.eu Jan 24, 2018 2/12
DEEP-Hybrid-DataCloud: context
● H2020 project, EINFRA-21 call
– Topic: Platorm-driven e-Infrastructure towards the European Open Science Cloud
– Scope: Computng e-infrastructure with extreme large datasets
● (…) prototypes to cope with very large data resources (...)
● (…) sofware layers supportng applicatons such as modelling, simulaton, patern recogniton, visualisaton, etc. (...)
● (…) supported by robust mathematcal methods and tools (...)
● (…) Clean slate approaches to high-performance computng and data management (…)
● DEEP-Hybrid-DatCloud: Designing and Enabling E-Infrastructures for intensive data Processing
in a Hybrid DataCloud
– Started as a spin-of project (together with XDC) from INDIGO-DataCloud technologies
– Submited in March 2017
– Started November 1st 2017
– Grant agreement number 777435
● Global objectie: Promote the use of intensiie computnn seriices by diferent research
communites and areas, an the support by the corresponding e-Infrastructure providers and
open source projects

3. https://deep-hybrid-datacloud.eu Jan 24, 2018 3/12
DEEP project objectives
● Focus on intensiie computnn techniques for the analysis of iery larne datasets considering demanding use cases
– Pilot applicatons from diferent research communites
– Three techniques of wide interest: deep learning, post processing and on-line analysis of data streams
– Improved list of requirements for e-Infrastructures → future generaton
● Eiolie up to producton level intensiie computnn seriices exploitng specialized hardware
– New solutons to beter interact with bare metal resources in the cloud
– Use of hardware accelerators such as GPUs and low-latency interconnects
● Intenrate intensive computng services under a hybrid cloud approach
– Assuring interoperability with existng EOSC
– Expanding over multple IaaS using high level networking technologies
– Enrich orchestraton tools for supportng multple services and providers
● Defne a “DEEP as a Seriice” soluton to ofer an adequate integraton path to developers of fnal applicatons
– Implement a catalon of the most useful services and applicatons as well defned buildinn blocks
– Ofer a DeiOps approach for the applicaton development
● Analyse the complementarity with other onnoinn projects targetng added value services for the cloud
– In partcular those related to the management of extremely large datasets
– Explore diferent e-Infrastructures and complementary services
– Identfcaton, integraton and/or co-development of missing functonalites

4. https://deep-hybrid-datacloud.eu Jan 24, 2018 4/12
DEEP pilot use cases
● Three techniques of wide interest, involving
– Large, heterogeneous data sets
– Intensive computng demands that would beneft from using hardware accelerators (GPUs, low
latency interconnects)
● Deep learning
– Pilot cases: stem cells, biodiversity applicatons, medical image
– Provide a general, distributed architecture and pipeline to train deep learning (and other)
models
● Post-processing
– Pilot cases: post-processing of HPC simulatons
– Flexible pipeline for the analysis of simulaton data generated at HPC resources
● On-line analysis of data streams
– Pilot case: intrusion detecton systems
– Provide an architecture able to analyze massive on-line data streams, also with historical records

5. https://deep-hybrid-datacloud.eu Jan 24, 2018 5/12
DEEP consortium
● Balanced set of partners
– Strong technological background on development, implementaton,
deployment and operaton of federated e-Infrastructures
● 9 academic partners
– CSIC, LIP, INFN, PSNC, KIT, UPV, CESNET, IISAS, HMGU
● 1 industrial partner
– Atos
● 6 countries
– Spain, Italy, Poland, Germany, Czech Republic, Slovakia

6. https://deep-hybrid-datacloud.eu Jan 24, 2018 6/12
WP leaders and deputies
● WP1 (NA): Project Management and Exploitaton
– Project oversight, quality management, admin, etc.
– Leader: CSIC, deputy: Atos
● WP2 (NA): Intensive Computng Pilot Applicatons
– Defniton and understanding of pilot usage scenarios
– Leader: HMGU, deputy: KIT
● WP3 (SA): Testbed and Integraton with EOSC services
– Project service actvites: testbeds, tools, integratons
– Leader: LIP, deputy: PSNC
● WP4 (JRA): Accelerated High Performance Computng in the Cloud
– Support for accelerators and HPC resources
– Leader: IISAS, deputy: INFN
● WP5 (JRA): High Level Hybrid Cloud solutons
– Platorm provisioning, delivering the executon platorm for WPP5
– Leader: INFN, deputy: CESNET
● WP6 (JRA): DEEP as a Service
– Deliver fnal soluton to the users
– Leader: CSIC, deputy: UPV
● WP7: Ethics requirements
– Leader: CSIC

7. https://deep-hybrid-datacloud.eu Jan 24, 2018 7/12
Project governance
● Steering Commitee
– Driving the project executon
● Consortum Assembly
– Legal project partners
– Strategic project decisions
● Cooperaton Open Board
– Partners and stakeholders from Europe and
relevant projects and initatves
– Interact with Steering Commitee to defne
the agenda, interests and proposals for
discussion
● Advisory Board
– External board
– Integrated by experts from relevant
stakeholders

8. https://deep-hybrid-datacloud.eu Jan 24, 2018 8/12
INDIGO Components and evolution (I)
● INDIGO Orchestrator
– Hybrid support on multple sites
– Support to specifying specialized computng hardware
● Infrastructure Mananer
– Hybrid Clouds support involving specialized computng hardware
● uDocker
– Support for GPUs and specialized hardware to be further developed
● Cloud Informaton System
– Missing informaton about accelerators or special hardware at a provider
– React faster to changes in the infrastructure (faster publicaton and propagaton
of informaton)

9. https://deep-hybrid-datacloud.eu Jan 24, 2018 9/12
INDIGO Components and evolution (I)
● OpenStack/OpenNebula
– Extensions be needed to properly support accelerators: improving scheduling strategies,
easier confguraton and improved documentaton.
● PaaS layer
– Accelerator support
● Docker
– Use as container technology for applicatons
● LXC
– Use as alternatve hypervisor
● Ansible
– Contextualizaton and confguraton tool, further development of modules
● INDIGO Virtual Router
– Improvements to reach producton level

10. https://deep-hybrid-datacloud.eu Jan 24, 2018 10/12
DEEP work programme
● WPe have defned diferent phases for the project
● Plan and requirements (Nov 2017 – Jan 2018)
● Inital desinn (Feb – Apr 2018)
● First prototype (May – Oct 2018)
– in-situ integraton meetng to take place: conclude the integraton of the frst testbed prototype, supportng
at least two inital pilot applicatons
– June 2018: Santander Meetng
● Second prototype
– Improvement of design and proposed solutons (frst quarter of 2019)
– Integraton towards a “second prototype” (mid 2019)
● Full Pilot testbed
– Integraton of all the Pilot applicatons and their tuning for high performance
● Promoton and exploitaton (2020)
– Improve the support and fnal quality of the solutons
– Promote the exploitaton in the EOSC framework, following the integraton actvites

11. https://deep-hybrid-datacloud.eu Jan 24, 2018 11/12
Contacts
WPeb page:
htps://deep-hybrid-datacloud.eu
Email:
info@deep-hybrid-datacloud.eu
htps://twiter.com/DEEPeeu

12. https://deep-hybrid-datacloud.eu
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 777435.
Thank you
Any Questions?