1.
Björn Backeberg
Senior User Community Support Officer
EGI Founda?on – advanced compu?ng for research
bjorn.backeberg@egi.eu
Japan Geoscience Union 2019, Chiba, Japan
Cloud Computing Needs for Earth Observation Data
Analysis: EGI and EOSC-hub

2.
2
European Open Science Cloud
European Cloud Initiative by
the European Commission (April 2016)
Problem Statement
1. How to maximise the incentives for sharing data and to
increase the capacity to exploit them?
2. How to ensure that data can be used as widely as possible,
across scientific disciplines and between the public and the
private sector?
3. How better to interconnect the existing and the new data
infrastructures across Europe?
4. How best to coordinate the support available to European
data infrastructures as they move towards exascale computing?
“…a trusted, open environment for the scientific community for storing, sharing and re-
using scientific data and results…by 2020…”

3.
3
H2020 projects:
1. eInfraCentral (2017-19)
2. EOSCpilot (2017-19)
3. EOSC-hub (2018-2020)
4. OpenAIRE-Advanced (2018-2020)
5. FREYA (2018-2020)
6. ESFRI Clusters (2019-2021)
7. INFRAEOSC-* (2019-2020/21)
What happened since then…
European Commission activities:
• EOSC summits (2017, 2018)
• 2nd EOSC HLEG (draft report)
è 4 Policy; 4 Governance; 7
implementation recommendations
• EOSC implementation Roadmap (April
2018)
è 6 action lines (Archi., Data, Services,
Access & Interf., Rules, Governance)
• EOSC Portal launch (Nov 2018)
• EOSC Executive Board (Nov 2018.
Members)

4.
Vision
100
partners
36
months
33
million
Euro
Researchers from all disciplines
have easy, integrated and open access to the
advanced digital services, scien6fic instruments,
data, knowledge and exper6se they need
to collaborate to achieve excellence
in science, research and innova6on

5.
Mission
EGI
Federation EUDAT
INDIGO-
DataCloud
Research
Infrastruct
ures
The EOSC-hub project mobilises providers of
European relevance offering services, software and
data for advanced data-driven research and
innovation.
These resources are offered via the Hub – the
integration and management system of the
European Open Science Cloud, acting as a
European-level entry point for all stakeholders.

6.
6
The project established ‘The Hub’
• Data
• Applications &
tools
• Baseline services
(storage, compute,
connectivity)…
• Training,
consultants
• Marketplace
• AAI
• Accounting
• Monitoring
• …
Usage according to
Rules of
Participation
From the
consortium AND
from external
contributors
• Lightweight
certification of
providers
• SLA negotiation
• Customer Relationship
Management
• …
Based on
FitSM
• Security regula?ons,
• Compliance to standards,
• Terms of use,
• FAIR implementa?on
guidelines
• …
• A system with: Federation & collaboration services; Processes & policies; Business models
& procurement experience; Strategy & Technical Service Roadmap, etc.
• Simplify access to a broad portfolio of products, resources and service provided by the
major pan-European and international organizations through an open and integrated
service catalogue
11 Feb 2019
https://marketplace.eosc-portal.eu

7.
7
Generic services:
Support to the Research Data Lifecycle
Processing & Analysis
Data Management, Curation &
Preservation
Access, DeposiZon & Sharing
Federation Services
● B2FIND (data)
● Marketplace (Services)
● Applications on Demand
● Federated HTC & Cloud Compute IaaS & PaaS
● Processing of sensitive data
● Jupyter Notebook
● Application DB (software & VM)
● B2DROP (data)
● B2Note (data)
● B2SHARE (data)
● DataHub
● Federated AAI. monitoring,
accounting
● SLA and order Management
● Security incident response and
policies
● Technical support & Training
● B2HANDLE
● B2SAFE
● European Certified Trusted
Repository
● Thematic data analytics
● Scientific Workflow Management,
Orchestration (DIRAC, PaaS Orchestrator)
1
2
3
4
Discover & Reuse
11 Feb 2019

8.
8
Browse & access services: https://eosc-portal.eu

9.
9
The Compute Platform: EGI
EGI is a federation of over 200 computing and data centres spread across
Europe and the rest of the world.
47 Countries
12 Integrated
e-Infrastructures
61,000 users
31 large-scale research
collaborations
1,700 Open Access
Publications in 2018
EGI delivers services for computing and data-intensive science for EOSC-hub.

10.
10
The EGI Federation (as of May 2019)
4.4 Billion
CPU core
hours (2018)
> 1 Million
computing
cores in 2019
> 740 PB
disk & tape
2,915 service
end-points

11.
11
2014: EGI Federated Cloud is launched
in Helsinki
Multi-cloud IaaS with Single Sign-On
Federation features:
• Common VM image catalogue
• Discovery, accounting, SLO monitoring
• Unified GUI dashboard
Cloud Compute
Cloud Container
Compute BETA
Training Infrastructure
Online Storage
Applications on
Demand BETA
Notebooks BETA
EGI Services powered by the Cloud Federation
https://marketplace.egi.eu

12.
IaaS
providers
Federation Services
Orchestration
Platforms
Check-in : Common AuthN and AuthZ across all layers
Research Platforms
Operators
Research Communities
Research Communities
The EGI Federated Cloud today
12

13.
13
Services for federating distributed resources
Services available to the EGI
Federation members facilitating
collaboration

14.
14
Federated services for research

15.
JupyterHub hosted in the EGI Cloud
• Offers Jupyter notebooks ‘as Service’
• One-click solution: login and start
using
Extra EGI Features:
• Login with the EGI AAI Check-In
service
• Persistent storage for notebooks
• Bring your own environments/kernels
• Use EGI computing and storage
resources from your notebooks
15
EGI Notebooks as a Service

16.
16
EGI Notebooks Target Scenario
GitHub
Your
repository
EGI Notebooks
& Binder
services
Zenodo
Your
laptop
Download ipynb file
Create repository
Upload ipynb file
Add requirements.txt
Specify GitHub repo
Generate DOI
Execute
Data repository
Re-execute
Obtain GitHub project reference
Provide GitHub project reference
Discover Notebook
(use DOI)
Fellow
researchers
Journal
paper
DOI
Distributed big
data
DataHub
B2DROP
++
GenerateDOI

17.
17
EGI assets for the implementation of EOSC
Interoperability
guidelines,
standards
Policies,
processes
tools for
federated
service
management
Competence
Centres Training
Compute and
storage facilities
(national and
European
infrastructures)
Validated
Middleware for
federation-level
interoperability
Centrally
managed services
(AAI, workload
orchestration, data
management,
marketplace)
Thematic services
(Data analytics
and scientific
Gateways)
Federating services, activities, know-how
Technical infrastructure

18.
EOSC-hub support for Earth Observation R&D
• Bring together services and research products towards
implementing Open Innovation and Open Science
• Harmonize services for accessing information and
knowledge derived from Earth Observations
• Support informed decision making for sustainable
development and policy making
• Provide the large computing and storage capabilities to
support Big Data Analytics
• Host the online Sentinel Long-Term Archive (in
discussion with European Commission)
EOSC Mission for
Earth Observation
Research
18

19.
EO Pillar
Thematic
Services
IaaS, EO Finder, Data
Collections Catalogue,
EO Browser
JupyterHub for global
Copernicus data & Data
Catalogue Service
Data Cube
Data
Exploitation
Platform
Earthquake Response and
Landslides Analysis 19
EOSC-hub Thematic Services
Thematic Services are scientific services (incl. data)
that provide discipline-specific capabilities for
researchers.
Examples include:
• Browse and download
data and apps,
• Develop workflows
• Execute workflows,
• Online analytics,
• Visualise results,
• Share results and
publications.
https://marketplace.eosc-portal.eu/

20.
A Hybrid Cloud to support exploitation of EO data
End-user access point
EOSC
OpenStack API
powered by libcloud
OpenStack API
powered by libcloud
.EC2
Powered by jclouds
PaaS
Terradue Ellip PaaS
PaaS layer
Develop workflows
interactively
Design scalable
processing chains
Deploy on
processing chains
Connect to custom
Geobrowser
Hybrid IaaS
layer
>18,000 Data Products generated/month

21.
• The European Open Science Cloud will provide capacity and capabilities,
fostering the exploitation of research data
• The EGI federated cloud IaaS delivers the compute platform for EOSC
• Generic and (science discipline) thematic services are made discoverable
and accessible through the EOSC Marketplace
• EOSC supports multi-disciplinary science and Earth Observation is one of
the major use cases
• Early Adopter Programme (15 June - deadline) and Webinar (29 May)
http://bit.ly/EOSC-hub_EAP_webinar
21
Conclusions

22.
Acknowledgements
• Co-authors & collaborators:
- Yin Chen (EGI/EOSC-hub)
- Tiziana Ferrari (EGI/EOSC-hub)
- Diego Scardaci (EGI/EOSC-hub)
- Gergely Sipos (EGI/EOSC-hub)
- Pedro Gonçalves (Terradue)
- Paolo Mazzeq (CNR-IIA)
- Anabela Oliveira (LNEC)
• EOSC-hub for providing travel support
22
Thank you for your attention!

23.
EOSC-hub receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 777536.
eosc-hub.eu
@EOSC_eu
23
Back-up slides

24.
24
EGI in numbers
NumberofinstalledCPUs/year
From 300,000 installed CPUs in
2011 to over 1 million in 2019
The EGI Federation offers online
storage (disk) and archive storage
(tape) to meet requirements for Data
Sharing - the capacity in 2019 is 696
Petabytes

25.
EGI FedCloud Architecture
25
EGI Federation services:
Accounting, Monitoring, Configuration Database, Information Discovery, VM Marketplace
EGI Check-in
IaaS Federated Access Tools
Community PlatformsAppDB VMOps
Cloud Management
Framework
IaaS API
Cloud Management
Framework
IaaS API
The EGI Federated Cloud is not just AppDB
Providers have their APIs that
can be used with EGI Check-in
accounts, opening the door to
automation of cloud-native
applications.
IaaS Federated Access Tools layer helps
users of the cloud to deal with the
heterogeneity in the IaaS API and EGI
Federation services
EGI Federated Cloud no longer mandates a single API for
every provider. OCCI still widely supported but sites are
moving to native APIs (mainly OpenStack!)

26.
26
EGI FedCloud in action
Since 2015 1.9 million
Virtual Machines consumed
over 10,000 years of
compute time

27.
27
EGI Notebooks Technology Stack
Kubernetes cluster powered by EGI Cloud Compute
EGI CheckIn
Kubernetes
Ingress
SSL Certificate
NFS Persistent
Storage
https://notebooks.egi.eu

28.
On-demand operational coastal circulation forecast service
28
OPENCoastS Service
Configure Forecast Systems
Manage Forecast Systems
Visualise outputs