EGI Federated Cloud relies on a federated Service Management System based on the FitSM standard, and is ISO 20k compliant. Since 2018 we are facing an increasing demand of multi-supply cloud services to meet big data analytics performance requirements. The EGI Federated Cloud is one of the cloud pillars of the "European Open Science Cloud", an initiative aiming to federate data and digital infrastructures for research projects of European relevance.
EGI Cloud Services in a Federated Multi-Supply Envirnment
1. www.egi.eu
@EGI_eInfra
The work of the EGI Foundation
is partly funded by the European Commission
under H2020 Framework Programme
EGI : Advanced Computing for Research
EGI Cloud Services in a Federated
Multi-Supply Environment
EGI Foundation
Tiziana Ferrari
(Tiziana.Ferrari@egi.eu)
3. 6/10/2019 3@EGI_eInfrawww.egi.eu
• 260+ data & computing centres
• Across 45+ countries
• 4.4 Billion CPU core wall time delivered in 2018
– 1, million computing cores
– 356 PB disk & 380 PB tape storage
• +1700 open access publications in 2018
• +41 new international projects
• 31 large scale ESFRI projects/landmarks
supported
EGI Factsheet - 2018
4. @EGI_eInfrawww.egi.eu 6/10/2019 4
EGI Federated Operations
National
Infrastru
cture
Operations
Centre
RC
RC
RC
National
Infrastru
cture
Operations
Centre
RC
RC
RC
Operations
Management
Board Federation Infrastructure
EIRO
Operations
Centre
RC
Research
Infrastruc
ture
Operations
Centre
RC
RC
RC
5. @EGI_eInfrawww.egi.eu 6/10/2019 5
EGI Service Portfolio
The list of services that EGI as a federation
offers for research & innovation
Compute Storage and Data Training Applications
Cloud Compute
Cloud Container
Compute BETA
High-Throughput
Compute
Archive Storage
Data Transfer
FitSM Training
Training Infrastructure
Online Storage
Security
Check-in BETA
ISO 27001 Training
Workload
Manager BETA
6. @EGI_eInfrawww.egi.eu 6/10/2019 6
EGI Internal Service Portfolio
The list of services delivered internally to the EGI federation
to enable the EGI providers to work together
Coordination
Communications
Strategy and Policy
Development
Project
Management
and Planning
Operations and
Support
Technology
Community
IT Service Management
Security
Operations
Configuration
Database
Helpdesk
Validated Software and
Repository
Operational
Tools
Collaboration Tools
Service Monitoring
Accounting
Security
Check-in
Attribute Management
Marketplace BETA
7. @EGI_eInfrawww.egi.eu 6/10/2019 7
• Integrated management system is the framework of policies, processes and
procedures used by EGI Foundation to ensure that it can fulfil all the tasks required to
achieve its objectives.
• The objective:
to ensure systematic and professional operation and delivery of EGI Foundation services.
to plan, implement, monitor and continually improve all business processes under the
responsibility of EGI Foundation.
• It integrates all of the distributed organization's systems and processes into one
complete framework, enabling an organization to work as a single unit with
unified objectives.
Integrated Management System
8. @EGI_eInfrawww.egi.eu 6/10/2019 8
IMS: General Processes
•Manage the service portfolio; alignment of new or changed services with organisation strategy
Service portfolio management
• Maintain a service catalogue; define, agree and monitor relevant agreements (SLA, OLA, UA)
Service level management
• Specify all service reports and ensure its production according to specifications in a timely manner
to support decision-making
Service reporting management
• Identify, record and analyse customer opportunities; manage service orders and maintain a good
relationship with customers
Customer relationship management
• Establish and maintain a healthy relations with suppliers supporting the services; ensure the
required capacity and monitor performance
Supplier & federated members relationship management
• Ensure effective management of budgeting, accounting for services
Budgeting & accounting management
9. @EGI_eInfrawww.egi.eu 6/10/2019 9
IMS: IT Processes
• Ensures sufficient capacities to meet agreed service levels and monitor performance requirements for
services
Capacity management
• Ensure sufficient service availability to meet agreed requirements and adequate service continuity in
case of exceptional situations
Service availability & continuity management
• Restore normal / agreed service operation in case of an incident; respond to user service requests
Incident & service request management
• Investigate the root causes of (recurring) incidents in order to avoid future recurrence of incidents
Problem management
• Provide and maintain an information about logical model of service components and its configuration
Configuration management
• Ensure changes are planned, approved, implemented and reviewed in a controlled manner
Change management
• Manage releases, so that changes can be tested and deployed to the live environment
Release & deployment management
10. @EGI_eInfrawww.egi.eu 6/10/2019 10
• Standards family for lightweight IT service management
• Suitable for IT service providers of any type and scale
• Main design principle: Keep it simple!
• All FitSM parts are freely released under Creative Commons licenses
• FitSM is operated and managed by ITEMO (non-profit)
• Certification provided by ICO-Cert and APMG International
What is FitSM
www.fitsm.eu
The development of FitSM was originally funded by the European Commission
through an EC-FP7 project "FedSM“
FitSM_Standard
12. @EGI_eInfrawww.egi.eu 6/10/2019 12
• Multi-cloud IaaS with Single Sign-On
• Federation features:
Common VM image catalogue
Discovery, accounting, SLO monitoring
Unified GUI dashboard
EGI Cloud Federation
Cloud Compute
Cloud Container
Compute BETA
Training Infrastructure
Online Storage
Applications on
Demand BETA
Notebooks BETA
EGI Services powered by the Cloud Federation
13. @EGI_eInfrawww.egi.eu 6/10/2019 13
EGI Cloud enables research oriented computing
IaaS
providers
Federation Services
Orchestration
Platforms
Check-in : Common AuthN and AuthZ across all layers
Research Platforms
Operators
Research Communities
Research Communities
15. @EGI_eInfrawww.egi.eu 6/10/2019 15
0.0E+00
5.0E+05
1.0E+06
1.5E+06
2.0E+06
2.5E+06
3.0E+06
3.5E+06
4.0E+06
4.5E+06
5.0E+06
2018 Apr 2018 May 2018 Jun 2018 Jul 2018 Aug 2018 Sep 2018 Oct 2018 Nov 2018 Dec 2019 Jan 2019 Feb 2019 Mar 2019 Apr
French NGI Vos EOSC-hub VOS fedcloud.egi.eu vo.lifewatch.eu geohazards.terradue.com
bioisi vo.access.egi.eu vo.emsodev.eu peachnote.com vo.nextgeoss.eu
vo.nbis.se d4science.org chipster.csc.fi ericll.org enmr.eu
biomed training.egi.eu
Usage
fedcloud.egi.eu
Last 12 Months:
26,4M CPU hours
500K VMs
16. @EGI_eInfrawww.egi.eu 6/10/2019 16
• Check-in provides:
Single Sign-On through eduGAIN, social media and other institutional or community-managed
identity providers
Harmonised authorisation information, aggregated from multiple sources
Industry Standard OpenID Connect technology allowing web and non-web access to services
• Integration:
Native support at all the EGI Cloud layers (IaaS providers, IaaS Orchestration, AppDB VMOps)
and at EGI services/platforms running on top: Notebooks, AoD, Container
FedCloud client https://aai.egi.eu/fedcloud for easily getting individual tokens for CLI/API access
New: Check-in & EGI Cloud
17. @EGI_eInfrawww.egi.eu 6/10/2019 17
Cloud Management
Framework
IaaS API
Cloud Management
Framework
IaaS API
Direct API
Access
Interfaces and Check-in
EGI Federation features:
Accounting, Monitoring, Conf. DB, Info Discovery,
AppDB
AppDB VMOpsGUI Access
IaaS Federated Access Tools
Federated
Access
Developers/
Advanced users
AAI: Check-in
GUI Users
21. @EGI_eInfrawww.egi.eu 6/10/2019 21
IdP/SP Proxy
•Implementation of the AARC blueprint
architecture
•Registered in eduGAIN as an SP complying
with REFEDS Research & Scholarship and
Sirtfi
•All community SPs can have one statically
configured IdP
•No need to run an IdP Discovery Service on
each community SP
•Connected SPs get consistent/harmonised
user identifiers and accompanying attribute
sets from different IdPs/AAs that can be
interpreted in a uniform way for
authorisation purposes
26. @EGI_eInfrawww.egi.eu 6/10/2019 26
• EGI Federated Cloud relies on a federated Service Management System based on the
FitSM standard, and is ISO 20k compliant
EGI federated cloud in line with the NIST Cloud Federation Reference Architecture
o Areas of future exploration: federation models and related service management system, federation
interoperability framework (standards, protocols and guidelines for interoperability)
BUT: Service management in a multi-supply environment being modelled by Service Integration and Management
(SIAM) is talking many of these aspects
o Federated Trust and Identity is not specific to a federated multi-supply environment, why mixing the two in a
single reference architecture?
EGI federated AAI solution is Check-in as relies on AARC best practices, policies and guidelines – applicable to any IT
capability
• Increasing demand of multi-supply cloud services to meet big data analytics performance
requirements many use cases are no production ready
• European Union launched the European Open Science Cloud in Nov 2018 as initiative to
federate data and digital infrastructures for research of European relevance (http://eosc-
portal.eu/)
EOSC Early Adopter Programme (https://www.eosc-hub.eu/eosc-early-adopter-programme)
Considerations
Notas del editor
E-Infrastructures are geographically distributed computing resources and data storage facilities linked by high-performance networks. They allow scientists to share information securely, analyse data efficiently and collaborate with colleagues worldwide. They are an essential part of modern scientific research and a driver for economic growth.
EGI was established in 2010 building on over a decade of investment by national governments and the European Commission. EGI is a European-wide federation of national computing and data storage resources. Its aim is to support cutting-edge research, innovation and knowledge transfer in Europe. EGI federates resources from various resource centres, mainly from research insitututes and universities. These centres provide computer clusters, storage servers, applications and human support services for secure access and sharing. EGI provides these services to European researchers and their international collaborators.
EGI is coordinated by EGI.eu, a not-for-profit foundation based in Amsterdam and owned by EGI’s participants, the National Grid Infrastructures (NGIs).
The Council is the supervisory authority and monitors the general course of affairs in the Foundation. It consists of participants and associated participants of the foundation. https://www.egi.eu/about/EGI.eu/council_members.html
List of services of EGI powered by the EGI Cloud Federation
IN2P3 highlighted just in case
User communities build either on top of orchestration tools that allow to deal with multiple providers in a homogeneous way or directly interact with the native APIs of the provides. Both cases they can use single sign-on thanks to Check-in.
A common GUI provided by AppDB VMOps brings a user-friendly dashboard to manage the resources at the distributed providers
The EGI federation services are integrated with the providers using their native APIs to deliver the extra features of EGI Cloud mentioned in previous slide
GUI access:
AppDB VMOps https://dashboard.appdb.egi.eu/vmops
API/CLI access:
Discovery: AppDB IS API (REST and GraphQL) https://wiki.egi.eu/wiki/Federated_Cloud_Discovery#AppDB
IaaS Federated Access Tools: https://wiki.egi.eu/wiki/Federated_Cloud_IaaS_Orchestration
Direct IaaS access, several APIs depending on the provider: https://wiki.egi.eu/wiki/Federated_Cloud_APIs_and_SDKs