Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Cloud Computing Needs for Earth Observation Data Analysis: EGI and EOSC-hub


Eche un vistazo a continuación

1 de 28 Anuncio

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a Cloud Computing Needs for Earth Observation Data Analysis: EGI and EOSC-hub (20)


Más reciente (20)

Cloud Computing Needs for Earth Observation Data Analysis: EGI and EOSC-hub

  1. 1. Björn Backeberg Senior User Community Support Officer EGI Founda?on – advanced compu?ng for research Japan Geoscience Union 2019, Chiba, Japan Cloud Computing Needs for Earth Observation Data Analysis: EGI and EOSC-hub
  2. 2. 2 European Open Science Cloud European Cloud Initiative by the European Commission (April 2016) Problem Statement 1. How to maximise the incentives for sharing data and to increase the capacity to exploit them? 2. How to ensure that data can be used as widely as possible, across scientific disciplines and between the public and the private sector? 3. How better to interconnect the existing and the new data infrastructures across Europe? 4. How best to coordinate the support available to European data infrastructures as they move towards exascale computing? “…a trusted, open environment for the scientific community for storing, sharing and re- using scientific data and results…by 2020…”
  3. 3. 3 H2020 projects: 1. eInfraCentral (2017-19) 2. EOSCpilot (2017-19) 3. EOSC-hub (2018-2020) 4. OpenAIRE-Advanced (2018-2020) 5. FREYA (2018-2020) 6. ESFRI Clusters (2019-2021) 7. INFRAEOSC-* (2019-2020/21) What happened since then… European Commission activities: • EOSC summits (2017, 2018) • 2nd EOSC HLEG (draft report) è 4 Policy; 4 Governance; 7 implementation recommendations • EOSC implementation Roadmap (April 2018) è 6 action lines (Archi., Data, Services, Access & Interf., Rules, Governance) • EOSC Portal launch (Nov 2018) • EOSC Executive Board (Nov 2018. Members)
  4. 4. Vision 100 partners 36 months 33 million Euro Researchers from all disciplines have easy, integrated and open access to the advanced digital services, scien6fic instruments, data, knowledge and exper6se they need to collaborate to achieve excellence in science, research and innova6on
  5. 5. Mission EGI Federation EUDAT INDIGO- DataCloud Research Infrastruct ures The EOSC-hub project mobilises providers of European relevance offering services, software and data for advanced data-driven research and innovation. These resources are offered via the Hub – the integration and management system of the European Open Science Cloud, acting as a European-level entry point for all stakeholders.
  6. 6. 6 The project established ‘The Hub’ • Data • Applications & tools • Baseline services (storage, compute, connectivity)… • Training, consultants • Marketplace • AAI • Accounting • Monitoring • … Usage according to Rules of Participation From the consortium AND from external contributors • Lightweight certification of providers • SLA negotiation • Customer Relationship Management • … Based on FitSM • Security regula?ons, • Compliance to standards, • Terms of use, • FAIR implementa?on guidelines • … • A system with: Federation & collaboration services; Processes & policies; Business models & procurement experience; Strategy & Technical Service Roadmap, etc. • Simplify access to a broad portfolio of products, resources and service provided by the major pan-European and international organizations through an open and integrated service catalogue 11 Feb 2019
  7. 7. 7 Generic services: Support to the Research Data Lifecycle Processing & Analysis Data Management, Curation & Preservation Access, DeposiZon & Sharing Federation Services ● B2FIND (data) ● Marketplace (Services) ● Applications on Demand ● Federated HTC & Cloud Compute IaaS & PaaS ● Processing of sensitive data ● Jupyter Notebook ● Application DB (software & VM) ● B2DROP (data) ● B2Note (data) ● B2SHARE (data) ● DataHub ● Federated AAI. monitoring, accounting ● SLA and order Management ● Security incident response and policies ● Technical support & Training ● B2HANDLE ● B2SAFE ● European Certified Trusted Repository ● Thematic data analytics ● Scientific Workflow Management, Orchestration (DIRAC, PaaS Orchestrator) 1 2 3 4 Discover & Reuse 11 Feb 2019
  8. 8. 8 Browse & access services:
  9. 9. 9 The Compute Platform: EGI EGI is a federation of over 200 computing and data centres spread across Europe and the rest of the world. 47 Countries 12 Integrated e-Infrastructures 61,000 users 31 large-scale research collaborations 1,700 Open Access Publications in 2018 EGI delivers services for computing and data-intensive science for EOSC-hub.
  10. 10. 10 The EGI Federation (as of May 2019) 4.4 Billion CPU core hours (2018) > 1 Million computing cores in 2019 > 740 PB disk & tape 2,915 service end-points
  11. 11. 11 2014: EGI Federated Cloud is launched in Helsinki Multi-cloud IaaS with Single Sign-On Federation features: • Common VM image catalogue • Discovery, accounting, SLO monitoring • Unified GUI dashboard Cloud Compute Cloud Container Compute BETA Training Infrastructure Online Storage Applications on Demand BETA Notebooks BETA EGI Services powered by the Cloud Federation
  12. 12. IaaS providers Federation Services Orchestration Platforms Check-in : Common AuthN and AuthZ across all layers Research Platforms Operators Research Communities Research Communities The EGI Federated Cloud today 12
  13. 13. 13 Services for federating distributed resources Services available to the EGI Federation members facilitating collaboration
  14. 14. 14 Federated services for research
  15. 15. JupyterHub hosted in the EGI Cloud • Offers Jupyter notebooks ‘as Service’ • One-click solution: login and start using Extra EGI Features: • Login with the EGI AAI Check-In service • Persistent storage for notebooks • Bring your own environments/kernels • Use EGI computing and storage resources from your notebooks 15 EGI Notebooks as a Service
  16. 16. 16 EGI Notebooks Target Scenario GitHub Your repository EGI Notebooks & Binder services Zenodo Your laptop Download ipynb file Create repository Upload ipynb file Add requirements.txt Specify GitHub repo Generate DOI Execute Data repository Re-execute Obtain GitHub project reference Provide GitHub project reference Discover Notebook (use DOI) Fellow researchers Journal paper DOI Distributed big data DataHub B2DROP ++ GenerateDOI
  17. 17. 17 EGI assets for the implementation of EOSC Interoperability guidelines, standards Policies, processes tools for federated service management Competence Centres Training Compute and storage facilities (national and European infrastructures) Validated Middleware for federation-level interoperability Centrally managed services (AAI, workload orchestration, data management, marketplace) Thematic services (Data analytics and scientific Gateways) Federating services, activities, know-how Technical infrastructure
  18. 18. EOSC-hub support for Earth Observation R&D • Bring together services and research products towards implementing Open Innovation and Open Science • Harmonize services for accessing information and knowledge derived from Earth Observations • Support informed decision making for sustainable development and policy making • Provide the large computing and storage capabilities to support Big Data Analytics • Host the online Sentinel Long-Term Archive (in discussion with European Commission) EOSC Mission for Earth Observation Research 18
  19. 19. EO Pillar Thematic Services IaaS, EO Finder, Data Collections Catalogue, EO Browser JupyterHub for global Copernicus data & Data Catalogue Service Data Cube Data Exploitation Platform Earthquake Response and Landslides Analysis 19 EOSC-hub Thematic Services Thematic Services are scientific services (incl. data) that provide discipline-specific capabilities for researchers. Examples include: • Browse and download data and apps, • Develop workflows • Execute workflows, • Online analytics, • Visualise results, • Share results and publications.
  20. 20. A Hybrid Cloud to support exploitation of EO data End-user access point EOSC OpenStack API powered by libcloud OpenStack API powered by libcloud .EC2 Powered by jclouds PaaS Terradue Ellip PaaS PaaS layer Develop workflows interactively Design scalable processing chains Deploy on processing chains Connect to custom Geobrowser Hybrid IaaS layer >18,000 Data Products generated/month
  21. 21. • The European Open Science Cloud will provide capacity and capabilities, fostering the exploitation of research data • The EGI federated cloud IaaS delivers the compute platform for EOSC • Generic and (science discipline) thematic services are made discoverable and accessible through the EOSC Marketplace • EOSC supports multi-disciplinary science and Earth Observation is one of the major use cases • Early Adopter Programme (15 June - deadline) and Webinar (29 May) 21 Conclusions
  22. 22. Acknowledgements • Co-authors & collaborators: - Yin Chen (EGI/EOSC-hub) - Tiziana Ferrari (EGI/EOSC-hub) - Diego Scardaci (EGI/EOSC-hub) - Gergely Sipos (EGI/EOSC-hub) - Pedro Gonçalves (Terradue) - Paolo Mazzeq (CNR-IIA) - Anabela Oliveira (LNEC) • EOSC-hub for providing travel support 22 Thank you for your attention!
  23. 23. EOSC-hub receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 777536. @EOSC_eu 23 Back-up slides
  24. 24. 24 EGI in numbers NumberofinstalledCPUs/year From 300,000 installed CPUs in 2011 to over 1 million in 2019 The EGI Federation offers online storage (disk) and archive storage (tape) to meet requirements for Data Sharing - the capacity in 2019 is 696 Petabytes
  25. 25. EGI FedCloud Architecture 25 EGI Federation services: Accounting, Monitoring, Configuration Database, Information Discovery, VM Marketplace EGI Check-in IaaS Federated Access Tools Community PlatformsAppDB VMOps Cloud Management Framework IaaS API Cloud Management Framework IaaS API The EGI Federated Cloud is not just AppDB Providers have their APIs that can be used with EGI Check-in accounts, opening the door to automation of cloud-native applications. IaaS Federated Access Tools layer helps users of the cloud to deal with the heterogeneity in the IaaS API and EGI Federation services EGI Federated Cloud no longer mandates a single API for every provider. OCCI still widely supported but sites are moving to native APIs (mainly OpenStack!)
  26. 26. 26 EGI FedCloud in action Since 2015 1.9 million Virtual Machines consumed over 10,000 years of compute time
  27. 27. 27 EGI Notebooks Technology Stack Kubernetes cluster powered by EGI Cloud Compute EGI CheckIn Kubernetes Ingress SSL Certificate NFS Persistent Storage
  28. 28. On-demand operational coastal circulation forecast service 28 OPENCoastS Service Configure Forecast Systems Manage Forecast Systems Visualise outputs