SlideShare una empresa de Scribd logo
1 de 26
Descargar para leer sin conexión
Project COLA: Use Case to
create a scalable application in
the cloud based on the MICADO
concept
Peter Kacsuk
SZTAKI
kacsuk@sztaki.hu
Introduction to the application
The application is Data Avenue:
• This application can
• transfer data from one data storage to another one
• Upload data from your computer to a data storage
• Download data from a data storage to your computer
• The data storage can be any of the following type:
• HTTP, HTTPS, SFTP, GSIFTP, SRM, iRODS and S3
Data Avenue services
3
FS1 FSn
Data Avenue Blacktop service
Openstack Amazon
Data Avenue
@ SZTAKI
Java or
other appli
code (JAVA
API or WS
API)
Data Avenue
Portlet
gateways
FS2OpenNebula
storage-related protocols
web service interface
Introduction to the application
Detailed objectives:
• We want to create a scalable service out of this application. It
means that users from all over the world can access this
service in order to transfer their data from one data storage to
another one provided that they have got access right to the
used data storages.
• The current service can be accessed at:
• https://data-avenue.eu/en_GB/
The Data Avenue service
How to use Data Avenue?
How to use Data Avenue?
Use-case1: DataAvenue@SZTAKI
> Select storage type (Protocol)
> Set storage host address (URL)
> Press Go
Use-case1: DataAvenue@SZTAKI
> Select authentication method
> Give credentials
> Press Go
The Data Avenue Concept
Copy between different storages
Progress bar
Directory contents: files,
directories
Problems with the current service
• It uses a single Data Avenue application and it becomes a
bottleneck in the following cases:
• One user initiates many transfers in parallel
• Many users initiate single transfers in parallel
• Many users initiate many transfers in parallel
Solution
• The Data Avenue application can run in several instances in parallel in
the cloud
• If a single Data Avenue (DA) becomes bottleneck new DA is instantiated
in the cloud and some of the required data transfers are automatically re-
directed to the new DA instance
• In order to organize all the services as a set of coordinated services we
need an information service => Consul service
• In order to observe if a data instance becomes overloaded we need a
monitoring tool => Prometheus service
• In order to deploy and manage all the services as a set of coordinated
services we need a cloud orchestrator service => Occopus service
• In order to evenly exploit the DA instances we need a load balancer
service that directs the users’ DA service calls evenly to the available DA
instances => haproxy service
Scale up
Scale down
Automatic scalable architecture for Data Avenue
More details for monitoring
Alert
Manager
Executor
Available as tutorial at the Occopus web page: http://occopus.lpds.sztaki.hu/tutorials
Soon will be available as tutorial at the COLA web page
CPU Usage
Number of VMs
15/14
Demonstrating scalability
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
numberofvirtualmachines
Measure every 40 minute, starting at 8:00 am.
One-day long test bench
16/14
Limitations
• It uses VMs for every service and hence the deployment time
of the whole infrastructure is 8-10 minutes in CloudSigma
• Good scalability performance results can be achieved only if
the used storage has a scalable access mechanism. For
example, S3 cloud storage enables scalability but https
does not.
Docker container based architecture
• In order to reduce deployment time we placed every service in
a separate docker container.
• The modification required:
• Dockerized version of all the services (HAProxy, Prometheus,
Consul)
• Dockerized version of the data node application (DA)
• Modification of the cloud-init file that contains the description of cloud-
related parameters (cloud-init file became much simpler)
General architecture adaptable for
other applications -> MICADO v0
• If the DA description is replaced by another service-oriented
application in the cloud-init file, then it can be used for other
applications, too.
• Therefore this architecture can be considered as the first
version of the MICADO architecture -> MICADO v0
Towards a more generic MICADO
architecture -> MICADO v1
• Goal:
• create a MICADO architecture in which the application can be
accommodated without modifying cloud-init
• To achieve this we introduce a docker cluster and the
applications will run within in docker containers within this
docker cluster
• Therefore this architecture can be considered as the improved
version of the MICADO v0 architecture -> MICADO v1
• MICADO v1 could have to alternatives:
• A. 2-layered MICADO architecture without load balancer layer
• B. 3-layered MICADO architecture with load balancer layer
2-layer MICADO v1/a architecture
Swarm
docker
cluster
Assessment of MICADO v1/a
Advantage:
• Adaptable for different applications without modifying the cloud-init file
• Scales up and down the number of data nodes according to the actual
load of the data nodes
• Guarantees the balanced usage of data nodes (responsibility of the
Docker Master)
Disadvantage:
• No guarantee that the data nodes are used in a balanced way to serve
user requests
• All the data nodes should run the same applications
3-layer MICADO v1/b architecture
Assessment of MICADO v1/b
Advantage:
• Adaptable for different applications without modifying the cloud-init file
• Scales up and down the number of data nodes according to the actual
load of the data nodes
• Guarantees the balanced usage of data nodes (responsibility of the
Docker Master)
• Guarantees that the data nodes are used in a balanced way to serve
user requests (responsibility of the HAProxy services)
Disadvantage:
• All the data nodes should run the same applications
Future work:
• MICADO v2 to enable the balanced and scalable usage of different
applications
Acknowledgement
• The development of MICADO is in strong collaboration
between the SZTAKI and UoW teams.
• SZTAKI team: Prof. Peter Kacsuk
• Dr. Jozsef Kovacs
• Enikő Nagy
• Attila Farkas
• Dr. Robert Lovas
• Dr. Zoltan Farkas
• UoW team: Dr. Tamas Kiss
• Prof. Gabor Terstyanszky
• Botond Rákoczi
• Gregoire Gesmier
• Gabriele Pierantoni
For more information please visit
www.project-cola.eu
twitter.com/projectCOLA
facebook.com/projectCOLA
Thank you!

Más contenido relacionado

La actualidad más candente

Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...
Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...
Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...Ocean Project
 
Decide general presentation 2017
Decide general presentation 2017Decide general presentation 2017
Decide general presentation 2017DECIDEH2020
 
CloudLightning: Self-Organising, Self-Managing Heterogeneous Cloud
CloudLightning: Self-Organising, Self-Managing Heterogeneous CloudCloudLightning: Self-Organising, Self-Managing Heterogeneous Cloud
CloudLightning: Self-Organising, Self-Managing Heterogeneous CloudCloudLightning
 
CloudLightning - Project Overview
CloudLightning - Project OverviewCloudLightning - Project Overview
CloudLightning - Project OverviewCloudLightning
 
CloudLightning - Multiclouds: Challenges and Current Solutions
CloudLightning - Multiclouds: Challenges and Current SolutionsCloudLightning - Multiclouds: Challenges and Current Solutions
CloudLightning - Multiclouds: Challenges and Current SolutionsCloudLightning
 
Slideshared 4. iucee-inpods cloud engineering
Slideshared 4. iucee-inpods cloud engineering Slideshared 4. iucee-inpods cloud engineering
Slideshared 4. iucee-inpods cloud engineering Ravindra Dastikop
 
Big Data Service Offerings from Cloud Vendors
Big Data Service Offerings from Cloud VendorsBig Data Service Offerings from Cloud Vendors
Big Data Service Offerings from Cloud VendorsMiteshN
 
Review of Cloud Computing Simulation Platforms and Related Environments
Review of Cloud Computing Simulation Platforms and Related EnvironmentsReview of Cloud Computing Simulation Platforms and Related Environments
Review of Cloud Computing Simulation Platforms and Related EnvironmentsRECAP Project
 
A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...
A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...
A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...RECAP Project
 
Day 3 Conference Welcome by Erik Weaver
Day 3 Conference Welcome by Erik WeaverDay 3 Conference Welcome by Erik Weaver
Day 3 Conference Welcome by Erik WeaverETCenter
 
Optimising Service Deployment and Infrastructure Resource Configuration
Optimising Service Deployment and Infrastructure Resource ConfigurationOptimising Service Deployment and Infrastructure Resource Configuration
Optimising Service Deployment and Infrastructure Resource ConfigurationRECAP Project
 
Cloud Computing Research Projects
Cloud Computing Research ProjectsCloud Computing Research Projects
Cloud Computing Research ProjectsPhdtopiccom
 
"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale
"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale
"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitalelostrettodigitale
 
Jugo: A Generic Architecture for Composite Cloud as a Service Auth
Jugo: A Generic Architecture for Composite Cloud as a Service AuthJugo: A Generic Architecture for Composite Cloud as a Service Auth
Jugo: A Generic Architecture for Composite Cloud as a Service AuthMahmud Hossain
 

La actualidad más candente (20)

Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...
Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...
Optimis - Cloud but better, Open Cloud Forum at Cloud Expo Europe, February 2...
 
IBM Cloud & Helix Nebula
IBM Cloud & Helix NebulaIBM Cloud & Helix Nebula
IBM Cloud & Helix Nebula
 
HNSciCloud Overview
HNSciCloud OverviewHNSciCloud Overview
HNSciCloud Overview
 
Hybrid cloud for science
Hybrid cloud for science Hybrid cloud for science
Hybrid cloud for science
 
Decide general presentation 2017
Decide general presentation 2017Decide general presentation 2017
Decide general presentation 2017
 
Helix Nebula Phase 1
Helix Nebula Phase 1Helix Nebula Phase 1
Helix Nebula Phase 1
 
CloudLightning: Self-Organising, Self-Managing Heterogeneous Cloud
CloudLightning: Self-Organising, Self-Managing Heterogeneous CloudCloudLightning: Self-Organising, Self-Managing Heterogeneous Cloud
CloudLightning: Self-Organising, Self-Managing Heterogeneous Cloud
 
CloudLightning - Project Overview
CloudLightning - Project OverviewCloudLightning - Project Overview
CloudLightning - Project Overview
 
CloudLightning - Multiclouds: Challenges and Current Solutions
CloudLightning - Multiclouds: Challenges and Current SolutionsCloudLightning - Multiclouds: Challenges and Current Solutions
CloudLightning - Multiclouds: Challenges and Current Solutions
 
Slideshared 4. iucee-inpods cloud engineering
Slideshared 4. iucee-inpods cloud engineering Slideshared 4. iucee-inpods cloud engineering
Slideshared 4. iucee-inpods cloud engineering
 
Big Data Service Offerings from Cloud Vendors
Big Data Service Offerings from Cloud VendorsBig Data Service Offerings from Cloud Vendors
Big Data Service Offerings from Cloud Vendors
 
Review of Cloud Computing Simulation Platforms and Related Environments
Review of Cloud Computing Simulation Platforms and Related EnvironmentsReview of Cloud Computing Simulation Platforms and Related Environments
Review of Cloud Computing Simulation Platforms and Related Environments
 
A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...
A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...
A Modelling Language for Defining Cloud Simulation Scenarios in RECAP Project...
 
Day 3 Conference Welcome by Erik Weaver
Day 3 Conference Welcome by Erik WeaverDay 3 Conference Welcome by Erik Weaver
Day 3 Conference Welcome by Erik Weaver
 
Optimising Service Deployment and Infrastructure Resource Configuration
Optimising Service Deployment and Infrastructure Resource ConfigurationOptimising Service Deployment and Infrastructure Resource Configuration
Optimising Service Deployment and Infrastructure Resource Configuration
 
Cloud Computing Research Projects
Cloud Computing Research ProjectsCloud Computing Research Projects
Cloud Computing Research Projects
 
Pilot phase Award Ceremony - RHEA
Pilot phase Award Ceremony - RHEAPilot phase Award Ceremony - RHEA
Pilot phase Award Ceremony - RHEA
 
"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale
"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale
"Big Data e vissuto quotidiano" - Massimo Villari per lo Stretto Digitale
 
HNSciCloud PILOT PLATFORM OVERVIEW
HNSciCloud PILOT PLATFORM OVERVIEWHNSciCloud PILOT PLATFORM OVERVIEW
HNSciCloud PILOT PLATFORM OVERVIEW
 
Jugo: A Generic Architecture for Composite Cloud as a Service Auth
Jugo: A Generic Architecture for Composite Cloud as a Service AuthJugo: A Generic Architecture for Composite Cloud as a Service Auth
Jugo: A Generic Architecture for Composite Cloud as a Service Auth
 

Similar a Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept

Azure Web App services
Azure Web App servicesAzure Web App services
Azure Web App servicesAlexey Bokov
 
Lightening the burden of cloud resources administration: from VMs to Functions
Lightening the burden of cloud resources administration: from VMs to FunctionsLightening the burden of cloud resources administration: from VMs to Functions
Lightening the burden of cloud resources administration: from VMs to FunctionsEUBrasilCloudFORUM .
 
basic concept of Cloud computing and its architecture
basic concept of Cloud computing  and its architecturebasic concept of Cloud computing  and its architecture
basic concept of Cloud computing and its architectureMohammad Ilyas Malik
 
Presentation on Cloud Computing
Presentation on Cloud ComputingPresentation on Cloud Computing
Presentation on Cloud ComputingHarpreetKaur1382
 
[WSO2Con Asia 2018] Architecting for Container-native Environments
[WSO2Con Asia 2018] Architecting for Container-native Environments[WSO2Con Asia 2018] Architecting for Container-native Environments
[WSO2Con Asia 2018] Architecting for Container-native EnvironmentsWSO2
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022HostedbyConfluent
 
Hybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerůHybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerůMarketingArrowECS_CZ
 
Twelve-Factor application pattern with Spring Framework
Twelve-Factor application pattern with Spring FrameworkTwelve-Factor application pattern with Spring Framework
Twelve-Factor application pattern with Spring Frameworkdinkar thakur
 
Presentation on Cloud Computing (CE).pptx
Presentation on Cloud Computing (CE).pptxPresentation on Cloud Computing (CE).pptx
Presentation on Cloud Computing (CE).pptxHarpreetKaur1382
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017Patrick Chanezon
 
AWS Webcast - Datacenter Migration to AWS
AWS Webcast - Datacenter Migration to AWSAWS Webcast - Datacenter Migration to AWS
AWS Webcast - Datacenter Migration to AWSAmazon Web Services
 
Open shift and docker - october,2014
Open shift and docker - october,2014Open shift and docker - october,2014
Open shift and docker - october,2014Hojoong Kim
 
Getting Started with Docker - Nick Stinemates
Getting Started with Docker - Nick StinematesGetting Started with Docker - Nick Stinemates
Getting Started with Docker - Nick StinematesAtlassian
 
Cloud broadcasting and computing
Cloud broadcasting and computing Cloud broadcasting and computing
Cloud broadcasting and computing AMEED KHAN
 
Building a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsBuilding a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsAvere Systems
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSSteve Wong
 

Similar a Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept (20)

Azure Web App services
Azure Web App servicesAzure Web App services
Azure Web App services
 
Lightening the burden of cloud resources administration: from VMs to Functions
Lightening the burden of cloud resources administration: from VMs to FunctionsLightening the burden of cloud resources administration: from VMs to Functions
Lightening the burden of cloud resources administration: from VMs to Functions
 
basic concept of Cloud computing and its architecture
basic concept of Cloud computing  and its architecturebasic concept of Cloud computing  and its architecture
basic concept of Cloud computing and its architecture
 
Presentation on Cloud Computing
Presentation on Cloud ComputingPresentation on Cloud Computing
Presentation on Cloud Computing
 
[WSO2Con Asia 2018] Architecting for Container-native Environments
[WSO2Con Asia 2018] Architecting for Container-native Environments[WSO2Con Asia 2018] Architecting for Container-native Environments
[WSO2Con Asia 2018] Architecting for Container-native Environments
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
 
Hybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerůHybridní cloud s F5 v prostředí kontejnerů
Hybridní cloud s F5 v prostředí kontejnerů
 
Twelve-Factor application pattern with Spring Framework
Twelve-Factor application pattern with Spring FrameworkTwelve-Factor application pattern with Spring Framework
Twelve-Factor application pattern with Spring Framework
 
cloud ppt 1.pptx
cloud ppt 1.pptxcloud ppt 1.pptx
cloud ppt 1.pptx
 
Presentation on Cloud Computing (CE).pptx
Presentation on Cloud Computing (CE).pptxPresentation on Cloud Computing (CE).pptx
Presentation on Cloud Computing (CE).pptx
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017
 
AWS Webcast - Datacenter Migration to AWS
AWS Webcast - Datacenter Migration to AWSAWS Webcast - Datacenter Migration to AWS
AWS Webcast - Datacenter Migration to AWS
 
Open shift and docker - october,2014
Open shift and docker - october,2014Open shift and docker - october,2014
Open shift and docker - october,2014
 
Cloud ppt
Cloud pptCloud ppt
Cloud ppt
 
Cloud computing
Cloud computingCloud computing
Cloud computing
 
E2E Services using Cloud Visitation Platforms
E2E Services using Cloud Visitation PlatformsE2E Services using Cloud Visitation Platforms
E2E Services using Cloud Visitation Platforms
 
Getting Started with Docker - Nick Stinemates
Getting Started with Docker - Nick StinematesGetting Started with Docker - Nick Stinemates
Getting Started with Docker - Nick Stinemates
 
Cloud broadcasting and computing
Cloud broadcasting and computing Cloud broadcasting and computing
Cloud broadcasting and computing
 
Building a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for AnalystsBuilding a Just-in-Time Application Stack for Analysts
Building a Just-in-Time Application Stack for Analysts
 
Introduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OSIntroduction to Apache Mesos and DC/OS
Introduction to Apache Mesos and DC/OS
 

Más de Project COLA

MiCADO framework by Project COLA
MiCADO framework by Project COLAMiCADO framework by Project COLA
MiCADO framework by Project COLAProject COLA
 
Rollup MiCADO by Project COLA
Rollup MiCADO by Project COLARollup MiCADO by Project COLA
Rollup MiCADO by Project COLAProject COLA
 
Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019
Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019
Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019Project COLA
 
Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019
Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019
Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019Project COLA
 
What is it (good for)? - MiCADO webinar No.1/4 - 09/2019
What is it (good for)? - MiCADO webinar No.1/4 - 09/2019What is it (good for)? - MiCADO webinar No.1/4 - 09/2019
What is it (good for)? - MiCADO webinar No.1/4 - 09/2019Project COLA
 
Auto-scaling deadline constrained workloads in containers in the cloud
Auto-scaling deadline constrained workloads in containers in the cloudAuto-scaling deadline constrained workloads in containers in the cloud
Auto-scaling deadline constrained workloads in containers in the cloudProject COLA
 
MiCADOscale presented at EGI conference 2019
MiCADOscale presented at EGI conference 2019MiCADOscale presented at EGI conference 2019
MiCADOscale presented at EGI conference 2019Project COLA
 

Más de Project COLA (7)

MiCADO framework by Project COLA
MiCADO framework by Project COLAMiCADO framework by Project COLA
MiCADO framework by Project COLA
 
Rollup MiCADO by Project COLA
Rollup MiCADO by Project COLARollup MiCADO by Project COLA
Rollup MiCADO by Project COLA
 
Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019
Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019
Scalable WordPress use case - MiCADO webinar No.3/4 - 09/2019
 
Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019
Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019
Building Cloud-Native Applications in MiCADO - MiCADO webinar No.2/4 - 09/2019
 
What is it (good for)? - MiCADO webinar No.1/4 - 09/2019
What is it (good for)? - MiCADO webinar No.1/4 - 09/2019What is it (good for)? - MiCADO webinar No.1/4 - 09/2019
What is it (good for)? - MiCADO webinar No.1/4 - 09/2019
 
Auto-scaling deadline constrained workloads in containers in the cloud
Auto-scaling deadline constrained workloads in containers in the cloudAuto-scaling deadline constrained workloads in containers in the cloud
Auto-scaling deadline constrained workloads in containers in the cloud
 
MiCADOscale presented at EGI conference 2019
MiCADOscale presented at EGI conference 2019MiCADOscale presented at EGI conference 2019
MiCADOscale presented at EGI conference 2019
 

Último

Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Shubham Pant
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteMavein
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024Jan Löffler
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsRoxana Stingu
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...APNIC
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdfShreedeep Rayamajhi
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSedrianrheine
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpressssuser166378
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfmchristianalwyn
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSlesteraporado16
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxnaveenithkrishnan
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilitiesalihassaah1994
 

Último (12)

Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024Check out the Free Landing Page Hosting in 2024
Check out the Free Landing Page Hosting in 2024
 
Computer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a WebsiteComputer 10 Lesson 8: Building a Website
Computer 10 Lesson 8: Building a Website
 
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
WordPress by the numbers - Jan Loeffler, CTO WebPros, CloudFest 2024
 
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced HorizonsVision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
Vision Forward: Tracing Image Search SEO From Its Roots To AI-Enhanced Horizons
 
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
Benefits of doing Internet peering and running an Internet Exchange (IX) pres...
 
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdfIntroduction to ICANN and Fellowship program  by Shreedeep Rayamajhi.pdf
Introduction to ICANN and Fellowship program by Shreedeep Rayamajhi.pdf
 
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDSTYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
TYPES AND DEFINITION OF ONLINE CRIMES AND HAZARDS
 
Presentation2.pptx - JoyPress Wordpress
Presentation2.pptx -  JoyPress WordpressPresentation2.pptx -  JoyPress Wordpress
Presentation2.pptx - JoyPress Wordpress
 
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdfLESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
LESSON 5 GROUP 10 ST. THOMAS AQUINAS.pdf
 
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASSLESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
LESSON 10/ GROUP 10/ ST. THOMAS AQUINASS
 
Bio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptxBio Medical Waste Management Guideliness 2023 ppt.pptx
Bio Medical Waste Management Guideliness 2023 ppt.pptx
 
Zero-day Vulnerabilities
Zero-day VulnerabilitiesZero-day Vulnerabilities
Zero-day Vulnerabilities
 

Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept

  • 1. Project COLA: Use Case to create a scalable application in the cloud based on the MICADO concept Peter Kacsuk SZTAKI kacsuk@sztaki.hu
  • 2. Introduction to the application The application is Data Avenue: • This application can • transfer data from one data storage to another one • Upload data from your computer to a data storage • Download data from a data storage to your computer • The data storage can be any of the following type: • HTTP, HTTPS, SFTP, GSIFTP, SRM, iRODS and S3
  • 3. Data Avenue services 3 FS1 FSn Data Avenue Blacktop service Openstack Amazon Data Avenue @ SZTAKI Java or other appli code (JAVA API or WS API) Data Avenue Portlet gateways FS2OpenNebula storage-related protocols web service interface
  • 4. Introduction to the application Detailed objectives: • We want to create a scalable service out of this application. It means that users from all over the world can access this service in order to transfer their data from one data storage to another one provided that they have got access right to the used data storages. • The current service can be accessed at: • https://data-avenue.eu/en_GB/
  • 5. The Data Avenue service
  • 6. How to use Data Avenue?
  • 7. How to use Data Avenue?
  • 8. Use-case1: DataAvenue@SZTAKI > Select storage type (Protocol) > Set storage host address (URL) > Press Go
  • 9. Use-case1: DataAvenue@SZTAKI > Select authentication method > Give credentials > Press Go
  • 10. The Data Avenue Concept Copy between different storages Progress bar Directory contents: files, directories
  • 11. Problems with the current service • It uses a single Data Avenue application and it becomes a bottleneck in the following cases: • One user initiates many transfers in parallel • Many users initiate single transfers in parallel • Many users initiate many transfers in parallel
  • 12. Solution • The Data Avenue application can run in several instances in parallel in the cloud • If a single Data Avenue (DA) becomes bottleneck new DA is instantiated in the cloud and some of the required data transfers are automatically re- directed to the new DA instance • In order to organize all the services as a set of coordinated services we need an information service => Consul service • In order to observe if a data instance becomes overloaded we need a monitoring tool => Prometheus service • In order to deploy and manage all the services as a set of coordinated services we need a cloud orchestrator service => Occopus service • In order to evenly exploit the DA instances we need a load balancer service that directs the users’ DA service calls evenly to the available DA instances => haproxy service
  • 13. Scale up Scale down Automatic scalable architecture for Data Avenue
  • 14. More details for monitoring Alert Manager Executor Available as tutorial at the Occopus web page: http://occopus.lpds.sztaki.hu/tutorials Soon will be available as tutorial at the COLA web page
  • 15. CPU Usage Number of VMs 15/14 Demonstrating scalability
  • 16. 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 numberofvirtualmachines Measure every 40 minute, starting at 8:00 am. One-day long test bench 16/14
  • 17. Limitations • It uses VMs for every service and hence the deployment time of the whole infrastructure is 8-10 minutes in CloudSigma • Good scalability performance results can be achieved only if the used storage has a scalable access mechanism. For example, S3 cloud storage enables scalability but https does not.
  • 18. Docker container based architecture • In order to reduce deployment time we placed every service in a separate docker container. • The modification required: • Dockerized version of all the services (HAProxy, Prometheus, Consul) • Dockerized version of the data node application (DA) • Modification of the cloud-init file that contains the description of cloud- related parameters (cloud-init file became much simpler)
  • 19. General architecture adaptable for other applications -> MICADO v0 • If the DA description is replaced by another service-oriented application in the cloud-init file, then it can be used for other applications, too. • Therefore this architecture can be considered as the first version of the MICADO architecture -> MICADO v0
  • 20. Towards a more generic MICADO architecture -> MICADO v1 • Goal: • create a MICADO architecture in which the application can be accommodated without modifying cloud-init • To achieve this we introduce a docker cluster and the applications will run within in docker containers within this docker cluster • Therefore this architecture can be considered as the improved version of the MICADO v0 architecture -> MICADO v1 • MICADO v1 could have to alternatives: • A. 2-layered MICADO architecture without load balancer layer • B. 3-layered MICADO architecture with load balancer layer
  • 21. 2-layer MICADO v1/a architecture Swarm docker cluster
  • 22. Assessment of MICADO v1/a Advantage: • Adaptable for different applications without modifying the cloud-init file • Scales up and down the number of data nodes according to the actual load of the data nodes • Guarantees the balanced usage of data nodes (responsibility of the Docker Master) Disadvantage: • No guarantee that the data nodes are used in a balanced way to serve user requests • All the data nodes should run the same applications
  • 23. 3-layer MICADO v1/b architecture
  • 24. Assessment of MICADO v1/b Advantage: • Adaptable for different applications without modifying the cloud-init file • Scales up and down the number of data nodes according to the actual load of the data nodes • Guarantees the balanced usage of data nodes (responsibility of the Docker Master) • Guarantees that the data nodes are used in a balanced way to serve user requests (responsibility of the HAProxy services) Disadvantage: • All the data nodes should run the same applications Future work: • MICADO v2 to enable the balanced and scalable usage of different applications
  • 25. Acknowledgement • The development of MICADO is in strong collaboration between the SZTAKI and UoW teams. • SZTAKI team: Prof. Peter Kacsuk • Dr. Jozsef Kovacs • Enikő Nagy • Attila Farkas • Dr. Robert Lovas • Dr. Zoltan Farkas • UoW team: Dr. Tamas Kiss • Prof. Gabor Terstyanszky • Botond Rákoczi • Gregoire Gesmier • Gabriele Pierantoni
  • 26. For more information please visit www.project-cola.eu twitter.com/projectCOLA facebook.com/projectCOLA Thank you!