SlideShare una empresa de Scribd logo
1 de 44
Descargar para leer sin conexión
Core Team
Riccardo Capecchi - Marco Careddu - Piermarco Zerbini
Mar 2017 Devops Day 2017
Cloud DB - strengths and weaknesses
Shopfully - Who We Are
Founded in 2010, ShopFully is the leading
platform used by over 25 million users
worldwide when getting ready to go shopping
in their neighborhood.
The platform contains a variety of information
including details on promotions, new products,
shops, opening times and contacts of the
main retailers and brands in each shopping
category, geolocated in one place and easily
accessible to users.
2
Shopfully - What We Do
ShopFully is the last mile media, the first
source of geolocalized information on
promotions, new products, shops, opening
times and contacts of the main retailers and
brands in all shopping categories
The services offered by ShopFully can be
accessed both online at www.shopfully.com
(or country specific URLs) through the website
as well as through the free app developed for
all major mobile platforms: iOS, Android,
Windows8, Amazon and BlackBerry.
3
Shopfully - We’re going to talk about
1. Some details on our old Database Infrastructure.
2. How we choose our Cloud DB.
3. The design of our application, mainly focused on the Database.
4. How we move move on a Cloud DB
5. The new Challenges and Benefits of a Cloud DB.
6. Conclusions, should you consider to go on a cloud DB ?
4
{ }
Moving to the Cloud
What about the Infrastructure ?
WHY we move on DB As Service
● 9 dedicated servers
● Galera cluster multi-master managed by severalnines’ cluster control framework
● Shared database infrastructure
Before Cloud Database
6
WHY we move on DB As Service
Before Cloud Database
Problems:
● High load on all nodes during traffic spikes
● Very high load on survivor nodes when recovering a broken node
Causes:
● Cluster capacity was near to its limit
Possible solutions:
● Horizontal scale up: unsafe because of high number of cluster nodes
● Vertical scale up replacing all dedicated servers: losing two nodes at same
time was insecure
7
WHY we move on DB As Service
Before Cloud Database - Vision
8
WHERE do we move on?
Let’s go on DB As a Service, but… what do we want?
Goals
● Zero Downtime
● Latest mysql engine version as
possible
● Reduce effort for database
management
9
WHERE do we move on?
Let’s go on DB As Service, but… what do we want?
Preferred providers
● Google Cloud
● AWS
10
WHERE do we move on?
Google Cloud SQL VS Amazon Web Service RDS
Both of them could import a live database, but….
11
WHERE do we move on?
Google Cloud SQL
First generation VS Second generation
Second generation features:
● Up to 7X throughput and 20X storage capacity of First Generation
instances
● Less expensive than First Generation for most use cases
● Option to add High Availability failover and read replication
● Mysql 5.7
COOL! But… second generation does not supports external replication
master.
12
WHERE do we move on?
AWS RDS
Mysql VS Aurora
Let’s move on Amazon RDS. But which database service?
Amazon Aurora (Aurora) is a fully managed, MySQL-compatible, relational
database engine that combines the speed and reliability of high-end commercial
databases with the simplicity and cost-effectiveness of open-source databases.
It delivers up to five times the performance of MySQL without requiring
changes to most of your existing applications.
Amazon Aurora makes it simple and cost-effective to set up, operate, and scale
your new and existing MySQL deployments, thus freeing you to focus on
your business and applications. We love Aurora! But... 13
HOW MUCH does it cost?
OVH vs AWS
The database infrastructure costs was due to:
● visible:
○ 9 dedicated servers
● invisible:
○ 3 virtual machines (load balancers)
○ 1 virtual machine (cluster monitor)
○ storage backup
○ sysadmin time worked
14
HOW MUCH does it cost?
OVH vs AWS
First look to AWS pricing:
● visible: x10 to Infrastructure
Balance for databases
15
HOW MUCH does it cost?
OVH vs AWS
First look to AWS pricing:
● visible: x10 to Infrastructure
Balance for databases
16
HOW MUCH does it cost?
Could we split databases and buy smaller instances?
17
HOW MUCH does it cost?
Query Analysis
Collected data from our Mysql Galera
Cluster highlights that:
● the “load” of each database
● for each database, around half of the
total queries are routed on the writer’s
endpoint of RDS Aurora, and the rest
on the reader’s endpoint
We mapped each database into an AWS
DB tier, in this way the total cost was
reduced from 10x to around 4x.
DB1
DB2
DB3
DB4
DB5
DB6
18
HOW MUCH does it cost?
OVH vs AWS - Round 2
Considering the followings:
● Our galera cluster was near its limits, and we should have paid more for maintenance and
new hardware.
● Amazon solution offers:
○ fully managed solution
○ data replication across availability zones
○ easy way to enlarge/reduce read replicas
○ cloudwatch
○ possibile automation
○ costing management by tag
19
{ }
Moving to the Cloud
What about the code ?
Prerequisites for the cloud: Application design + challenges
Is your application ready for the Cloud?
● Follow some simple rules to simplify the configuration of your app
○ Twelve-Factor app is your friend (3rd factor in particular)
● Our application became almost ‘twelvefactored’ in previous
iterations, to anticipate eventual cloud migration
○ it helped a lot in the migration for extremely centralized
configuration
21
Prerequisites for the cloud: Application design + challenges
Is your application ready for a cloud DB ?
● Keep a simple design
○ No DB triggers or stored procedures
■ In our case we were able to substitute the first with async
application jobs and to avoid the second altogether
○ Rare use of specific MySql features
● The day you will want to change DB vendor or upgrade to a new
major release, you will also thank yourself
22
Prerequisites for the cloud: Application design + challenges
What we had
● Multi-master (Galera)
○ DB Read Write split at the application level using CakePHP ORM
■ a simple 'sticky' master after write, to mitigate inherent deadlocks
of multi-master model
What we needed
● Master-slave (Aurora)
○ improve buggy DB Read Write split
■ moving to master-slave we discovered split was imperfect,
‘leaking’ write queries to slaves
■ bug hidden in the previous multi-master architecture
23
Prerequisites for the cloud: Application design + challenges
Scale for the cloud
● Using proper dimensioned clusters pushed our application to the
limits
○ Lessons learned
■ OLD (but gold): don’t forget to periodically check your DB
indexes (or lack of) usage
■ Use any kind of shielding you can
● CDNs, Application Caches etc.
■ Async, async everywhere
24
{ }
Moving to the Cloud
How do it with Zero Downtime ?
How do we move on?
From Galera Cluster to Aurora
GOAL:
migrate db one at a time
26
How do we move on?
From Galera Cluster to Aurora
GOAL:
migrate db one at a time
Problem:
binlog-do-db option is not supported
by OUR Galera Cluster.
27
How do we move on?
From Galera Cluster to Aurora
Galera Cluster: Replication master for all databases.
Mysql:
1 Slave Replica for all databases.
28
How do we move on?
From Galera Cluster to Aurora
Mysql as “Washing Machine”:
Activates binlog for a single DB
29
How do we move on?
From Galera Cluster to Aurora
Mysql as “Washing Machine”:
Works as external replication master
for Aurora
30
How do we move on?
From Galera Cluster to Aurora
31
How do we move on?
From Galera Cluster to Aurora
Load Balancer
Writer endpoint Reader endpoint
Webservers
32
How do we move on?
From Galera Cluster to Aurora
Load Balancer
Writer endpoint Reader endpoint
Webservers
33
How do we move on?
From Galera Cluster to Aurora
Writer endpoint Reader endpoint
Webservers
Load Balancer
34
New Challenges and Benefits
Great we are now with our DB on the Cloud, but how this changes our lives ?
Performance, Price and Availability are now more interconnected than ever, we want responsive and
quick services that use at their best the DB Instances to reduce AWS cost,
35
New Challenges and Benefits - Autoscale the DB
AWS services are famous for their auto-scale capability… but not on RDS.
But “something” that turn instances on and off could really be useful to us because most of our
traffic is predictable.
As first try we used the aws cli with some simple scheduled tasks on Jenkins to programmatically
turn on/off the DB Instances for the different countries at their wake up/sleep time.
This was a good change, but sometimes the load was higher or lower of the expected so we wrote
some small bash utilities that periodically (every 2 minutes) check the CPU usage of our replica
instances and if the average it’s over or under a threshold it takes an action to scale up or down
that cluster.
36
New Challenges and Benefits - Autoscale the DB
This is much better as the number of instances dynamically changes based on the load, but …
● Adding an instance take up to 10 minutes.
● Removing an instance causes failed connection for our users.
37
New Challenges and Benefits
Some Benefits of using a DB on AWS that we have found include:
- Easily create new instances and replica of them, this means that this tasks can be done now
also by less skilled (on the DB) members of the team.
- Easily manage snapshot and restore an instance from them (this can be good also to scratch
your staging/dev environment and start fresh every day or week).
- Don’t worry anymore about DR, with Aurora the data is automatically replicated across
Availability Zones, optionally you could also have a replica on a different region.
- Easily change the class of your DB servers if you have over or underestimated your load.
- Cloudwatch add value with a good range of metrics ready out-of-the-box to be used.
- Support Center is your friend: we experienced positive and proactive interaction with them
when we had a development cluster crash experimenting advanced new features (Aurora’s
spatial indexes implementation)
38
New Challenges and Benefits
So it’s all wonderful when you stay on cloud ? Not exactly.
These are a few drawbacks of using a DB on AWS for our uses:
- AWS It’s “fully managed” but you have to understand and setup VPC/subnet/security group,
this require some costs on time or a consultant.
- RDS it’s “fully managed” but you have to understand how parameter groups works and the
slightly difference in Aurora engine.
- RDS it’s great if you have a simple infrastructure but it’s also harder if you want to achieve a 0
downtime service.
- If you don’t plan it wisely the costs can easily grow (i.e. reserved instances).
39
{ }
Moving the DB to the Cloud
Is this enough ?
Conclusions
Moving the database as first thing could not be your best option, latency could be an
application killer and in general it’s best to start with an application that it’s totally on cloud,
much better if it’s new and you can plan it from scratch to work on cloud and all the services
you can find there.
Told that, if you have to plan a big change or upgrade for your DB infrastructure a cloud
provider could be a great option as it gives a way to start quickly and over time change the
capacity without too much hassle.
We have now moved our API to AWS as well and this has increased the performance and
lowered the response time … but this is maybe good for another talk ...
41
THANK YOU
FOR YOUR ATTENTION
We are hiring
https://corporate.doveconviene.it/lavora-con-noi/
42
Contacts
● Riccardo Capecchi
○ https://about.me/riccardocapecchi
● Marco Careddu
○ careddu.m85@gmail.com
○ https://www.linkedin.com/in/marco-careddu-33707632
○ https://twitter.com/sgradix
● Piermarco Zerbini
○ piermarco.zerbini@gmail.com
○ https://it.linkedin.com/in/piermarcozerbini
○ https://github.com/Snafrutz
43
References
● SeveralNines - Cluster Control
● Google Cloud - Replication with External Cluster
● Google Cloud - 1st Vs 2nd Gen
● AWS Guides - Mysql importing External Replica
● Aurora Overview
● Our bash scripts to autoscale Aurora
● Twelve Factor App
44

Más contenido relacionado

La actualidad más candente

DCSF19 CMD and Conquer: Containerizing the Monolith
DCSF19 CMD and Conquer: Containerizing the Monolith  DCSF19 CMD and Conquer: Containerizing the Monolith
DCSF19 CMD and Conquer: Containerizing the Monolith Docker, Inc.
 
Pluggable Infrastructure with CI/CD and Docker
Pluggable Infrastructure with CI/CD and DockerPluggable Infrastructure with CI/CD and Docker
Pluggable Infrastructure with CI/CD and DockerBob Killen
 
Kubernetes laravel and kubernetes
Kubernetes   laravel and kubernetesKubernetes   laravel and kubernetes
Kubernetes laravel and kubernetesWilliam Stewart
 
Docker for any type of workload and any IT Infrastructure
Docker for any type of workload and any IT InfrastructureDocker for any type of workload and any IT Infrastructure
Docker for any type of workload and any IT InfrastructureDocker, Inc.
 
DCSF 19 Accelerating Docker Containers with NVIDIA GPUs
DCSF 19 Accelerating Docker Containers with NVIDIA GPUsDCSF 19 Accelerating Docker Containers with NVIDIA GPUs
DCSF 19 Accelerating Docker Containers with NVIDIA GPUsDocker, Inc.
 
Taking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideTaking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideDocker, Inc.
 
Your journey into the serverless world
Your journey into the serverless worldYour journey into the serverless world
Your journey into the serverless worldRed Hat Developers
 
Docker Bday #5, SF Edition: Introduction to Docker
Docker Bday #5, SF Edition: Introduction to DockerDocker Bday #5, SF Edition: Introduction to Docker
Docker Bday #5, SF Edition: Introduction to DockerDocker, Inc.
 
Commit to excellence - Java in containers
Commit to excellence - Java in containersCommit to excellence - Java in containers
Commit to excellence - Java in containersRed Hat Developers
 
The Fairy Tale of the One Command Build Script
The Fairy Tale of the One Command Build ScriptThe Fairy Tale of the One Command Build Script
The Fairy Tale of the One Command Build ScriptDocker, Inc.
 
DCSF19 How To Build Your Containerization Strategy
DCSF19 How To Build Your Containerization Strategy  DCSF19 How To Build Your Containerization Strategy
DCSF19 How To Build Your Containerization Strategy Docker, Inc.
 
Webinar container management in OpenStack
Webinar container management in OpenStackWebinar container management in OpenStack
Webinar container management in OpenStackCREATE-NET
 
Bare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefBare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefMatt Ray
 
How to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker BuildHow to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker BuildDocker, Inc.
 
Advanced Access Control with Docker EE
Advanced Access Control with Docker EEAdvanced Access Control with Docker EE
Advanced Access Control with Docker EEDocker, Inc.
 
DCSF19 How Docker Simplifies Kubernetes for the Masses
DCSF19 How Docker Simplifies Kubernetes for the Masses  DCSF19 How Docker Simplifies Kubernetes for the Masses
DCSF19 How Docker Simplifies Kubernetes for the Masses Docker, Inc.
 
Orchestrating Docker with OpenStack
Orchestrating Docker with OpenStackOrchestrating Docker with OpenStack
Orchestrating Docker with OpenStackErica Windisch
 
Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...
Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...
Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...Docker, Inc.
 
Beyond Ingresses - Better Traffic Management in Kubernetes
Beyond Ingresses - Better Traffic Management in KubernetesBeyond Ingresses - Better Traffic Management in Kubernetes
Beyond Ingresses - Better Traffic Management in KubernetesMark McBride
 
Docker?!?! But I'm a SysAdmin
Docker?!?! But I'm a SysAdminDocker?!?! But I'm a SysAdmin
Docker?!?! But I'm a SysAdminDocker, Inc.
 

La actualidad más candente (20)

DCSF19 CMD and Conquer: Containerizing the Monolith
DCSF19 CMD and Conquer: Containerizing the Monolith  DCSF19 CMD and Conquer: Containerizing the Monolith
DCSF19 CMD and Conquer: Containerizing the Monolith
 
Pluggable Infrastructure with CI/CD and Docker
Pluggable Infrastructure with CI/CD and DockerPluggable Infrastructure with CI/CD and Docker
Pluggable Infrastructure with CI/CD and Docker
 
Kubernetes laravel and kubernetes
Kubernetes   laravel and kubernetesKubernetes   laravel and kubernetes
Kubernetes laravel and kubernetes
 
Docker for any type of workload and any IT Infrastructure
Docker for any type of workload and any IT InfrastructureDocker for any type of workload and any IT Infrastructure
Docker for any type of workload and any IT Infrastructure
 
DCSF 19 Accelerating Docker Containers with NVIDIA GPUs
DCSF 19 Accelerating Docker Containers with NVIDIA GPUsDCSF 19 Accelerating Docker Containers with NVIDIA GPUs
DCSF 19 Accelerating Docker Containers with NVIDIA GPUs
 
Taking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and DecideTaking Docker to Production: What You Need to Know and Decide
Taking Docker to Production: What You Need to Know and Decide
 
Your journey into the serverless world
Your journey into the serverless worldYour journey into the serverless world
Your journey into the serverless world
 
Docker Bday #5, SF Edition: Introduction to Docker
Docker Bday #5, SF Edition: Introduction to DockerDocker Bday #5, SF Edition: Introduction to Docker
Docker Bday #5, SF Edition: Introduction to Docker
 
Commit to excellence - Java in containers
Commit to excellence - Java in containersCommit to excellence - Java in containers
Commit to excellence - Java in containers
 
The Fairy Tale of the One Command Build Script
The Fairy Tale of the One Command Build ScriptThe Fairy Tale of the One Command Build Script
The Fairy Tale of the One Command Build Script
 
DCSF19 How To Build Your Containerization Strategy
DCSF19 How To Build Your Containerization Strategy  DCSF19 How To Build Your Containerization Strategy
DCSF19 How To Build Your Containerization Strategy
 
Webinar container management in OpenStack
Webinar container management in OpenStackWebinar container management in OpenStack
Webinar container management in OpenStack
 
Bare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefBare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and Chef
 
How to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker BuildHow to Improve Your Image Builds Using Advance Docker Build
How to Improve Your Image Builds Using Advance Docker Build
 
Advanced Access Control with Docker EE
Advanced Access Control with Docker EEAdvanced Access Control with Docker EE
Advanced Access Control with Docker EE
 
DCSF19 How Docker Simplifies Kubernetes for the Masses
DCSF19 How Docker Simplifies Kubernetes for the Masses  DCSF19 How Docker Simplifies Kubernetes for the Masses
DCSF19 How Docker Simplifies Kubernetes for the Masses
 
Orchestrating Docker with OpenStack
Orchestrating Docker with OpenStackOrchestrating Docker with OpenStack
Orchestrating Docker with OpenStack
 
Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...
Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...
Using Docker for GPU-accelerated Applications by Felix Abecassis and Jonathan...
 
Beyond Ingresses - Better Traffic Management in Kubernetes
Beyond Ingresses - Better Traffic Management in KubernetesBeyond Ingresses - Better Traffic Management in Kubernetes
Beyond Ingresses - Better Traffic Management in Kubernetes
 
Docker?!?! But I'm a SysAdmin
Docker?!?! But I'm a SysAdminDocker?!?! But I'm a SysAdmin
Docker?!?! But I'm a SysAdmin
 

Destacado

Quality Assurance Digital Education: Lessons from the Maltese Experience
Quality Assurance Digital Education: Lessons from the Maltese ExperienceQuality Assurance Digital Education: Lessons from the Maltese Experience
Quality Assurance Digital Education: Lessons from the Maltese ExperienceAnthony Fisher Camilleri
 
Financial Management- Comparison Two MNCS in FMCG Sector
Financial Management- Comparison Two MNCS in FMCG SectorFinancial Management- Comparison Two MNCS in FMCG Sector
Financial Management- Comparison Two MNCS in FMCG Sectorsdusane1
 
"Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ...
"Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ..."Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ...
"Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ...ExternalEvents
 
L IGP Agneau de Sisteron
L IGP Agneau de SisteronL IGP Agneau de Sisteron
L IGP Agneau de SisteronExternalEvents
 
Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17
Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17
Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17StartUps.be
 
Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17
Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17
Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17StartUps.be
 
JAWS DAYS 2017 lunch session
JAWS DAYS 2017 lunch sessionJAWS DAYS 2017 lunch session
JAWS DAYS 2017 lunch session陽平 山口
 
владислав володин семинар Prom.ua - хмельницкий 2017 февраль
владислав володин   семинар Prom.ua - хмельницкий 2017 февральвладислав володин   семинар Prom.ua - хмельницкий 2017 февраль
владислав володин семинар Prom.ua - хмельницкий 2017 февральVlad Volodin
 
Walter russell’s cosmology byJohn EchEl
Walter russell’s cosmology byJohn EchElWalter russell’s cosmology byJohn EchEl
Walter russell’s cosmology byJohn EchElNano Nate
 
Data Warehouse Design Project
Data Warehouse Design ProjectData Warehouse Design Project
Data Warehouse Design ProjectPradeep Yamala
 
阿里研究院- 新零售研究报告
阿里研究院- 新零售研究报告阿里研究院- 新零售研究报告
阿里研究院- 新零售研究报告Tony Ho
 

Destacado (16)

Quality Assurance Digital Education: Lessons from the Maltese Experience
Quality Assurance Digital Education: Lessons from the Maltese ExperienceQuality Assurance Digital Education: Lessons from the Maltese Experience
Quality Assurance Digital Education: Lessons from the Maltese Experience
 
Arquitectura del Quattrocento.
Arquitectura del Quattrocento.Arquitectura del Quattrocento.
Arquitectura del Quattrocento.
 
Financial Management- Comparison Two MNCS in FMCG Sector
Financial Management- Comparison Two MNCS in FMCG SectorFinancial Management- Comparison Two MNCS in FMCG Sector
Financial Management- Comparison Two MNCS in FMCG Sector
 
"Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ...
"Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ..."Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ...
"Projeto Piloto - Brasil Primeiro Seminário Regional – Projeto FAO Guayaquil ...
 
L IGP Agneau de Sisteron
L IGP Agneau de SisteronL IGP Agneau de Sisteron
L IGP Agneau de Sisteron
 
Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17
Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17
Harold Grondel: Intrapreneurship - Corporate innovation - TSD 17
 
Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17
Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17
Philippe Mouchard: Intrapreneurship - Corporate innovation - TSD 17
 
JAWS DAYS 2017 lunch session
JAWS DAYS 2017 lunch sessionJAWS DAYS 2017 lunch session
JAWS DAYS 2017 lunch session
 
Ikhtiologi riska
Ikhtiologi riskaIkhtiologi riska
Ikhtiologi riska
 
Documento di Buona Causa
Documento di Buona CausaDocumento di Buona Causa
Documento di Buona Causa
 
владислав володин семинар Prom.ua - хмельницкий 2017 февраль
владислав володин   семинар Prom.ua - хмельницкий 2017 февральвладислав володин   семинар Prom.ua - хмельницкий 2017 февраль
владислав володин семинар Prom.ua - хмельницкий 2017 февраль
 
Así es hoy el cv perfecto
Así es hoy el cv perfectoAsí es hoy el cv perfecto
Así es hoy el cv perfecto
 
Walter russell’s cosmology byJohn EchEl
Walter russell’s cosmology byJohn EchElWalter russell’s cosmology byJohn EchEl
Walter russell’s cosmology byJohn EchEl
 
Introduction to investigation
Introduction to investigationIntroduction to investigation
Introduction to investigation
 
Data Warehouse Design Project
Data Warehouse Design ProjectData Warehouse Design Project
Data Warehouse Design Project
 
阿里研究院- 新零售研究报告
阿里研究院- 新零售研究报告阿里研究院- 新零售研究报告
阿里研究院- 新零售研究报告
 

Similar a Idi2017 - Cloud DB: strengths and weaknesses

Serverless Compose vs hurtownia danych
Serverless Compose vs hurtownia danychServerless Compose vs hurtownia danych
Serverless Compose vs hurtownia danychThe Software House
 
Cloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs GoogleCloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs GooglePatrick Pierson
 
Database Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big DataDatabase Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big Dataexponential-inc
 
Migrating a multi tenant app to Azure (war biopic)
Migrating a multi tenant app to Azure (war biopic)Migrating a multi tenant app to Azure (war biopic)
Migrating a multi tenant app to Azure (war biopic)★ Akshay Surve
 
Containerizing couchbase with microservice architecture on mesosphere.pptx
Containerizing couchbase with microservice architecture on mesosphere.pptxContainerizing couchbase with microservice architecture on mesosphere.pptx
Containerizing couchbase with microservice architecture on mesosphere.pptxRavi Yadav
 
Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...
Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...
Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...Continuent
 
Data Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWSData Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWSJohn McCormack
 
Altoros using no sql databases for interactive_applications
Altoros using no sql databases for interactive_applicationsAltoros using no sql databases for interactive_applications
Altoros using no sql databases for interactive_applicationsJeff Harris
 
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)Amazon Web Services
 
Introduction to Google Cloud Platform
Introduction to Google Cloud PlatformIntroduction to Google Cloud Platform
Introduction to Google Cloud Platformdhruv_chaudhari
 
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part20812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2Raul Chong
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQLbalwinders
 
Percona Live 2014 - Scaling MySQL in AWS
Percona Live 2014 - Scaling MySQL in AWSPercona Live 2014 - Scaling MySQL in AWS
Percona Live 2014 - Scaling MySQL in AWSPythian
 
Idera live 2021: Managing Databases in the Cloud - the First Step, a Succes...
Idera live 2021:   Managing Databases in the Cloud - the First Step, a Succes...Idera live 2021:   Managing Databases in the Cloud - the First Step, a Succes...
Idera live 2021: Managing Databases in the Cloud - the First Step, a Succes...IDERA Software
 
Save 60% of Kubernetes storage costs on AWS & others with OpenEBS
Save 60% of Kubernetes storage costs on AWS & others with OpenEBSSave 60% of Kubernetes storage costs on AWS & others with OpenEBS
Save 60% of Kubernetes storage costs on AWS & others with OpenEBSMayaData Inc
 
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)Rick Hwang
 
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...Amazon Web Services
 
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...Amazon Web Services
 

Similar a Idi2017 - Cloud DB: strengths and weaknesses (20)

Serverless Compose vs hurtownia danych
Serverless Compose vs hurtownia danychServerless Compose vs hurtownia danych
Serverless Compose vs hurtownia danych
 
Cloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs GoogleCloud comparison - AWS vs Azure vs Google
Cloud comparison - AWS vs Azure vs Google
 
Database Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big DataDatabase Virtualization: The Next Wave of Big Data
Database Virtualization: The Next Wave of Big Data
 
Migrating a multi tenant app to Azure (war biopic)
Migrating a multi tenant app to Azure (war biopic)Migrating a multi tenant app to Azure (war biopic)
Migrating a multi tenant app to Azure (war biopic)
 
Build A Scalable Mobile App
Build A Scalable Mobile App Build A Scalable Mobile App
Build A Scalable Mobile App
 
Containerizing couchbase with microservice architecture on mesosphere.pptx
Containerizing couchbase with microservice architecture on mesosphere.pptxContainerizing couchbase with microservice architecture on mesosphere.pptx
Containerizing couchbase with microservice architecture on mesosphere.pptx
 
Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...
Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...
Picking the Right Clustering for MySQL - Cloud-only Services or Flexible Tung...
 
Data Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWSData Scotland 2019: You can run SQL Server on AWS
Data Scotland 2019: You can run SQL Server on AWS
 
Altoros using no sql databases for interactive_applications
Altoros using no sql databases for interactive_applicationsAltoros using no sql databases for interactive_applications
Altoros using no sql databases for interactive_applications
 
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
 
Introduction to Google Cloud Platform
Introduction to Google Cloud PlatformIntroduction to Google Cloud Platform
Introduction to Google Cloud Platform
 
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part20812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
0812 2014 01_toronto-smac meetup_i_os_cloudant_worklight_part2
 
IBM - Introduction to Cloudant
IBM - Introduction to CloudantIBM - Introduction to Cloudant
IBM - Introduction to Cloudant
 
Introduction to NoSQL
Introduction to NoSQLIntroduction to NoSQL
Introduction to NoSQL
 
Percona Live 2014 - Scaling MySQL in AWS
Percona Live 2014 - Scaling MySQL in AWSPercona Live 2014 - Scaling MySQL in AWS
Percona Live 2014 - Scaling MySQL in AWS
 
Idera live 2021: Managing Databases in the Cloud - the First Step, a Succes...
Idera live 2021:   Managing Databases in the Cloud - the First Step, a Succes...Idera live 2021:   Managing Databases in the Cloud - the First Step, a Succes...
Idera live 2021: Managing Databases in the Cloud - the First Step, a Succes...
 
Save 60% of Kubernetes storage costs on AWS & others with OpenEBS
Save 60% of Kubernetes storage costs on AWS & others with OpenEBSSave 60% of Kubernetes storage costs on AWS & others with OpenEBS
Save 60% of Kubernetes storage costs on AWS & others with OpenEBS
 
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
Study Notes - Architecting for the cloud (AWS Best Practices, Feb 2016)
 
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
 
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
Consolidate MySQL Shards Into Amazon Aurora Using AWS Database Migration Serv...
 

Último

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 

Último (20)

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 

Idi2017 - Cloud DB: strengths and weaknesses

  • 1. Core Team Riccardo Capecchi - Marco Careddu - Piermarco Zerbini Mar 2017 Devops Day 2017 Cloud DB - strengths and weaknesses
  • 2. Shopfully - Who We Are Founded in 2010, ShopFully is the leading platform used by over 25 million users worldwide when getting ready to go shopping in their neighborhood. The platform contains a variety of information including details on promotions, new products, shops, opening times and contacts of the main retailers and brands in each shopping category, geolocated in one place and easily accessible to users. 2
  • 3. Shopfully - What We Do ShopFully is the last mile media, the first source of geolocalized information on promotions, new products, shops, opening times and contacts of the main retailers and brands in all shopping categories The services offered by ShopFully can be accessed both online at www.shopfully.com (or country specific URLs) through the website as well as through the free app developed for all major mobile platforms: iOS, Android, Windows8, Amazon and BlackBerry. 3
  • 4. Shopfully - We’re going to talk about 1. Some details on our old Database Infrastructure. 2. How we choose our Cloud DB. 3. The design of our application, mainly focused on the Database. 4. How we move move on a Cloud DB 5. The new Challenges and Benefits of a Cloud DB. 6. Conclusions, should you consider to go on a cloud DB ? 4
  • 5. { } Moving to the Cloud What about the Infrastructure ?
  • 6. WHY we move on DB As Service ● 9 dedicated servers ● Galera cluster multi-master managed by severalnines’ cluster control framework ● Shared database infrastructure Before Cloud Database 6
  • 7. WHY we move on DB As Service Before Cloud Database Problems: ● High load on all nodes during traffic spikes ● Very high load on survivor nodes when recovering a broken node Causes: ● Cluster capacity was near to its limit Possible solutions: ● Horizontal scale up: unsafe because of high number of cluster nodes ● Vertical scale up replacing all dedicated servers: losing two nodes at same time was insecure 7
  • 8. WHY we move on DB As Service Before Cloud Database - Vision 8
  • 9. WHERE do we move on? Let’s go on DB As a Service, but… what do we want? Goals ● Zero Downtime ● Latest mysql engine version as possible ● Reduce effort for database management 9
  • 10. WHERE do we move on? Let’s go on DB As Service, but… what do we want? Preferred providers ● Google Cloud ● AWS 10
  • 11. WHERE do we move on? Google Cloud SQL VS Amazon Web Service RDS Both of them could import a live database, but…. 11
  • 12. WHERE do we move on? Google Cloud SQL First generation VS Second generation Second generation features: ● Up to 7X throughput and 20X storage capacity of First Generation instances ● Less expensive than First Generation for most use cases ● Option to add High Availability failover and read replication ● Mysql 5.7 COOL! But… second generation does not supports external replication master. 12
  • 13. WHERE do we move on? AWS RDS Mysql VS Aurora Let’s move on Amazon RDS. But which database service? Amazon Aurora (Aurora) is a fully managed, MySQL-compatible, relational database engine that combines the speed and reliability of high-end commercial databases with the simplicity and cost-effectiveness of open-source databases. It delivers up to five times the performance of MySQL without requiring changes to most of your existing applications. Amazon Aurora makes it simple and cost-effective to set up, operate, and scale your new and existing MySQL deployments, thus freeing you to focus on your business and applications. We love Aurora! But... 13
  • 14. HOW MUCH does it cost? OVH vs AWS The database infrastructure costs was due to: ● visible: ○ 9 dedicated servers ● invisible: ○ 3 virtual machines (load balancers) ○ 1 virtual machine (cluster monitor) ○ storage backup ○ sysadmin time worked 14
  • 15. HOW MUCH does it cost? OVH vs AWS First look to AWS pricing: ● visible: x10 to Infrastructure Balance for databases 15
  • 16. HOW MUCH does it cost? OVH vs AWS First look to AWS pricing: ● visible: x10 to Infrastructure Balance for databases 16
  • 17. HOW MUCH does it cost? Could we split databases and buy smaller instances? 17
  • 18. HOW MUCH does it cost? Query Analysis Collected data from our Mysql Galera Cluster highlights that: ● the “load” of each database ● for each database, around half of the total queries are routed on the writer’s endpoint of RDS Aurora, and the rest on the reader’s endpoint We mapped each database into an AWS DB tier, in this way the total cost was reduced from 10x to around 4x. DB1 DB2 DB3 DB4 DB5 DB6 18
  • 19. HOW MUCH does it cost? OVH vs AWS - Round 2 Considering the followings: ● Our galera cluster was near its limits, and we should have paid more for maintenance and new hardware. ● Amazon solution offers: ○ fully managed solution ○ data replication across availability zones ○ easy way to enlarge/reduce read replicas ○ cloudwatch ○ possibile automation ○ costing management by tag 19
  • 20. { } Moving to the Cloud What about the code ?
  • 21. Prerequisites for the cloud: Application design + challenges Is your application ready for the Cloud? ● Follow some simple rules to simplify the configuration of your app ○ Twelve-Factor app is your friend (3rd factor in particular) ● Our application became almost ‘twelvefactored’ in previous iterations, to anticipate eventual cloud migration ○ it helped a lot in the migration for extremely centralized configuration 21
  • 22. Prerequisites for the cloud: Application design + challenges Is your application ready for a cloud DB ? ● Keep a simple design ○ No DB triggers or stored procedures ■ In our case we were able to substitute the first with async application jobs and to avoid the second altogether ○ Rare use of specific MySql features ● The day you will want to change DB vendor or upgrade to a new major release, you will also thank yourself 22
  • 23. Prerequisites for the cloud: Application design + challenges What we had ● Multi-master (Galera) ○ DB Read Write split at the application level using CakePHP ORM ■ a simple 'sticky' master after write, to mitigate inherent deadlocks of multi-master model What we needed ● Master-slave (Aurora) ○ improve buggy DB Read Write split ■ moving to master-slave we discovered split was imperfect, ‘leaking’ write queries to slaves ■ bug hidden in the previous multi-master architecture 23
  • 24. Prerequisites for the cloud: Application design + challenges Scale for the cloud ● Using proper dimensioned clusters pushed our application to the limits ○ Lessons learned ■ OLD (but gold): don’t forget to periodically check your DB indexes (or lack of) usage ■ Use any kind of shielding you can ● CDNs, Application Caches etc. ■ Async, async everywhere 24
  • 25. { } Moving to the Cloud How do it with Zero Downtime ?
  • 26. How do we move on? From Galera Cluster to Aurora GOAL: migrate db one at a time 26
  • 27. How do we move on? From Galera Cluster to Aurora GOAL: migrate db one at a time Problem: binlog-do-db option is not supported by OUR Galera Cluster. 27
  • 28. How do we move on? From Galera Cluster to Aurora Galera Cluster: Replication master for all databases. Mysql: 1 Slave Replica for all databases. 28
  • 29. How do we move on? From Galera Cluster to Aurora Mysql as “Washing Machine”: Activates binlog for a single DB 29
  • 30. How do we move on? From Galera Cluster to Aurora Mysql as “Washing Machine”: Works as external replication master for Aurora 30
  • 31. How do we move on? From Galera Cluster to Aurora 31
  • 32. How do we move on? From Galera Cluster to Aurora Load Balancer Writer endpoint Reader endpoint Webservers 32
  • 33. How do we move on? From Galera Cluster to Aurora Load Balancer Writer endpoint Reader endpoint Webservers 33
  • 34. How do we move on? From Galera Cluster to Aurora Writer endpoint Reader endpoint Webservers Load Balancer 34
  • 35. New Challenges and Benefits Great we are now with our DB on the Cloud, but how this changes our lives ? Performance, Price and Availability are now more interconnected than ever, we want responsive and quick services that use at their best the DB Instances to reduce AWS cost, 35
  • 36. New Challenges and Benefits - Autoscale the DB AWS services are famous for their auto-scale capability… but not on RDS. But “something” that turn instances on and off could really be useful to us because most of our traffic is predictable. As first try we used the aws cli with some simple scheduled tasks on Jenkins to programmatically turn on/off the DB Instances for the different countries at their wake up/sleep time. This was a good change, but sometimes the load was higher or lower of the expected so we wrote some small bash utilities that periodically (every 2 minutes) check the CPU usage of our replica instances and if the average it’s over or under a threshold it takes an action to scale up or down that cluster. 36
  • 37. New Challenges and Benefits - Autoscale the DB This is much better as the number of instances dynamically changes based on the load, but … ● Adding an instance take up to 10 minutes. ● Removing an instance causes failed connection for our users. 37
  • 38. New Challenges and Benefits Some Benefits of using a DB on AWS that we have found include: - Easily create new instances and replica of them, this means that this tasks can be done now also by less skilled (on the DB) members of the team. - Easily manage snapshot and restore an instance from them (this can be good also to scratch your staging/dev environment and start fresh every day or week). - Don’t worry anymore about DR, with Aurora the data is automatically replicated across Availability Zones, optionally you could also have a replica on a different region. - Easily change the class of your DB servers if you have over or underestimated your load. - Cloudwatch add value with a good range of metrics ready out-of-the-box to be used. - Support Center is your friend: we experienced positive and proactive interaction with them when we had a development cluster crash experimenting advanced new features (Aurora’s spatial indexes implementation) 38
  • 39. New Challenges and Benefits So it’s all wonderful when you stay on cloud ? Not exactly. These are a few drawbacks of using a DB on AWS for our uses: - AWS It’s “fully managed” but you have to understand and setup VPC/subnet/security group, this require some costs on time or a consultant. - RDS it’s “fully managed” but you have to understand how parameter groups works and the slightly difference in Aurora engine. - RDS it’s great if you have a simple infrastructure but it’s also harder if you want to achieve a 0 downtime service. - If you don’t plan it wisely the costs can easily grow (i.e. reserved instances). 39
  • 40. { } Moving the DB to the Cloud Is this enough ?
  • 41. Conclusions Moving the database as first thing could not be your best option, latency could be an application killer and in general it’s best to start with an application that it’s totally on cloud, much better if it’s new and you can plan it from scratch to work on cloud and all the services you can find there. Told that, if you have to plan a big change or upgrade for your DB infrastructure a cloud provider could be a great option as it gives a way to start quickly and over time change the capacity without too much hassle. We have now moved our API to AWS as well and this has increased the performance and lowered the response time … but this is maybe good for another talk ... 41
  • 42. THANK YOU FOR YOUR ATTENTION We are hiring https://corporate.doveconviene.it/lavora-con-noi/ 42
  • 43. Contacts ● Riccardo Capecchi ○ https://about.me/riccardocapecchi ● Marco Careddu ○ careddu.m85@gmail.com ○ https://www.linkedin.com/in/marco-careddu-33707632 ○ https://twitter.com/sgradix ● Piermarco Zerbini ○ piermarco.zerbini@gmail.com ○ https://it.linkedin.com/in/piermarcozerbini ○ https://github.com/Snafrutz 43
  • 44. References ● SeveralNines - Cluster Control ● Google Cloud - Replication with External Cluster ● Google Cloud - 1st Vs 2nd Gen ● AWS Guides - Mysql importing External Replica ● Aurora Overview ● Our bash scripts to autoscale Aurora ● Twelve Factor App 44