ElasticSearch on AWS: Continuously Deployed, Immutable and Stateful

1. ElasticSearch on AWS
Continuously Deployed,
Immutable and Stateful

2. About me
● Philipp Garbe (@pgarbe)
● Husband
● Father
● AutoScout24
● Former Microsoft-Fanboy
● Now Docker-Fanboy ;)

3. ● Use case: Logging for AS24
● Challenge: How to handle the state?
○ Immutable
○ Continuously Deployed
○ Stateful
● Next challenges
What to expect

4. Project
Tatsu

5. Unified
Logs
CC

6. STRATEGIC
GOALS
Goals of the business side
ARCHITECTURAL
PRINCIPLES
High-Level Principles
DESIGN AND DELIVERY
PRINCIPLES
Tactical measures
REDUCE TIME TO MARKET
Speed, Fast Feedback
COST EFFICIENCY
Collect metrics to allow decisions cost vs. value.
SUPPORT DATA-DRIVEN DECISIONS
Listen to users and validate hypothesis.
Provide as many relevant metrics & data as possible.
YOU BUILT IT, YOU RUN IT
The team is responsible for shaping, building, running and
maintaining its products. Fast feedback from live and
customers helps us to continuously improve.
ORGANIZED AROUND BUSINESS CAPABILITIES
Build teams around products not projects. Follow the
domain and respect bounded contexts. Inverse Conway
Maneuver.
LOOSELY COUPLED
By default avoid sharing and tight coupling, except for the
big things in common. Don’t create the next monolith.
MACRO AND MICRO ARCHITECTURE
Clear separation. Autonomous micro services within the
rules and constraints of the macro architecture.
AWS FIRST
Favor AWS platform service over managed service, over
self-hosted OSS, over self-rolled solutions.
DATA-DRIVEN / METRIC-DRIVEN
Collect metrics from processes and applications. Analyze,
alert and act on them.
ELIMINATE ACCIDENTAL COMPLEXITY
Strive to keep it simple. Focus on essential complexity.
You build one, you delete one.
AUTONOMOUS TEAMS
Make fast local decisions. Be responsible. Know your
boundaries. Share findings.
INFRASTRUCTURE AS CODE
Automate everything: Reproducible, traceable and tested.
Immutable servers over snowflake servers.
COLLABORATION CULTURE
Engineers from all backgrounds work together in
collaborative teams as engineers and share
responsibilities. No silos.
BE BOLD
Go into production early. Value monitoring over tests.
Recover and learn. Optimize for MTTR not MTBF.
SECURITY, COMPLIANCE AND DATA PRIVACY
Security must be included from the beginning and
everybody’s concern. Keep data-privacy in mind.
CONTAINMENT AND BOUNDARIES
Align blast radius and vendor lock-in with the boundaries
of the organization or business capabilities.
Version 1.0
Icons made by Freepik from www.flaticon.com are licensed under CC BY 3.0
https://github.com/autoscout24/scout24-it-principles

7. INFRASTRUCTURE AS CODE
Automate everything: Reproducible, traceable and tested.
Immutable servers over snowflake servers.

8. Some numbers
12 nodes (m4.4xlarge, EBS gp2 á 2000 GB)
9 indices (one per day)
648 shards
7 Billion docs (new documents peak: 2,500 docs / sec)
16.99 TB of data

9. Challenge: How to handle the state?

10. Immutable Servers
Rake
CloudFormation
ElasticSearch configuration

11. Continuously Deployed
● One CloudFormation stack per node
● Rolling updates through rake
● Every commit goes to production

13. Stateful
Keep the data!

15. Monitoring
● CloudWatch
○ Cron-Lambda sends /_cluster/health metrics to CloudWatch
○ CloudWatch alarms based on these metrics
● DataDog
○ Deep Dive

16. Next challenges
No AvailabilityZone awareness

17. Next challenges
Mapping issues

18. Next challenges
Long-running deployments (2-8h)

19. ?Questions
Philipp Garbe
http://garbe.io
@pgarbe
https://github.com/pgarbe