in comic strip

1. How we cooked Elasticsearch, Consul,
Haproxy and DNS-recursor
In comic strip
Authors:
Shcherbakov S.
Sin A.
Tokarev O.

2. DNS
I resolve my elasticsearch
location by DNS
And send request to it
Gimme my data!
WhereismyElasticsearch?

3. What happens if Elasticsearch down?

4. DNS
WhereismyElasticsearch?

5. Ok. I will hide my Elasticsearch(es) behind HAproxy!!!

6. Data request
Service monitoring

7. DNS
Ok. Now I can access my data
even if one of elasticsearch is down!
Gimme my data!
WhereismyElasticsearch?

8. DNS
WhereismyElasticsearch?
^#%$*&#%*%
@*&%##*&$%399493

9. Ok. Let's add extra balancer for redundancy!

10. DNS
Ok. Now DNS points to one of balancer.
But what will happen
if one of balancer down??
Gimme my data!
{OR}
WhereismyElasticsearch?

11. DNSWhereismyElasticsearch?
Gimme my data!

12. DNS
WhereismyElasticsearch?
G
im
m
e
m
y
data!

13. DNS
WhereismyElasticsearch?
Gimme my data!
It makes me crazy
that DNS continues give me
An address of broken
server

14. How to make DNS forget broken balancer server
ASAP?!?

15. Make Consul cluster monitor your balancers health
and inform your NS about IPs of health ones
(Yep, Consul can speak in DNS language)

16. DNS
Forward zone
Consul cluster
Ok! Now if one of my balancer down
Consul will return IPs of health ones only
Consul is authoritative for a zone
with my critical services

17. Not enough thermometers???

18. DNS
Consul clusterNotify Admin!
Retrieve services IPs

19. Three safety loops
Node level
Disaster notifications
(zabbix)
Application lo-level
Automated application cluster load balancing
(HAproxy)
Application hi-level
Automated application address discovering
(Consul, DNS recursor)

20. Do you thing that it is the DNS server
that is the Single Point of Failure?

21. ZABBIX!!!
But it does not affect any component of production system directly
Neither customers are affected
As for DNS servers they can be readily deployed in required amount
with resolv.conf updated with new servers IPs

22. DNS
Still works..
NOT BAD!!

24. HAproxy
/etc/haproxy/haproxy.cfg
frontend elasticsearch
bind *:9200
default_backend elasticsearch
backend elasticsearch
server es01 123.123.123.1:9200 check inter 1000 rise 10 fall 5
server es02 123.123.123.2:9200 check inter 1000 rise 10 fall 5
server es03 123.123.123.3:9200 check inter 1000 rise 10 fall 5
option httpchk GET /_cluster/health
http-check expect ! string "status":"red"

25. Consul agent (on HAproxy nodes)
/opt/consul/config/srv-es-lb.json
{
"service": {
"name": "es-lb",
"port": 9200,
"checks": [
{
"interval": "5s",
"http": "http://localhost:9200",
"timeout": "1s"
}
]
}
}

26. Consul looks like as NS from outside
[root@consul01 ~]# dig @consul01 -p 8600 es-lb.service.martyshka +short
185.66.251.35
185.66.251.34
[root@consul01 ~]#

27. PDNS recursor
/etc/pdns-recursor/recursor.conf
packetcache-ttl=0
/etc/pdns-recursor/conf.d/forward-zones.conf
forward-zones=martyshka=consul01:8600;

28. Resolver (all nodes)
/etc/resolv.conf
search openprovider.nl hosting-concepts.nl
nameserver <recursor 1 IP>
nameserver <recursor 2 IP>
nameserver <provider NS 1 IP>
nameserver <provider NS 2 IP>
nameserver 8.8.8.8

29. DNS
Consul clusterNotify Admin!
Retrieve services IPs

30. ● Cloud
● Cluster
● Fail-over
● Redundancy
● Fault tolerance
“fashionable stylish for youth”