Geneva Docker Meetup

1. The Elastic Stack on Docker
Elastic in a Box

2. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

3. Getting acquainted - Who am I?
● Valentin Crettaz (mailto:valentin.crettaz@consulthys.com)
○ https://www.linkedin.com/in/valentincrettaz/
○ https://twitter.com/consulthys
● Developing Java software since 1996
● Fell in love with Elasticsearch in 2010 (v0.9.0)
○ https://www.elastic.co/blog/you-know-for-search
● Running Elasticsearch meetups in Switzerland since early 2016
○ https://www.meetup.com/fr-FR/elasticsearch-switzerland
● Active open-source contributor
○ https://github.com/consulthys
● Active Stack Overflow contributor
○ http://stackoverflow.com/users/4604579/val

4. Getting acquainted - Who are you?
● How many of you have already…
○ … heard of Elasticsearch?
○ … downloaded Elasticsearch?
○ … installed/run Elasticsearch?
○ … extended Elasticsearch?
● Your background?
● Define Elasticsearch with your own words

5. Getting acquainted - Disclaimer

6. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

7. Ecosystem - The Big Picture

8. Ecosystem - An Even Bigger Picture

9. Ecosystem - Logstash
2013: https://www.elastic.co/blog/welcome-jordan-logstash

10. Ecosystem - Kibana
2013: https://www.elastic.co/blog/welcome-drew-rashid

11. Ecosystem - Beats
2015: https://www.elastic.co/blog/welcome-packetbeat-tudor-monica

12. Ecosystem - Monitoring

13. Ecosystem - Machine Learning
2016: https://www.elastic.co/blog/welcome-prelert-to-the-elastic-team

14. Ecosystem - Graph

15. Ecosystem - Elastic Cloud
2015: https://www.elastic.co/blog/welcome-found

16. Ecosystem - APM
2017: https://www.elastic.co/blog/welcome-opbeat-to-the-elastic-family

17. Ecosystem - Site + Application Search
2017: https://www.elastic.co/blog/swiftype-joins-forces-with-elastic

18. Ecosystem - Semantic Source Search
2018: https://www.elastic.co/blog/welcome-insight-io-to-the-elastic-team

19. Ecosystem - and more...
● Watcher (for alerting)
● Shield (for security)
● Report (for reporting)
● Canvas (for infographics)
● ...

20. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

21. Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine capable of
solving a growing number of use cases.

22. Use cases
● Full-text search
● Application search
● Enterprise search
● Business analytics
● Metrics analytics
● Security analytics
● Operational logs analytics
● Anomaly detection
● ...

23. Features overview
● Distributed and scalable
● Resilient
● Fault tolerant
● High availability
● RESTful interface
● Document-oriented
● Schema free
● Multi-tenancy
● Extensible
● Growing and active community
● Query DSL
● Aggregations
● Full-text search (Lucene)
● Structured search
● Geo-spatial search
● Suggesters
● Highlighters
● Percolation
● Profiling
● Client libraries in 10+ languages
● ...

24. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

25. What are we talking about?

26. What are we talking about?

27. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

28. Installation (Vanilla) - https://elastic.co/downloads
1. Download and unzip (or yum/dpkg/msi it)
2. ./bin/elasticsearch
3. curl http://localhost:9200/
4. That’s all folks, now you can brag about it !

29. Installation (Docker) - https://www.docker.elastic.co/

30. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

31. Running Official Docker Images (1/2)
docker run docker.elastic.co/elasticsearch/elasticsearch:6.4.1
docker run -e "node.name=es_node1"
docker.elastic.co/elasticsearch/elasticsearch:6.4.1
docker run -e "node.name=es_node1"
-e ES_JAVA_OPTS="-Xmx2g -Xms2g"
-e "cluster.name=my_es_cluster"
docker.elastic.co/elasticsearch/elasticsearch:6.4.1
...

32. Running Official Docker Images (2/2)
...
docker run -e "node.name=es_node1"
-e ES_JAVA_OPTS="-Xmx2g -Xms2g"
-e "cluster.name=my_es_cluster"
-v /elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
docker.elastic.co/elasticsearch/elasticsearch:6.4.1
docker run -e "node.name=es_node1"
-e ES_JAVA_OPTS="-Xmx2g -Xms2g"
-e "cluster.name=my_es_cluster"
-v /elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
-v /mnt/es-data:/usr/share/elasticsearch/data
-p 9201:9200
docker.elastic.co/elasticsearch/elasticsearch:6.4.1

33. Creating custom images using Dockerfile (1/5)
FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1
ENV ES_JAVA_OPTS="-Xmx2g -Xms2g"
COPY /conf/elasticsearch.yml /usr/share/elasticsearch/config/
VOLUME /mnt/es-data
EXPOSE 9200 9300
HEALTHCHECK CMD curl --fail http://localhost:9200 || exit 1

34. $> docker build . -t custom_es641
Sending build context to Docker daemon 45.44MB
Step 1/6 : FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1
---> 96dd1575de0f
Step 2/6 : ENV ES_JAVA_OPTS "-Xmx2g -Xms2g"
---> Using cache
---> d9b744d2ce1a
Step 3/6 : COPY /conf/elasticsearch.yml /usr/share/elasticsearch/config/
---> Using cache
---> 920f451da5b0
Step 4/6 : VOLUME /mnt/es-data
---> Using cache
---> 9b1e957f4820
Step 5/6 : EXPOSE 9200 9300
---> Using cache
---> 8b62712bc1e0
Step 6/6 : HEALTHCHECK CMD curl --fail http://localhost:9200 || exit 1
---> Using cache
---> c673dde527e4
Successfully built c673dde527e4
Successfully tagged custom_es641:latest
Creating custom images using Dockerfile (2/5)

35. $> docker run -p 9200:9200 custom_es641:latest
$> curl localhost:9200
{
"name" : "Slo94R2",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "ccJTixm4QOmtUQn8wnWa7Q",
"version" : {
"number" : "6.4.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "e36acdb",
"build_date" : "2018-09-13T22:18:07.696808Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Creating custom images using Dockerfile (3/5)
elasticsearch
Slo94R2

36. Creating custom images using Dockerfile (4/5)
docker run -e "node.name=es1" --name es1 -p 9200:9200 -p 9300:9300
-e "discovery.zen.ping.unicast.hosts= es1,es2,es3"
-e "cluster.name=3_node_cluster"
custom_es641:latest
docker run -e "node.name=es2" --name es2 -p 9201:9200 -p 9301:9300
-e "discovery.zen.ping.unicast.hosts= es1,es2,es3"
-e "cluster.name=3_node_cluster"
custom_es641:latest
docker run -e "node.name=es3" --name es3 -p 9202:9200 -p 9302:9300
-e "discovery.zen.ping.unicast.hosts= es1,es2,es3"
-e "cluster.name=3_node_cluster"
custom_es641:latest

37. Creating custom images using Dockerfile (5/5)
$ curl localhost:9200/_cat/nodes?v
ip heap.% ram.% cpu load_1m load_5m load_15m node.role master name
172.17.0.4 36 58 42 1.47 0.59 0.46 mdi - es3
172.17.0.3 56 58 53 1.47 0.59 0.46 mdi - es2
172.17.0.2 42 58 30 1.47 0.59 0.46 mdi * es1
3_node_cluster
es2es1* es3

38. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

39. Kibana on Docker (1/2)
docker run docker.elastic.co/kibana/kibana:6.4.1
docker run --rm -p 5601:5601 -e "ELASTICSEARCH_URL=http://es1:9200"
docker.elastic.co/kibana/kibana:6.4.1
docker run --rm -p 5601:5601 -e "ELASTICSEARCH_URL=http://es1:9200"
-e "SERVER_NAME=my-kibana"
docker.elastic.co/kibana/kibana:6.4.1
...

40. Kibana on Docker (2/2)
FROM docker.elastic.co/kibana/kibana:6.4.1
ENV ELASTICSEARCH_URL="http://es1:9200"
COPY /conf/kibana.yml /usr/share/kibana/config/
EXPOSE 5601
HEALTHCHECK CMD curl --fail http://localhost:5601 || exit 1

41. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

42. Logstash on Docker (1/2)
docker run docker.elastic.co/logstash/logstash:6.4.1
docker run --rm -p 9600:9600 -e "PIPELINE_WORKERS=3"
docker.elastic.co/logstash/logstash:6.4.1
docker run --rm -p 9600:9600 -e "PIPELINE_WORKERS=3"
-e "LOG_LEVEL=DEBUG"
docker.elastic.co/logstash/logstash:6.4.1
...

43. Logstash on Docker (2/2)
FROM docker.elastic.co/logstash/logstash:6.4.1
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD pipeline/ /usr/share/logstash/pipeline/
ADD config/ /usr/share/logstash/config/
EXPOSE 9600

44. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

45. Tying it all together using docker-compose (1/2)
version: "3.3"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
volumes:
-
./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./jvm.options:/usr/share/elasticsearch/config/jvm.options
- ./data:/usr/share/elasticsearch/data
- ./snapshots:/tmp/es6_dev_repo
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx2g -Xms2g"
networks:
- elk
...
...
kibana:
image: docker.elastic.co/kibana/kibana:6.4.1
volumes:
- ./kibana/config/:/usr/share/kibana/config
- ./kibana/cache/:/usr/share/kibana/optimize
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
...

46. Tying it all together using docker-compose (2/2)
...
logstash:
image: docker.elastic.co/logstash/logstash:6.4.1
volumes:
- ./logstash/config/:/usr/share/logstash/config/
ports:
- "9600:9600"
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
driver: bridge
$ docker-compose up

47. What are we talking about?

48. Tying it all together https://www.elastic.co/blog/a-full-stack-in-one-command
https://github.com/elastic/examples/tree/master/Miscellaneous/docker/full_stack_example

49. Agenda
● Getting acquainted
● Ecosystem
● Features overview
● What are we talking about?
● Installation
● ES on Docker
● Kibana on Docker
● Logstash on Docker
● Tying it all together
● What’s next?

50. ● Docs: https://www.elastic.co/guide/index.html
● Videos: https://www.elastic.co/videos
● Slides: https://speakerdeck.com/elastic/
● Blog: https://www.elastic.co/blog
● Source: https://github.com/elastic
● Meetups: https://www.meetup.com/fr-FR/elastic-switzerland/
● Conference: https://www.elastic.co/elasticon/conf/2018/sf
● Discuss:
■ https://stackoverflow.com/questions/tagged/elasticsearch
■ https://discuss.elastic.co/
● Articles:
■ https://www.elastic.co/blog/a-full-stack-in-one-command
■ https://www.elastic.co/blog/docker-networking
■ https://sematext.com/blog/elasticsearch-in-docker/
■ https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
■ https://www.elastic.co/guide/en/kibana/current/docker.html
■ https://www.elastic.co/guide/en/logstash/current/docker.html
What’s next?

51. Q&A