1. Igor Motov
igor@motovs.org
twitter: @imotov
github: imotov

3. Sonian Inc.
•Cloud-based email archiving
•Founded in 2007
•Headquarters: Newton, MA

4. Small team of about15
developers distributed
from Campinas, Brazil to
Vancouver, Canada

5. Using elasticsearch since
June 2010, v0.8.0

6. We have about
6 billion
records indexed in elasticsearch

7. 100,000
Netflix DVD Titles

8. 3,000,000
Pages in en.wikipedia.org

9. 22,000,000
Books in Library of Congress catalog

10. 150,000,000
Linked-in profiles

11. 3,000,000,000
Estimated bing.com index size

12. 6,000,000,000
Sonian Inc. index size

13. 50,000,000,000
Estimated google.com
index size

14. Infrastructure

15. http://www.sonian.com/awssonian-technical-diagram/

16. Ingestion (safe): Clojure
Search Engine: elasticsearch
Web App: Ruby on Rail
Deployment: Chef
Monitoring: Sensu

17. 10 clusters
6 AWS Regions
2-17 nodes in each cluster

18. Custom version of
elasticsearch
based on 0.19.9
with several plugins

19. jetty plugin
• jetty-based http transport
• SSL support
• Authentication
• Request logging (json, plain)

20. Request logs are also indexed
in elasticsearch

21. Open source
https://github.com/sonian/elasti
csearch-jetty

22. Zookeeper plugin
Zookeeper-based discovery
Replacement for zen discovery
Experimental!

23. Open source
https://github.com/sonian/elasti
csearch-zookeeper

24. Valve plugin
•Custom jetty plugin filter
•Rejects bulk indexing requests
if cluster is overloaded

25. Lessons learned in
the last two years
or

26. Proper Care and
Feeding of
Elasticsearch Nodes

27. Rule1: Give nodes plenty of
space
Running out of disk space or
memory is the simplest way to
corrupt your index.

28. Make sure elasticsearch
doesn’t swap
It reduces performance and
causes nodes to leave clusters

29. elasticsearch.yml
bootstrap.mlockall: true

30. Increase the number of open
file descriptors to 64k.

31. Rule 2: Distributed but well
connected
All nodes should be able to talk
to each other all the time

32. Otherwise your cluster might
get split-brain syndrome

33. Consider setting
discovery.zen.minimum_master_nodes

34. Rule 3: Throttle the bulk
indexing load
Asynchronous architecture
makes es scalable and fast, but
susceptible to running out of
memory under excessive bulk
indexing load.

35. Rule 4: Try to make all shards
approximately the same size
Elasticsearch allocates shards
based on the number of
shards. It doesn’t consider
shard sizes or available disk
space.

36. 4 rules for happy elasticsearch
1. Give nodes plenty of space
2. Distributed but well
connected
3. Throttle the load
4. Make all shards the same
size

37. Questions?

38. More Information
Latest stable release: 0.19.10
Web Site: http://www.elasticsearch.org/
Follow @elasticsearch on twitter
IRC: #elasticsearch on irc.freenode.net
GitHub: https://github.com/elasticsearch/elasticsearch
Mailing list: elasticsearch on http://groups.google.com/
Stackoverflow tag: elasticsearch