HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat. Zuhaib covers cover of the issues the HipChat team has overcome using elasticsearch with AWS and CouchDB.

1. Wednesday, November 20, 13

2. Heavy Lifting
ZUHAIB SIDDIQUE
Wednesday, November 20, 13
•
@ZUHAIB
•
ENGINEER

3. Wednesday, November 20, 13

4. 1. Persistent chat rooms
Wednesday, November 20, 13

5. 2. File sharing & Chat history
Wednesday, November 20, 13

6. 3. @mentions + notifications
Wednesday, November 20, 13

7. 4. Integrations and API
Wednesday, November 20, 13

8. 5. Runs on everything
Wednesday, November 20, 13

9. 5. Runs on everything
Wednesday, November 20, 13

10. 5. Runs on everything
Wednesday, November 20, 13

11. Group chat and IM
for teams.
Wednesday, November 20, 13

12. 1,000,000,000
1,200,000,000
messages sent
Wednesday, November 20, 13

13. What is a HipChat Message?
{
}
Wednesday, November 20, 13
"id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"from": {
"nick": "Zuhaib Siddique",
"group_id": 1,
"user_id": 1
},
"room": {
"group_id": 1,
"name": "testing",
"id": 1
},
"stanza_type": "muc_message",
"date": "2013-11-14T23:20:13Z 424967",
"stanza_data": {
"body": "ElasticSearch FTW"
}

14. In the beginning there was …
Wednesday, November 20, 13

15. Then we added...
https://github.com/rnewson/couchdb-lucene
Wednesday, November 20, 13

16. CouchDB + CouchDB-Lucene
•Started to tip over at
200M Docs
•m2.4xlarge
Wednesday, November 20, 13

17. CouchDB + CouchDB-Lucene
m2.4xlarge
m2.4xlarge
Worked well for the first 500M
Wednesday, November 20, 13

18. m2.4xlarge
Wednesday, November 20, 13
hs1.8xlarge

19. But it still wasn’t big enough.
Wednesday, November 20, 13

20. Dont F*ck the Customer
Wednesday, November 20, 13

21. First Attempt at Scaling
Wednesday, November 20, 13

22. Wednesday, November 20, 13

23. Why ElasticSearch for HipChat
Wednesday, November 20, 13

24. Why ElasticSearch for HipChat
{
"query_string" : {
"default_field" : "content",
"query" : "this AND that OR thus"
}
}
Wednesday, November 20, 13

25. Why ElasticSearch for HipChat
Wednesday, November 20, 13

26. Search Error Rate
Wednesday, November 20, 13

27. Wednesday, November 20, 13

28. Don’t Forget about CouchDB
AWS Max
EBS size
Wednesday, November 20, 13

29. Stop the CouchDB bleeding
Wednesday, November 20, 13

30. CouchDB end of the road
Wednesday, November 20, 13

31. Wednesday, November 20, 13

32. (allthethings) ElasticSearch
ElasticSearch
{
"_index": "muc-2013.11",
"_type": "document",
"_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"_version": 1,
"exists": true,
"_source": {
"from": {
"nick": "Zuhaib Siddique",
"group_id": 1,
"user_id": 1
},
"room": {
"group_id": 1,
"name": "testing",
"id": 1
},
"stanza_type": "muc_message",
"date": "2013-11-14T23:20:13Z 424967",
"_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"stanza_data": {
"body": "ElasticSearch FTW"
}
}
}
Wednesday, November 20, 13
CouchDB
{
"id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"from": {
"nick": "Zuhaib Siddique",
"group_id": 1,
"user_id": 1
},
"room": {
"group_id": 1,
"name": "testing",
"id": 1
},
"stanza_type": "muc_message",
"date": "2013-11-14T23:20:13Z 424967",
"stanza_data": {
"body": "ElasticSearch FTW"
}
}

33. (allthethings) ElasticSearch
ElasticSearch
{
"_index": "muc-2013.11",
"_type": "document",
"_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"_version": 1,
"exists": true,
"_source": {
"from": {
"nick": "Zuhaib Siddique",
"group_id": 1,
"user_id": 1
},
"room": {
"group_id": 1,
"name": "testing",
"id": 1
},
"stanza_type": "muc_message",
"date": "2013-11-14T23:20:13Z 424967",
"_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"stanza_data": {
"body": "ElasticSearch FTW"
}
}
}
Wednesday, November 20, 13
CouchDB
{
"id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"from": {
"nick": "Zuhaib Siddique",
"group_id": 1,
"user_id": 1
},
"room": {
"group_id": 1,
"name": "testing",
"id": 1
},
"stanza_type": "muc_message",
"date": "2013-11-14T23:20:13Z 424967",
"stanza_data": {
"body": "ElasticSearch FTW"
}
}

34. CouchDB vs ElasticSearch
CouchDB Spike
to 800ms!
Avg 100ms with ES
Wednesday, November 20, 13

35. Running ElasticSearch
HipChat Customer
master
Wednesday, November 20, 13
clients
Thanks Philip O’Toole - Loggly

36. $ fab create_esindex:hostname=esindex-d4
Wednesday, November 20, 13

37. Behind the Scenes
• Spin up EC2 Instance
• Install Oracle Java 7
• Create Provision IOPs EBS
• Install and configure ElasticSearch
• ???
• Profit!
Wednesday, November 20, 13

38. # Elasticsearch
nodes = partial_search(:node, "role:elasticsearch_server AND
chef_environment:#{node.chef_environment}",
:keys => {
'hostname' => [ 'hostname' ]
})
if nodes.empty?
Chef::Log.error "site.json - No Elasticsearch servers found"
else
es_nodes = nodes.map {|n|
{
'host' => n['hostname'],
'port' => 9200
}
}
data["elasticsearch"] = {
"connections" => es_nodes.sort_by {|n| n['host'] }
}
end
Wednesday, November 20, 13

39. site.json
"elasticsearch": {
"connections": [
{
“host": "esindex-c1",
"port": 9200
},
...
{
"host": "esindex-e3",
"port": 9200
}
]
}
Wednesday, November 20, 13

40. You
Wednesday, November 20, 13

41. Running Elasticsearch
Wednesday, November 20, 13

42. Running ElasticSearch
Single Node OOM!
Wednesday, November 20, 13

43. Monitor, Monitor, Monitor!
Wednesday, November 20, 13

44. Backup
Wednesday, November 20, 13

45. 100,000,000,000?
To infinity and beyond!
Wednesday, November 20, 13

46. Questions?
Wednesday, November 20, 13