Description of how Sematext SPM Performance Monitoring service is built and how it works. Originally presented at Berlin Buzzwords 2012.

1. SPM: Large Scale Performance Monitoring
for
ElasticSearch HBase Solr & Friends
#spmbuzz #bbuzz
Otis Gospodnetić – Sematext International
@otisg ◦ @sematext ◦ sematext.com

2. Agenda
Introductions
SPM Architecture
SPM Client Side
SPM Receiver & Flume
SPM & HBase
SPM Front End
Lessons
Interesting Metrics
2
Copyright 2012 Sematext Int'l. All rights reserved.

3. About Otis Gospodnetić
ASF Member: Lucene, Solr, Nutch, Mahout
Author: Lucene in Action 1 & 2
Entrepreneur: Sematext, Simpy
3
Copyright 2012 Sematext Int'l. All rights reserved.

4. About Sematext
Products & Services
Consulting, Development, Tech Support:
Search; Lucene, Solr, ElasticSearch...
Big Data Analytics; Hadoop, HBase, Voldemort...
Web Crawling; Nutch, Droids
Machine Learning; Mahout
4
Copyright 2012 Sematext Int'l. All rights reserved.

5. Sematext Metrics
100% organic: keine GMO / VC
5 years old
< 10 people
7 countries
3 timezones
3 continents
> 100 customers
5
Copyright 2012 Sematext Int'l. All rights reserved.

6. Scalable Performance Monitoring
Multi-tenant APM SaaS
With: HBase, Hadoop, Flume, Jetty, Esper...
For: ElasticSearch, HBase, Solr, Sensei...
Amazon Cloud
apps.sematext.com
ganz frei
6
Copyright 2012 Sematext Int'l. All rights reserved.

7. SPM Functionality
Performance graphs for any time period
No loss of metric precision/granularity
Slice & dice metrics with filters
Compare two time periods
Integrated alerts
Integrated email reports
Monitor multiple (types of) systems
Metric arithmetic (ganz soon!)
7
Copyright 2012 Sematext Int'l. All rights reserved.

8. Bird's Eye View
8
Copyright 2012 Sematext Int'l. All rights reserved.

9. SPM: Client Side
In-JVM Agent – app stats collector
Collectd – system stats collector
Out-of-JVM Sender – collector & sender
JMX, bytecode instrumentation, Flume,
HttpClient, local buffering
9
Copyright 2012 Sematext Int'l. All rights reserved.

10. SPM: Client Side
apt-get install collectd
run SPM client installer
-Dcom.sun.management.jmxremote
-javaagent:/spm/spm-monitor/lib/spm-
monitor-es-1.5.0-withdeps.jar=/spm/spm-
monitor/conf/spm-monitor-config-token-here-
default.xml
/etc/init.d/spm-sender start
10
Copyright 2012 Sematext Int'l. All rights reserved.

11. SPM: Receiver & Flume
Apache + Jetty + KeepAlive
Flume tail + HBase sink
In-memory metric pre-aggregation - HBaseHUT
Flume tail + Esper sink = Performance Alerts
11
Copyright 2012 Sematext Int'l. All rights reserved.

12. SPM: HBase
Raw metrics table
Aggregated metrics table
Aggregations via MapReduce
Aggregations on the fly - HBaseHUT
12
Copyright 2012 Sematext Int'l. All rights reserved.

13. SPM: HBase Aggregations
Generic aggregations framework
Large dataset report building (HDFS/HBase)
13
Copyright 2012 Sematext Int'l. All rights reserved.

14. SPM: Front End
JavaScript UI, jQuery, Flot
Flot → HighCharts → d3
Filters:
Time, Host, JVM
Core, Request Handler (Solr)
Node, Index, and Shard (ElasticSearch)
Network interface
Disk
...
14
Copyright 2012 Sematext Int'l. All rights reserved.

15. ElasticSearch Indexing Perf.
15
Copyright 2012 Sematext Int'l. All rights reserved.

16. SPM Numbers
>1000 data points/second
6 HBase nodes
BB raw rows & MM aggregate rows
16
Copyright 2012 Sematext Int'l. All rights reserved.

17. Lessons
EC2
Linux
KeepAlives
HBase
17
Copyright 2012 Sematext Int'l. All rights reserved.

18. Lessons: EC2
Agreement?
Benchmarking impossible
Spot instance prices spike
Are you feeling lucky?
18
Copyright 2012 Sematext Int'l. All rights reserved.

19. Lessons: swapiness
19
Copyright 2012 Sematext Int'l. All rights reserved.

20. Lessons: nofile
> ulimit -n
Default – often way too low
1024
> sudo vim /etc/security/limits.conf
Bump it up!
spmsender - nofile 32000
> sudo vim /etc/pam.d/su
session required pam_limits.so
20
Copyright 2012 Sematext Int'l. All rights reserved.

21. Lessons: KeepAlives
KeepAlives matter at high request rates
21
Copyright 2012 Sematext Int'l. All rights reserved.

22. Lessons: HBase Keys
Bad keys can hotspot RegionServers
Solution: HBaseWD
https://github.com/sematext/HBaseWD
22
Copyright 2012 Sematext Int'l. All rights reserved.

23. Lessons: Append-only Updates
Read-Modify-Write can be slow
Solution: HBaseHUT
http://github.com/sematext/HBaseHUT
23
Copyright 2012 Sematext Int'l. All rights reserved.

24. Lessons: HTable vs. asynchbase
HTable is blocking and serial
Solution:
https://github.com/stumbleupon/asynchbase
See:
https://issues.apache.org/jira/browse/HBASE-5539
http://tsunanet.net/~tsuna/asynchbase/benchmark/viz.html
24
Copyright 2012 Sematext Int'l. All rights reserved.

25. Poll: Search Monitoring
25
Copyright 2012 Sematext Int'l. All rights reserved.

26. Interesting Metrics
Solr
ElasticSearch
HBase
26
Copyright 2012 Sematext Int'l. All rights reserved.

27. Buzzworthiness
Big Data Large Scale
Search Multi-tenant
Analytics SaaS
Real-time Cloud
27
Copyright 2012 Sematext Int'l. All rights reserved.

28. We're Hiring
Dig Search?
Dig Analytics?
Dig Big Data?
Dig Performance?
Dig working with and in open-source?
We're hiring world-wide!
http://sematext.com/about/jobs.html
28
Copyright 2012 Sematext Int'l. All rights reserved.

29. Contact
sematext.com
blog.sematext.com
github.com/sematext
@sematext
@otisg
otis@sematext.com
Stop by our booth
29
Copyright 2012 Sematext Int'l. All rights reserved.