齋藤 智之 (LINE株式会社 開発3室) ログ収集プラットフォーム開発におけるElasticsearchの運用 LINE Developer Meetup in Tokyo #27 -Elastic-の発表資料です https://line.connpass.com/event/76226/

1. Elasticsearch( , tomoyuki.saito@linecorp.com)

2. l
l Site Reliability Engineer
l
l LINE
l
l Elasticsearch

3. • Project
• Elasticsearch Issues
Agenda

4. •
•
• Elasticsearch Clusters
Project

5. LINE

6. talk-server:
OBS:
Auth:
LINE

7. talk-server
9TB/day200 /day 40 /s

10. •
•
• Elasticsearch Clusters
Project

12. Front-end Kafka
l Apache Kafka:
l
l :

13. Sink
l Kafka consumer application
l
l Elasticsearch: JSON bulk indexing

14. l Protocol Buffers message
l Table( unit)
l message :
l Kafka topic:
l :
l Kafka
message RequestLog {
string method_name = 1;
string url = 2;
…}

15. l Custom logback appender (Logback: Java )
l protobuf message
l Kafka
l Kafka
l library
l logback
Logging SDK

16. l Kafka, Elasticsearch, Hadoop
l Sink
l Logging
l

17. •
•
• Elasticsearch Clusters
Project

18. Elasticsearch Clusters
100+3
160+ docs
18+TB

19. 3,000+
3,700+ docs
400+TB
17,000+

20. •
•
• Elasticsearch Clusters
Project

21. •
• Index template
• Shard sizing
• Indexing performance
Issues

22. •
• Index template
• Shard sizing
• Indexing performance
Issues

23. 1Gbps

24. 1Gbps 10Gbps

25. 1Gbps 10Gbps

26. Performance degradation
: Multiple Elasticsearch Clusters or a Monster Cluster?

27. -

28. •
• Index template
• Shard sizing
• Indexing performance
Issues

29. l "template" index
l
l number_of_shards
l number_of_replicas
l
l type
Index template
{ "template": "talk.request_log-*",
"settings": {
"number_of_shards": 32,
"number_of_replicas": 1,
},
"mappings": {
"_default_": {
"properties": {
"method_name": {
"type": "keyword",
},
"ip": {
"type": "ip",
},
…
…}

30. Template
{ "template": "talk.request_log-*",
"settings": {
"number_of_shards": 32,
"number_of_replicas": 1,
},
"mappings": {
"_default_": {
"properties": {
"method_name": {
"type": "keyword",
},
"ip": {
"type": "ip",
},
…
…}
message RequestLog {
string method_name = 1;
string ip = 2 [(EsMapping.type)="ip"];
…}
Protobuf
elasticsearch:
number_of_shards: 32
number_of_replication: 1

31. •
• Index template
• Shard sizing
• Indexing performance
Issues

32. l
l resource overhead
l
l Indexing
Shard sizing
How many shards should I have in my Elasticsearch cluster?
“How many shards should I have?” and “How large should my shards be?”

33. indexing
50GB/shard index rolling period
Shard sizing
: How many shards should I have in my Elasticsearch cluster?

34. à Rollover Index API size-base rolling?
Shard sizing -
: And the big one said "Rollover" — Managing Elasticsearch time-based indices efficiently

35. •
• Index template
• Shard sizing
• Indexing performance
Issues

36. 2.2x
speed up
25 K
docs/s
55 K
docs/s
ES Indexing

37. l Elasticsearch
l Version: 5.6.1
l 40 CPU cores, 256 GB memory, 8 TB HDD x 12ea JBOD, 10Gbps network
l disk write throughput: 200 MB/s
l
l auto-generated document id
l processors
l shard 1 à 2)

38. l document id
l Index API:
l document
l auto-generated id document id
l Index API:
l 1.3x speed up
auto-generated document id
: [5.6]Tune for indexing speed#Use auto-generated ids, [5.6.1]InternalEngine#planIndexingAsPrimary()

39. l _nodes/os
l os.available_processors: 40
l os.allocated_processors: 32
l processor
l elastic/elasticsearch#20874
l processors: 38 bulk request
l 1.3x speed up
processors
: [5.6]Thread Pool#Processors setting

40. indexing throttling
Disk
segment merge
in-flight merge
indexing throttling

41. l disk
l 1.4x speed up
l Elasticsearch
l CPU usage: 50% ~ 80%
l Disk write: 125 MB/s x 2
shard

42. l 2.2x speed up
l auto document id
l processors
l CPU
l
l document mapping parameter
l disk (SSD? RAID?)
Indexing performance

43. •
• Index template
• Shard sizing
• Indexing performance
Issues

44. LINE Elasticsearch
Petabyte

45. THANK YOU