Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Logstash family introduction
Similar a Logstash family introduction (20)
Último
Último (20)
Logstash family introduction
- 2. What is a log • Oxford Dic4onary – a thick piece of wood that is cut from or has fallen from a tree – (also logbook) an official record of events during a par4cular period of 4me, especially a journey on a ship • 4me + data
- 3. In theory, life cycle of log Record Transmit Analyze Store Delete
- 4. In design, life cycle of log Record Transmit Store Delete
- 5. In fact, life cycle of log Record Delete
- 6. Problems • Logging to a database or filesysytem • Logging has placed a load on the database and filesystem • Mul4ple log formats • No easy way to search logs • No easy method to gather sta4s4cs
- 7. Find the logs of 16 computers 6 months ago ?
- 8. Why use Logstash? • A lot choices! • But we want a free & high-integrality & easy to use solution • splunk (finding your faults, just like mom) • facebookarchive/scribe (2682 ★) • Graylog2(Server+WUI 1683 ★) • fluentd (2038 ★) • logstash (2689 ★)
- 9. logstash and other things hRps://www.youtube.com/watch?v=RuUFnog29M4
- 10. Logstash • Open Source, Apache Licence • WriRen in JRuby, Runs on JVM • Plugins easily wriRen in Ruby • Process mul4ple format ( input, output ) • Logstash Family! ( Elas4cSearch , Kibana)
- 12. Elas4cSearch • A response to the claim : “Search is hard” • Powerful indexing & search tool • search & index data available Rescully as JSON over HTTP
- 13. Kibana
- 14. All-‐in-‐one!
- 15. How logstash works? • logstash process events, not (only) loglines! • “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads • “Inputs generate events, filters modify them, outputs ship them elsewhere.” • -- [the life of an event in logstash]
- 16. In my thinking, Event Life Cycle Input filter output
- 17. In fact, Event Life Cycle event (Input -‐> output) event -‐-‐-‐-‐-‐-‐-‐-‐-‐ input filter output
- 18. Logstash is a wooden tube Input Input Input filter output codec filter filter output output
- 19. Logstash plugins Workflow • inputs – How events get into LogStash. • codecs – convert an incoming format into an internal representa4on • filters – processing ac4ons on events : modify events or drop events • outputs – How output events from LogStash
- 20. Logstash plugins
- 21. What is an event!? • A @4mestamp (ISO 8601 4mestamp) • A messsage field ( data ) • A @version • host ( the host of sender) • type( syslog, irc, etc)
- 22. Exersice: Hello Word! java -‐jar logstash-‐1.1.12-‐flatjar.jar agent -‐f hello.conf java -‐jar logstash.jar agent -‐f hello.conf
- 23. Input • tcp • udp • unix • file • syslog • redis • logstash-‐fowarder(former Lumberjack)
- 24. Codecs • plain • json • rubydebug • mul4line
- 25. Outputs • mongodb • elas4cSearch • email • file • jira
- 26. Exercise: Mul4ple input & output
- 27. logstash-‐forwader • ♫ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! ♫ • WriRen in Go • lumberjack is reserved for protocol • Resource Usage Concerns • Need an SSL CA to verify the server
- 28. lumberjack • Encryp4on & Authen4ca4on (TLS) • Compression ( reduce bandwidth) • Sequence & ack behavior like TCP • Low latency • Reliable Aplica4on-‐Level message transport
- 29. Forwarder Sample
- 30. Filters • date • grok • drop • geoIP • mutate • mul4line
- 32. filter config
- 33. powerful grok • Parse arbitrary text and structure it. • The syntax for a grok paRern is – %{SYNTAX:SEMANTIC} • 55.3.244.1 GET /index.html 15824 – %{IP:client} – %{WORD:method} – %{URIPATHPARAM:request} – %{NUMBER:bytes} • hRps://github.com/elas4csearch/logstash/blob/ v1.4.2/paRerns/grok-‐paRerns
- 34. grok sample
- 35. drop
- 36. mutate • Muta4ons on fields. – rename – remove – replace – join – split – upper – lower
- 37. mul4line • Codecs & filter
- 38. Reference • hRps://www.digitalocean.com/community/ tutorials/how-‐to-‐use-‐logstash-‐and-‐kibana-‐to-‐ centralize-‐and-‐visualize-‐logs-‐on-‐ubuntu-‐14-‐04 • hRp://www.vmdoh.com/blog/centralizing-‐ logs-‐lumberjack-‐logstash-‐and-‐elas4csearch • hRp://jpmens.net/2012/08/09/i-‐grok-‐how-‐to-‐ mutate-‐a-‐file-‐with-‐logstash/ • hRp://gleenders.blogspot.tw/2014/02/ logstash-‐glassfish.html