5. v10
• Mainly for log forwarding
• with good performance
• working in many productions reliably
• Various plugins are released
• There are 200+ plugins!
• Mainly for CRuby
5
Sunday, January 26, 14
6. Why v11?
• We want more useful features, but...
• Changing the internal architecture is hard
• Keeping the external API is even harder
• Originally, v10 was started as a prototype
• Need drastic changes for the future :)
6
Sunday, January 26, 14
7. What’s new?
• ServerEngine based reliable architecture
• Zero-downtime restarting
• Filter / Label for flexible record handling
• Improved configuration
• Improved plugin development
7
Sunday, January 26, 14
8. ServerEngine based
• Robust signal handling
• Put a signal into Queue first
• Built-in supervisor
• Multiprocess support
• No need for in_multiprocess plugin
8
Sunday, January 26, 14
10. Zero downtime restart
• SocketManager shares resources with
workers
Supervisor
1. Listen to TCP socket
TCP
10
Sunday, January 26, 14
11. Zero downtime restart
• SocketManager shares resources with
workers
Supervisor
TCP
1. Listen to TCP socket
2. Pass its socket to worker
heartbeat
Worker
TCP
11
Sunday, January 26, 14
12. Zero downtime restart
• SocketManager shares resources with
workers
Supervisor
TCP
heartbeat
Worker
1. Listen to TCP socket
2. Pass its socket to worker
3. Do same action
at worker restarting
with keeping TCP socket
Worker
TCP
12
Sunday, January 26, 14
13. Filter / Label support
• No more tag-related tricks!
• add_tag_xxx, remove_tag_xxx, etc...
• Redirect events to another group
• Much easier to group and share plugins
13
Sunday, January 26, 14
14. Filter
• <match> can have nested <match>
• Configuration format is not fixed!
v10:
<match access.**>
type flowcounter
add_tag_prefix counted
</match>
<match counted.**>
typo growthforecast
</match>
v11:
<match access.** copy>
type flowcounter
<match **>
typo growthforecast
</match>
</match>
14
Sunday, January 26, 14
15. Label
• <label> can contain multiple <match>
• out_redirect can forward events to <label>
<match access.**>
type rewrite_tag_filter
...
<match bang.**>
type redirect
to_label blackhole
</match>
...
</match>
<label blackhole>
<match **>
type null
</match>
</label>
bang’s record go away!
15
Sunday, January 26, 14
16. Improved configuration
• Can use Hash, Array and others
• No need for “,” or similar tricks
• You can write Ruby directly
• Worker pragma
• Separate plugins in each process
• Support the v10 compatible mode
16
Sunday, January 26, 14
17. New parameter types
• Can write complex values without DSL!
• Can use Ruby code for configuration
Hash, Array, etc:
Embedded Ruby code:
<source>
type my_tail
keys ["k1", "k2", "k3"]
</source>
<match **>
typo my_filter
add_keys {"k1" : "v1"}
</match>
Sunday, January 26, 14
<match ** copy>
type my_filter
env "#{ENV['KEY']}"
</match>
•
•
•
Socket.gethostname
`command`
etc...
17
18. Improved plugin
• Actor
• Abstraction layer for several actions
• Error stream
• with @ERROR label
• Log level per plugin
• No more global API, Engine.emit, $log, etc...
18
Sunday, January 26, 14
19. Actor
• Easy to write popular routines
v10:
v11:
class TimerWatcher <
Coolio::TimerWatcher
...
end
def start
@loop = Coolio::Loop.new
@timer = ...
@loop.attach(@timer)
@thread = ...
end
Sunday, January 26, 14
actor.every(@interval) {
event_router.emit(...)
}
19
20. Error stream
• Can handle an error at each record level
Input
{"event":1, ...}
chunk1
{"event":2, ...}
{"event":3, ...}
{"event":4, ...}
Output
OK
ERROR!
OK
OK
…
{"event":5, ...}
{"event":6, ...}
chunk2
ERROR!
OK
Sunday, January 26, 14
Error stream
<label @ERROR>
<match **>
type file
...
</match>
</label>
Built-in @ERROR is used
when error occurred in “emit”
20
21. Lastly...
• Super alpha status!
• Adding / Removing features may occur
• There are several “to be implemented” features
• The release date is not fixed yet
• The initial release should be useful from day one
• Need feedbacks!
21
Sunday, January 26, 14