Modern Black Mages Fighting in the Real World

Modern Black Mages
Fighting in the Real World
Sep 9, 2016 in RubyKaigi 2016
@tagomoris
Satoshi "Moris" Tagomori

Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, ...
Treasure Data, Inc.

https://github.com/tagomoris/msgpack-inspect

http://docs.ﬂuentd.org/articles/logo
http://www.ﬂuentd.org/

Fluentd
• What is Fluentd?
• Open Source Log Collector
• Pluggable, Reliable, Less resource usage, Ease to use
• Versions of Fluentd
• v0.12: stable versions (2014/12 - Now)
• v0.14: versions for next stable (2016/05 - Now)

http://docs.ﬂuentd.org/articles/logo

What's about this session?
• Introduce some patterns of "Black Magic"s (a.k.a. meta
programming) in Ruby
• Show you some PRAGMATIC use of Black Magics

Fluentd v0.14 API Update
• Everything changed :)
• Plugin namespace
• before: Fluent::* (Top level classes even for plugins!)
• after: Fluent::Plugin::*
• Plugin base class for common methods
• Inconsistent Output plugin hierarchy
• Plugin must call `super` in common methods
http://www.slideshare.net/tagomoris/ﬂuentd-v014-plugin-api-details

Classes hierarchy (v0.12)
Fluent::Input F::Filter
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output F::Buffer
F::Parser
F::Formatter
3rd party plugins

Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::Buffer
F::P::Parser
F::P::Formatter
F::P::Storage
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin

diff v0.12 v0.14
F::P::Output
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output
Super classes by

how to buﬀer data
All output plugins

are just "Output"

Basic Black Magics:
Class and Mixin in Ruby

Class and Subclass in Ruby
class A
#bar
class B
#bar
super
B.new.bar

class A
#bar
class B
#bar
super
B.new.bar
module M
#bar
Introducing Methods by Mixin

class A
#bar
class B
#bar
super
B.new.bar
module M
#bar
Singleton Class of Ruby
#bar
B.new.singleton_class

class A
#bar
class B
#bar
super
b=B.new
b.singleton_class.include M2
b.bar
module M
#bar
Adding Methods on An Instance (1)
#bar
M2
#bar

class A
#bar
class B
#bar
super
b=B.new
b.extend M2
b.bar
module M
#bar
Adding Methods on An Instance (2)
#bar
M2
#bar

Fluentd v0.12 Fluent::Output
class Fluent::Output
#emit(tag, es, chain)
MyOutput
Engine calls plugin.emit(tag, es, chain)
@buﬀer

Fluentd v0.12 Fluent::BufferedOutput (1)
class Fluent::Outputclass BuﬀeredOutput
#emit(tag, es, chain, key)
MyOutput
super(tag, es, chain, any_key)
@buﬀer
#emit(key, data, chain)
#format(tag,time,record)
#format_stream(tag,es)

Fluentd v0.12 Fluent::BufferedOutput (2)
MyOutput
super(tag, es, chain, any_key)
@buﬀer

Fluentd v0.12 Fluent::TimeSlicedOutput
MyOutput
@buﬀer
#emit(tag, es, chain) class TimeSlicedOutput

Fluentd v0.12 Fluent::ObjectBufferedOutput
MyOutput
@buﬀer
#emit(tag, es, chain) class ObjectBuﬀeredOutput

Fluentd v0.12 Fluent::BufferedOutput
class Fluent::Outputclass BuﬀeredOutputMyOutput
@buffer calls #write in OutputThread
@buﬀer
chunk
#write(chunk)
OutputThread
#pop

Fluentd v0.12 Fluent::TimeSlicedOutput
@buﬀer
MyOutput
class TimeSlicedOutput
OutputThread
#write(chunk)
#write calls chunk.key
chunk
#pop

Fluentd v0.12 Fluent::ObjectBufferedOutput
@buﬀer
MyOutput
class ObjectBuﬀeredOutput
OutputThread
#write(chunk)
#write(chunk)
#write_object(chunk_key, chunk)
chunk
#pop

Fluentd v0.12 API Problems
• Entry point method is implemented by Plugin subclasses
• Fluentd core cannot add any processes
• counting input events
• hook arguments/return values to update API
• Fluentd core didn't show ﬁxed API
• Plugins have different call stacks
• It's not clear what should be implemented for authors
• It's not clear what interfaces are supported for
arguments/return values

How can we solve this problem?

Fluent::Plugin::Output (v0.14)

Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
#process(tag, es)
Engine calls plugin.emit_events(tag, es)
@buﬀer
#write
#emit_events(tag, es)
#format(tag, time, record)
#write(chunk)
#try_write(chunk)
#emit_sync(tag, es)
#emit_buffered(tag, es)

Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
Output calls plugin.write (or try_write)
@buﬀer
chunk
#write(chunk)
#try_write(chunk)
ﬂush thread
#process(tag, es)

Fluentd v0.14 Design Policy
• Separate entry points from implementations
• Methods in superclass control everything
• Do NOT override these methods!
• Methods in subclass do things only for themselves
• not for data ﬂow, control ﬂow nor others
• Plugins have simple/straightforward call stack
• Easy to understand/maintain

How about existing
v0.12 plugins?

Requirement:
(Almost)
All Existing Plugins SHOULD 
Work Well
WITHOUT ANY MODIFICATION

• Fluent::Compat namespace for compatibility layer
v0.14 Plugins & Compat Layer
F::P::Output
F::P::Base
v0.14 Plugins
Fluent::
Compat::
Output
F::C::
Buffered
Output
F::C::
TimeSliced
Output
F::C::
ObjectBuffered
Output
Fluent::Output
F::
Buffered
Output
F::
TimeSliced
Output
F::
ObjectBuffered
Output
v0.12 Plugins

Double Decker Compat Layer?
• Existing plugins inherits Fluent::Output or others
• No more codes in Fluent top level :-(
• Separate code into Fluent::Compat
• and import it into Fluent top level

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
v0.12 Plugins via Compat Layer: Best case (virtual)
Compat::BufferedOutput
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#write(chunk)
flush thread

@buﬀer
#write
v0.12 Plugins via Compat Layer: Best case (real)
#write(chunk)
ﬂush thread

@buﬀer
#write
When plugin overrides #format_stream
#write(chunk)
ﬂush thread

@buﬀer
#write
When plugin overrides #format_stream
#write(chunk)
ﬂush thread
default implementation
for calling "super"

@buﬀer
#write
When plugin overrides #emit
#emit(tag, es, chain) #emit(tag, es, chain, key)
#write(chunk)
ﬂush thread

@buﬀer
#write
#write(chunk)
ﬂush thread

@buﬀer
#write
#write(chunk)
ﬂush thread
This call doesn't happen, in fact
#emit doesn't return values!

@buﬀer
#write
#write(chunk)
ﬂush thread
#emit calls @buffer.emit
→ NoMethodError !

@buﬀer
#write
#write(chunk)
ﬂush thread
#emit
1. #emit calls @buffer.emit with data to be written in buffer
0. plugin calls @buffer.extend to add #emit
2. @buffer.emit stores arguments into plugin's attribute
3. get stored data
4. call @buffer.write with data

@buﬀer
#write
Thinking about "chunk" instance ...
#write(chunk)
ﬂush thread
#write may call "chunk.key",
but v0.14 chunk doesn't have #key !

@buﬀer
#write
#write(chunk)
ﬂush thread
"chunk" has #metadata, and values
of #key can be created via #metadata
Let's "chunk.extend" !
Where to do so?
?

Fluent::Plugin::OutputMyOutput
@buffer
#write
C::BufferedOutput
#write(chunk) flush thread
#write(chunk)
BufferedChunkMixin
plugin.extend BufferedChunkMixin
in #configure

Similar hacks for
TimeSlicedOutput and
ObjectBufferedOutput ...

Plugin Lifecycle Updated
Methods(v0.12)
• #configure
• #start
• #before_shutdown
• #shutdown
v0.12 Plugins often
doesn't call "super"!
Methods(v0.14)
• #configure
• #start
• #stop
• #before_shutdown
• #shutdown
• #after_shutdown
• #close
• #terminate
In v0.14, these methods MUST call "super"
• #configured?
• #started?
• #stopped?
• #before_shutdown?
• #shutdown?
• #after_shutdown?
• #closed?
• #terminated?

For Example: shutdown compat plugins
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
It doesn't call "super"! We want to call this...

What We Want To Do:
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. Fluentd core calls #shutdown
2. call #shutdown? to check "super" is called or not
3. call #shutdown of superclass forcedly!

What We Want To Do:
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
How to make this point?

One More Magic!
Module#prepend

class A
#bar
class B
#bar
super
B.new.bar
Wrapping Methods on a Class (1)
#bar

class A
#bar
class B
#bar
super
B.new.bar
module M
Wrapping Methods on a Class (2)
#bar
#bar
Using extend is powerful,
but it should be done for all instances
How about wrapping methods for
all instances of the class?

class A
#bar
class B
#bar
super
module M;def bar;super;end;end
B.prepend M
B.new.bar
module M
Wrapping Methods on a Class (3): Module#prepend
#bar
#bar
module M wraps B, and
M#bar is called at ﬁrst

What We Want To Do:
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
THIS ONE !!!

What We Got :-)
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. prepend CallSuperMixin at ﬁrst
2. call #shutdown? to check "super" is called or not
3. if not, get method of superclass, bind self with it, then call it
Thank you @unak -san!

Testing: Capturing return values by Test Driver
OutputMyOutput
@buﬀer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver
Create plugin instances
Feed test data into plugin

Testing: Capturing return values by Test Driver
OutputMyOutput
@buﬀer
#write
#emit_events
#format
#emit_buffered
We want to assert this return value!

Using #prepend to capture return values
OutputMyOutput
@buﬀer
#write
#emit_events
#format
#emit_buffered
module

M
#format
Store return value of "super"

Using #prepend ... doesn't work :-(
OutputMyOutput
@buﬀer
#write
#emit_events
#format
#emit_buffered
#format
Test code sometimes overwrites methods for many reasons :P
singleton

class
#format
😱
module

M

One Another Magic:
Stronger Than Anything

class A
#bar
class B
#bar
super
b=B.new
b.singleton_class.module_eval{define_method(:bar){"1"}}
b.bar
Another Study: How To Wrap Singleton Method?
#bar
module M
#bar

class A
#bar
class B
#bar
super
module P
Another Study: How To Wrap Singleton Method?
#bar
b=B.new
b.singleton_class.module_eval{define_method(:bar){"1"}}
b.singleton_class.prepend P
b.bar
#bar
module M
#bar
Singleton class is a class,
so it can be prepended :)
It's actually done in Test Driver implementation...

Using #prepend on singleton_class: Yay!
OutputMyOutput
@buﬀer
#write
#emit_events
#format
#emit_buffered
#format
singleton

class
#format😃
module

M
Prepending modules on singleton_class overrides everything!

IS BUILT ON A TOP OF
BUNCH OF BLACK MAGICS :P

Do Whatever You Can
For Users!
It Makes Everyone Happier!
... Except for Maintainers :(

Modern Black Mages Fighting in the Real World

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (11)

Similar a Modern Black Mages Fighting in the Real World

Similar a Modern Black Mages Fighting in the Real World (20)

Más de SATOSHI TAGOMORI

Más de SATOSHI TAGOMORI (19)

Último

Último (20)

Modern Black Mages Fighting in the Real World