SlideShare una empresa de Scribd logo
1 de 22
Descargar para leer sin conexión
Fluentd v11
Jan 23, 2014
Masahiro Nakagawa
Treasure Data, Inc

www.treasuredata.com/
Sunday, January 26, 14

1
Who are you?
•

Masahiro Nakagawa

•
•

Treasure Data, Inc.

•
•
•

@repeatedly

Senior Software Engineer
Fluentd, td-agent, etc...

Dlang, MessagePack, ...

2
Sunday, January 26, 14
Structured logging
http://fluentd.org/

Reliable forwarding
Pluggable architecture

Sunday, January 26, 14
MxN→M+N
Access logs

Alerting

Apache

Nagios

App logs

Analysis

Frontend

MongoDB

Backend

MySQL
Hadoop

System logs
syslogd
buffer / buffer / routing

Databases

Archiving
Amazon S3
4

Sunday, January 26, 14
v10
• Mainly for log forwarding
• with good performance
• working in many productions reliably
• Various plugins are released
• There are 200+ plugins!
• Mainly for CRuby
5
Sunday, January 26, 14
Why v11?
• We want more useful features, but...
• Changing the internal architecture is hard
• Keeping the external API is even harder
• Originally, v10 was started as a prototype
• Need drastic changes for the future :)
6
Sunday, January 26, 14
What’s new?
• ServerEngine based reliable architecture
• Zero-downtime restarting
• Filter / Label for flexible record handling
• Improved configuration
• Improved plugin development
7
Sunday, January 26, 14
ServerEngine based
• Robust signal handling
• Put a signal into Queue first
• Built-in supervisor
• Multiprocess support
• No need for in_multiprocess plugin
8
Sunday, January 26, 14
Multi Process
Supervisor

Worker

Worker

Worker

<worker>
input tail
output mongo
</worker>

<worker>
input forward
output webhdfs
</worker>

<worker>
input foo
output bar
</worker>

Separate stream pipelines in one instance!
9
Sunday, January 26, 14
Zero downtime restart
• SocketManager shares resources with
workers

Supervisor

1. Listen to TCP socket

TCP

10
Sunday, January 26, 14
Zero downtime restart
• SocketManager shares resources with
workers

Supervisor
TCP

1. Listen to TCP socket
2. Pass its socket to worker

heartbeat

Worker
TCP

11
Sunday, January 26, 14
Zero downtime restart
• SocketManager shares resources with
workers

Supervisor
TCP
heartbeat

Worker

1. Listen to TCP socket
2. Pass its socket to worker
3. Do same action
at worker restarting
with keeping TCP socket

Worker
TCP

12
Sunday, January 26, 14
Filter / Label support
• No more tag-related tricks!
• add_tag_xxx, remove_tag_xxx, etc...
• Redirect events to another group
• Much easier to group and share plugins
13
Sunday, January 26, 14
Filter
• <match> can have nested <match>
• Configuration format is not fixed!
v10:
<match access.**>
type flowcounter
add_tag_prefix counted
</match>
<match counted.**>
typo growthforecast
</match>

v11:
<match access.** copy>
type flowcounter
<match **>
typo growthforecast
</match>
</match>
14

Sunday, January 26, 14
Label
• <label> can contain multiple <match>
• out_redirect can forward events to <label>
<match access.**>
type rewrite_tag_filter
...
<match bang.**>
type redirect
to_label blackhole
</match>
...
</match>

<label blackhole>
<match **>
type null
</match>
</label>

bang’s record go away!
15

Sunday, January 26, 14
Improved configuration
• Can use Hash, Array and others
• No need for “,” or similar tricks
• You can write Ruby directly
• Worker pragma
• Separate plugins in each process
• Support the v10 compatible mode
16
Sunday, January 26, 14
New parameter types
• Can write complex values without DSL!
• Can use Ruby code for configuration
Hash, Array, etc:

Embedded Ruby code:

<source>
type my_tail
keys ["k1", "k2", "k3"]
</source>
<match **>
typo my_filter
add_keys {"k1" : "v1"}
</match>
Sunday, January 26, 14

<match ** copy>
type my_filter
env "#{ENV['KEY']}"
</match>

•
•
•

Socket.gethostname
`command`
etc...
17
Improved plugin
• Actor
• Abstraction layer for several actions
• Error stream
• with @ERROR label
• Log level per plugin
• No more global API, Engine.emit, $log, etc...
18
Sunday, January 26, 14
Actor
• Easy to write popular routines
v10:

v11:

class TimerWatcher <
Coolio::TimerWatcher
...
end
def start
@loop = Coolio::Loop.new
@timer = ...
@loop.attach(@timer)
@thread = ...
end
Sunday, January 26, 14

actor.every(@interval) {
event_router.emit(...)
}

19
Error stream
• Can handle an error at each record level
Input
{"event":1, ...}

chunk1

{"event":2, ...}
{"event":3, ...}

{"event":4, ...}

Output
OK
ERROR!
OK

OK

…

{"event":5, ...}
{"event":6, ...}

chunk2

ERROR!
OK

Sunday, January 26, 14

Error stream

<label @ERROR>
<match **>
type file
...
</match>
</label>
Built-in @ERROR is used
when error occurred in “emit”
20
Lastly...
• Super alpha status!
• Adding / Removing features may occur
• There are several “to be implemented” features
• The release date is not fixed yet
• The initial release should be useful from day one
• Need feedbacks!
21
Sunday, January 26, 14
We’re Hiring!
Jan 23, 2014

www.treasuredata.com/
Sunday, January 26, 14

22

Más contenido relacionado

La actualidad más candente

Mitmproxy usage v20141216
Mitmproxy usage v20141216Mitmproxy usage v20141216
Mitmproxy usage v20141216Jingchao Di
 
Deploying WO on Windows
Deploying WO on WindowsDeploying WO on Windows
Deploying WO on WindowsWO Community
 
Django rest framework tips and tricks
Django rest framework   tips and tricksDjango rest framework   tips and tricks
Django rest framework tips and tricksxordoquy
 
Open stack and_vagrant-os-meetup-2015
Open stack and_vagrant-os-meetup-2015Open stack and_vagrant-os-meetup-2015
Open stack and_vagrant-os-meetup-2015yfauser
 
The WordPress REST API as a Springboard for Website Greatness
The WordPress REST API as a Springboard for Website GreatnessThe WordPress REST API as a Springboard for Website Greatness
The WordPress REST API as a Springboard for Website GreatnessWP Engine UK
 
My Top 5 Favorite Gems
My Top 5 Favorite GemsMy Top 5 Favorite Gems
My Top 5 Favorite GemsJimmy Ngu
 
Sonata Block Bundle - sfPot March 2014
Sonata Block Bundle - sfPot March 2014Sonata Block Bundle - sfPot March 2014
Sonata Block Bundle - sfPot March 2014th0masr
 
Developing OpenResty Framework
Developing OpenResty FrameworkDeveloping OpenResty Framework
Developing OpenResty FrameworkOpenRestyCon
 
Ansible module development 101
Ansible module development 101Ansible module development 101
Ansible module development 101yfauser
 
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...Alberto Perdomo
 
CPAN Dependency Heaven
CPAN Dependency HeavenCPAN Dependency Heaven
CPAN Dependency HeavenOpusVL
 
React server side rendering performance
React server side rendering performanceReact server side rendering performance
React server side rendering performanceNick Dreckshage
 
Automation and Ansible
Automation and AnsibleAutomation and Ansible
Automation and Ansiblejtyr
 
Django rest framework
Django rest frameworkDjango rest framework
Django rest frameworkBlank Chen
 
Getting Started with Ansible
Getting Started with AnsibleGetting Started with Ansible
Getting Started with AnsibleAhmed AbouZaid
 
Put a Button on It: Removing Barriers to Going Fast
Put a Button on It: Removing Barriers to Going FastPut a Button on It: Removing Barriers to Going Fast
Put a Button on It: Removing Barriers to Going FastOSCON Byrum
 
Building RESTful APIs
Building RESTful APIsBuilding RESTful APIs
Building RESTful APIsSilota Inc.
 
High Performance Solution for PHP7
High Performance Solution for PHP7High Performance Solution for PHP7
High Performance Solution for PHP7Xinchen Hui
 

La actualidad más candente (20)

Mitmproxy usage v20141216
Mitmproxy usage v20141216Mitmproxy usage v20141216
Mitmproxy usage v20141216
 
Deploying WO on Windows
Deploying WO on WindowsDeploying WO on Windows
Deploying WO on Windows
 
Django rest framework tips and tricks
Django rest framework   tips and tricksDjango rest framework   tips and tricks
Django rest framework tips and tricks
 
Open stack and_vagrant-os-meetup-2015
Open stack and_vagrant-os-meetup-2015Open stack and_vagrant-os-meetup-2015
Open stack and_vagrant-os-meetup-2015
 
Mojolicious mvc
Mojolicious mvcMojolicious mvc
Mojolicious mvc
 
The WordPress REST API as a Springboard for Website Greatness
The WordPress REST API as a Springboard for Website GreatnessThe WordPress REST API as a Springboard for Website Greatness
The WordPress REST API as a Springboard for Website Greatness
 
My Top 5 Favorite Gems
My Top 5 Favorite GemsMy Top 5 Favorite Gems
My Top 5 Favorite Gems
 
Sonata Block Bundle - sfPot March 2014
Sonata Block Bundle - sfPot March 2014Sonata Block Bundle - sfPot March 2014
Sonata Block Bundle - sfPot March 2014
 
Developing OpenResty Framework
Developing OpenResty FrameworkDeveloping OpenResty Framework
Developing OpenResty Framework
 
Ansible module development 101
Ansible module development 101Ansible module development 101
Ansible module development 101
 
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
Strangers In The Night: Ruby, Rack y Sinatra - Herramientas potentes para con...
 
Ansible presentation
Ansible presentationAnsible presentation
Ansible presentation
 
CPAN Dependency Heaven
CPAN Dependency HeavenCPAN Dependency Heaven
CPAN Dependency Heaven
 
React server side rendering performance
React server side rendering performanceReact server side rendering performance
React server side rendering performance
 
Automation and Ansible
Automation and AnsibleAutomation and Ansible
Automation and Ansible
 
Django rest framework
Django rest frameworkDjango rest framework
Django rest framework
 
Getting Started with Ansible
Getting Started with AnsibleGetting Started with Ansible
Getting Started with Ansible
 
Put a Button on It: Removing Barriers to Going Fast
Put a Button on It: Removing Barriers to Going FastPut a Button on It: Removing Barriers to Going Fast
Put a Button on It: Removing Barriers to Going Fast
 
Building RESTful APIs
Building RESTful APIsBuilding RESTful APIs
Building RESTful APIs
 
High Performance Solution for PHP7
High Performance Solution for PHP7High Performance Solution for PHP7
High Performance Solution for PHP7
 

Similar a Fluentd v11 at tokuben

Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.Junichi Ishida
 
Presto conferencetokyo2019
Presto conferencetokyo2019Presto conferencetokyo2019
Presto conferencetokyo2019wyukawa
 
Fluentd at HKOScon
Fluentd at HKOSconFluentd at HKOScon
Fluentd at HKOSconN Masahiro
 
Drupal Course 2012 - Code Driven Development
Drupal Course 2012 - Code Driven DevelopmentDrupal Course 2012 - Code Driven Development
Drupal Course 2012 - Code Driven DevelopmentAttila Cs. Nagy
 
Ship It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemShip It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemYi-Ting Cheng
 
Magento 2 Workflows
Magento 2 WorkflowsMagento 2 Workflows
Magento 2 WorkflowsRyan Street
 
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20Michael Lihs
 
Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)Vincent Biret
 
Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)Vincent Biret
 
Tracing and profiling my sql (percona live europe 2019) draft_1
Tracing and profiling my sql (percona live europe 2019) draft_1Tracing and profiling my sql (percona live europe 2019) draft_1
Tracing and profiling my sql (percona live europe 2019) draft_1Valerii Kravchuk
 
Build automation best practices
Build automation best practicesBuild automation best practices
Build automation best practicesCode Mastery
 
SPS calgary 2017 introduction to azure functions microsoft flow
SPS calgary 2017 introduction to azure functions microsoft flowSPS calgary 2017 introduction to azure functions microsoft flow
SPS calgary 2017 introduction to azure functions microsoft flowVincent Biret
 
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkFluentd v1 and future at techtalk
Fluentd v1 and future at techtalkN Masahiro
 
CubeJS: eBay’s Node.js Adoption Journey
CubeJS: eBay’s Node.js Adoption JourneyCubeJS: eBay’s Node.js Adoption Journey
CubeJS: eBay’s Node.js Adoption JourneyPatrick Steele-Idem
 
Client Side Performance for Back End Developers - Cambridge .NET User Group -...
Client Side Performance for Back End Developers - Cambridge .NET User Group -...Client Side Performance for Back End Developers - Cambridge .NET User Group -...
Client Side Performance for Back End Developers - Cambridge .NET User Group -...Bart Read
 
Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65N Masahiro
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton Araf Karsh Hamid
 

Similar a Fluentd v11 at tokuben (20)

Fluentd v1 and Roadmap
Fluentd v1 and RoadmapFluentd v1 and Roadmap
Fluentd v1 and Roadmap
 
Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.
 
Presto conferencetokyo2019
Presto conferencetokyo2019Presto conferencetokyo2019
Presto conferencetokyo2019
 
Fluentd at HKOScon
Fluentd at HKOSconFluentd at HKOScon
Fluentd at HKOScon
 
Drupal Course 2012 - Code Driven Development
Drupal Course 2012 - Code Driven DevelopmentDrupal Course 2012 - Code Driven Development
Drupal Course 2012 - Code Driven Development
 
Ship It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemShip It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails Ecosystem
 
Magento 2 Workflows
Magento 2 WorkflowsMagento 2 Workflows
Magento 2 Workflows
 
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
 
PhpStorm for WordPress
PhpStorm for WordPressPhpStorm for WordPress
PhpStorm for WordPress
 
Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)Vincent biret azure functions and flow (toronto)
Vincent biret azure functions and flow (toronto)
 
Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)Vincent biret azure functions and flow (ottawa)
Vincent biret azure functions and flow (ottawa)
 
Tracing and profiling my sql (percona live europe 2019) draft_1
Tracing and profiling my sql (percona live europe 2019) draft_1Tracing and profiling my sql (percona live europe 2019) draft_1
Tracing and profiling my sql (percona live europe 2019) draft_1
 
Build automation best practices
Build automation best practicesBuild automation best practices
Build automation best practices
 
SPS calgary 2017 introduction to azure functions microsoft flow
SPS calgary 2017 introduction to azure functions microsoft flowSPS calgary 2017 introduction to azure functions microsoft flow
SPS calgary 2017 introduction to azure functions microsoft flow
 
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkFluentd v1 and future at techtalk
Fluentd v1 and future at techtalk
 
CubeJS: eBay’s Node.js Adoption Journey
CubeJS: eBay’s Node.js Adoption JourneyCubeJS: eBay’s Node.js Adoption Journey
CubeJS: eBay’s Node.js Adoption Journey
 
1. MySql plugins
1. MySql plugins1. MySql plugins
1. MySql plugins
 
Client Side Performance for Back End Developers - Cambridge .NET User Group -...
Client Side Performance for Back End Developers - Cambridge .NET User Group -...Client Side Performance for Back End Developers - Cambridge .NET User Group -...
Client Side Performance for Back End Developers - Cambridge .NET User Group -...
 
Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65
 
CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton CI-CD Jenkins, GitHub Actions, Tekton
CI-CD Jenkins, GitHub Actions, Tekton
 

Más de Treasure Data, Inc.

GDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for MarketersGDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for MarketersTreasure Data, Inc.
 
AR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and MarketAR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and MarketTreasure Data, Inc.
 
Introduction to Customer Data Platforms
Introduction to Customer Data PlatformsIntroduction to Customer Data Platforms
Introduction to Customer Data PlatformsTreasure Data, Inc.
 
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowHands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowTreasure Data, Inc.
 
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and AppsBrand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and AppsTreasure Data, Inc.
 
How to Power Your Customer Experience with Data
How to Power Your Customer Experience with DataHow to Power Your Customer Experience with Data
How to Power Your Customer Experience with DataTreasure Data, Inc.
 
Why Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without DataWhy Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without DataTreasure Data, Inc.
 
Connecting the Customer Data Dots
Connecting the Customer Data DotsConnecting the Customer Data Dots
Connecting the Customer Data DotsTreasure Data, Inc.
 
Harnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company SuccessHarnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company SuccessTreasure Data, Inc.
 
Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017Treasure Data, Inc.
 
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)Treasure Data, Inc.
 
Introduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of HivemallIntroduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of HivemallTreasure Data, Inc.
 
Scaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big DataScaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big DataTreasure Data, Inc.
 
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data:  Move your data from MySQL to Redshift with (not much more tha...Treasure Data:  Move your data from MySQL to Redshift with (not much more tha...
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...Treasure Data, Inc.
 
Treasure Data From MySQL to Redshift
Treasure Data  From MySQL to RedshiftTreasure Data  From MySQL to Redshift
Treasure Data From MySQL to RedshiftTreasure Data, Inc.
 
Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudTreasure Data, Inc.
 

Más de Treasure Data, Inc. (20)

GDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for MarketersGDPR: A Practical Guide for Marketers
GDPR: A Practical Guide for Marketers
 
AR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and MarketAR and VR by the Numbers: A Data First Approach to the Technology and Market
AR and VR by the Numbers: A Data First Approach to the Technology and Market
 
Introduction to Customer Data Platforms
Introduction to Customer Data PlatformsIntroduction to Customer Data Platforms
Introduction to Customer Data Platforms
 
Hands On: Javascript SDK
Hands On: Javascript SDKHands On: Javascript SDK
Hands On: Javascript SDK
 
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD WorkflowHands-On: Managing Slowly Changing Dimensions Using TD Workflow
Hands-On: Managing Slowly Changing Dimensions Using TD Workflow
 
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and AppsBrand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
Brand Analytics Management: Measuring CLV Across Platforms, Devices and Apps
 
How to Power Your Customer Experience with Data
How to Power Your Customer Experience with DataHow to Power Your Customer Experience with Data
How to Power Your Customer Experience with Data
 
Why Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without DataWhy Your VR Game is Virtually Useless Without Data
Why Your VR Game is Virtually Useless Without Data
 
Connecting the Customer Data Dots
Connecting the Customer Data DotsConnecting the Customer Data Dots
Connecting the Customer Data Dots
 
Harnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company SuccessHarnessing Data for Better Customer Experience and Company Success
Harnessing Data for Better Customer Experience and Company Success
 
Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017Packaging Ecosystems -Monki Gras 2017
Packaging Ecosystems -Monki Gras 2017
 
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
글로벌 사례로 보는 데이터로 돈 버는 법 - 트레저데이터 (Treasure Data)
 
Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14
 
Introduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of HivemallIntroduction to New features and Use cases of Hivemall
Introduction to New features and Use cases of Hivemall
 
Scalable Hadoop in the cloud
Scalable Hadoop in the cloudScalable Hadoop in the cloud
Scalable Hadoop in the cloud
 
Using Embulk at Treasure Data
Using Embulk at Treasure DataUsing Embulk at Treasure Data
Using Embulk at Treasure Data
 
Scaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big DataScaling to Infinity - Open Source meets Big Data
Scaling to Infinity - Open Source meets Big Data
 
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
Treasure Data:  Move your data from MySQL to Redshift with (not much more tha...Treasure Data:  Move your data from MySQL to Redshift with (not much more tha...
Treasure Data: Move your data from MySQL to Redshift with (not much more tha...
 
Treasure Data From MySQL to Redshift
Treasure Data  From MySQL to RedshiftTreasure Data  From MySQL to Redshift
Treasure Data From MySQL to Redshift
 
Unifying Events and Logs into the Cloud
Unifying Events and Logs into the CloudUnifying Events and Logs into the Cloud
Unifying Events and Logs into the Cloud
 

Último

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_planJamie (Taka) Wang
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?SANGHEE SHIN
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 

Último (20)

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_plan
 
Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?Do we need a new standard for visualizing the invisible?
Do we need a new standard for visualizing the invisible?
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 

Fluentd v11 at tokuben

  • 1. Fluentd v11 Jan 23, 2014 Masahiro Nakagawa Treasure Data, Inc www.treasuredata.com/ Sunday, January 26, 14 1
  • 2. Who are you? • Masahiro Nakagawa • • Treasure Data, Inc. • • • @repeatedly Senior Software Engineer Fluentd, td-agent, etc... Dlang, MessagePack, ... 2 Sunday, January 26, 14
  • 4. MxN→M+N Access logs Alerting Apache Nagios App logs Analysis Frontend MongoDB Backend MySQL Hadoop System logs syslogd buffer / buffer / routing Databases Archiving Amazon S3 4 Sunday, January 26, 14
  • 5. v10 • Mainly for log forwarding • with good performance • working in many productions reliably • Various plugins are released • There are 200+ plugins! • Mainly for CRuby 5 Sunday, January 26, 14
  • 6. Why v11? • We want more useful features, but... • Changing the internal architecture is hard • Keeping the external API is even harder • Originally, v10 was started as a prototype • Need drastic changes for the future :) 6 Sunday, January 26, 14
  • 7. What’s new? • ServerEngine based reliable architecture • Zero-downtime restarting • Filter / Label for flexible record handling • Improved configuration • Improved plugin development 7 Sunday, January 26, 14
  • 8. ServerEngine based • Robust signal handling • Put a signal into Queue first • Built-in supervisor • Multiprocess support • No need for in_multiprocess plugin 8 Sunday, January 26, 14
  • 9. Multi Process Supervisor Worker Worker Worker <worker> input tail output mongo </worker> <worker> input forward output webhdfs </worker> <worker> input foo output bar </worker> Separate stream pipelines in one instance! 9 Sunday, January 26, 14
  • 10. Zero downtime restart • SocketManager shares resources with workers Supervisor 1. Listen to TCP socket TCP 10 Sunday, January 26, 14
  • 11. Zero downtime restart • SocketManager shares resources with workers Supervisor TCP 1. Listen to TCP socket 2. Pass its socket to worker heartbeat Worker TCP 11 Sunday, January 26, 14
  • 12. Zero downtime restart • SocketManager shares resources with workers Supervisor TCP heartbeat Worker 1. Listen to TCP socket 2. Pass its socket to worker 3. Do same action at worker restarting with keeping TCP socket Worker TCP 12 Sunday, January 26, 14
  • 13. Filter / Label support • No more tag-related tricks! • add_tag_xxx, remove_tag_xxx, etc... • Redirect events to another group • Much easier to group and share plugins 13 Sunday, January 26, 14
  • 14. Filter • <match> can have nested <match> • Configuration format is not fixed! v10: <match access.**> type flowcounter add_tag_prefix counted </match> <match counted.**> typo growthforecast </match> v11: <match access.** copy> type flowcounter <match **> typo growthforecast </match> </match> 14 Sunday, January 26, 14
  • 15. Label • <label> can contain multiple <match> • out_redirect can forward events to <label> <match access.**> type rewrite_tag_filter ... <match bang.**> type redirect to_label blackhole </match> ... </match> <label blackhole> <match **> type null </match> </label> bang’s record go away! 15 Sunday, January 26, 14
  • 16. Improved configuration • Can use Hash, Array and others • No need for “,” or similar tricks • You can write Ruby directly • Worker pragma • Separate plugins in each process • Support the v10 compatible mode 16 Sunday, January 26, 14
  • 17. New parameter types • Can write complex values without DSL! • Can use Ruby code for configuration Hash, Array, etc: Embedded Ruby code: <source> type my_tail keys ["k1", "k2", "k3"] </source> <match **> typo my_filter add_keys {"k1" : "v1"} </match> Sunday, January 26, 14 <match ** copy> type my_filter env "#{ENV['KEY']}" </match> • • • Socket.gethostname `command` etc... 17
  • 18. Improved plugin • Actor • Abstraction layer for several actions • Error stream • with @ERROR label • Log level per plugin • No more global API, Engine.emit, $log, etc... 18 Sunday, January 26, 14
  • 19. Actor • Easy to write popular routines v10: v11: class TimerWatcher < Coolio::TimerWatcher ... end def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ... end Sunday, January 26, 14 actor.every(@interval) { event_router.emit(...) } 19
  • 20. Error stream • Can handle an error at each record level Input {"event":1, ...} chunk1 {"event":2, ...} {"event":3, ...} {"event":4, ...} Output OK ERROR! OK OK … {"event":5, ...} {"event":6, ...} chunk2 ERROR! OK Sunday, January 26, 14 Error stream <label @ERROR> <match **> type file ... </match> </label> Built-in @ERROR is used when error occurred in “emit” 20
  • 21. Lastly... • Super alpha status! • Adding / Removing features may occur • There are several “to be implemented” features • The release date is not fixed yet • The initial release should be useful from day one • Need feedbacks! 21 Sunday, January 26, 14
  • 22. We’re Hiring! Jan 23, 2014 www.treasuredata.com/ Sunday, January 26, 14 22