At #ChefConf 2012, Dhruv Bansal and Nathaniel Eliot delivered a talk and demo on Ironfan, our Chef orchestration layer. Spin up clusters when you need them, kill them when you don't, so you can spend your time, money, and engineering focus on finding insights, not getting your machines ready.

1. Ironfan: Build a Hadoop
Cluster in 20 Minutes
Dhruv Bansal Nathaniel Eliot
@dhruvbansal @temujin9
Saturday, May 19, 2012

2. Saturday, May 19, 2012

3. Outline
Saturday, May 19, 2012

4. Why Ironfan?
Outline
• Why we love Chef
• Why we don’t
• Why we wrote Ironfan
Saturday, May 19, 2012

5. Why Ironfan?
Outline
• Why we love Chef
• Why we don’t
• Why we wrote Ironfan
Ironfan in Action
• Build a Hadoop cluster
• Run a Hadoop job
• Put that cluster away
Saturday, May 19, 2012

6. Why Ironfan?
Outline
• Why we love Chef
• Why we don’t
• Why we wrote Ironfan
Ironfan in Action
• Build a Hadoop cluster
• Run a Hadoop job
• Put that cluster away
Ironfan roadmap
• Better environments
• Cloud agnosticism
• Continuous integration
• Stacks
Saturday, May 19, 2012

7. Why we love Chef
Infrastructure as code:
Version control Testable
Shareable Recapitulable
Saturday, May 19, 2012

8. Why we love Chef
Modular, composable units:
MySQL Nginx SOLR
My Application
Saturday, May 19, 2012

9. Why we love Chef
Saturday, May 19, 2012

10. Why we don’t love Chef
Anything is possible
Nothing is simple
And there’s much
repetition (not DRY)
Saturday, May 19, 2012

11. Why we don’t love Chef
Write Upload Compile Converge
Development Machine Managed Server
Hard Coded Dynamic
Too much is hard-coded at
development/upload time!
Saturday, May 19, 2012

12. Why we don’t love Chef
How do we make @server_ips dynamic?
Saturday, May 19, 2012

13. Enter Ironfan
We wrote Ironfan to
Saturday, May 19, 2012

14. Enter Ironfan
We wrote Ironfan to
• Simplify, unify, and standardize our usage of
the Chef toolset
Saturday, May 19, 2012

15. Enter Ironfan
We wrote Ironfan to
• Simplify, unify, and standardize our usage of
the Chef toolset
• Build further abstractions on top of Chef
Saturday, May 19, 2012

16. Enter Ironfan
We wrote Ironfan to
• Simplify, unify, and standardize our usage of
the Chef toolset
• Build further abstractions on top of Chef
• Give us superpowers that Chef doesn’t have
yet
Saturday, May 19, 2012

17. Enter Ironfan
We wrote Ironfan to
• Simplify, unify, and standardize our usage of
the Chef toolset
• Build further abstractions on top of Chef
• Give us superpowers that Chef doesn’t have
yet
http://github.com/infochimps-labs/ironfan
Saturday, May 19, 2012

18. What does Ironfan do?
Saturday, May 19, 2012

19. What does Ironfan do?
Chef
Saturday, May 19, 2012

20. What does Ironfan do?
Ironfan
Simple helpers in the
silverware cookbook abstract
common Chef patterns and
keep things DRY
Chef
Saturday, May 19, 2012

21. What does Ironfan do?
Saturday, May 19, 2012

22. What does Ironfan do?
Dynamic service discovery:
Saturday, May 19, 2012

23. What does Ironfan do?
Dynamic service discovery:
Saturday, May 19, 2012

24. What does Ironfan do?
Saturday, May 19, 2012

25. What does Ironfan do?
A simple DSL for
defining clusters
of machines
Saturday, May 19, 2012

26. DEMO TIME!
Saturday, May 19, 2012

27. Thanks for coming!
Learn more about what
Ironfan and Infochimps can sales@infochimps.com
do for you.
Come work at a fast-
growing, cutting-edge
careers@infochimps.com
technology company based
in Austin, TX.
http://github.com/infochimps-labs/ironfan
http://github.com/infochimps-labs/wukong
Saturday, May 19, 2012