slides from the talk we gave in Copenhagen at cufp 2012.

1. Who are ’we’?
How and why we bootstrapped Functional Programming
Evaluation
Summary
Functional Programming @ Ghent IT Valley
R. Slootmaekers N. Trangez
Incubaid Research Lab
Commercial Users of Functional Programming, 2012
R. Slootmaekers, N. Trangez FP@GhentITV

2. Who are ’we’?
How and why we bootstrapped Functional Programming
Evaluation
Summary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional Programming
Amplidata’s unbreakable storage
our pain
2.0 plan and execution
3 Evaluation
Were we Happy?(2010)
Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV

3. Who are ’we’?
How and why we bootstrapped Functional Programming
Evaluation
Summary
Incubaid is a technology incubator, active in datacenter
and cloud computing
Prior exits through Terremark, Telenet (Belgian telco),
Veritas/Symantec, Sun Microsystems,. . .
some of our current involvements: Amplidata , Awingu ,
CloudFounders , Dacentec , Racktivity
R. Slootmaekers, N. Trangez FP@GhentITV

4. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional Programming
Amplidata’s unbreakable storage
our pain
2.0 plan and execution
3 Evaluation
Were we Happy?(2010)
Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV

5. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
Amplidata’s Dispersed Storage System
20 000ft view
diskA0
StorageNodeA diskA1
encoder
decoder
Client ........ diskA...
provisioning
policies
StorageNodeZ
metadata
client streams object data
encoder generates equations : Ax = b or x = A−1 b
policies determine how many disks | equations
provisioning determines which disks
send equations to StorageNodes
record metadata
R. Slootmaekers, N. Trangez FP@GhentITV

6. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional Programming
Amplidata’s unbreakable storage
our pain
2.0 plan and execution
3 Evaluation
Were we Happy?(2010)
Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV

7. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
Pain
Pain: resource management
Pain: sockets
Pain: threads
Pain: data corruption
Pain: objects
development was grinding to a halt
R. Slootmaekers, N. Trangez FP@GhentITV

8. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
Pain
Pain: resource management
Pain: sockets
Pain: threads
Pain: data corruption
Pain: objects
development was grinding to a halt
R. Slootmaekers, N. Trangez FP@GhentITV

9. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
The 1.6 Codebase (Core)
Time (Blue):
Pareto applies
1 60% is IO
30% is encoding/decoding
size (Red):
0 > 50 kloc
StIO MdIO Codec Rest Codec ∼ 1 kloc
IO < 1 kloc
R. Slootmaekers, N. Trangez FP@GhentITV

10. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
The Bet
I placed a bet to rewrite the StorageNode component in OCaml
Functionally identical
Same Performance
in 2 days. . .
1.5 day later the drop in replacement was demoed, and we
were allowed to Investigate alternatives to C++ for version 2.0
R. Slootmaekers, N. Trangez FP@GhentITV

11. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
The Bet
I placed a bet to rewrite the StorageNode component in OCaml
Functionally identical
Same Performance
in 2 days. . .
1.5 day later the drop in replacement was demoed, and we
were allowed to Investigate alternatives to C++ for version 2.0
R. Slootmaekers, N. Trangez FP@GhentITV

12. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
How is this possible?
component is very simple
perfect specs
existing component test suite
performance is mostly determined by network & disk speed
OCaml has object orientation.
R. Slootmaekers, N. Trangez FP@GhentITV

13. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional Programming
Amplidata’s unbreakable storage
our pain
2.0 plan and execution
3 Evaluation
Were we Happy?(2010)
Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV

14. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
2.0 Estimates & plan
// in IO ⇒ 30 → 6
recycle codec
do the rest in comfort ⇒ 10 → 20
maybe improve metadata IO (?)
⇒ (time) 100% → 86%
⇒ (size) 100% → 40%
R. Slootmaekers, N. Trangez FP@GhentITV

15. Who are ’we’?
Amplidata’s unbreakable storage
How and why we bootstrapped Functional Programming
our pain
Evaluation
2.0 plan and execution
Summary
What happened?
We decided on Functional programming & picked OCaml.
use Light Weight Threads library.
Blinded by success.
rewrote codec (in C)
metadata → Arakoon
feature creep
⇒ (time) 100% → 40%
⇒ (size) 100% → 100% (90% OCaml, 9% C, 1% Asm)
R. Slootmaekers, N. Trangez FP@GhentITV

16. Who are ’we’?
How and why we bootstrapped Functional Programming Were we Happy?(2010)
Evaluation Are we still happy?(2012)
Summary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional Programming
Amplidata’s unbreakable storage
our pain
2.0 plan and execution
3 Evaluation
Were we Happy?(2010)
Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV

17. Who are ’we’?
How and why we bootstrapped Functional Programming Were we Happy?(2010)
Evaluation Are we still happy?(2012)
Summary
Were we Happy? (2010)
No:
Yes: Tools? what tools?
Type Inference rocks 32/64 bit issues
Superfast compiler Multicore issues
Fast Enough Code OCaml(build) learning curve
Nice C integration Emacs only
Refactoring OO does not fit well
Architecture?
R. Slootmaekers, N. Trangez FP@GhentITV

18. Who are ’we’?
How and why we bootstrapped Functional Programming Were we Happy?(2010)
Evaluation Are we still happy?(2012)
Summary
Outline
1 Who are ’we’?
2 How and why we bootstrapped Functional Programming
Amplidata’s unbreakable storage
our pain
2.0 plan and execution
3 Evaluation
Were we Happy?(2010)
Are we still happy?(2012)
R. Slootmaekers, N. Trangez FP@GhentITV

19. Who are ’we’?
How and why we bootstrapped Functional Programming Were we Happy?(2010)
Evaluation Are we still happy?(2012)
Summary
Are we still happy?(2012)
No:
OCaml language evolution.
Yes:
OCaml runtime evolution.
FP strategy works.
OCaml library evolution.
people & process mistakes.
R. Slootmaekers, N. Trangez FP@GhentITV

20. Who are ’we’?
How and why we bootstrapped Functional Programming
Evaluation
Summary
Summary
Doing everything in 1 programming language is plain silly.
(But you need pain and some guerilla tactics to convince
people)
Functional programming and distributed systems match
R. Slootmaekers, N. Trangez FP@GhentITV