1. Taming Pythons with ZooKeeper
PyCon Finland 2012

2. @nailor

3. Content Squad

4. Day to day operations
• Consume boatloads of XML
• Build database out ofit
• Transcode audio
• Enrich data
• Build indexes
• Ship indexes

5. Shipping

6. Pre-defined order

7. The naïve solution
• Central orchestrator
• Assume all machines are running
• Run remote commands
• We need to know all the quirks
• Fail fast

8. Guess what? It breaks

9. ...and replacing isn't simple

10. New tool is needed
• No central orchestrator
• Assume some machines are always down
• Run locally
• Shift responsibility to system owners
• Fail gracefully

11. CAP Theorem

12. The CAP Theorem
1.Consistency
2.Availability
3.Partition tolerance

13. Pick two. Any two will do.

14. We went for CA

15. Why?
• Users are mostly contained in single DC
• Inside a single DC connections are quite robust
• Remember the order? We need consistency
• Availability is everything

16. How?

18. Apache ZooKeeper

19. What?
• A distributed tree-like data structure
• Simple primitives
• Automatic leader elections
• Guaranteed hard consistency
• Ephemeral nodes

20. Tree-like structure
/
/dir/ /dir2/
/subdir/

21. Simple primitives
●
Guaranteed atomic operations
●
Counters
●
Change notifications

22. Automatic leader election
●
Nodes know who is the most up to date
●
If no leader can be picked, ZooKeeper refuses to work

23. Guaranteed hard consistency
●
Every change is sent to every node!
●
Quorum for all operations is always required

24. Ephemeral nodes
●
Node is present only if the client is alive

25. Library: zkPython

26. The Good:
●
Thin
●
Comes with ZooKeeper
●
Maintained by the Apache ZooKeeper project

27. The bad:
●
Thin
●
C bindings only, no PyPy for you

28. The ugly:
●
No documentation :(

29. There are others: Kazoo

30. The Good:
●
Pure Python
●
Recipes implemented
●
Used by many (Quora, Mozilla, reddit, Zope)

31. The bad:
●
Not much recipes done
●
Not owned by the mainline

32. The ugly:
●
Own implementation of the protocol

33. Dos and Don'ts

34. Don't ship large chunks

35. Monitor the ZooKeeper

36. Don't write there all the time

37. Stay in one DC

38. Spotify & ZooKeeper

39. Summary time!

40. Concurrency == hard

41. Distributed consistency == hard

42. No partitions? Go ZooKeeper!

43. Pick your weapon library

44. Remember tradeoffs

45. Thank you