Presentation from my talk at WebWorkersCamp 2010, probably a bit hard to follow with the slides only.

1. Click & Tune
What’s new in our architectures ?
Olivier Gutknecht
olg@no-distance.net / twitter.com/olg

2. Me
Apple alumnus
Fotopedia co-founder & alumnus
Author, french ebook - introduction to REST
http://izibook.eyrolles.com/

3. Me
Apple alumnus
Fotopedia co-founder & alumnus
Author, french ebook - introduction to REST
http://izibook.eyrolles.com/

4. Click & Tune
?

6. START

8. Getting in tune

9. Getting in tune
We have a lot of sliders, faders and
buttons to tweak.
And we often ignore most of them.
Especially at architecture level.

10. What changed ?

11. What changed ?
Concepts | Context | Implementations

12. Concepts | Context | Implementations
“The Google File System” / Google 2003
“MapReduce: Simplified Data Processing on Large
Clusters” / Google 2004
“Life beyond Distributed Transactions: an
Apostate’s Opinion” / Pat Helland, Amazon 2007
[ZFS, Dynamo, PNUTS, CAP, REST,...] papers

13. Concepts | Context | Implementations
Client and apps evolution:
html5, native apps, COMET, web of [docs,apps], ...
Modeling problems:
tags, social graphs, walls, feeds, ...
“Like the big boys” - brought legitimacy

14. Concepts | Context | Implementations
Nutch - 2003
Memcached - 2003
Rails - 2004
CouchDB - 2005
S3 & EC2 - 2006
Redis, RabbitMQ,
MongoDB, Node.JS...

16. Sliders ?
# of backend servers
cache size
fsync / checkpointing
invalidation delay

17. interesting, but...

18. Sliders
Consistency level
Sync/Async mix
R/W actions balance
Model heterogeneity

19. Trade-offs

21. It’s the app, really.
And use cases.
And business cases.

22. Latency
“Disk is the new tape”
Cache ? invalidation & cold starts
...

23. Read
Reads are a memory & locality problem.
Caches. Partitions. Dupes. Smart shards.

24. Write
Write now. Work later. (maybe not on reads)
Divide and Conquer. And cheat.
Our frameworks are not ready yet.

25. Trade-offs

26. Acceptable Latency
Mandatory consistency (and transactions)
Synchronous actions
Maintainability
(Cost)

27. NoSQL is not about No SQL.
It’s about revisiting assumptions.
CAP ? ACID ? Choose your poisons.
(and enjoy your sliders, yay for Cassandra)

28. Consistency
Poison #1

29. Partition
Tolerance
Poison #2

30. Availability

31. Ditto for async.

32. The state of art is not
good enough.
(we’ve been stuck in the
same local minima for
several years)
Ilya Grigorik

33. Yay for node.js - and Grand Central - and
EM - and Goliath - and JRuby - and ...

34. Ditto for batches.

35. Hadoop. Pig. Hive. Sqoop. Flume. HBase.

36. So ?

37. Full Stack

40. HTTP everywhere ?
& AMQP ?
& Thrift ? ProtoBufs ? Avro ?

41. AUT
OMA

42. Because our stacks are
state of the art
(or bleeding edge)
Beyond Packages
Global config
Change
Do you really want to ssh to your servers ?

43. “But we have very
specific needs”

44. You’re 10% unique
Jesse Robbins, Opscode CEO

45. You’re 10% unique
(and it’s probably the
things you did wrong)
Jesse Robbins, Opscode CEO

46. The Cloud
What’s great about it ?
Scalability ?
On-demand capacity ?
Reduced CapEx ?

47. “Everything fails all the time”
Werner Vogels (Amazon CTO)

48. The Cloud
The best way to discover
new problems.
(and it’s great)

49. The Cloud
Think stateless. Think
automation. Think latency. Think
crappy machines. Think lost
servers. Thinks lost disks. Think
offline datacenters. Think ACL.
Think security. Think autoconf.

50. “MyPAAS”

51. Think
Holistic

52. Speed.
Availability.
Maintainability.

53. Know the
limits of your
tools

54. Episode IV: a New Hope
C h e f , P u p p e t , D r i z z l e , C a s s a n d ra ,
MongoDB, Redis, HTTP, Rails, Node.js,
EventMachine, Twisted, Heroku, SimpleDB,
EC2, GAE, REST, Rack, CouchDB, Redis,
Bitcask, Voldemort, Varnish, Pig, Hadoop,
HBase, Thrift, Protobufs, Avro, ESI, Mahout,
SolR, AMQP, Grand Central Dispatch,
WebSockets, LINQ, Reconnoiter, Nginx, ...

55. Thanks.
Creative Contents Images by
Big Red Button by thirtyfootscrew - http://www.flickr.com/photos/thirtyfootscrew/3324101122/
The Steam Pipeline by stuck in customs - http://www.flickr.com/photos/stuckincustoms/3780481716
Digidesign Icon Faders by CJ Sorg - http://www.flickr.com/photos/cjsorg/3692124539/
Pro Control 24 by aud1073ch - qhttp://www.flickr.com/photos/aud1073ch/34438156/
Pipe Dream by Rishi Menon - http://www.flickr.com/photos/rxmflickr/4102530508/