Slides from a talk I gave at erlounge Wellington about a workflow system using RabbitMQ and CouchDB.

1. one system to rule
them all

3. the challenge

4. • write a backend that scales well
• handle various frontends (web, iPhone, ...)
• runs “somewhere in the cloud”
• easy to maintain and extend

5. the solution

6. RabbitMQ
+CouchDB
=Awesome

7. Part I

8. RabbitMQ
• AMQP based Message Queue
• lots of additional transports (XMPP, HTTP,
STOMP, ...)
• erlang based

9. frontend frontend frontend
daemon RabbitMQ
daemon daemon daemon

10. • every functionality gets a backed daemon
• every daemon has a queue associated
• communication via a JSON message that
gets passed around

11. the JSON message

13. {
"response": {
"expiry": 710,
"ts": "1234425918",
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "en",
"reply_to": "B312-E16003D4-BEA23D9AB789",
"platform": "iwmn"
}
}

14. works great for ...
• RPC style communication
• if the flow through the system is static
• fast transactions

15. gets fiddly when ...
• there are asynchronous callbacks from
outside
• message flow can change based on various
factors
• tasks can take days to complete

16. Part II

17. workflows

19. ... again ...

20. what we have ...
• various frontends
• backend daemons
• JSON messages

21. what we need
• a way to describe message flow
• a easy flexible way to write and update it

22. CouchDB

23. ... to the rescue

25. workflow
• really only a path definition through
backends
• some sort of condition management
• a backend definition

26. {
"_id": "f7a4408898e5...a05b4181045",
"_rev": "3-633060011",
"name": "billing_info",
"type": "workflow",
"user": "iwmn",
"path": [
"logger",
"billing",
"billing_iwmn_cc"
]
}

27. ... and the backend
definition

28. {
"_id": "33adf2e3efc3...f423929f057ac",
"_rev": "3-3507820969",
"conn": "rpc",
"type": "worker",
"name": "billing_iwmn_cc",
"location": "billing.cc",
"description": "the CC interface",
"command": "billing_verify_cc"
}

29. now we need a
workflow daemon

30. frontend frontend frontend
daemon RabbitMQ workflow
daemon daemon daemon

31. where are we now?
• we can call backends as before
• we can call workflows via the workflow
queue
• we can manage workflows in CouchDB

32. Part III

33. asynchronous callbacks

35. problem
• we start a workflow
• we call some external API
• the external API somehow notifies us
(Mail, Callback URL, ...) about progress
• we need to continue in the workflow

36. we need a persistent
message store

37. {
"_id": "22cd37afd06c82129adb665d3150f570",
"_rev": "3-2669449653",
"status": "done",
"last_update": 1252296819,
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"message": {
"response": {
"expiry": 710,
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "de",
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB",
"reply_to": "workflow",
"user": "iwmn",
"log": [
"logger",
"billing",
"billing.iwmn.info"
],
"platform": "iwmn"
}
},
"created": 1252296815,
"type": "bill"
}

38. one more problem
how do we correlate messages

39. ... with some CouchDB
awesome ...

40. • the backend daemon for the callback polls a
view in CouchDB
• we can search for any value in the request
we sent
• all we need is a view that searches for the
very value this external API uses as
correlation ID

41. external frontend frontend frontend
callback
daemon RabbitMQ workflow
daemon daemon daemon

42. me
Lenz Gschwendtner
CTO iWantMyName
@norbu09

43. credits
• RabbitMQ
• CouchDB
• Erlang and Perl

44. more credits
• http://www.flickr.com/photos/djpd
• http://www.flickr.com/photos/lilyapp
• http://www.flickr.com/photos/b-tal
• http://www.flickr.com/photos/mikebaird
• http://www.flickr.com/photos/eyetwist