Slides from my PgConf NYC 2014 talk on integrating PostgreSQL and RabbitMQ.

1. Integrating PostgreSQL with RabbitMQ
Gavin M. Roy
April 4th, 2014

2. About Me
VP of Architecture
AWeber Communications
!
Blog: http://gavinroy.com
Twitter: @Crad
GitHub: https://github.com/gmr
!
RabbitMQ in Depth (MEAP)
http://manning.com/roy

3. RabbitMQ Primer

4. About RabbitMQ
• Message Oriented Middleware
• Primary identity is defined by AMQP 0-9-1
• Supports other protocols with Plugins:
AMQP 1.0, HTTP, MQTT, STOMP, XMPP, UDP,
WebStomp, and more
• Written in Erlang/OTP, is Open Source (MPL), and
is developed/maintained by Pivotal

5. Why RabbitMQ?
• Create loosely coupled architectures
• Decouple database write operations
• Communicate across application platforms
• Tap into pre-existing message flow for new purposes
• Scale-out clustering for growth, throughput & HA
• Federation for WAN latencies & network partitions

6. Native AMQP Clients
C
Clojure
Cobol
Common Lisp
Delphi
Erlang
Go
Groovy
Haskell
Java
JavaScript
.NET
OCaml
Perl
PHP
Python
Ruby
Scala

7. Who Uses It?
Agora Games
Chef
Google AdMob
Instagram
MeetMe
Mercado Libre
Mozilla
Nasa
New York Times
NSF
Openstack
Reddit

8. RabbitMQ Concepts

9. AMQP Message
Body
Properties

10. Message Properties
app-id
content-encoding
content-type
correlation-id
delivery-mode
expiration
headers
message-id
priority
reply-to
timestamp
type
user-id
Body
Properties

11. Publishers and Consumers
CP

12. Multiple Publishers
P
C
P
P

13. Multiple Consumers
P
C
C
C

14. AMQ Model
Exchange
defines routing behavior
Queue
stores messages in memory and
optionally on disk
Binding
defines relationship between
exchanges and queues
X
Queue
Binding

15. Routing Keys
• Provided when publishing a message
• Compared against binding keys by exchanges
• Example uses of a Routing Key:
• Connotate the type of message
• Designate the destination of a message
• Categorize the content in the message

16. Built-In Exchange Types
Direct
String matching on Routing Key
Fanout
No routing key, messages sent to all bound queues
Topic
Pattern matching in Routing Key
Headers
No routing key, string matching in the message headers
property

17. Topic Exchange Binding Keys
namespace.delimited.keys
#
Receive all messages
namespace.#
Receive all messages in the namespace
namespace.delimited.*
Receive all namespace.delimited messages
namespace.*.keys
Receive all namespace messages ending in keys

18. Example Exchange Plugins
Consistent Hashing
Distribute messages via
routing key hashed value
Pulse
Publish internal RabbitMQ
metrics via AMQP
Random
Distribute messages across
all bound queues randomly
Recent History
Stores last 20 messages to
any bound queue
Riak Storage
Store received messages in
Riak"
Script Exchange
Calls out to external scripts
for routing behavior

19. Exchange to Exchange Bindings
X
X
X
Queue
Queue
Queue

20. to RabbitMQ

21. pg_amqp
• PostgreSQL Extension
• User-defined functions to publish via AMQP
• Developed by OmnTI
• Invoke from user-defined function
• Available on PGXN

22. Using pg_amqp
• Has table of AMQP broker connections
• Publishes body only AMQP messages
• Transactional publishing via amqp.publish
• Non-transactional via amqp.autonomous_publish
• Needs some love for additional features

23. https://github.com/gmr/On-Rabbits-and-ElephantsExample:
SELECT amqp.publish(broker_id,
'name',
'routing-key',
'message');
Publishing a Message

24. PgSQL Listen Exchange
• Exchange that issues LISTEN and publishes received
notifications to bound queues
• When routing messages, bindings are checked for the
routing key to match the NOTIFY channel
• Unobtrusive to PostgreSQL environment, native
constructs for publishing messages
• Requires PostgreSQL 9.0 and greater
• Body only messages*

25. Sending a Notification
psql (9.3.2)
Type "help" for help.
!
postgres=# timing
Timing is on.
postgres=# NOTIFY channel_name,
‘Test notification';
NOTIFY
Time: 0.160 ms

26. Receiving a Notification
psql (9.3.2)
Type "help" for help.
!
postgres=# LISTEN channel_name;
!
(after NOTIFY issued)
!
Asynchronous notification "channel_name"
with payload "Test notification" received
from server process with PID 16749.

28. Example Notification Message

29. to PostgreSQL

30. No direct integrations yet

31. Chapter 14
Creating a pgsql-storage
RabbitMQ plugin

32. Other ways?
via Consumer apps in the language of your choice
via Apache Flume with RabbitMQ Source,
PostgreSQL Sink

33. Questions?
rabbitmq-pgsql-listen-exchange
https://github.com/aweber/rabbitmq-pgsql-listen-exchange
pg_ampq
http://pgxn.org/dist/pg_amqp/
RabbitMQ in Depth
44% off code pgcf (includes other books as well)
http://manning.com/roy/
Follow me on Twitter
@Crad