08448380779 Call Girls In Friends Colony Women Seeking Men
Real time system_performance_mon
1. Real time system
performance monitoring
AMQP and
CatalystX::JobServer
Tomas (t0m) Doran
São Paulo.pm perl workshop 2010
2. Aka the “my code
doesn’t work yet” talk
• I wrote half the slides for this at 6am Friday
(São Paulo time)
• The title is a bit misleading, as I wrote the
talk abstract in March, then wrote entirely
different software.
• Sorry about that.
3. Message queueing
• Lets not talk about the (not working) code
just yet...
• This talk is about Message Queueing.
• What is Message Queueing?
• Why would I want to use it?
4. What is message
queueing.
• In it’s simplest form, it’s just a list.
• 1 (or more) ‘producers’ (writers)
• 1 (or more) ‘consumers’ (readers)
• Queue if rate of production > rate of
consumption
5. Why do I want it?
• Decouples producers and consumers
(probably across the network).
• Lets you manage load, and spikes in load
(only n consumers).
• In a web environment - lets you serve
more pages, quicker.
6. Other cases
• View as a list a little simplistic.
• Depending on your application, you may
want:
• One to one
• One to many
• One to many (broadcast)
• Many to all of the above
• More complex topology?
8. AMQP
AMQP enables complete
interoperability for messaging
middleware, both the networking
protocol and the semantics of broker
services are defined in AMQP.
10. AMQP
• Full abstraction of all the mechanics of
messaging across the network.
• Serialization and framing of messages.
• Wiring of message routing is part of the
protocol.
11. AMQP
• So all your clients know (at least half) of
the wiring.
• Different topologies depending on routing
configuration.
• Can specify other options such as durability
• Nice when your server dies - no ‘current
config’
13. Concepts - Exchanges
• Named
• Messages are published (sent) to one
exchange
• Can be durable (lasts till deleted), or auto-
deleted
14. Queues
• Queues can be named or (or named by the
server - private queues).
• Queues are bound to one (or more)
exchanges.
• Queues can have 0 or more clients
• Queues may persist with 0 clients or not.
• FIFO (if you have 1 consumer)
• Message never delivered to > 1 client
16. Traditional queue
• Named exchange
• Bound to 1 Named queue
• 0 or more listeners get round-robin
messages
• Messages queue when nobody listens / if
consumers are slow
17. Broadcast
• Named exchange
• Each client creates an anonymous
ephermeral queue
• Client binds the queue to that exchange
• All clients get all messages
• Messages go to /dev/null if no clients
18. Others
• Different exchange types - direct, topic,
fanout
• You can do a lot using just these and a mix
of named and anonymous queues
• More complex topologies possible
19. I accidentally wrote
software
• I try to avoid doing this
• I’m not very good at it ;)
20. I blame Net::RabbitFoot
• It’s written using AnyEvent and Coro
• Which I hadn’t used before, but looked
good for this type of thing.
• I felt my way around writing simple things
standalone.
• Got all the web servers logging into Rabbit
21. Logging into message
queuing
• Good example of broadcast
• Want to aggregate logs to files
• And be able to ‘tail’ them
• Logging directly from the application
• Also tailing (normal) log files to message
queue
25. Dumping them to a file
• That’s pretty simple after that..
• Except:
• Log rotation
• Not flushing to disk once per message
• etc...
26. Viewing them live
• Someone wrote an AMQP client in flash
• AMQP security model not useful publicly
• Cute prototype
• (Sorry, no demo - refuses to work locally)
27. Queueing Jobs
• Next application, simple job queue.
• 1 worker, serialized.. (Crap, good enough)
• Log JSON into RabbitMQ (again), inflate,
call the ->run method.
• Works!
28. Job Server
• I now have several scripts, all doing bits
with queuing. All duplicating code.
• I want to aggregate messages
• Send messages of aggregates at regular
intervals
• Need something more generic
29. Job server
• Wrote a simple abstraction on getting
channels, making exchanges and queues, etc
• Was still going to end up with a load of
scripts/jobs that needed running and
managing.Yuk.
• Inspecting the state of each job ‘service’
hard / boring..
30. But wait
• Each ‘thing’ (timer, listener, logger, emitter,
worker etc) can be an instance of a class
with attributes for the state.
• Construct a set from some config file.
• Lets me aggregate jobs together / share
code (Can use AnyEvent/Coro for threads).
• Less processes to manage. Working out the
state just got harder still.
31. But wait
• Make all the classes use MooseX::Storage
• Catalyst makes instances of things from
config...
• So, I could adapt all these instances into
Catalyst models, write a controller, done!
• Erm, but Catalyst, and message queueing,
and what?!?!?
32. I blame Miyagawa
• For Plack, and Catalyst::Engine::Plack, and
Corona (the Coro PSGI server).
• I had this INSANE idea one night.
• I tried it.
• IT WORKED.
• WOAH.
33. I blame clkao
• Next step - browser kicks off a job.
• Status via long poll?
• He did it already!
• With long hair!
34. Hippie
• Async pipe to the browser.
• Abstracts all the nasty ajax details (also
does long poll, or websockets)
• Applications:
• Automatically updated monitor of app
components
• Start a job and then get an AJAX update
screen for the job progress.
35. Useable?
• The basics work.
• Running it production at work
• (Since WedneThursday....)
• If you need a high volume, simple,
production ready job scheduler right
now, use Gearman.
36. Useable?
• Needs my github versions of
Catalyst::Engine::PSGI and Web::Hippie
• Running jobs from the web app doesn’t
work/exist yet.
• All the pieces are there - next week?
38. Next steps?
• Production battle scars ;)
• Hippie for watching jobs run, not just
component status updates.
• Docs would be good...
• Hippe::Pipe for bidirectional
• More traits / plugins / components
39. What to go look at
• http://www.rabbitmq.com
• Docs, setup info, FAQ
• Slideshare - lots of good and more detailed
AMQP introductions.
40. What to go look at
(CPAN)
• Net::RabbitFoot
• App::RabbitTail
• Plack
• Web::Hippie
• CatalystX::JobServer (github)
41. Bonus slide!
• I forgot to mention this originally...
• DO NOT FORK AFTER USING
Net::RabbitFoot (and then try to use it in
the child process)
• This will go horribly horribly wrong.
43. Thanks!
• Questions?
• Anyone interested in playing, grab me on irc
• Happy to hand-hold as I need more people
using this.
• I’ll love you forever if you write docs.