Logging is an absolute must for any API or web application, but when starting out, questions such as "how can we do it without disrupting everything else" and "what is the easiest way to log" often come up. We’re going to examine a tried and tested method to carry out high-performance, low-latency logging using the power of RabbitMQ to ensure minimal impact to the performance of your runtime application. The talk will show you that a really great logging architecture is a low-cost investment in your application that will definitely pay off in the long run.

1. Low Latency Logging
with RabbitMQ
PHP London
4th September 2014

2. Who is this guy?
James Titcumb
www.jamestitcumb.com
www.protected.co.uk
www.phphants.co.uk
@asgrim

3. Let’s go back to basics...

4. Errors

5. source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/

10. error_reporting(0);

11. They look rubbish!

12. @

13. Log to file only

14. Ways Around
// Will raise E_WARNING
fopen($somefile, 'r');

15. Ways Around
// No error! :)
if (file_exists($somefile)) {
fopen($somefile, 'r');
} else {
// nice handling...
// maybe throw exception...
}

16. Exceptions

17. Jargon Buster
● throw
Triggering
● try
Try to run
● catch
Handle it
● finally
Run code after try/catch

18. source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/

19. C♯/C++

20. Catchable

21. Turn into fatal errors
if not caught

22. Classy

23. Descriptive exceptions

24. Example (exception class)
class DivisionByZeroException
extends LogicException
{
}

25. Example (throw)
class Mathematics {
public function divide($a, $b) {
if ($b == 0) {
$msg = sprintf(“Divide %s by zero”, $a);
throw new DivisionByZeroException($msg);
}
return ($a / $b);
}
}

26. Example (catch)
$maths = new Mathematics();
try
{
$result = $maths->divide(5, 0);
}
catch (DivisionByZeroException $exception)
{
$logger->warning($exception->getMessage());
}

27. Logging

29. What is “logging”?

30. What is “logging”?
Keeping a record of all events...

31. What is “logging”?
Keeping a record of all events...
exceptions, errors, warnings, info, debug

32. Logging is a paper trail

33. Log like you just don’t care

35. Log wherever you like

36. What about Apache’s error_log?
source: http://up-ship.com/blog/?p=20903

37. Doin’ it rightwrong…
// Magic file that makes your entire project work perfectly
@ini_set('display_errors', 0);
@error_reporting(0);
function __globalErrorHandler() {
return true;
}
@set_error_handler('__globalErrorHandler');
@set_exception_handler('__globalErrorHandler');
@register_shutdown_function(function() {
if(error_get_last())
echo "Script executed successfully!";
}); https://github.com/webarto/boostrap.php

38. Requirements (for everyone)
● Fire & forget
● Minimum or zero latency
● Highly available
● Log everything
● PSR-3 compatible

39. PSR-3

40. Logging Options
● monolog (PSR-3)
● Drupal - PSR-3 Watchdog
● phpconsole
● log4php
● RavenPHP + Sentry
● FirePHP (dev environment)
● Roll your own

41. source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html

42. Capturing Logging
Use “capture methods”, send to $logger
● set_exception_handler()
○ Handles all uncaught exceptions
● set_error_handler()
○ Handles most errors
● register_shutdown_function()
○ Handles fatal errors

43. Sending Log Messages
● Handler/Adapter translates
● However you want…
● Monolog has loads:
○ syslog-compatible / error_log
○ Email, HipChat
○ AMQP, Sentry, Zend Monitor, Graylog2
○ Redis, MongoDB, CouchDB

44. Typical PSR-3 Compatible Design
Capture Method
Logger (PSR-3)
Handler / Adapter
Data Storage

45. MonologErrorHandler
->handleException()
MonologLogger
->log()
MonologHandler
->handle()
Monolog

46. Low Latency
High Performance

47. Slow Logging
Browser Application Log Server
HTTP request
Send log message to database
Error!
Acknowledge message
HTTP response to client

48. Zero Latency Logging (ideal)
Browser Application Log Server
HTTP request
Send log message to database
Error!
HTTP response to client

49. UDP?

50. Low Latency Logging (balance)
Browser Application Log Server
HTTP request
Send log message to database
Error!
HTTP response to client

51. …so how?

52. Say hello to RabbitMQ

53. RabbitMQ - Basic
test_queue
1 2 3 4 5
Producer Consumer
source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html

54. RabbitMQ - Exchanges
Exchange
Consumer test_queue
1 2 3 4 5
source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html
Consumer
Consumer
Consumer
Consumer

55. Using Queues === Fast!
Add RabbitMQ to logging architecture...

56. Low Latency (using AMQP)
EdLogHandlerErrorHandler
->handleException()
EdLogLogger
->log()
EdLogPublisherAmqpPublisher
->publish()
RabbitMQ JSON payload
Logging Server

57. Low Latency Logging (with AMQP)
Browser Application Log Server
Fetch message
HTTP request
JSON via AMQP
Error!
HTTP response
RabbitMQ

58. Why bother?
● Scalability
RabbitMQ
Application A
Application B
Application C
Log Worker
Log Worker
Log Worker
Log Worker

59. Single Point of Failure...
● RabbitMQ can do HA or clustering
RabbitMQ
Node 1
RabbitMQ
Node 3
RabbitMQ
Node 2
RabbitMQ
Node 4
RabbitMQ
Node 6
RabbitMQ
Node 5

60. Questions?

61. Thanks for watching!
James Titcumb
www.jamestitcumb.com
www.protected.co.uk
www.phphants.co.uk
@asgrim