Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Working with Asynchronous Events

291 visualizaciones

Publicado el

Talk given at the PHP user group Frankfurt regarding asynchronous events in Symfony 2 and the RabbitMQ library. We demonstrate some ups and downs as well as best practices and lessons learned.

Publicado en: Software
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Working with Asynchronous Events

  1. 1. Working with Asynchronous Events Jan Gregor Emge-Triebel (@jan0707) Dennis Oehme (@dennisoehme)
  2. 2. Garden of Concepts GmbH ● Hanau / Munich ● In house CMS “.stone” ● In house CRM “Elvis” ● Various individual customer projects ● Symfony based development, supported by: ○ MongoDB, ElasticSearch, MySQL, RabbitMQ, Redis, AngularJS, Twig, Jade, Gitlab, Docker, Vagrant, Ansible ● Say hi @goc_mediastudio
  3. 3. Requests == Event “Symfony is a request based framework.” Fabien Potencier, Symfony Live 2014 @fabpot
  4. 4. Requests Request Controller Logic 1 Logic 2 Response Request Controller Logic 1 Logic 2 Response Synchronous Asynchronous
  5. 5. Why do we need asynchronous event processing ? ● Move long running logic ● Cut down reaction times ● Transfer load ● Microservices ● Exchange data/messages with other (sub-)systems ● Parallelization ● Scaling
  6. 6. Our journey ● Gearman ● (PHP) Resque ○ Demo ● RabbitMQ
  7. 7. Gearman ● Gearman Job Server (written in C) ● Runs as a Linux daemon ● Requires PECL extension for worker API ● Worker APIs exists for other Languages (C, Perl,...)
  8. 8. Gearman http://gearman.org/examples/send-emails/
  9. 9. Gearman ● Latest PECL-Extension version is 1.1.2 (August 2013) ● Latest Server-Release (spring 2014) ● No more process is being made ● Gearman daemon and extension are often incompatible
  10. 10. (PHP) Resque ● Is a Resque port to php: ○ Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. ● Needs Redis(-Cluster) ● Redis is capable of atomic transactions (FIFO)
  11. 11. Demo https://github.com/Jan0707/async-symfony
  12. 12. (PHP) Resque ● Hardly any commits since 2012 ● Last commit to Master on 2015-05-12 ● chrisboulton/php-resque & michelsalib/BCCResqueBundle must be presumed dead ?
  13. 13. RabbitMQ Or why this talk should be named: “Working with Asynchronous Events in RabbitMQ”
  14. 14. RabbitMQ ● Message Broker written in Erlang ● AMQP (Advanced Message Queuing Protocol) ● Open, but binary network protocoll ● Routing, message distribution ● Transactions ● Plus, more nice features ○ TTL ○ DLX (Dead Letter eXchange) ○ PreFetching
  15. 15. Of publishers and consumers ... Publisher Publish Consumer Ex- change QueueRoutes Consumes
  16. 16. There is a Bundle for it ● https://github.com/php-amqplib/RabbitMqBundle ● Consumer and Producer as Symfony services ● Every queue Consumer runs as a Symfony command ● Configuration via app/config.yml ○ Better: Create app/config/rabbitmq.yml and import in app/config/config.yml
  17. 17. app/config/rabbitmq.yml
  18. 18. Symfony Producer
  19. 19. Symfony Consumer
  20. 20. Lessons Learned: Consumer ● PHP threads are not meant to be long running ○ New problems: memory leaks, ... ● Use supervisor to check and restart Consumers ● Open database connections, timeouts ○ Better quit when inactive ● Cap maximum runtime per thread ○ Automatically shutdown after processign x messages ● Memory limit: RabbitMQ-Bundle utilizes “soft-quota” ● Booting consumers takes rather long ● Prefetching (Warning: Order!)
  21. 21. Lessons Learned: Message ● Compact/short messages ● Queues eat up a lot of storage / RAM ● Common format such as JSON or XML ○ Do not serialize PHP objects or arrays ○ Ensure interoperability with other systems and languages
  22. 22. Lessons Learned: Message Transformation ● Use JMS/Serializer to convert objects to JSON ○ Deserialize JSON to PHP objects in consumer ○ Objects can and should be validated (with symfony forms and asserts) ○ Reload entities / documents to keep them up to date (doctrine/unit of work) ● Alternative approach: PHPs JsonSerializable Interface
  23. 23. Lessons Learned: Message Header ● Use message “Header” for content-type and version ○ Message formats can change ○ consumers must remain compatible ● Unqiue IDs make logging / debugging / live easier ● Other meta data: ○ timestamp ○ user / session ○ App id ○ ...
  24. 24. Tip: Custom Consumer-/Producer-Classes ● Message transformation ● Events ● Logging
  25. 25. Transactions ● Messages must be brokered to exactly one Consumer ● Consumer must explicitly answer a message with “acknowledge” ● If not, the message will be marked “rejected” and be requeued ○ Beware of requeuing: ■ Can lead to toxic messages ■ Can change the order of messages ○ Beware of final rejection: ■ Can lead to loss of data
  26. 26. Toxic Messages ● If a message is not explicitly being answered, it will be requeued. ● If that happens multiple times : ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop
  27. 27. Dead Letter Exchange ● RabbitMQ comes with a Dead Letter Exchange (DLX) ● Every queue determines its own or shared DLX ● DLX is a separate queue, storing all rejected messages ○ (Reject, TTL, Prefetch)
  28. 28. app/config/rabbitmq.yml
  29. 29. Signal Handling ● Graceful Shutdown ● pcntl_signal / _dispatch ● SIGTERM ● FATAL ERROR abfangen ● Logging ● Toxic Message-Handling Shutdown
  30. 30. Deployment ● Consumers need an up to date code base ● Consumers need a graceful shutdown ● Consumers must only be shutdown when idling ○ Otherwise : Loss of message / data
  31. 31. Other challenges ● Significant change of infrastructure ○ Choose your libraries and dependencies carefully! ● Moving load is not removing load ● Rethinking necessary ○ Display of status / intermediate result ○ Websockets or Push instead of Polling ● Interdependencies of messages ○ Especially when running parallel consumers! ● Performance (Doctrine ORM / ODM)
  32. 32. Questions ? Thank you :) Find this and other talks at: https://www.gardenofconcepts. com/talks/ Feedback is always welcome & please rate our talk !

×