Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Data Migration at Scale with RabbitMQ and Spring Integration

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Cargando en…3
×

Eche un vistazo a continuación

1 de 53 Anuncio
Anuncio

Más Contenido Relacionado

Similares a Data Migration at Scale with RabbitMQ and Spring Integration (20)

Anuncio

Más de Alvaro Videla (20)

Más reciente (20)

Anuncio

Data Migration at Scale with RabbitMQ and Spring Integration

  1. 1. Data Migration at Scale with RabbitMQ and Spring Integration By Álvaro Videla and Félix López Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
  2. 2. Álvaro Videla • Developer Advocate at Pivotal / RabbitMQ • Co-Author of RabbitMQ in Action • Creator of the RabbitMQ Simulator • Blogs about RabbitMQ Internals: http://videlalvaro.github.io/ internals.html • @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2
  3. 3. Álvaro Videla Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 3 Co-authored ! RabbitMQ in Action http://bit.ly/rabbitmq
  4. 4. What is RabbitMQ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 4
  5. 5. RabbitMQ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 5
  6. 6. RabbitMQ • Multi Protocol Messaging Server Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 6
  7. 7. RabbitMQ • Multi Protocol Messaging Server • Open Source (MPL) Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 7
  8. 8. RabbitMQ • Multi Protocol Messaging Server • Open Source (MPL) • Polyglot Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 8
  9. 9. RabbitMQ • Multi Protocol Messaging Server • Open Source (MPL) • Polyglot • Written in Erlang/OTP Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 9
  10. 10. Multi Protocol Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 10 http://bit.ly/rmq-protocols
  11. 11. Polyglot Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11
  12. 12. Polyglot • Java Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 12
  13. 13. Polyglot • Java • node.js Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 13
  14. 14. Polyglot • Java • node.js • Erlang Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 14
  15. 15. Polyglot • Java • node.js • Erlang • PHP Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 15
  16. 16. Polyglot • Java • node.js • Erlang • PHP • Ruby Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 16
  17. 17. Polyglot • Java • node.js • Erlang • PHP • Ruby • .Net Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 17
  18. 18. Polyglot • Java • node.js • Erlang • PHP • Ruby • .Net • Haskell Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 18
  19. 19. Polyglot Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 19 Even COBOL!!!11
  20. 20. http://www.rabbitmq.com/download.html Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 20 Unix - Mac - Windows
  21. 21. Messaging with RabbitMQ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 21 A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator
  22. 22. RabbitMQ Simulator Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 22
  23. 23. http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput- latency-and-bandwidth/ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 23
  24. 24. RabbitMQ Basic Qos Simulator Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 24
  25. 25. Prevent Unbound Buffers Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 25 λ = mean arrival time μ = mean service rate if λ > μ what happens?
  26. 26. Prevent Unbound Buffers Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 26 λ = mean arrival time μ = mean service rate if λ > μ what happens? Queue length goes to infinity over time.
  27. 27. Prevent Unbound Buffers λ = mean arrival time μ = mean service rate if λ > μ what happens? Queue length goes to infinity over time. Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 27 https://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers- with-rabbitmq/
  28. 28. Recommended Reading Performance Modeling and Design of Computer Systems: Queueing Theory in Action Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 28
  29. 29. What you get out of the box Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 29
  30. 30. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 30
  31. 31. Some RabbitMQ Features • Persistent Messages Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 31
  32. 32. Some RabbitMQ Features • Persistent Messages • Publisher Confirms Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 32
  33. 33. Some RabbitMQ Features • Persistent Messages • Publisher Confirms • Message Acknowledgment Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 33
  34. 34. Some RabbitMQ Features • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 34
  35. 35. Some RabbitMQ Features • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 35
  36. 36. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 36
  37. 37. Some RabbitMQ Features • Dead Letter Exchanges Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 37
  38. 38. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38
  39. 39. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 39
  40. 40. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 40
  41. 41. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 41
  42. 42. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation • Shovel Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 42
  43. 43. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 43
  44. 44. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 44 • SSL Connections
  45. 45. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 45 • SSL Connections • User Authentication
  46. 46. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 46 • SSL Connections • User Authentication • Per resource authorisation
  47. 47. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 47 • SSL Connections • User Authentication • Per resource authorisation • Different kinds of users
  48. 48. Some RabbitMQ Features Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 48 • SSL Connections • User Authentication • Per resource authorisation • Different kinds of users • Various authentication mechanisms via plugins (LDAP)
  49. 49. Management Interface / REST API Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 49
  50. 50. RabbitMQ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 50
  51. 51. RabbitMQ Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 51 Flexibility
  52. 52. Questions? Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 52
  53. 53. Learn More. Stay Connected @old_sound @flopezluis @springcentral | spring.io/video Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 53

×