RabbitMQ: Message queuing that works

•

1 like•693 views

"RabbitMQ: Message queuing that works" by Gabriele Santomaggio. In this talk I will show how to use RabbitMQ for two different applications. For the first one, how to setup a simple messaging application using different clients as Java, Java-Script, Python, Erlang, .Net, etc. For the second one, how to distribute long batch processes between more consumers in high-availability configuration. In both examples, we will see how to scale the application in terms of clients and brokers, we will see a few standard configurations for local cluster and geographics one. At the end of this talk, you will have walked through all the basics of RabbitMQ.

Technology

www.erlang-solutions.com
RABBITMQ
Message
queuing that
works

www.erlang-solutions.com
Gabriele Santomaggio
gabriele.santomaggio@erlang-solutions.com
www.erlang-solutions.com
▸ Working at Erlang Solutions
▸ RabbitMQ contributor
▸ Co-author of RabbitMQ Cookbook
▸ @gsantomaggio

www.erlang-solutions.com
What is RabbitMQ ?
▸ Multi-protocol Messaging broker
▸ Written in Erlang
Producer
Consumer
Exchange
Queue
Server
Virtual host
Binding

www.erlang-solutions.com
Protocols
▸ AMQP 0-9-1
▸ MQTT
▸ STOMP
▸ AMQP 1.0
▸ HTTP

www.erlang-solutions.com
RabbitMQ CLIENTS
▸ Java
▸ .NET/C# Client
▸ Python
▸ PHP
▸ Erlang
▸ Node.js
▸ iPhone (Dev in progress)
▸ Others…

www.erlang-solutions.com
1. Create team
("my_company")
2. Create different rooms
("tech.programming", "tech.networking","
marketing")
3. Send message to the room
4. Send private messages
5. Store messages
6. Clustering / HA
Messaging - example
GOALS

www.erlang-solutions.com
What we need
my_company
App. client 1
App. client 2
Db - service 1
App 1
App 2
Db - service 2
Exchange
Persistent Queue
Temporary Queues
Db - service n

www.erlang-solutions.com
R1: Create team
exchange_declare(exchange="my_company",
exchange_type="topic")

www.erlang-solutions.com
R2: Room subscribe
queue_declare(exclusive=True,
auto_delete=True)
queue_bind(exchange="my_company",
queue=queue,
routing_key="tech.programming")
basic_consume(on_message,
queue=queue,
no_ack=True)

$www.erlang-solutions.com headers = { 'sender_user':my_user_name, 'sent': datetime.datetime.now(), 'is_private': False } properties = pika.BasicProperties( headers=headers) channel.publish(exchange="my_company", routing_key="tech.programming" properties=properties, body="I love Erlang!") R3: Send message to the room$

$www.erlang-solutions.com headers = { 'sender_user':my_user_name, 'sent': datetime.datetime.now(), 'is_private': True } properties = pika.BasicProperties( headers=headers) channel.publish(exchange="my_company", routing_key="gabriele", properties=properties, body="how are you? :)!") R4: Send private message$

www.erlang-solutions.com
R5: Store messages
channel.queueDeclare("store.messages",
true, false, false, null);
channel.queueBind("store.messages",
"my_company", "#");
channel.basicQos(1);// round robin
channel.basicConsume("store.messages",
new DefaultConsumer()...

www.erlang-solutions.com
Oh no… seems
so complex!!
R6: Distributed RabbitMQ

www.erlang-solutions.com
▸ rabbitmqctl stop_app
▸ rabbitmqctl reset
▸ rabbitmqctl join_cluster
rabbit@server
▸ rabbitmqctl start_app
Clustering!

www.erlang-solutions.com
High Availability
Healthy Cluster
One Node Down

www.erlang-solutions.com
R6: Done!
We finished the app

www.erlang-solutions.com
What about WAN configuration?

www.erlang-solutions.com
Federation/Shovel - Plugin
▸ Loose coupling
▸ WAN-friendly
▸ Tolerant to network failures
Node 1 Node 2
AMQP client

www.erlang-solutions.com
Cloud scaling...
Db - Service Db - Service Db - Service
store.messages -
HA
Client 1 Client 2 Client..n

www.erlang-solutions.com
Exchange messages

www.erlang-solutions.com
Full example
▸ Full description(soon...I hope:))
https://www.erlang-solutions.
com/blog.html
▸ Source code (other languages)
https://github.
com/Gsantomaggio/rabbitmqexample

www.erlang-solutions.com
THANK YOU!
Any questions?
gabriele.santomaggio@erlang-solutions.com
www.erlang-solutions.com
@gsantomaggio

Viewers also liked

Introduction to ZeroMQ - eSpace TechTalk

Mahmoud Said

zeromq

Rajan Bhatt

RabbitMq

Ahmad Saif

Zeromq anatomy & jeromq

Dongmin Yu

Software Architecture over ZeroMQ

pieterh

ZeroMQ: Super Sockets - by J2 Labs

James Dennis

RabbitMQ

Lenz Gschwendtner

Erlang latest version & opensource projects

Digikrit

FOSDEM 2011 - 0MQ

pieterh

The RabbitMQ Message Broker

Martin Toshev

The Future of Messaging: RabbitMQ and AMQP

Eberhard Wolff

RabbitMQ Operations

Michael Klishin

12-Step Program for Scaling Web Applications on PostgreSQL

Konstantin Gredeskoul

CouchDB Day NYC 2017: Full Text Search

IBM Cloud Data Services

Using Erlang in an Embedded and Cross-Compiled World

Frank Hunleth

Using Mikko Koppanen's PHP ZMQ extension we will look at how you can easily distribute work to background processes, provide flexible service brokering for your next service oriented architecture, and manage caches efficiently and easily with just PHP and the ZeroMQ libraries. Whether the problem is asynchronous communication, message distribution, process management or just about anything, ZeroMQ can help you build an architecture that is more resilient, more scalable and more flexible, without introducing unnecessary overhead or requiring a heavyweight queue manager node.

ZeroMQ Is The Answer

Ian Barber

CouchDB Day NYC 2017: Using Geospatial Data in Cloudant & CouchDB

IBM Cloud Data Services

High powered messaging with RabbitMQ

James Carr

Etomidate ketamine

Harith Daggupati

Viewers also liked (19)

Introduction to ZeroMQ - eSpace TechTalk

zeromq

RabbitMq

Zeromq anatomy & jeromq

Software Architecture over ZeroMQ

ZeroMQ: Super Sockets - by J2 Labs

RabbitMQ

Erlang latest version & opensource projects

FOSDEM 2011 - 0MQ

The RabbitMQ Message Broker

The Future of Messaging: RabbitMQ and AMQP

RabbitMQ Operations

12-Step Program for Scaling Web Applications on PostgreSQL

CouchDB Day NYC 2017: Full Text Search

Using Erlang in an Embedded and Cross-Compiled World

ZeroMQ Is The Answer

CouchDB Day NYC 2017: Using Geospatial Data in Cloudant & CouchDB

High powered messaging with RabbitMQ

Etomidate ketamine

Similar to RabbitMQ: Message queuing that works

PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story

vanphp

Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...

Jimmy DeadcOde

Top 13 best front end web development tools to consider in 2021

Samaritan InfoTech

Entrepreneurship3

Yenwen Feng

joomla webhosting op joomladagen 2009

Gruus Van Woerkom

Joomla Hosting and Hosting Trends - Joomla!Days NL 2009 #jd09nl

Joomla!Days Netherlands

Us 17-krug-hacking-severless-runtimes

Ravishankar Somasundaram

Michael Tucker, GoDaddy_How GoDaddy used Spree to build their Online Store pr...

Spark Solutions

Import golang; struct microservice

Giulio De Donato

The Art of Evolutionary Algorithms Programming

Juan J. Merelo

CharlesSweetResume06155122015

Charlie Sweet

In this session, Robert will take the audience through the real-life learnings he has gained in building microservices for a large UK retailer using Kotlin, Micronaut, DataStore and running it on GCP. This session is meant to be practical in its advice and is targetted at those new to Kotlin and microservice development and attendees can expect to walk away knowing how to get started in this space. While GCP, in particular, DataStore, Storage, and GKE will be mentioned they are not the core focus with about 10min of the talk focused on that (and the bulk of that being on DataStore). First shared at DevFest 2019

Building a µservice with Kotlin, Micronaut & GCP

Robert MacLean

"Xapi-lang For declarative code generation" By James Nelson

GWTcon

WAF protections and bypass resources

Antonio Costa aka Cooler_

Low latency microservices in java QCon New York 2016

Peter Lawrey

Sterowniki .NET i C++ dla Apache Cassandra

Cognitum

Transactional File System In Java - Commons Transaction

Guo Albert

Do you use Java in your XPages applications? Want to have a better way to access Domino data? Over the last year, more and more developers have started using the OpenNTF Domino Java API. In this webinar, Paul Withers and Jesse Gallagher, part of the team of developers working on this open source software project, explain why you should be using the Domino API. Get a jumpstart into the Domino API and improve your XPages development experience!

Getting Started with the OpenNTF Domino API

Teamstudio

Microservices for performance - GOTO Chicago 2016

Peter Lawrey

Flamingo - Inspiring Commerce Frontend made in Go - Meet Magento 2019

AOE

Similar to RabbitMQ: Message queuing that works (20)

PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story

Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...

Top 13 best front end web development tools to consider in 2021

Entrepreneurship3

joomla webhosting op joomladagen 2009

Joomla Hosting and Hosting Trends - Joomla!Days NL 2009 #jd09nl

Us 17-krug-hacking-severless-runtimes

Michael Tucker, GoDaddy_How GoDaddy used Spree to build their Online Store pr...

Import golang; struct microservice

The Art of Evolutionary Algorithms Programming

CharlesSweetResume06155122015

Building a µservice with Kotlin, Micronaut & GCP

"Xapi-lang For declarative code generation" By James Nelson

WAF protections and bypass resources

Low latency microservices in java QCon New York 2016

Sterowniki .NET i C++ dla Apache Cassandra

Transactional File System In Java - Commons Transaction

Getting Started with the OpenNTF Domino API

Microservices for performance - GOTO Chicago 2016

Flamingo - Inspiring Commerce Frontend made in Go - Meet Magento 2019

More from Codemotion

Fuzz-testing: A hacker's approach to making your code more secure | Pascal Ze...

Codemotion

Pompili - From hero to_zero: The FatalNoise neverending story

Codemotion

Pastore - Commodore 65 - La storia

Codemotion

Rivivere l'ebbrezza di progettare un vecchio computer o una consolle da bar è oggi possibile sfruttando le FPGA, ovvero logiche programmabili che consentono a chiunque di progettare il proprio hardware o di ricrearne uno del passato. In questa sessione si racconta come dal reverse engineering dell'hardware di vecchie glorie come il Commodore 64 e lo ZX Spectrum sia stato possibile farle rivivere attraverso tecnologie oggi alla portata di tutti.

Pennisi - Essere Richard Altwasser

Codemotion

There's a lot of talk about blockchain, but how does the technology behind it actually work? For developers, getting some hands-on experience is the fastest way to get familiair with new technologies. So let's build a blockchain, then! In this session, we're going to build one in plain old Java, and have it working in 40 minutes. We'll cover key concepts of a blockchain: transactions, blocks, mining, proof-of-work, and reaching consensus in the blockchain network. After this session, you'll have a better understanding of core aspects of blockchain technology.

Michel Schudel - Let's build a blockchain... in 40 minutes! - Codemotion Amst...

Codemotion

When was the last time you were truly lost? Thanks to the maps and location technology in our phones, a whole generation has now grown up in a world where getting lost is truly a thing of the past. Location technology goes far beyond maps in the palm of our hand, however. In this talk, we will explore how a ridesharing app works. How do we discover our destination?How do we find the closest driver? How do we display this information on a map? How do we find the best route?To answer these questions,we will be learning about a variety of location APIs, including Maps, Positioning, Geocoding etc.

Richard Süselbeck - Building your own ride share app - Codemotion Amsterdam 2019

Codemotion

Eward Driehuis, SecureLink's research chief, will guide you through the bumpy ride we call the cyber threat landscape. As the industry has over a decade of experience of dealing with increasingly sophisticated attacks, you might be surprised to hear more attacks slip through the cracks than ever. From analyzing 20.000 of them in 2018, backed by a quarter of a million security events and over ten trillion data points, Eward will outline why this happens, how attacks are changing, and why it doesn't matter how neatly or securely you code.

Eward Driehuis - What we learned from 20.000 attacks - Codemotion Amsterdam 2019

Codemotion

IoT revolution is ended. Thanks to hardware improvement, building an intelligent ecosystem is easier than never before for both startups and large-scale enterprises. The real challenge is now to connect, process, store and analyze data: in the cloud, but also, at the edge. We’ll give a quick look on frameworks that aggregate dispersed devices data into a single global optimized system allowing to improve operational efficiency, to predict maintenance, to track asset in real-time, to secure cloud-connected devices and much more.

Francesco Baldassarri - Deliver Data at Scale - Codemotion Amsterdam 2019 -

Codemotion

What if Virtual Reality glasses could transform your environment into a three-dimensional work of art in realtime in the style of a painting from Van Gogh? One of the many interesting developments in the field of Deep Learning is the so called "Style Transfer". It describes a possibility to create a patchwork (or pastiche) from two images. While one of these images defines the the artistic style of the result picture, the other one is used for extracting the image content. A team from TNG Technology Consulting managed to build an AI showcase using OpenCV and Tensorflow to realize such goggles.

Martin Förtsch, Thomas Endres - Stereoscopic Style Transfer AI - Codemotion A...

Codemotion

Blockchain (and Cryptocurrency) is an evolution of 20-year old research from scientists like Chaum, Lamport, and Castro & Liskov. Due to the current hype, it's hard to distinguish beneficial aspects of the technology from a desire for a "silver bullet" for device security, verifiable logistics, or "saving democracy". The problem: blockchain introduces new security challenges - and blind adoption without understanding reduces overall security. In this talk, Melanie Rieback and Klaus Kursawe explain the pitfalls and limits of blockchain, so you can avoid making your applications LESS secure.

Melanie Rieback, Klaus Kursawe - Blockchain Security: Melting the "Silver Bul...

Codemotion

Networking is a core part of computing in the digital world we inhabit. But, how well do you know how it works? Do you understand all the moving parts of the OSI stack inside your computer, and how the network is actually put together? How can this ever work? This guided safari of layers, standards, protocols, and happenstance will bring us close to the copper wire, and up through the layers of CDMA/CD, ARP, routing and HTTP. We will make a few excursions through patchworks that still work forty years later, and cleverly designed mechanisms that show that simplicity is the only way to last.

Angelo van der Sijpt - How well do you know your network stack? - Codemotion ...

Codemotion

Performance tests are not only an important instrument for understanding a system and its runtime environment. It is also essential in order to check stability and scalability – non-functional requirements that might be decisive for success. But won't my cloud hosting service scale for me as long as I can afford it? Yes, but… It only operates and scales resources. It won't automatically make your system fast, stable and scalable. This talk shows how such and comparable questions can be clarified with performance tests and how DevOps teams benefit from regular test practise.

Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...

Codemotion

Sascha will demonstrate the opportunities and challenges of Conversational AI learned from the practice. Both Technology and User Experience will be covered introducing a process finding micro-moments, writing happy paths, gathering intents, designing the conversational flow, and finally publishing on almost all channels including Voice Services and Chatbots. Valuable for enterprises, developers, and designers. All live on stage in just minutes and with almost no code.

Sascha Wolter - Conversational AI Demystified - Codemotion Amsterdam 2019

Codemotion

A key challenge we face at Pacmed is quickly calibrating and deploying our tools for clinical decision support in different hospitals, where data formats may vary greatly. Using Intensive Care Units as a case study, I’ll delve into our scalable Python pipeline, which leverages Pandas’ split-apply-combine approach to perform complex feature engineering and automatic quality checks on large time-varying data, e.g. vital signs. I’ll show how we use the resulting flexible and interpretable dataframes to quickly (re)train our models to predict mortality, discharge, and medical complications.

Michele Tonutti - Scaling is caring - Codemotion Amsterdam 2019

Codemotion

Coolblue is a proud Dutch company, with a large internal development department; one that truly takes CI/CD to heart. Empowerment through automation is at the heart of these development teams, and with more than 1000 deployments a day, we think it's working out quite well. In this session, Pat Hermens (a Development Managers) will step you through what enables us to move so quickly, which tools we use, and most importantly, the mindset that is required to enable development teams to deliver at such a rapid pace.

Pat Hermens - From 100 to 1,000+ deployments a day - Codemotion Amsterdam 2019

Codemotion

Quantum computers can use all of the possible pathways generated by quantum decisions to solve problems that will forever remain intractable to classical compute power. As the mega players vie for quantum supremacy and Rigetti announces its $1M "quantum advantage" prize, we live in exciting times. IBM-Q and Microsoft Q# are two ways you can learn to program quantum computers so that you're ready when the quantum revolution comes. I'll demonstrate some quantum solutions to problems that will forever be out of reach of classical, including organic chemistry and large number factorisation.

James Birnie - Using Many Worlds of Compute Power with Quantum - Codemotion A...

Codemotion

Chinese food exploded across America in the early 20th century, rapidly adapting to local tastes while also spreading like wildfire. How was it able to spread so fast? The GY6 is a family of scooter engines that has achieved near total ubiquity in Europe. It is reliable and cheap to manufacture, and it's made in factories across China. How are these factories able to remain afloat? Chinese-American food and the GY6 are both riveting studies in product-market fit, and both are the product of a distributed open source-like development model. What lessons can we learn for open source software?

Don Goodman-Wilson - Chinese food, motor scooters, and open source developmen...

Codemotion

Pieter Omvlee - The story behind Sketch - Codemotion Amsterdam 2019

Codemotion

Would you fly in a plane designed by a craftsman or would you prefer your aircraft to be designed by engineers? We are learning that science and empiricism works in software development, maybe now is the time to redefine what “Software Engineering” really means. Software isn't bridge-building, it is not car or aircraft development either, but then neither is Chemical Engineering. Engineering is different in different disciplines. Maybe it is time for us to begin thinking about retrieving the term "Software Engineering" maybe it is time to define what our "Engineering" discipline should be.

Dave Farley - Taking Back “Software Engineering” - Codemotion Amsterdam 2019

Codemotion

What is the job of a CTO and how does it change as a startup grows in size and scale? As a CTO, where should you spend your focus? As an engineer aspiring to be a CTO, what skills should you pursue? In this inspiring and personal talk, I describe my journey from early Red Hat engineer to CTO at Bloomon. I will share my view on what it means to be a CTO, and ultimately answer the question: Should the CTO be coding?

Joshua Hoffman - Should the CTO be Coding? - Codemotion Amsterdam 2019

Codemotion

More from Codemotion (20)