RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021

•

0 recomendaciones•64 vistas

RabbitMQ on Pulsar is a protocol handler developed by Tencent Cloud, which is fully equivalent to the open source RabbitMQ on Pulsar. Functionally, it can support RabbitMQ's authentication, dynamic topology, financial level, dynamic routing and other functional features. The architecture supports cloud-native features such as multi-tenancy, resource isolation, and automatic load balancing. It allows RabbitMQ users to get on the cloud-native message queue train without changing the code, and enjoy the bonuses brought by cloud-native. In this session, we will share the practice of RabbitMQ on Pulsar in Tencent Cloud.

Internet

RabbitMQ on Pulsar的实践和架构演进之路
林宇强@腾讯云高级研发工程师

背景
腾讯云TDMQ，公有云场景，多租户，协议和功能完全对齐
RabbitMQ，希望在Apache Pulsar上架设一套RabbitMQ的协议适配
器。
能够让原生RabbitMQ的用户不需要改动代码，使用上Pulsar的生
态，提供一套高性能、云原生、免运维的RabbitMQ云上替代方案。
l Apache Pulsar：云原生、高性能、多租户、横向扩展等
l RabbitMQ：存量生态多、动态拓扑、金融级消息、多语言支持等

议程
探索的起始：基于社区版本
元数据抽离：Compact Topic
功能的完善：RabbitMQ功能集
架构的终点：Gateway模型

1. 多租户
2. 资源管理
3. AMQP Admin
4. 异常处理完善
架构

1. 单Bundle
2. Declare不支持
3. 消费模型不完善
4. 无认证和鉴权
5. 架构羸弱
6. Tracing和Metrics生态无法复用
7. AMQP Admin需要重定向
发现问题

架构
Compact Topic
1. Compact Topic
2. pulsar-client生产
3. 重写Replicator
4. 发布确认
5. 死信Exchange
6. 延迟消息
7. AMQP认证和鉴权

发现问题
1. 单Bundle
2. Declare不支持
3. 消费模型不完善
4. 架构羸弱
5. Tracing和Metrics生态无法复用
6. 配额管理、资源隔离、策略控制失效

发现问题
1. 单Bundle
2. 架构羸弱
3. Tracing和Metrics生态无法复用
4. 配额管理、资源隔离、策略控制失效
5. Compact Topic和元数据一致性
6. 回顾和Gateway模式的优劣对比

与Gateway模式的对比
事项 Protocol handler插件 Gateway模式
部署方式和Pulsar broker同进程部署独立进程部署
性能损耗可减少RPC和编解码次数，提高性能增加1次RPC和2次编解码，增加服务器成本
资源配额功能需要重做适配pulsar-client的异常处理，复用现有功能
操作空间可操作Pulsar broker API，操作空间大仅基于pulsar-client和pulsar-admin，操作空间受限
周边功能 Metrics、Tracing、告警等功能重做 Metrics、Tracing、告警等功能重做，有少量可复用
现有生态
可运维性待插件的broker和不带插件的broker在运
维上要当成两个东西来对待
Gateway背后的Pulsar原生且无任何变动，可以和
Pulsar协议的集群一视同仁

总结
1. 架构演进和探索过程
2. 协议适配关键点与思考
3. RabbitMQ功能支持
4. Gateway和Protocol Handler的平衡

Más contenido relacionado

La actualidad más candente

Openstack网络规划

OpenCity Community

Kafka cluster best practices

Rico Chen

Micro service

rfyiamcool

Pm 04 华胜天成openstack实践汇报-20120808

OpenCity Community

Openstack neutron 原理详解

Yong Luo

Kubernetes project update and how to contribute

inwin stack

Openstack的研究与实践

OpenCity Community

Train.IO 【第六期－OpenStack 二三事】

inwin stack

OpenStack and Docke Integration V6

Guangya Liu

AWS lambda

CKmates

The Application of OpenStack in 360buy, by Bin He

Hui Cheng

跳过私有云建设的“坑” 私有云建设经验教训以及IBM PMC2.0 简介

Hardway Hou

The Tencent Angel PowerFL is a fast-growing platform for privacy-preserving computation, which provides a full-stack solution for achieving the great potential in federated learning and analytics. The Angel PowerFL platform enables multiple participants to jointly build machine learning models while keeping their data confidential from each other, and it has superior performance in terms of security, efficiency and reliability. In this talk, we will introduce a novel federated communication framework, which is built on Apache Pulsar. We use Apache Pulsar as the fundamental component to build the secure communication channels, upon which a federation SDK is built as the communication protocol for multiparty privacy-preserving computation. We will also show that our framework can support real-time federated model serving and applications.

PowerFL meets Pulsar: A Novel Communication Framework for Privacy-Preserving ...

StreamNative

Build scalable microblog qcon beijing 2010

Tim Y

Oh K8s Is Swag - Kubernetes Basics

Okis Chuang

深入浅出Netty l.t

oleone

微服務自己動手做

YenChen Liu 劉晏辰

Traffic server overview

qianshi

Kube-OVN Introduction

梦馨刘

re:Invent Keynote Recap Speaker: James Chiang, Solutions Architect, AWS 亞馬遜 AWS 於 2018 年 11 月底在美國拉斯維加斯所舉辦的第七屆 AWS re:Invent 2018 大會，在 AWS 客戶、合作夥伴、媒體人士、產業分析師及 AWS 員工共襄盛舉下，與會人數再創新高，超過 5 萬人。會中 AWS 發布超過 20 款雲端方案，且一半以上專攻雲端 AI、機器學習、物聯網，包括對 SageMaker 強化更多進階功能，推出第一款專用的機器學習推論晶片、加入深度的機器學習運算法支援，及其他包括儲存、資料庫、混合雲、邊緣運算 IoT 等解決方案。而具備微型機器學習能力的迷你自駕遙控車 DeepRacer 的現身，驚人之舉不僅抓人眼球，深入客戶體驗的用心，更成功抓住全球使用者的心。為讓您與全球先進技術同步，共享最新趨勢資訊，解決您開發機器學習和發展 AIoT 所遇到的難題，AWS 台灣團隊將於 2019 年 1 月 31 日 (四) 舉辦《AWS re:Invent 2018 Recap 台北》，特別嚴選最適切國內諸位先進和企業需求的內容，從「技術創新」、「AIoT」兩大分組議程，發表 AWS 的新服務和新方案。大會除了邀請亞馬遜 AWS 大中華區首席雲計算企業顧問 (Principal Evangelist) 張俠博士分享 AWS 的解決方案藍圖外，眾多 AWS 資深專家也將分享包含機器學習、深度學習推理加速等新方案，完全託管的文件系統、資料庫，無伺服器、容器技術與安全性，以及大數據與分析、物聯網服務應用、儲存方案等最新技術。歡迎您親臨會場，全方位體驗 AWS 新服務將能為您創造的驚人創新之效益。

AWS reInvent 2018 Recap - Solutions Updates part 1

Amazon Web Services

La actualidad más candente (20)

Openstack网络规划

Kafka cluster best practices

Micro service

Pm 04 华胜天成openstack实践汇报-20120808

Openstack neutron 原理详解

Kubernetes project update and how to contribute

Openstack的研究与实践

Train.IO 【第六期－OpenStack 二三事】

OpenStack and Docke Integration V6

AWS lambda

The Application of OpenStack in 360buy, by Bin He

跳过私有云建设的“坑” 私有云建设经验教训以及IBM PMC2.0 简介

PowerFL meets Pulsar: A Novel Communication Framework for Privacy-Preserving ...

Build scalable microblog qcon beijing 2010

Oh K8s Is Swag - Kubernetes Basics

深入浅出Netty l.t

微服務自己動手做

Traffic server overview

Kube-OVN Introduction

AWS reInvent 2018 Recap - Solutions Updates part 1

Similar a RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021

CloudConnect 云计算大会 China 2016 Brocade 网络创新平台框架 • 开场: 云网锦绣 - 连接云的力量（张宇峰，Brocade大中华区CTO） • “大云“之端 – 中国移动OpenStack公有云虚拟应用交付的集成编排（李睿，Brocade Sr. SE) • 开放创新 – 中国银联对下一代金融云平台中SDN/NFV架构及应用探索（周雍恺中国银联研究院博士后研究院，张泽鹏 Brocade Sr. SE） • 应用为王 – 互联港湾混合云/多云环境下的虚拟应用交付部署及自动扩展（任志远互联港湾CEO，余欣 Brocade Sr. SWN SE） • vEPC & NVA网络可视分析 – 慧岭科技在SDN/NFV使能移动虚拟运营商的初步实践（智勇慧岭科技 CTO） • 敏捷之魂 – LinkedIn和Netflix钟爱的事件驱动云数据中心工作流级自动化（储子瑞 Brocade资深软件网络工程师，梁博九州云资深架构师及培训负责人） • MANO驱动的NFV/SDN服务编排 – 运营商云网合一开放生态实战方案分享（张宇峰，龚永生九州云资深架构师） • Panel：DevOps – 颠覆文化，开放边界，蜕变产业，演进生态（任志远，余欣，智勇，梁博，龚永生，方辉 SDNLAB联合创始人, 张宇峰） • 尾声: 回到未来 – 软件，机器，赛跑，学习（张宇峰）

云网锦绣 SDN实战研讨会

Hardway Hou

Cloud Computing Introduction

guest90f660

Ibm dnt-dcos-v9-3

Guangya Liu

Open stack中国社区开放测试平台（ostp）

OpenCity Community

IBM Cloud Private Introduction

Guangya Liu

OpenStack Quantum项目介绍和演示

OpenCity Community

Openstack高度自动化持续交付

Bill Zhong Qibin

微博平台混合云实践 - Docker全架构

Chen Fei

LVS

mysqlops

使用LVS集群架设高可扩展的网络服务

Wensong Zhang

章文嵩：使用Lvs集群架设高可扩展的网络服务

drewz lin

Planet Lab

Yishuai Chen

腾讯马志强虚拟化环境下网络朋务器平台的协作经验

colderboy17

腾讯马志强虚拟化环境下网络朋务器平台的协作经验

2015中国软件技术大会－开放云介绍

Hacmp5.x

大鱼架构演进

51 cto下载 51cto信息图：openshift vs cloudfoundry

Hong Cai

Hacking Nginx at Taobao

Joshua Zhu

徐晓 Qq空间技术架构之峥嵘岁月

drewz lin

Similar a RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021 (20)

云网锦绣 SDN实战研讨会

Cloud Computing Introduction

Ibm dnt-dcos-v9-3

Open stack中国社区开放测试平台（ostp）

IBM Cloud Private Introduction

OpenStack Quantum项目介绍和演示

Openstack高度自动化持续交付

微博平台混合云实践 - Docker全架构

LVS

使用LVS集群架设高可扩展的网络服务

章文嵩：使用Lvs集群架设高可扩展的网络服务

Planet Lab

腾讯马志强虚拟化环境下网络朋务器平台的协作经验

2015中国软件技术大会－开放云介绍

Hacmp5.x

大鱼架构演进

51 cto下载 51cto信息图：openshift vs cloudfoundry

Hacking Nginx at Taobao

徐晓 Qq空间技术架构之峥嵘岁月

Más de StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Distributed Database Design Decisions to Support High Performance Event Strea...

StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

StreamNative

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

StreamNative

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

StreamNative

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

StreamNative

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

StreamNative

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022

StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...

StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

StreamNative

Más de StreamNative (20)