Enviar búsqueda
Cargar
Serverless Event Streaming with Pulsar Functions-xiaolong
•
3 recomendaciones
•
210 vistas
StreamNative
Seguir
Google DevFest 2019 in Beijing
Leer menos
Leer más
Datos y análisis
Denunciar
Compartir
Denunciar
Compartir
1 de 45
Descargar ahora
Descargar para leer sin conexión
Recomendados
Opening Remarks - Pulsar Summit Asia 2021
Opening Remarks - Pulsar Summit Asia 2021
StreamNative
Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021
Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021
StreamNative
Kafka的设计与实现
Kafka的设计与实现
wang xing
The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021
The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021
StreamNative
RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021
RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021
StreamNative
lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用
hugo
Large-Scale Cluster Mangement & Kubernetes Under The Hood
Large-Scale Cluster Mangement & Kubernetes Under The Hood
Lei (Harry) Zhang
基于OpenResty的百万级长连接推送
基于OpenResty的百万级长连接推送
OpenRestyCon
Recomendados
Opening Remarks - Pulsar Summit Asia 2021
Opening Remarks - Pulsar Summit Asia 2021
StreamNative
Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021
Improvements Made in KoP 2.9.0 - Pulsar Summit Asia 2021
StreamNative
Kafka的设计与实现
Kafka的设计与实现
wang xing
The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021
The Evolution History of RoP(RocketMQ-on-Pulsar) - Pulsar Summit Asia 2021
StreamNative
RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021
RabbitMQ on Pulsar's Practice in Tencent Cloud - Puslar Summit Asia 2021
StreamNative
lua & ngx_lua 的介绍与应用
lua & ngx_lua 的介绍与应用
hugo
Large-Scale Cluster Mangement & Kubernetes Under The Hood
Large-Scale Cluster Mangement & Kubernetes Under The Hood
Lei (Harry) Zhang
基于OpenResty的百万级长连接推送
基于OpenResty的百万级长连接推送
OpenRestyCon
Kafka in Depth
Kafka in Depth
YI-CHING WU
Kafka cluster best practices
Kafka cluster best practices
Rico Chen
Ansible 101
Ansible 101
YI-CHING WU
Monitor is all for ops
Monitor is all for ops
琛琳 饶
Openstack nova
Openstack nova
Yong Luo
The Application of OpenStack in 360buy, by Bin He
The Application of OpenStack in 360buy, by Bin He
Hui Cheng
深入浅出Netty l.t
深入浅出Netty l.t
oleone
基于Fuel的超融合一体机
基于Fuel的超融合一体机
EdwardBadBoy
Using armeria to write your RPC
Using armeria to write your RPC
koji lin
Docker
Docker
旭 張
Traffic server overview
Traffic server overview
qianshi
高性能Web服务器Nginx及相关新技术的应用实践
高性能Web服务器Nginx及相关新技术的应用实践
rewinx
20081128 http caching_in_php
20081128 http caching_in_php
Hunter Wu
DAE
DAE
Qiangning Hong
DoubleBinlog方案
DoubleBinlog方案
Lixun Peng
Openstack neutron 原理详解
Openstack neutron 原理详解
Yong Luo
Oh K8s Is Swag - Kubernetes Basics
Oh K8s Is Swag - Kubernetes Basics
Okis Chuang
How to upgrade_ror3_to_ror4
How to upgrade_ror3_to_ror4
Wei-Yi Chiu
JCConf2015: groovy to gradle
JCConf2015: groovy to gradle
Ching Yi Chan
Ansible introduction
Ansible introduction
Chyi-Shian Jiang
Ali-tomcat
Ali-tomcat
wang hongjiang
未来网络技术发展探梦 - 开篇
未来网络技术发展探梦 - 开篇
Yao-Wei Ou
Más contenido relacionado
La actualidad más candente
Kafka in Depth
Kafka in Depth
YI-CHING WU
Kafka cluster best practices
Kafka cluster best practices
Rico Chen
Ansible 101
Ansible 101
YI-CHING WU
Monitor is all for ops
Monitor is all for ops
琛琳 饶
Openstack nova
Openstack nova
Yong Luo
The Application of OpenStack in 360buy, by Bin He
The Application of OpenStack in 360buy, by Bin He
Hui Cheng
深入浅出Netty l.t
深入浅出Netty l.t
oleone
基于Fuel的超融合一体机
基于Fuel的超融合一体机
EdwardBadBoy
Using armeria to write your RPC
Using armeria to write your RPC
koji lin
Docker
Docker
旭 張
Traffic server overview
Traffic server overview
qianshi
高性能Web服务器Nginx及相关新技术的应用实践
高性能Web服务器Nginx及相关新技术的应用实践
rewinx
20081128 http caching_in_php
20081128 http caching_in_php
Hunter Wu
DAE
DAE
Qiangning Hong
DoubleBinlog方案
DoubleBinlog方案
Lixun Peng
Openstack neutron 原理详解
Openstack neutron 原理详解
Yong Luo
Oh K8s Is Swag - Kubernetes Basics
Oh K8s Is Swag - Kubernetes Basics
Okis Chuang
How to upgrade_ror3_to_ror4
How to upgrade_ror3_to_ror4
Wei-Yi Chiu
JCConf2015: groovy to gradle
JCConf2015: groovy to gradle
Ching Yi Chan
Ansible introduction
Ansible introduction
Chyi-Shian Jiang
La actualidad más candente
(20)
Kafka in Depth
Kafka in Depth
Kafka cluster best practices
Kafka cluster best practices
Ansible 101
Ansible 101
Monitor is all for ops
Monitor is all for ops
Openstack nova
Openstack nova
The Application of OpenStack in 360buy, by Bin He
The Application of OpenStack in 360buy, by Bin He
深入浅出Netty l.t
深入浅出Netty l.t
基于Fuel的超融合一体机
基于Fuel的超融合一体机
Using armeria to write your RPC
Using armeria to write your RPC
Docker
Docker
Traffic server overview
Traffic server overview
高性能Web服务器Nginx及相关新技术的应用实践
高性能Web服务器Nginx及相关新技术的应用实践
20081128 http caching_in_php
20081128 http caching_in_php
DAE
DAE
DoubleBinlog方案
DoubleBinlog方案
Openstack neutron 原理详解
Openstack neutron 原理详解
Oh K8s Is Swag - Kubernetes Basics
Oh K8s Is Swag - Kubernetes Basics
How to upgrade_ror3_to_ror4
How to upgrade_ror3_to_ror4
JCConf2015: groovy to gradle
JCConf2015: groovy to gradle
Ansible introduction
Ansible introduction
Similar a Serverless Event Streaming with Pulsar Functions-xiaolong
Ali-tomcat
Ali-tomcat
wang hongjiang
未来网络技术发展探梦 - 开篇
未来网络技术发展探梦 - 开篇
Yao-Wei Ou
【Ask maclean技术分享oracle数据库优化】awr鹰眼系列awr报告全面指标分析
【Ask maclean技术分享oracle数据库优化】awr鹰眼系列awr报告全面指标分析
maclean liu
02.python.开发最佳实践
02.python.开发最佳实践
Na Lee
基于Symfony框架下的快速企业级应用开发
基于Symfony框架下的快速企业级应用开发
mysqlops
探索 ISTIO 新型 DATA PLANE 架構 AMBIENT MESH - GOLANG TAIWAN GATHERING #77 X CNTUG
探索 ISTIO 新型 DATA PLANE 架構 AMBIENT MESH - GOLANG TAIWAN GATHERING #77 X CNTUG
YingSiang Geng
Avm2虚拟机浅析与as3性能优化
Avm2虚拟机浅析与as3性能优化
Harvey Zhang
Angular 4 網站開發最佳實務 (Modern Web 2017)
Angular 4 網站開發最佳實務 (Modern Web 2017)
Will Huang
Linux binary Exploitation - Basic knowledge
Linux binary Exploitation - Basic knowledge
Angel Boy
How do we manage more than one thousand of Pegasus clusters - backend part
How do we manage more than one thousand of Pegasus clusters - backend part
acelyc1112009
Weibo lamp improvements
Weibo lamp improvements
Xinchen Hui
广告技术部自动化测试介绍.pdf
广告技术部自动化测试介绍.pdf
bj_qa
轻量级Flash服务器开发框架(刘恒)
轻量级Flash服务器开发框架(刘恒)
FLASH开发者交流会
Web请求异步处理和海量数据即时分析在淘宝开放平台的实践
Web请求异步处理和海量数据即时分析在淘宝开放平台的实践
mysqlops
Gwt rpc
Gwt rpc
Roy Chen
Pptv lb日志实时分析平台
Pptv lb日志实时分析平台
drewz lin
纵览Loadrunner核心功能
纵览Loadrunner核心功能
beiyu95
My sql 5.6新特性深入剖析——innodb引擎
My sql 5.6新特性深入剖析——innodb引擎
frogd
LabView with Lego NXT
LabView with Lego NXT
CAVEDU Education
千呼萬喚始出來的 Java SE 7
千呼萬喚始出來的 Java SE 7
Justin Lin
Similar a Serverless Event Streaming with Pulsar Functions-xiaolong
(20)
Ali-tomcat
Ali-tomcat
未来网络技术发展探梦 - 开篇
未来网络技术发展探梦 - 开篇
【Ask maclean技术分享oracle数据库优化】awr鹰眼系列awr报告全面指标分析
【Ask maclean技术分享oracle数据库优化】awr鹰眼系列awr报告全面指标分析
02.python.开发最佳实践
02.python.开发最佳实践
基于Symfony框架下的快速企业级应用开发
基于Symfony框架下的快速企业级应用开发
探索 ISTIO 新型 DATA PLANE 架構 AMBIENT MESH - GOLANG TAIWAN GATHERING #77 X CNTUG
探索 ISTIO 新型 DATA PLANE 架構 AMBIENT MESH - GOLANG TAIWAN GATHERING #77 X CNTUG
Avm2虚拟机浅析与as3性能优化
Avm2虚拟机浅析与as3性能优化
Angular 4 網站開發最佳實務 (Modern Web 2017)
Angular 4 網站開發最佳實務 (Modern Web 2017)
Linux binary Exploitation - Basic knowledge
Linux binary Exploitation - Basic knowledge
How do we manage more than one thousand of Pegasus clusters - backend part
How do we manage more than one thousand of Pegasus clusters - backend part
Weibo lamp improvements
Weibo lamp improvements
广告技术部自动化测试介绍.pdf
广告技术部自动化测试介绍.pdf
轻量级Flash服务器开发框架(刘恒)
轻量级Flash服务器开发框架(刘恒)
Web请求异步处理和海量数据即时分析在淘宝开放平台的实践
Web请求异步处理和海量数据即时分析在淘宝开放平台的实践
Gwt rpc
Gwt rpc
Pptv lb日志实时分析平台
Pptv lb日志实时分析平台
纵览Loadrunner核心功能
纵览Loadrunner核心功能
My sql 5.6新特性深入剖析——innodb引擎
My sql 5.6新特性深入剖析——innodb引擎
LabView with Lego NXT
LabView with Lego NXT
千呼萬喚始出來的 Java SE 7
千呼萬喚始出來的 Java SE 7
Más de StreamNative
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022
StreamNative
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
StreamNative
Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...
Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...
StreamNative
Distributed Database Design Decisions to Support High Performance Event Strea...
Distributed Database Design Decisions to Support High Performance Event Strea...
StreamNative
Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022
Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022
StreamNative
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
StreamNative
Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...
Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...
StreamNative
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
StreamNative
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
StreamNative
Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...
Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...
StreamNative
Understanding Broker Load Balancing - Pulsar Summit SF 2022
Understanding Broker Load Balancing - Pulsar Summit SF 2022
StreamNative
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
StreamNative
Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022
Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022
StreamNative
Event-Driven Applications Done Right - Pulsar Summit SF 2022
Event-Driven Applications Done Right - Pulsar Summit SF 2022
StreamNative
Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022
Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022
StreamNative
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
StreamNative
Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022
Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022
StreamNative
Welcome and Opening Remarks - Pulsar Summit SF 2022
Welcome and Opening Remarks - Pulsar Summit SF 2022
StreamNative
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...
StreamNative
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
StreamNative
Más de StreamNative
(20)
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022
Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...
Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...
Distributed Database Design Decisions to Support High Performance Event Strea...
Distributed Database Design Decisions to Support High Performance Event Strea...
Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022
Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...
Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...
Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...
Understanding Broker Load Balancing - Pulsar Summit SF 2022
Understanding Broker Load Balancing - Pulsar Summit SF 2022
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...
Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022
Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022
Event-Driven Applications Done Right - Pulsar Summit SF 2022
Event-Driven Applications Done Right - Pulsar Summit SF 2022
Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022
Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022
Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022
Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022
Welcome and Opening Remarks - Pulsar Summit SF 2022
Welcome and Opening Remarks - Pulsar Summit SF 2022
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...
Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...
Serverless Event Streaming with Pulsar Functions-xiaolong
1.
Serverless Event Streaming With
Pulsar Functions Xiaolong Ran @wolfstudy 2019/12/08
2.
• 开源项⽬爱好者,Go 语⾔爱好者 •
TiDB -> BitMain -> StreamNative • Apache Pulsar committer • Go Functions • pulsar-client-go • pulsarctl • … • https://github.com/wolfstudy Who am I
3.
⽬录 Apache Pulsar 简介 Apache
Pulsar 架构 Go Function 的设计 Go Function 的实现 Go Function 使⽤示例 Pulsar Function Runtime Pulsar Function Worker
4.
What is Apache
Pulsar
5.
为什么要“造轮⼦” • 没有系统能够解决遇到的问题和规模 • 多租户
- 百万 Topics - 低延时 - 持久化 - 跨地域复制 • 已有的系统存在的问题 • 分区模型紧耦合(存储和计算)- 不是云原⽣的设计 • 存储模型过于简单,强依赖⽂件系统 • I/O 不隔离:消费者在清除 backlog 的时候会影响其它⽣产者和消费者 • 运维太痛苦 - 替换机器、服务扩容需要强制 reblance 导致服务对外不可⽤ • 可能出现消息丢失的情况
6.
Flexible Messaging
7.
Flexible Messaging • One
data, different ways to consume • Queuing (aka stateless messaging) • Shared (* RabbitMQ) • Streaming (aka stateful messaging) • Exclusive • Failover (* Kafka) • Key_Shared
8.
Multi Tenancy
9.
Geo-Replication
10.
Cloud-native Architecture 计算存储分离 •独⽴扩展 • 瞬时失效恢复 •
扩容⽆需数据均衡
11.
Pulsar View -
Topic
12.
Pulsar View -
Partition
13.
Pulsar View -
Segment
14.
Event Stream is
the right foundation for your data
15.
Apache pulsar ⽣态
16.
When Event Streaming Meets
Serverless
17.
Introduce Pulsar Functions
18.
Pulsar Functions • A
serverless event streaming framework • Lightweight computation • Event - first, Stream - first • Multi languages • Multi runtimes • SDK - less && SDK
19.
Pulsar Functions
20.
Pulsar Functions
21.
Pulsar Functions
22.
Pulsar Functions 反射 Plugin
23.
Pulsar Functions 反射 Plugin • Go
是⼀⻔静态类型的语⾔ • 反射⽀持的相对较弱 • 注⼊⾮法模块,会带来⼀定安全隐患 • 如果模块出现问题,会导致服务奔溃 • 给版本管理带来了困难 • 社区不成熟
24.
Go functions User
Rule func () func () error func (input) error func () (output, error) func (input) (output, error) func (context.Context) error func (context.Context, input) error func (context.Context) (output, error) func (context.Context, input) (output, error)
25.
Go functions User
Rule Start() 函数的⼊参必须是函数类型 ⽤户函数的输⼊参数可以是 0-2 个 如果有两个参数,第⼀个必须是 context.Context 如果只有⼀个输⼊参数,必须是 context.Context ⽤户函数的输出参数可以是 0-2 个 如果返回两个参数,第⼆个参数必须是 error 如果只返回⼀个参数,必须是 error
26.
Go functions Context
27.
Go functions -
Use Case
28.
Go functions -
Use Case
29.
Pulsar Functions -
CLI
30.
Go functions -
Close channel 没有简单易⾏的⽅法去检查管道是否没有通过改变它的状态来 关闭。 关闭⼀个已经关闭的管道会触发 panic,所以,关闭者不知道 管道是否关闭仍去关闭它,这是⼀个危险的⾏为。 发送数据到⼀个关闭的管道会触发 panic, 所以,发送者不知 道管道是否关闭仍去发送消息给它,这是⼀个危险的⾏为。 原则:要在 producer 端关闭 channel
31.
Pulsar functions 使⽤场景 ETL Data
Enrichment Data Filtering Dynamic Routing Alerts and Threholds
32.
Flexible Runtime Colocate with
Broker - Thread & Process Managed Function Workers - Thread & Process External Schedulers - Container Kubernetes
33.
Colocate with Brokers
34.
Manager Function Workers
35.
External Schedulers -
Kubernetes
36.
Auto load balancing Pulsar
Functions use Pulsar’s auto-balancing mechanism on consumer Shared Subscription Failover Subscription Key-Shared Subscription (feature)
37.
Pulsar Function worker
38.
Pulsar Function worker •
存放元数据信息 • 接收并存储 Function 所有的状态请求 • 为了防⽌ topic 变⼤,写⼊时会进⾏压缩 • 所有数据使⽤ FQFN 作为 key 写⼊
39.
Pulsar Function worker •
存放调度分配信息 • 接收并存储 leader 监听到的所有状态请求 • 使⽤垃圾收集,防⽌ topic 变⼤ • 所有 worker 会订阅 assignment topic
40.
Pulsar Function worker •
管理 Functions Worker 的 Membership • 每⼀个 Functions Worker 都会订阅 Coordination Topic • Pulsar Broker 会跟踪处于活跃状态的 consumer • 通过查询活跃状态的 consumer,获取所有处于活跃状 态的 Functions Worker
41.
Complex Event Processing
Pipelines
42.
Community • Mailing lists: dev@pulsar.apache.org users@pulsar.apache.org •
Github: https://github.com/apache/pulsar https://github.com/apache/pulsar-translation • Slack: https://apache-pulsar.slack.com/ https://apache-pulsar.herokuapp.com/ • WeChat
43.
Apache Pulsar Users(China)
44.
Apache Pulsar Users(global)
45.
Thanks
Descargar ahora