At Yelp we run hundreds of Flink jobs to power a wide range of applications: push notifications, data replication, ETL, sessionizing and more. Routine operations like deploys, restart, and savepointing for so many jobs would take quite a bit of developers’ time without the right degree of automation. The latest addition to our toolshed is a Kubernetes operator managing the deployment and the lifetime of Flink clusters on PaaSTA, Yelp’s Platform As A Service. We replaced our deployment framework launching Flink clusters on top of AWS EMR with a Kubernetes operator managing fully Docker-ized Flink clusters. Compared to EMR, this architecture allowed us to both drastically reduce the deployment time of our Flink clusters and to share our hardware resources more efficiently. In addition, we now offer to our developers the same interface they are used to for running REST services, batch jobs and many other workloads on PaaSTA. This talk will give a brief overview of Yelp’s PaaSTA before diving into the details of how the Kubernetes operator has been implemented and how it has been integrated with Yelp developers’ workflow (deploy, logs, savepoints, upgrades, etc), to end with a glimpse of the future features we are planning for the operator (Flink as a library, autoscaling, etc.).

1. Kubernetes + Operator + PaaSTA =
Flink@Yelp
Oct 9, 2019

2. Yelp’s Mission
Connecting
people with great
local businesses

3. What you’ll see

4. What you’ll see What Flink at Yelp looks like
What Yelp uses Flink for and what using Flink at Yelp
looks like
WHAT YOU’LL SEE

5. What you’ll see What Flink at Yelp looks like
What Yelp uses Flink for and what using Flink at Yelp
looks like
How Kubernetes can power Flink
How Kubernetes and Operators can be used to power
Flink clusters deployment and operations
WHAT YOU’LL SEE

6. What you’ll see What Flink at Yelp looks like
What Yelp uses Flink for and what using Flink at Yelp
looks like
How Kubernetes can power Flink
Why platform integration matters
How Kubernetes and Operators can be used to power
Flink clusters deployment and operations
Why integrating Flink with Yelp’s platform as a service
(PaaSTA) is the key to unlock value for the users
WHAT YOU’LL SEE

7. Flink@Yelp

8. FLINK@YELP
Powering Data Enrichment and Transformation as a Service
StreamSQL manipulations and multi-stream
unwindowed joins as a service

9. FLINK@YELP
Powering Data Enrichment and Transformation as a Service
StreamSQL manipulations and multi-stream
unwindowed joins as a service
Real-time Notifications
Customized push notification to suggest relevant
businesses nearby

10. FLINK@YELP
Powering Data Enrichment and Transformation as a Service
StreamSQL manipulations and multi-stream
unwindowed joins as a service
Real-time Notifications
User Activity Sessions
Customized push notification to suggest relevant
businesses nearby
Multi-platform user activity sessions out of event logs

11. FLINK@YELP
Powering
Connectors

12. FLINK@YELP
Powering
Connectors

13. FLINK@YELP
Powering
Connectors

14. FLINK@YELP
Powering
Connectors

15. FLINK@YELP
Powering
Connectors

16. FLINK@YELP
Powering
Connectors

17. FLINK@YELP
The scale ~10 apps

18. FLINK@YELP
The scale ~10 apps
~50 clusters

19. FLINK@YELP
~1000 jobs
The scale ~10 apps
~50 clusters

20. The Status Quo

21. THE STATUS QUO
Flink on
AWS EMR

22. THE STATUS QUO
Meh. Both complex and slow
Running a dockerized Puppet monolith, 15 minutes
boot time and depending on AWS for Flink updates

23. THE STATUS QUO
Meh. Both complex and slow
Running a dockerized Puppet monolith, 15 minutes
boot time and depending on AWS for Flink updates
Still pretty manual
Each cluster needs trained operators to manually
deploy new versions or scale up resources

24. THE STATUS QUO
Meh. Both complex and slow
Running a dockerized Puppet monolith, 15 minutes
boot time and depending on AWS for Flink updates
Still pretty manual
Just different
Each cluster needs trained operators to manually
deploy new versions or scale up resources
Different UX and infrastructure from the rest of Yelp led
to high barrier to entry and knowledge impedance

25. Meet
Kubernetes

26. MEET KUBERNETES
Hello, I’m... an open-source system for automating deployment, scaling, and
management of containerized applications.
(The Internet)

27. MEET KUBERNETES
I like... Horizontal scaling
Scale applications up and down with a simple
command or automatically based on CPU usage

28. MEET KUBERNETES
I like... Horizontal scaling
Scale applications up and down with a simple
command or automatically based on CPU usage
Self-healing systems
Restart containers that fails, reschedule them when
nodes die, support user-defined health-checks

29. MEET KUBERNETES
I like... Horizontal scaling
Scale applications up and down with a simple
command or automatically based on CPU usage
Self-healing systems
Powerful primitives
Restart containers that fails, reschedule them when
nodes die, support user-defined health-checks
Pods, ReplicaSets, Services, Jobs and friends can be
used to model complex applications and workflows

30. MEET KUBERNETES
My hobbies are... Automatic bin packing
Place containers based on their requirements and
constraints, to drive up utilization and save resources

31. MEET KUBERNETES
My hobbies are... Automatic bin packing
Place containers based on their requirements and
constraints, to drive up utilization and save resources
Service discovery and load balancing
Give pods their own IP and a single DNS name for a set
of Pods and can load-balance across them

32. MEET KUBERNETES
My hobbies are... Automatic bin packing
Place containers based on their requirements and
constraints, to drive up utilization and save resources
Service discovery and load balancing
Storage orchestration
Give pods their own IP and a single DNS name for a set
of Pods and can load-balance across them
Automatically mount the storage system of your choice
and maintain state across application restarts

33. Assembling
Flink
Clusters

36. ASSEMBLING FLINK CLUSTERS
Job Manager is a Deployment of a Pod

37. Job Manager
Pod
Co-located group of containers with shared storage,
network and a spec for how to run the containers
is a Deployment of a Pod
ASSEMBLING FLINK CLUSTERS

38. Job Manager
Pod
Co-located group of containers with shared storage,
network and a spec for how to run the containers
is a Deployment of a Pod
Deployments
Provides declarative updates for Pods and ReplicaSets
to automate containers deployments and rollbacks
ASSEMBLING FLINK CLUSTERS

40. ASSEMBLING FLINK CLUSTERS
Task Managers are a Deployment of a ReplicaSet

41. Task Managers are a Deployment of a ReplicaSet
ReplicaSets
Maintain a stable set of identical Pods running at any
given time
ASSEMBLING FLINK CLUSTERS

43. Static IPs or DNS are replaced by a Service and a Proxy
ASSEMBLING FLINK CLUSTERS

44. Static IPs or DNS
Service
Exposes an application running on a set of Pods as a
network service regardless of their ephemeral IPs
are replaced by a Service and a Proxy
ASSEMBLING FLINK CLUSTERS

45. Static IPs or DNS
Service
Exposes an application running on a set of Pods as a
network service regardless of their ephemeral IPs
are replaced by a Service and a Proxy
Kube-proxy
Network proxy running on each node reflecting
Services and doing port-forwarding and round-robin
ASSEMBLING FLINK CLUSTERS

47. Flink jobs are deployed by the Supervisor
ASSEMBLING FLINK CLUSTERS

48. Flink jobs
Flink Supervisor
Yelp’s in-house daemon responsible of deployment,
state management and monitoring of Flink jobs on EMR
are deployed by Supervisor
ASSEMBLING FLINK CLUSTERS

50. Cluster shutdown is signaled via a Job
ASSEMBLING FLINK CLUSTERS

51. Cluster shutdown
Jobs
Create Pods and ensure that a specified number of
them successfully terminate.
is signaled via a Job
ASSEMBLING FLINK CLUSTERS

52. Kubernetes
Operators

53. software extensions to Kubernetes that make use of custom
resources to manage applications and their components.
(The Internet)
Operators are...
KUBERNETES OPERATORS

56. Human VS K8s
manages a service or a
set of services
KUBERNETES OPERATORS
manages a service or a
set of services
Kubernetes OperatorHuman Operator

57. Human VS K8s
manages a service or a
set of services
KUBERNETES OPERATORS
manages a service or a
set of services
Kubernetes OperatorHuman Operator
has deep knowledge of
how the system is
expected to behave
has deep knowledge of
how the system is
expected to behave

58. Human VS K8s
manages a service or a
set of services
KUBERNETES OPERATORS
manages a service or a
set of services
Kubernetes OperatorHuman Operator
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
has deep knowledge of
how the system is
expected to behave
knows how to deploy it

59. Human VS K8s
manages a service or a
set of services
KUBERNETES OPERATORS
manages a service or a
set of services
Kubernetes OperatorHuman Operator
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
knows how to react if
there are problems
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
knows how to react if
there are problems

60. Human VS K8s
manages a service or a
set of services
KUBERNETES OPERATORS
manages a service or a
set of services
Kubernetes OperatorHuman Operator
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
knows how to react if
there are problems
automates repetitive
tasks
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
knows how to react if
there are problems
uses automation for
repetitive tasks

61. Human VS K8s
manages a service or a
set of services
KUBERNETES OPERATORS
manages a service or a
set of services
Kubernetes OperatorHuman Operator
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
knows how to react if
there are problems
automates repetitive
tasks
has deep knowledge of
how the system is
expected to behave
knows how to deploy it
knows how to react if
there are problems
uses automation for
repetitive tasks
can only manage a
limited number of
instances
can manage a very
high number of
instances

64. Flink Custom
Resource
Declarative model
Model the configuration and the deployment of a Flink
cluster
KUBERNETES OPERATORS

65. Flink Custom
Resource
Declarative model
Model the configuration and the deployment of a Flink
cluster
State representation
Used by the operator to keep track of the state of any
Flink cluster
KUBERNETES OPERATORS

66. Flink Custom
Resource
Declarative model
Model the configuration and the deployment of a Flink
cluster
State representation
Labels and Annotations
Used by the operator to keep track of the state of any
Flink cluster
Used for selecting the components to update or to signal
that the user requested a shutdown
KUBERNETES OPERATORS

69. Flink Dashboard is accessible via an Ingress rule
ASSEMBLING FLINK CLUSTERS

70. Flink Dashboard
Ingress
Exposes HTTP and HTTPS routes from outside the
cluster to services within the cluster
is accessible via an Ingress rule
ASSEMBLING FLINK CLUSTERS

71. Flink Dashboard
Ingress
Exposes HTTP and HTTPS routes from outside the
cluster to services within the cluster
is accessible via an Ingress rule
ASSEMBLING FLINK CLUSTERS
Ingress Controller
Ingresses and ingress rules are managed by their own
“operator”

73. Yelp PaaSTA

74. YELP PAASTA
PaaSTA is...
a highly-available, distributed system for building, deploying, and
running services using containers and Apache Mesos.
(Yelp)

75. YELP PAASTA
PaaSTA is...
a highly-available, distributed system for building, deploying, and
running services using containers and Apache Mesos Kubernetes.
(Yelp)

76. YELP PAASTA
Why integrating? Consistent interface
Every PaaSTA user knows how to interact with any
service regardless of its nature

77. YELP PAASTA
Why integrating? Consistent interface
Every PaaSTA user knows how to interact with any
service regardless of its nature
Infrastructure as a Service
Whether it is a Web server, a Cassandra cluster or a
Flink job, to the user everything is a service

78. YELP PAASTA
Why integrating? Consistent interface
Every PaaSTA user knows how to interact with any
service regardless of its nature
Infrastructure as a Service
Platform engineers are users too
Whether it is a Web server, a Cassandra cluster or a
Flink job, to the user everything is a service
Shared infrastructure and tools are exposed as
services, libraries and CLIs to platform developers

81. main:
job_type: stateful
checkpoint_interval_ms : 30000
deploy_group: prod
taskmanager:
cpus: 2.0
mem: 10G
instances: 3
checkpoint_path : s3://flink-state/service/main/checkpoints
savepoint_path : s3://flink-state/service/main/savepoints
flink_conf:
taskmanager.network.detailed-metrics : "true"
env.java.opts.taskmanager : "-XX:+UseConcMarkSweepGC"

82. main:
job_type: stateful
checkpoint_interval_ms : 30000
deploy_group: prod
taskmanager:
cpus: 2.0
mem: 10G
instances: 3
checkpoint_path : s3://flink-state/service/main/checkpoints
savepoint_path : s3://flink-state/service/main/savepoints
flink_conf:
taskmanager.network.detailed-metrics : "true"
env.java.opts.taskmanager : "-XX:+UseConcMarkSweepGC"
Custom
Resource
Definition

88. YELP PAASTA
User Interaction Check status
paasta status -s service -i instance -r region

89. paasta logs -s service -i instance -n 100
YELP PAASTA
User Interaction Check status
paasta status -s service -i instance -r region
Read logs

90. paasta logs -s service -i instance -n 100
YELP PAASTA
User Interaction Check status
paasta status -s service -i instance -r region
Read logs
Deploy a new version
Different UX and infrastructure from the rest of Yelp led
to high barrier to entry and knowledge impedance
git commit && git push origin master

91. The Future

92. THE FUTURE
Python
on Beam
on Flink
on Kubernetes

93. THE FUTURE
Pipeline
Builder

94. THE FUTURE
Pipeline
Builder

95. THE FUTURE
Pipeline
Builder

96. THE FUTURE
Pipeline
Builder

97. THE FUTURE
Pipeline
Builder

98. THE FUTURE
Pipeline
Builder

99. What’s next Job Oriented Deployment
More isolation, faster restarts and simpler deployment
by running a single job per Flink cluster
THE FUTURE

100. What’s next Job Oriented Deployment
More isolation, faster restarts and simpler deployment
by running a single job per Flink cluster
Reactive Container Mode and Autoscaling
Flink will automatically react to new resources available
in K8s by rescaling the job (FLINK-10407)
THE FUTURE

101. What’s next Job Oriented Deployment
More isolation, faster restarts and simpler deployment
by running a single job per Flink cluster
Reactive Container Mode and Autoscaling
Thinner Supervisor
Flink will automatically react to new resources available
in K8s by rescaling the job (FLINK-10407)
Move savepoints, jobs lifecycle and configuration
management from the Supervisor to the Operator
THE FUTURE

102. Should I do it?

103. Let’s do it!
SHOULD I DO IT?
O(1) people for O(N) clusters
A K8s operator allows you to scale up your number of
Flink clusters without adding more human operators

104. Let’s do it! O(1) people for O(N) clusters
A K8s operator allows you to scale up your number of
Flink clusters without adding more human operators
Operators to codify knowledge
Codifying operational knowledge is easier than passing
it all down to new hires
SHOULD I DO IT?

105. Let’s do it! O(1) people for O(N) clusters
A K8s operator allows you to scale up your number of
Flink clusters without adding more human operators
Operators to codify knowledge
A catalyst for users
Codifying operational knowledge is easier than passing
it all down to new hires
Once integrated with your platform, users don’t have to
learn how to deploy or configure a Flink job anymore
SHOULD I DO IT?

106. Or maybe not The Kubernetes Tax
Embedding Kubernetes into your platform requires a
pretty solid effort, if you haven’t done it yet
SHOULD I DO IT?

107. Or maybe not The Kubernetes Tax
Embedding Kubernetes into your platform requires a
pretty solid effort, if you haven’t done it yet
(Build ∨ Buy) → Time
It takes some time to write your own operator or to fit
an existing one into your platform
SHOULD I DO IT?

108. Or maybe not The Kubernetes Tax
Embedding Kubernetes into your platform requires a
pretty solid effort, if you haven’t done it yet
(Build ∨ Buy) → Time
It takes some time to write your own operator or to fit
an existing one into your platform
SHOULD I DO IT?
There is always the cloud
Cloud providers are starting to offer managed platforms
based on Kubernetes operators

109. www.yelp.com/careers/
We're Hiring!

110. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp

111. Questions/Suggestions?
antonio@yelp.com

112. Thank you.