In this talk, we will go over the configurations knobs and all the virtual isolation features to run multiple clients in their assigned resource space without interfering with one another. At nutanix we run a single physical pulsar cluster shared by multiple use cases and applications. The biggest challenge in such a setup is an unintended denial of service for all client just because one of the clients breached their quotas and exceeded load expectations. In traditional distributed apps, one would do that with multiple physical clusters which is easier on day 1 but the operational complexity on day 2 or 3 can very quickly overwhelm a small team. Adding to its multi-tenant architecture, Pulsar provides a lot of features to restrict usage from different clients ranging from quotas, ttls, retention with simple configurations to more advanced features in namespace isolation, failure domains, anti affinity for namespaces, bookie groups and affinity between those etc. In this talk we will first present the available options followed by our experience with them. As a bonus we will add a story where we migrated our whole cluster and the learnings in the journey.

1. 1

2. Who are We?
● Leading Cloud data Platform Team
● Loves distributed Systems, open Source
● Data geek (data stores, stream, analytics etc)
● Pulsar & MySql contributor
● Developer by passion
● Love Distributed Systems, Streaming Platform
● Pulsar Enthusiast
Shivji Jha
SME for Pulsar,
Nutanix
Sourabh Agrawal
Pulsar Ninja,
Nutanix
2

3. Catalogue
● Quick Intro: Apache Pulsar
● Pulsar Isolation
○ What
○ Why
○ How
● Isolating by clusters
● Isolating brokers
● Isolating bookies
● Demo
3

4. About Pulsar
Apache Pulsar is a cloud-native, distributed, open-source pub-sub messaging and streaming
platform. Originally developed by Yahoo and contributed to the Apache Software Foundation in
2016
Provides :
● Stateless Brokers
● Horizontally Scalable
● Isolate Read and Writes
● Multi-tenant System
● Geo-Replication
● Active Community Support
4

5. Multi-tenancy
5

6. What is Pulsar Isolation ?
● Scale storage and Serving layer independently.
● Can separate resources (brokers & bookies) for specific use cases.
● Scale up/down resources as per need for specific namespaces by keeping rest of the
cluster untouched.
● Localized resource allocation
6

7. Why Pulsar Isolation ?
● Avoid Sharing resources within teams
○ Prevent unexpected consequences.
● Dedicated resources for a namespace.
● Allocate Localized resource in multi-region setup.
● Scale up/down nodes as per load on a namespace.
● More Secure (Permissions managed at namespace level)
● Easy setup
7

8. Isolation Categories
● Hard Isolation
○ Different physical Pulsar clusters (or nodes) for your isolation units
■ Failure Domains
■ Anti affinity Groups
■ Broker Isolation using Namespace Isolation Policy
■ Bookie Isolation using Affinity Groups
● Soft Isolation
○ Same Physical clusters but restrictions on tenants / namespaces by configuration
■ Disk Quotas
■ Throttling
8

9. Isolating by Configuration
9

10. Soft isolation
● Limit by configurations
○ Namespace:
● encryptionRequired
● Auth_policies
● Backlog_quota
● retention
■ Topic:
● maxProducersPerTopic
● maxConsumersPerTopic
● maxSubscriptionsPerTopic
● Limits by resources (capped per broker):
■ loadBalancerBrokerMaxTopics
■ loadBalancerBrokerOverloadedThresholdPercentage
10

11. Isolating by Cluster
11

12. Geo Replication
eu-west-2 eu-west-3
Geo
Replication
On
12
New Hotel company starts operating in London
Expand business to Paris
Enable HA

13. Isolating Brokers
13

14. 14
https://github.com/apache/pulsar/wiki/PIP-7:-Pulsar-Failure-domain-and-Anti-affinity-namespaces
● Failure Domain
○ Group brokers into failure domains
○ Reduces blast radius
○ Example
■ Fix a bug / patch
■ Upgrade brokers by failure domain
■ Anti affinity of namespaces
Failure Domains

15. 15
https://github.com/apache/pulsar/wiki/PIP-7:-Pulsar-Failure-domain-and-Anti-affinity-namespaces
● Anti affinity Group
○ Group of namespaces that should stay far.
○ Best effort
■ Prefer different failure domain
■ Prefer different broker
Anti affinity Groups

16. 16
What is “Namespace Isolation Policy”?

17. Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
17

18. Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
18

19. Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
19

20. Namespace Isolation Policy: Internals
min_availability=1
Secondary = Broker 5
Primary = Broker 1,2
20

21. Namespace Isolation Policy: Setup
Configure Isolation Policy
Unload the namespace to take effect for existing topics in namespaces or restart brokers.
./pulsar-admin ns-isolation-policy set --auto-failover-policy-type min_available
--auto-failover-policy-params min_limit=?,usage_threshold=80 --namespaces <tenant/namespace>
--primary broker-node-1,broker-node2 my-cluster <policy-name> --secondary broker-node-5
./pulsar-admin namespaces unload <tenant/namespace>
./pulsar-admin topics lookup <tenant/namespace/topic>
21

22. Isolating Bookies
22

23. Bookie Affinity Groups
23
● Isolate storage at namespace
● Leverage data durability configs in
isolated bookies
● Leverage
ZkIsolatedBookieEnsemblePlacement
Policy
● Scale bookies for namespace
https://streamnative.io/uploads/images/blogs/isolation-3.png

24. Bookie Isolation using
Affinity Group
Setup Bookie Rack
Bookie Affinity Group
./pulsar-admin bookies set-bookie-rack --bookie 127.0.0.1:3181 --hostname 127.0.0.1:3181
--group group1 --rack rack1
./pulsar-admin namespaces set-bookie-affinity-group <tenant/namespace> --primary-group
<anti-affinity-group-name> --secondary-group <anti-affinity-group-name>
./pulsar-admin namespaces set-bookie-affinity-group public/default --primary-group group1
24
Configure Bookie Group
Configure Bookie Group with secondary

25. Scaling Up/Down
Broker
● When scaling up brokers, change the isolation groups and add newly added broker as
primary or secondary broker for required namespaces.
● When scaling down, make sure existing isolation groups have enough brokers.
Bookie
● When scaling up bookies, change the bookie affinity group and add newly added bookies as
primary or secondary group for required namespaces.
● When scaling down, make sure existing affinity groups have enough bookies.
25

26. Demo : Namespace Isolation Policy
26

27. References
Namespace Isolation: https://pulsar.apache.org/docs/en/administration-isolation/#broker-isolation
RackAware Ensemble Policy:
https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePl
acementPolicy.html
Bookie Affinity Group:
https://pulsar.apache.org/docs/en/administration-isolation/#bookie-isolation,
https://github.com/apache/pulsar/pull/3933
Crash course on pulsar :
https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
Soft Isolation:
https://streamnative.io/en/blog/tech/2021-03-02-taking-an-in-depth-look-at-how-to-achieve-isolat
ion-in-pulsar/
27

28. Questions?
Pulsar Slack
https://apache-pulsar.slack.com/
Pulsar Mail Group:
users@pulsar.apache.org
dev@pulsar.apache.org
LinkedIn :
https://www.linkedin.com/in/sourabhaggrawal/
https://www.linkedin.com/in/shivjijha/
28