These slides were used during a meetup in Wellington, hosted by Catalyst IT. Pani Networks presented their Romana project: Cloud native, pure L3 networking for OpenStack and Kubernetes clusters.

1. Simplifying the network stack with
Romana
Pani Networks
OpenStack / Kubernetes Meetup, Wellington, May 2016

2. romana.io Simplifying the network stack with Romana @romanaproject
Agenda
● “Cloud native”, why does it matter?
● A better network for cloud native architectures
● Demos

3. romana.io Simplifying the network stack with Romana @romanaproject
About us
● Team background:
– Data center networks
– Low-level traffic management
● Created L2 overlay network startup
– Bought by Cisco
● OpenStack networking
● There's got to be a better way
– Time is right

4. What is 'cloud native'?

5. romana.io Simplifying the network stack with Romana @romanaproject
The past: Enterprise networking
● Full control
● Applications need L2 and L3
– May need hard-wired IP addresses
– Broadcasts
● Servers are pets, not cattle: “Careful!”
– VM migration
● Complex!

6. romana.io Simplifying the network stack with Romana @romanaproject
Cloud native applications
● Automate all the things!
– Infrastructure as code
– Cattle, not pets: “Meh... just kill it.”
– Workloads come and go quickly
– Build for resiliance
● IP is all you need
– No hardcoded IP addresses, discovery
– No special network requirements
– Basic IP connectivity

7. The problem

8. romana.io Simplifying the network stack with Romana @romanaproject
We have a mismatch
● Building cloud native applications…
● … on top of enterprise networking
– SDN controllers use overlay L2 domains
– VLAN, VXLAN, OVS, etc.
● Complexity and brittleness
– Lose benefits of simplicity
– Lose performance (encap, blinded hardware)
– Difficult to maintain and trouble shoot

9. romana.io Simplifying the network stack with Romana @romanaproject
The price you pay: Complexity
VXLAN Decap
VXLAN Decap
VXLAN Encap
VXLAN Encap
2 Top of Rack Round
Trips
East/West Traffic
Per Instance Security

10. romana.io Simplifying the network stack with Romana @romanaproject
The price you pay: Performance
Router
Endpoint A Endpoint B
Router
L2 overlay A
L2 overlay B
VRouter

11. romana.io Simplifying the network stack with Romana @romanaproject
Why do we do this to ourselves?
● We don't need any L2 features
● Except maybe traffic segmentation
– Multi tenancy
– Tiers and policies

12. The solution

13. romana.io Simplifying the network stack with Romana @romanaproject
Networking the way it was intended
● Use native L3 capabilities
● No overlays
● De-emphasize IP address ranges
● Still provide segmentation, multi tenancy
● Simple, clear and scalable network setup

14. romana.io Simplifying the network stack with Romana @romanaproject
Truly cloud native networking
● Project Romana
● Open source
● Apache 2.0 license
● Mostly written in Go
● Kubernetes and OpenStack

15. romana.io Simplifying the network stack with Romana @romanaproject
Truly cloud native networking
● Use only IP routing
– No overlays
– All workload addresses are 'real'
– Simplicity!
● Use smart addressing
– Encode tenant or segment in IP address
– Assign “virtual” addresses with host prefixes
– Massive (!) collapse of route table
● Routes are static
– No route updates, no broadcasts for new endpoint

16. romana.io Simplifying the network stack with Romana @romanaproject
Romana Architecture
● On each host: Agent
– Configures routes
– Connects endpoint interfaces
– Sets policy implementations
● Controller: Cooperating microservices
– Each service with RESTful interface
– Specialized for different tasks
● Environment: Different integration points
– APIs, drivers for various parts of OpenStack or
Kubernetes

17. romana.io Simplifying the network stack with Romana @romanaproject
Romana Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
Environment (OpenStack or Kubernetes)
Policy

18. Beautifully simple networking

19. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
Host B
eth0:
192.168.8.22
Host C
eth0:
192.168.8.33

20. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16

21. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6

22. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6

23. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6

24. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Routes:
10.1/16 → 192.168.8.22
10.2/16 → 192.168.8.33
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Routes:
10.0/16 → 192.168.8.11
10.2/16 → 192.168.8.33
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
Routes:
10.0/16 → 192.168.8.11
10.1/16 → 192.168.8.22

25. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
Rack A Rack B

26. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10

27. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
Host A2 Routes
0.0.0.0 192.168.1.200→
10.68/14 192.168.1.1→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→

28. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
ToR A Routes
10.128/10 192.168.2.200→
10.68/14 192.168.1.1→
10.72/14 192.168.1.2→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→
Host A2 Routes
0.0.0.0 192.168.1.200→
10.68/14 192.168.1.1→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→

29. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: Full L3
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
ToR A Routes
10.128/10 192.168.2.200→
10.68/14 192.168.1.1→
10.72/14 192.168.1.2→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→
Host Routes
0.0.0.0 192.168.1.200→

30. Scalable distributed firewall
and
traffic policies

31. romana.io Simplifying the network stack with Romana @romanaproject
Romana: Traffic segmentation
● Tenant traffic separated:
– Tenants don't get whole CIDR prefix or L2 domain
– But fully isolated from other tenants' traffic
● Tenants can define segments:
– Like tiers, provide isolation and policies
● Use segment and tenant bits in IP addresses:
– Apply policies (iptables) based on that
– Segments can stretch across hosts

32. romana.io Simplifying the network stack with Romana @romanaproject
Semantic and topological addressing
3
1
3
0
2
9
2
8
2
7
2
6
2
5
2
4
2
3
2
2
2
1
2
0
1
9
1
8
1
7
1
6
1
5
1
4
1
3
1
2
1
1
1
0
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
The network prefix.
In this example, we
are using the 10/8
address space.
6
Host ID Segment ID
We currently
store tenant ID in
upper bits of
segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.

33. romana.io Simplifying the network stack with Romana @romanaproject
Semantic and topological addressing
3
1
3
0
2
9
2
8
2
7
2
6
2
5
2
4
2
3
2
2
2
1
2
0
1
9
1
8
1
7
1
6
1
5
1
4
1
3
1
2
1
1
1
0
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
The network prefix.
In this example, we
are using the 10/8
address space.
6
Host ID Segment ID
We currently
store tenant ID in
upper bits of
segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.
Encode the
tenant ID

34. romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Allowing traffic within tenant
10.0.0.5 10.1.0.12
iptables:
check src/dst addrs
“tenant/segment bits
must match”
Src: 10.0.0.5
Dst: 10.1.0.12
Same
tenant/segment bits

35. romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Isolating tenant traffic: Default
10.0.0.5 10.1.128.9
iptables:
check src/dst addrs
“tenant/segment bits
must match”
Src: 10.0.0.5
Dst: 10.1.128.9
Different
tenant/segment bits
Different
tenant

36. romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Apply network policy between
segments (full isolation as default)
10.0.0.5 10.1.1.9
iptables:
Does policy chain
exist?
Otherwise: DROP
Src: 10.0.0.5
Dst: 10.1.1.9
Same tenant,
different segment
policy-chain:
From segment 0?
Protocol TCP?
To port 80?

37. Demo 1:
Kubernetes + Romana cluster
on top of Catalyst OpenStack cloud

38. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview

39. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Jump host with
public IP address

40. romana.io Simplifying the network stack with Romana @romanaproject

41. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo

42. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Install OpenStack
command line tools

43. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
$ neutron port-update
e925b70e-031e-4ef7-a27c-583b4b775290
--allowed-address-pairs type=dict list=true
mac_address=fa:16:3e:e1:df:59,ip_address=10.0.0.0/8

44. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
$ git clone https://github.com/romana/romana
$ cd romana/romana-install
$ ./romana-setup -p static -i my-inventory -s kubernetes install

45. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Romana
installer

46. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Kubernetes + Romana
Romana cluster
address range:
10/8

47. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Kubernetes + Romana
Pods
with containers.
Pods have Romana
IP addresses.

48. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - What you will see
● Creation of pods
● Network configuration
● Application of network policies

49. Demo 2:
Mixing containers with legacy workloads

50. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana

51. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
vm-workload
Legacy application
in VM

52. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
vm-workload
Direct connection:
- No gateway
- No encap/decap
- No NAT

53. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - What you will see
● Creation of pods
● Contact pod from VM
● See the packet route

54. Demo 3:
Romana + Kubernetes cluster
on top of Romana + OpenStack cluster

55. romana.io Simplifying the network stack with Romana @romanaproject
Baking layered cakes
● Kubernetes on OpenStack? Why?
– On demand clusters
– Full tenant isolation
● Really nice with fully routed networking
– No double encapsulation
– Logical, efficient packet forwarding
● Not all workloads fit into containers
– Seamless connection between pods and VMs

56. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4

57. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
$ ./romana-setup -p static -i hw-inventory -s devstack install

58. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
OpenStack + Romana
Romana cluster 1
address range:
10/8

59. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
OpenStack VMs
VMs have
IP addresses
of
Romana cluster 1

60. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
$ ./romana-setup -p static -i vm-inventory -s kubernetes install

61. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3
Kubernetes + Romana
VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
Romana cluster 2
address range:
172.16/12

62. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3
Kubernetes + Romana
VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
Pods
with containers.
Pods have
IP addresses
of
Romana cluster 2

63. romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4

64. romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
Remember this one?
2 Top of Rack
Round Trips
East/West
Traffic
Per Instance
Security
Without pure L3 network
layered clusters
would be even more
complex.

65. romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
But with Romana, networking
even in layered clusters becomes
really easy...

66. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - What you will see
● Creation of pods
● Pods and VMs with fully routable addresses
● Ease of use showcase: Trouble shooting

67. romana.io Simplifying the network stack with Romana @romanaproject
Conclusion
● Cloud native architectures simplify things
● Need cloud native networking to enjoy benefits
● Romana:
– Cloud native without compromises
– Native network performance
– Mostly static config: Solid network
– Very easy to work with and understand
● Easy to try:
– Simple installers for Kubernetes and OpenStack

68. romana.io Simplifying the network stack with Romana @romanaproject
Thank you!
● Romana Links
– http://romana.io - Project home
– http://romana.io/blog - Blog
– https://github.com/romana/romana - Sources
● Contact
– @romanaproject - Twitter
– info@romana.io - Email
– https://romana.slack.com/ - Slack channel