1. 1
Open source SD-WAN
One man’s decent into Madness
Dave Phelan - APNIC

2. 2
Who Am I?
• Dave Phelan
– Network and Infrastructure engineer for a LONG time
– Trainer at APNIC
– Parent to 2 Human children and 2 Fur Children
– Likes Cat memes
• [Date[
• [xx]

3. 3
3
What are we going to talk about?
• Why am I talking about ANOTHER overlay network?
• What is SD-WAN?
• What are the “standards” for this?
• What are my FOSS options?
• How do I do it?
• Should I do it (or what problem am I solving)?

4. 4
Why am I talking about this?
• Post training surveys
– Most requested content
• Finding solutions that don’t have a vendor lock in
– This is HARD…Or is it?
• Like it or not, SD-WAN is being deployed

5. 5
What is SD-WAN
• Defined in MEF-70(07/2019) Updated MEF-
70.1(11/2021)
• SD-WAN Is a Virtual Overlay Network
• Operates over one or more underlay (Layer 3)
services
• Centralised Management and Orchestration
– Usually via a Vendor Portal
• Provisions for Flexible routing
– Application based routing (YMMV)
– Load balanced/Preferred/failover etc

6. 6
What is SD-WAN – Components - 1
• Underlay Network
– The network that our SD-WAN sits over the top of.
– Can be any form of connectivity as long as we have L3
• LTE/Ethernet/MPLS/Commodity BB etc
• Overlay Network
– Virtual Tunnels for our SD-WAN Network
• SD-WAN Edge Device
– Serve as endpoints for connectivity to the Virtual Fabric
– Encapsulate and forward the traffic based on Policies

7. 7
What is SD-WAN – Components - 2
• SD-WAN Controller
– Manages and Orchestrates the Overlay Network
– Policy/routing definition is done here
• Management and Orchestration
– UI into the controller
– Allows for configuration of our Edges and creation of policy/routing

8. 8
What is SD-WAN – Components - 3
MEF-70.1 Page 13

9. 9
What are the Standards?
• MEF-70.1
– It defines the components, features, and Framework
• https://www.mef.net/resources/mef-70-1-sd-wan-service-attributes-and-service-
framework/
– Vendor Interop is questionable(non-existent)
• You have to drink the kool-aid
– It covers off the “Do” and “Don’t”, “Must”, and “Should”
– It’s all about the Subscriber and Supplier
• Not so much about the “how”

10. 10
What are my FOSS Options?
• There are now many options
– Zero-Tier
– Headscale(Based on Tailscale)
– Flexiwan
– Zevenet
– VyOS
– Others……
• BUT They all have drawbacks
– Limited Options for self-hosted controller/UI
– Still broken interop
– Not all SD-WAN Features are implemented
– Some still require you to create an account(phone home)

11. 11
How do I do it?
• Choose an Open source Option
• Install the required software on your network devices
– This is where the problems start
• Configure your routing policy
– This is where more problems occur
• Join your network devices to your Virtual Network
• Magic Magic…
• Packets go from A to B

12. 12
How DID I do it?
• Problem 1 – Network Hardware
– Low Cost, but flexible
– SD-WAN parts already there
• or that I can Modify
• Problem 2
– Which FOSS solution to use?
– Does my SW Choice drive my hardware or Visa-Versa
– What features am I missing?
– What can I do without?
– I don’t want to have to create a login with a Vendor!

13. 13
How DID I do it?
• Hardware/Network OS
– Mikrotik ROS7
– ARM Hardware has options for Zero-Tier by additional NPK
– X86 (CHR) supports docker containers
• Custom Rolled my own Docker ZeroTier Container
– More to come on this…
• Software
– ZeroTier (https://www.zerotier.com/)
– Many Deployment options
• Clients for Windows/Mac/Android/iPhone/Linux
– Can Be run as a docker container
– Doesn’t need to connect to the Mother ship (Planet servers)

14. 14
How DID I do it?
• Other Options could be
– OpenWRT
– Teltonika
– Protectli(Running OpenWRT)
• Still investigating these Options
• Still Investigating the other Software as well

15. 15
How DID I do it?
• Caveats
– This Method breaks the ties to the ZeroTier Roots
• You CAN’T do this if you run Android/iOS clients
• You CAN do this if you are running a docker/linux/wrt image
• IF you want to use Android/iOS clients, you will need create a ZT login and
NOT REMOVE/DISABLE the planets
– Packet Processing is done in CPU
• No HW offload

16. 16
How DID I do it?
• Challenges
– Primarily for a LAB
– Finding a good UI for the users
– Emulating as MUCH functionality as possible
• At what point do I “Draw the line”
• Still building some of this

17. 17
How DID I do it?
• Some Zero-Tier Terminology
– Planet
• Zero-Tier Root Servers
– Moon
• User Defined Root Server
– Leaf
• SD-WAN Endpoints
• Controllers

18. 18
How DID I do it?
• ZT Docker image
– Unable to run an ARM image as a VM
• Had to go x86(CHR)
• Sits off to the side of the rest of the routing engine
– Missing some tooling
• jq – Parsing JSON queries from the Mikrotik API
• curl – execute the API queries
– Preinstall my “Moon” files
• Still unsure if I can even do this on ARM_64 MT
– Based on the original image
• https://hub.docker.com/r/zerotier/zerotier

19. 19
How DID I do it?
• Step 1
– Create some new ROOT servers
• At least 2 Recommended
• Tooling is built-in to do this
– https://docs.zerotier.com/zerotier/moons/
• Step 2
– Block access to the planet servers
• IPTABLES rules/firewall rules should be sufficient
• Step 3
– Install the “MOONS” on your client nodes
• Details included in the above

20. 20
How DID I do it?
• Step 4
– Install a Node to use as a controller
• Step 5
– Choose a GUI
• https://github.com/dec0dOS/zero-ui
• https://github.com/key-networks/ztncui
– They have their Pros and Cons
• Step 6
– Setup your networks and Join your clients
– Configure any routing required on your end nodes

21. 21
Should I do it?
• What Problem am I trying to solve?
– Cost?
– Service Availability?
– Splitting services?
– User Self-Management?
• They All have different answers
– Can I do it another way?
– Will a standard VPN do the same thing?

22. 22
Should I do it?
• Use Case #1
MEF-70.1 Page 98

23. 23
Should I do it?
• Use Case #2
MEF-70.1 Page 99

24. 24
Demo Topology

25. 25
Quick 2 Minute Demo

26. 26
Questions?