Symantec deployed an SDN using OpenStack with the following key aspects: 1. They created different "Classes of Service" including a development environment and a production environment to onboard teams and manage workloads. 2. They provided self-service user onboarding through Horizon with automatic network creation to hide complexities. 3. They offered load balancing as a service using HA Proxy with various optimizations to achieve high performance. 4. They attached baremetal servers to the overlay network by launching them in network namespaces. 5. They aimed for over 99.95% control plane availability using a distributed controller and Cassandra setup with automation and monitoring.

1. Symantec SDN Deployment
Jasmeet Sidhu, Rudrajit Tapadar
Cloud Platform Engineering

2. Class of Service
Copyright © 2015 Symantec Corporation 2

3. Class of Service
• Dev
– For developers to get familiar with OpenStack cloud
– Each developer has a project
• Production
– For teams to onboard their members
– Each team has a project
– Manage user roles
– Manage production workloads
Copyright © 2015 Symantec Corporation
3

4. Self-Service User Onboarding
Copyright © 2015 Symantec Corporation 4

5. Self-Service User Onboarding
• Zero tickets for user onboarding
– Provide sign up capabilities on Horizon
• Provide easy networking on Dev CoS
– Hide all complexities
– Automatically create network
– Allocate routable subnets by using Contrail VNC APIs
– Create security group with proper rules
– Create unique domain names for instances by using Designate for routable
IPs
Copyright © 2015 Symantec Corporation
5

6. Load Balancer as a Service
Copyright © 2015 Symantec Corporation 6

7. Load Balancer as a Service
•Out of the box
– Icehouse, v1 APIs
– Launch HA Proxy service instances on a single AZ
– SSL Support: Wildcard cert
• Symantec fixes
–Multiple AZ, SSL Passthrough, Stats and Metrics
• Performance:
–~6.5 Gbps throughput with 10K parallel connections, VIP with 2 members
–20K HTTPS requests/sec for 10K parallel connections with 1 million requests, 1K response
size
• Tuning - haproxy.cfg: maxconn 50K, nbproc 4, ulimit-n 200K, Cipher
• Pain points
–No control over ha proxy cfg
–No control over resource allocations (cpu, etc)
Copyright © 2015 Symantec Corporation
7

8. Baremetal on Overlay
Copyright © 2015 Symantec Corporation 8

9. Baremetal on Overlay
•Applications that run on baremetal but needs to be on the
overlay
– Example: swift proxy and data nodes
– Launch them inside network namespaces
– Plug them to the vRouter
– East-West Traffic
• Manual Setup via scripts
– Nova is not aware but Contrail is.
– Multiple nics sitting on multiple networks
– Static IPs
Copyright © 2015 Symantec Corporation
9

10. Availability::Control Plane
Copyright © 2015 Symantec Corporation 10

11. Control Plane Availability
• Goal - 99.95% Availability
• 5 SDN controller VMs distributed over 3 racks
• 5 Cassandra database baremetal nodes distributed over 3 racks
– RF of 3 for analytics
– RF of 5 for config
– Compaction throughput 256 Mbps
• Deployment Automation: Puppet
• Issues seen: DB Timeouts, Version mismatch, admin token
Copyright © 2015 Symantec Corporation
11

12. Failed Customer Interactions
Copyright © 2015 Symantec Corporation 12

13. Failed Customer Interactions
• Measure the control plane availability
• Use Symantec’s Logging-Monitoring-Metering as a Service to parse Neutron logs
• Compare response codes: 5XX counted as failures
• Dashboards!
Copyright © 2015 Symantec Corporation
13

14. Availability::Data Plane
Copyright © 2015 Symantec Corporation 14

15. Data Plane Availability
• Work in progress..
–FIP Availability
–vDNS
–Link Local
–Private Network
Copyright © 2015 Symantec Corporation
15

16. Seamless Upgrades
Copyright © 2015 Symantec Corporation 16

17. Upgrade 1.20 to 2.0.1
• Goal - Zero Downtime
• Controller upgrades
– No in-place upgrades
– Build a parallel control plane with new release
– Add them to the VIP pool and gradually decommission old controllers
• Database upgrades
– Add new DB nodes one by one to the existing cluster
– Repair the DB
– Decommission old DB node one by one
• Compute upgrades
– Automate unloading and loading of kernel module in all computes
Copyright © 2015 Symantec Corporation
17

18. Health Monitoring
Copyright © 2015 Symantec Corporation 18

19. Health Monitoring
• Volta
–Logging
•Logstash
•Elasticsearch
–Metrics
•InfluxDB
•Statsd
•Collectd)
–RESTful APIs make it easy:
•Response Codes, Bytes Transfered, Time, Verb, etc.
• OpsView / Zabbix
Copyright © 2015 Symantec Corporation
19

20. Troubleshooting
Copyright © 2015 Symantec Corporation 20

21. Troubleshooting
• Most incidents are trivial
– Known issues
– Trivial fixes/workarounds
• Some incidents are complex
– RCA is very involved
– Might have to wait for next code release for a fix
– Quick and dirty solution – use auto healing scripts for workarounds
•Periodically check system health (Synthetic Transactions)
•Remediate known bugs
•Fix problems as they are detected, Save pagers, run 24x7!
(MX Encapsulation, Dead processes, etc.)
Copyright © 2015 Symantec Corporation
21

22. Thank you!
Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its
affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or
implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.