View IT operations as a flow of data (Sources of Truth) thru work-cells (automation processes) to deliver value to the customer. There should be only one source of truth for every piece of configuration data. Device configurations are poor source of truth.

1. Copyright © 2018 World Wide Technology, Inc. All rights reserved.
Super-NetOps Source of Truth
August 2018
Joel W. King Engineering and Innovations
Network Solutions

2. Goal

3. Links and Bio
Slides and demo playbooks are available at
github.com/joelwking/supernetops
www.linkedin.com/in/programmablenetworks/
Joined WWT: August 2013 based in Research Triangle Park, NC
Prior Work History
AMP Incorporated, Network Architect
Cisco, Developed Cisco Validated Designs (CVDs)
NetApp, Big Data: Video Surveillance Storage
Education
BBA Temple University
CCIE 1846 (ret.)
FUN FACT
Inducted to the Phantom Cyber Hall of Fame

4. Agenda
• Goals and Overview
Deliver value to your customers
• Inventory
Identify your assets
• Source(s) of Truth
Configuration inputs come from many sources
• Demonstration
Update firewall ACLs to support LTM VIPs

5. Goals and Overview

6. Traditional Flow of Work
CLI
NETWORK
ENGINEER
TICKETING
SYSTEM
RUN BOOK
SERVICE LEVEL AGREEMENT
PROCESS TIME
RUN BOOK
SERVICE LEVEL AGREEMENT
PROCESS TIME
SERVICE LEVEL AGREEMENT
PROCESS TIME
SERVICE LEVEL AGREEMENT
PROCESS TIME
REQUEST
COMPLETE
GUI
WAIT
TIME
WAIT
TIME
WAIT
TIME
WAIT
TIME
A
P
P
R
O
V
A
L
S

7. Super-NetOps Flow of Work
F5
ADC & SECURITY
No SQL
Nexus 9000
ACI
VARIABLE(S)
YAML , CSV
PLAYBOOKS
ACI
IPAM
Source Control
System
ASA
PALO ALTO
CHECKPOINT
NETWORK
CONFIGURATION DATA
Inventory
TICKETING
SYSTEM
CHATOPS
INFRASTRUCTURE

8. Source of Truth
http://blog.ipspace.net/2017/01/device-configurations-are-not-good.html
• One source of truth for configuration data
 IPAM data for hostnames, IP addressing
• Data should be programmatically consumable
 Application Program Interface (API)
 URL (Git Repo)
• Device configuration generated from templates
 Jinja Templating
• Global configurations
 Version controlled YAML file or simple object-oriented database
• ChatOps to capture organizational knowledge
• Automate change verification, metrics, back out plan

9. Goals and Overview: Structured Data

10. Journey
TRIBAL KNOWLEDGE
CSV EXCEL
STRUCTURED
DATA
YAML, JSON, XML
IPAM
SERVICENOW
BMC REMEDY
OPEN SOURCE
NSOT NETBOX
RELATIONAL DB
NOSQL DB
CHATOPS
VERSION
CONTROL
SYSTEM

11. Unstructured Data
CLI is not Structured Data
http://alsa21.blogas.lt/tag/data-center-solution
ASA-5585-99543# show run
: Saved
:
: Serial Number: JAD1742009H
: Hardware: ASA5585-SSP-60, 24576 MB RAM, CPU Xeon 5600 series 2400
MHz, 2 CPUs (24 cores)
:
ASA Version 9.2(4)14
!
hostname ASA-5585-99543
domain-name sandbox.wwtatc.local
!
interface TenGigabitEthernet0/9
shutdown
no nameif
no security-level
no ip address
!
ftp mode passive
dns server-group DefaultDNS
domain-name sandbox.wwtatc.local
pager lines 24

12. Data Serialization Formats
Pass Structured Data between Sources and Sinks
JSON
Java-Script Object Notation
XML
eXtensible Markup Language
YAML
YAML Ain't Markup Language
{ "imdata":
{
"firewall":
{"group": "rtp_wan_edge",
"acl_name": "INBOUND“
},
"tags": [ "white", "roan", "flint"]
}
}
<?xml version="1.0" encoding="UTF-8" ?>
<imdata>
<firewall>
<group>rtp_wan_edge</group>
<acl_name>INBOUND</acl_name>
</firewall>
<tags>white</tags>
<tags>roan</tags>
<tags>flint</tags>
</imdata>
imdata:
firewall:
group: rtp_wan_edge
acl_name: INBOUND
tags:
- white
- roan
- flint

13. Data Modeling
Data modeling is the process of documenting a complex software system design as an easily understood
diagram, using text and symbols to represent the way data needs to flow…
A data model can be thought of as a flowchart that illustrates the relationships between data.
Super-NetOps Engineer

14. Inventory

15. Inventory
If you can’t measure it, you can’t manage it.
Peter Drucker
management consultant, educator, and author
Many companies struggle to obtain an
accurate inventory of devices in the network.

16. Sources of Inventory
DISCOVERY
MANUAL
admin@flint:~$
NMAP
admin@flint:~$
AWS EC2
VMWARE_FACTS
DISCOVERY
ENRICHMENT
admin@flint:~$
query by
serial no.
CISCO TETRATION
admin@flint:~$

17. Service Now - Inventory source
github.com/ServiceNowITOM/ansible-sn-inventory
“Discovery finds computers, servers, printers, and a variety of IP-enabled devices, and the
applications that run on them. It can then update the CIs in your CMDB with the data it
collects.”
administrator@flint:~/ansible/playbooks$ ansible-playbook debug.yml -i ./inventory/now/now.py --list-hosts
playbook: debug.yml
play #1 (all): debug TAGS: []
pattern: [u'all']
hosts (6):
f5-demo-test1.sandbox.wwtatc.local
f5-egg-prod1.sandbox.wwtatc.local
f5-egg-qa2.sandbox.wwtatc.local
f5-demo-test2.sandbox.wwtatc.local
f5-egg-prod2.sandbox.wwtatc.local
f5-egg-qa1.sandbox.wwtatc.local
table = 'cmdb_ci_server‘
table = 'cmdb_ci_ip_router'
table = 'cmdb_ci_datacenter'
table = 'cmdb_ci_lb'
FOR YOUR REFERENCE

18. Sources of Truth: IP Address Management (IPAM)

19. • One source of truth for configuration data
 IPAM data for hostnames, IP addressing
• … but what about Ephemeral environments?
 COT (the Common OVF Tool) is a tool for editing OVA/OVF
https://cot.readthedocs.io/en/latest/introduction.html
BIG-IP version 13.1+ Static MGMT IP addresses
 Ansible module vsphere_guest doesn’t return
DHCP IP addresses
 Use vmware_vm_facts
IP Address Management (IPAM)
www.ansible.com/f5

20. IP address of a virtual edition BIG-IP
vars:
target: 'f5-demo-test1.sandbox.wwtatc.local'
tasks:
- name: Query vCenter for management IP address
vmware_vm_facts:
hostname: "{{ vcenter.server }}"
username: "{{ vcenter.username }}"
password: "{{ vcenter.password }}"
validate_certs: no
register: vm_facts
- debug:
msg: "{{ target }} {{ vm_facts.virtual_machines[target] | to_nice_json(indent=4) }}"
when: vm_facts.virtual_machines[target] is defined
- assert:
msg: "Invalid or no IP address"
that: vm_facts.virtual_machines[target].ip_address | ipaddr
TASK [debug] *****************
ok: [localhost] => {}
MSG:
f5-demo-test1.sandbox.wwtatc.local {
"esxi_hostname": "10.255.40.137",
"guest_fullname": "Other (64-bit)",
"ip_address": "10.255.111.171",
"mac_address": [
"00:50:56:af:94:b4",
"00:50:56:af:5c:06",
"00:50:56:af:ce:56",
"00:50:56:af:f1:10"
],
"power_state": "poweredOn",
"uuid": "422ffb59-42cd-859f-4930-91b46e3a6134",
"vm_network": {}
}
FOR YOUR REFERENCE
vmware_vm_facts return DHCP IP address of F5 VE BIG-IP

21. Update inventory with DHCP assigned IP addrs
https://docs.ansible.com/ansible/latest/modules/add_host_module.html
Use variables to create new hosts and groups in inventory for use in later plays of the
same playbook.
PLAY 1
PLAY 2

22. Sources of Truth: Database

23. Technology Comparison
Relational Databases
Data stored in tables
NoSQL Databases
Data stored in collections
of independent objects
Use SQL, Structured Query Language,
English like, easy to create queries
Strong Consistency
Common usage means easy
integration with enterprise systems
Flexible: dynamic schema, fields can be added
to documents
Scalable and high performance
Always-on for global deployments
MySQL
MariaDB
PostgreSQL
Amazon DynamoDB
Redis
Apache CouchDB

24. Under the covers
PostgreSQL as its database. This remote PostgreSQL can be a
server you manage, or can be provided by a cloud service
such as Amazon RDS.
ServiceNow has also moved from MySQL to MariaDB when
opting for new instances/ Helsinki Release.

25. MongoDB Compass and shell
FOR YOUR REFERENCE

26. Sources of Truth: Demo

27. Firewall Changes to support F5 VIP(s)
MID server
MongoDB
1
2
3
4
7
Retrieve configuration data from MongoDB
Apply firewall changes
Retrieve running configuration
Create documentation of the change
request and running configuration
5
6Store the updated
configuration artifacts
Update ticket with database ObjectId
Initiate playbook passing ObjectId, of database and collection
of configuration data
playbook

28. Requirements for Configuration Management
RFC 3139
… provide expiration time and effective time capabilities to
configuration data. It is required that some configuration
data items be set to expire, and other items be set to never expire …
BIGIPS SUPPORTING VIPS
SUPPORTING FIREWALL(S)
WHEN DOES IT EXPIRE?
CHANGE WINDOW TO IMPLEMENT

29. Service Delivery
CHANGE REQUESTAUDITDATA
PROCESS
INFRASTRUCTURE
Super-NetOps is the practice of
delivering network-based services as
programmable infrastructure to support
DevOps practices and methodologies.

30. Audit Database
 Compare
configuration
change(s) across all
firewalls in the group,
for a given change
request
 Walk configuration
changes for each
firewall over time

31. Reporting from an Audit Database
ServiceNowFirewall ACLs | VIPS

32. https://www2.wwt.com/all-blog/super-netops-source-truth/

33. Key-takeaways
 View IT operations as a flow of data (Sources of
Truth) thru work-cells (automation processes) to
deliver value to the customer.
 There should be only one source of truth for every
piece of configuration data.
 Device configurations are poor source of truth.
Slides and demo playbooks are available at github.com/joelwking/supernetops

35. Reference Material

36. Service Now as Inventory Source
'cmdb_ci_lb’ DYNAMIC
INVENTORY
f5_drift.yml
when: item.version != desired_version
https://github.com/joelwking/supernetops/blob/master/Agility2018/playbooks/f5_drift.yml

37. Firewall Groups
Managed independently from the service request
INBOUND
INBOUND
"firewall": {
"group": "rtp_wan_edge",
"acl_name": "INBOUND"}
rtp_wan_edgertp_wan_edge:
hosts:
csr1000v-1.sandbox.wwtatc.local: {}
csr1000v-2.sandbox.wwtatc.local: {}
inventory.yml
FOR YOUR REFERENCE

38. Resources
 Toyota Production System
http://missiontps.blogspot.com/p/14-principles.html
https://www.slideshare.net/akshayjain186590/opc-tps
 From Zero to Network Programmability in 120 minutes (CiscoLive 2018 BRKNMS-2935)
https://clnv.s3.amazonaws.com/2018/usa/pdf/BRKNMS-2935.pdf
 Introduction to NoSQL
https://resources.mongodb.com/getting-started-with-mongodb/back-to-basics-1-introduction-to-nosql
 PyMongo Tutorial
http://api.mongodb.com/python/current/tutorial.html
 REQUIREMENTS FOR CONFIGURATION MANAGEMENT OF IP-BASED NETWORKS
HTTPS://TOOLS.IETF.ORG/HTML/RFC3139
SEE 3.0.9 – SET EXPIRATION DATE FOR CONFIGURATION ELEMENTS

39. https://www2.wwt.com/all-blog/devnet-create-2018/