Speaker: Ramabathiran Arumugam, Sr Database Architect, Cisco Systems | Nayeem Khaja, Sr. Database Architect , Cisco

1. Ramabathiran Arumugam
Nayeem Khaja
Sr Database Administrator/Architect
Cisco Systems Inc.
MongoDB World, Jun 2018

2. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Objectives
3 Why did we choose MongoDB
4 Migration, Architecture, Benefits, Maintenance & Security
1 About Cisco and e-Commerce
5 Lesson Learned, Best Practices & What’s next
2 Challenges faced with the Platform

3. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
About Cisco and e-Commerce

4. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco at a Glance
13K+
UCS Blades
56K+
Virtual Machines
107+ PB
Overall Raw
Storage
482
Offices
94
Countries
4K+
Routers
6K+
LAN Switches
201K+
Connected
User Devices
SJC
45%
RTP
14%
AM Other
6%
EU/EM 7%
India
21%
AP Other
7%
Global
Distribution
of IT Staff
RCDN
2%

5. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco – Products & Services
Cisco Services

6. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco Commerce
(New, Follow-on and Renewal)
Customer
Experience
Web Mobile
(iOS, Android)
B2B
(Web Services, EDI/XML, Punch-outs)
Data Center /
Virtualization
Integrated
Capabilities/
Workflow
RealUserExperience/AppPerformanceMgmt
(AppDynamics,Akamai,ELK)
Configure Price Quote
Deal
Approval
Order
Capture
Order
Orchestration
Cisco Commerce Architecture with RDBMS
PassiveActive
UCS-B200 M2 UCS-B200 M2
~30 Miles
RDBMS
Delivery Platform
Web/Application Servers
Richardson, TX - DC1 Allen, TX - DC2

7. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Orders Booked Hits Per Day
300+ Services 4M Lines of Code
Unique Users
$40 B 4-6M 140K
36 Applications
Commerce Platform by numbers

8. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Challenges faced with RDBMS

9. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Challenges with the Platform
Downtime due to Infrastructure Changes/Outages.
10-12 hour downtimes for code deployment
6+ month effort for major software/hardware upgrades
Lack of agility due to monolithic code base
Performance issue due to complex code

10. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Why did we choose MongoDB ?

11. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MongoDB – Consistency
CAP
Theorem
Pick – Two
Availability
CP
APCA
MongoDB
Hbase
RDBMS
MySQL
Cassandra
CouchDB
Consistency
• All users see same data
• No Data Discrepancy
• Master Slave Architecture
Mission Critical Data
4-6 M hits 140K users 36+ Apps

12. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
1 2 3 4
Zero downtime for
software upgrades
and code
deployments
Increased Agility -
move towards API
based Architecture
5X performance
improvement &
scalability
Fault tolerance for
hardware change
with high availability
for application
Why did we choose MongoDB ?
1
Always On Performance Flexibility

13. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Migration
 Architecture
 Operational (Infra) & Application Benefits
 Maintenance & Security

14. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How did we migrate?

15. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How did we migrate?
RDBMS
Order Capture
DC1 - Tomcat
Order Capture
Addresses Items
Preferences
Roles
Contacts Logging
DC2 - Tomcat
Web/Application Servers
Transaction
s
MongoDB
Deploy new 5 node MongoDB
Replica Set
Developed & Built
Data Sync Tool
RDBMS >> MongoDB
Initial Full
Data Sync
Delta Data
Sync
Migrate Data
IO Benchmark test
Performance tuning
Order Capture
DC1 - Tomcat
Order Capture
Addresses Items
Preferences
Roles
Contacts Logging
DC2 - Tomcat
Web/Application Servers
Setup parallel Web & App Servers
Redesign Tables to JSON
collections
Rewrite RDBMS(SQL) codes to
Java
Transaction
s
Final Cutover
& Go-live

16. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Migration by the Numbers
Code
Rewrote 120 K lines
of back-end code.
Touched 600k lines
of existing java code
Brownfield
Delivering 26 Biz
capabilities (10H,
7 M, 9 L) along
with this change
X-Functional
Dependencies
Changed 110
backend integrations
to Rest API calls
Leveraged Kafka to
publish downstream
system
Delivery
TimeLine
7 months without
additional head count
Risk Mitigation
Backward Compatibility
Extensive shadow
testing for B2B Orders
Phased rollout
RDBMS MongoDB
Tables : 14 Collection : 1
Indexes : 60 Indexes : 7
Sequences: 16 Sequences : 6
1 Typical Application

17. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Architecture

18. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Caching/
Security
Cisco Commerce
(New, Follow-on and Renewal)
Delivery
Platform
User
Experience
Customer
Experience
Web Mobile
(iOS, Android)
B2B
(Web Services, EDI/XML, Punch-outs)
Data Center /
Virtualization
Integrated
Capabilities/
Workflow
RealUserExperience/AppPerformanceMgmt
(AppDynamics,Akamai,ELK)
Configure Price Quote
Deal
Approval
Order
Capture
Order
Orchestration
Cisco Commerce Architecture after MongoDB
Web/Application Servers

19. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
CCW Order Capture Business Benefits
• Powerful Cisco UCS cluster provides more power and stability
• Read and Write operations are 20 times faster than existing process
Cisco e-commerce Architecture with MongoDB
Infrastructure – Cisco on Cisco
UCS
 Chassis 5108, IOM2208, FI6296
 B200 M3 24x378GB
Storage
 NetApp All Flash Array (SAN)
OS : RHEL 7.x
DC1
DC2
DC3
 2 Nodes in DC1
 2 Nodes in DC2
 1 Node in DC3

20. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Operational (Infra) & Application benefits

21. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Infrastructure & DB Activities with Zero downtime
No major downtime for more than a year
Firmware
upgrade
Fabric
Interconnect (FI)
upgrade on UCS
Domains.
- MongoDB Upgrade to
3.2.12
- OS upgrade from
RHEL 6 to 7.
- NetApp All Flash
Storage
- xfs migration.
Mongo DB
upgrade to
3.4.6.
LDAP
Authentication
setup
Set Version
Compatibility
Intel Meltdown/
Spectre patch
• Tasks were done during week days/ weekend.
• Zero Downtime. One Node at time.
• No impact to business or development activities.
Ops Manager: Cache Usage Chart of 1 Node (last 1 year
chart)
Apr 17 May 17 Jun 17 Jul 17 Aug 17
Jul 17
Oct 17Sep 17 Nov 17 Dec 17 Jan 18 Feb 18 Mar 18 Apr 18 May 18 Jun 18
200GB
150GB
100GB
50GB
0
CacheUsage
Planned instance restart
Other node becomes primary
Timelin
e

22. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Completed four+
quarters without
issues
Software
Upgrades in
minutes without
downtime
Zero Downtime
Response Times
reduced from 3-5
sec to < 1 sec
5x Performance
Improvement
Eliminated
~120,000 lines
of BE code
Architecture
Optimization
Post Go-Live: Application Benefits

23. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How did we get 5x performance improvement?
What is access
together, should be
stored together
De-normalize
data Do not use
Joints
Use less
sequences
Use less
Indexes

24. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Monitoring & Security

25. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MongoDB Monitoring Systems
OPS Manager
Monitoring Dashboard
Monitoring Tool
• ReplicaSet Status
• Agent Status
• Log Management
• Corruption/ Error Alert
• CPU, Memory, FS Alert
• Replica/Shared Info
• User Connection
• Cache/Memory Usage
• Host, CPU, Memory
• Disk I/O, Latency
• Database Load & Trends
• Top Query/ Sessions
• Wait Stats & Top session
• Drill down session
• Topology/ mapping
App Dynamics

26. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Security
o Authentication enabled
o Username/ Password is
mandatory
o Prevents local login
without password
Authentication
o Enabled TLS/SSL
o Clients with SSL
certificate only be
allowed.
o Encrypts data during
Transit
TLS/SSL
o MongoDB do not
enforce complex
password
o Authentication using
LDAP Active Directory
o Enforces Password
complexity
o Alert to reset password
every 90 days
Complex password
o Enable auditing to track
system activities
o Data Encryption at Rest
Auditing / Encryption
• Changing password in AD system takes
some time.
• Create 2 set of AD users to avoid impact
during password reset.
• Reset 2nd user password
• Swap and update in Application & Mongo
configuration files
• Change in a rolling mode (node by node)
to avoid connectivity issue
SOX
Control
Do not use
default port#
27017

27. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Lesson learned & other best practices

28. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Database
Application
Infra
Lessons learned/ Best Practices
• Set sufficient Cache size based
on working set
• Set OpLog size to hold at-least 3
days of replication window
• Upgrade MongoDB to latest
version
• Use indexes on key fields to avoid
full collection scan and reduce cache
usage
• Create indexes in background after
bulk data migration
• For Linux, use RHEL 7 with xfs file
system. Use Flash Drive/ SSD
storage
• For critical databases use 5 nodes
or more
• Limit physical storage size less
than 2TB per replicaSet. This is to
reduce re-sync, backup/ restore
times.
• If Size > 2TB or IOPs > 15k,
sharding is preferred.
• Use compression to save storage
and improve physical I/O

29. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Clone production data to non-production for testing major releases.
 Backup / Restore method to clone small/ medium size database.
 For large database:
o Take File System snapshot copy from standby
o Clone the data to target to optimize time
Cloning/Refresh
 mongodump /restore for small databases
 Ops Manager backup for critical & large databases
 Block store: Small/Medium size. For large database takes long
time for backup/restore
 File system store : Preferred for large databases.
Backup
Best Practices for other Operational Activity

30. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
What’s Next…

31. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
OpenStac
k/Mongod
b
Auto Provision & Scale
Automated
Provisioning
API Driven
Distribute
d
Elastic &
Scalable
Increase in
Demand
Enhance
Customer
Experience
Adopt
Industry
trending
Technology

32. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Automated MongoDB Provisioning Flow
VM Provisioning
Using Terraform
Failure
Post OS
Setup
(Manual)
Client
Ordering/Costing
Cloud DBInfrastructure
Pre Validation
check
MongoDB
Provisioning
& Configuration
thru ansible
Provision
Add-on DB
features
Post Validation
• Install Mongodb, Replicaset, Admin users
• Add Replica set members and create user for db
• Integration with in-house tools
• Security
• Backup
• Monitoring
• Client communication
• Case Creation – Resolution
Database
Advisory
API
UI
Yes
EN
D
API
API
Automated Provisioning
start
Succes
s
Yes
• Email Notification to the client
• Update Cloud Repository

33. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Questions?
Always On Performance Agility
Key Take Away