This document discusses Cisco's implementation and operation of MongoDB sharding. It provides an overview of Cisco's data centers and private cloud platform. It then discusses a use case of using MongoDB sharding for Cisco's order processing system to improve performance, scalability and availability. The document outlines the architecture of MongoDB sharding, how data is partitioned and distributed across shards, and how applications connect to the sharded cluster. It also discusses how Cisco migrated their order processing application to utilize MongoDB sharding and the benefits they achieved including faster performance, increased scalability and reduced downtime. Lastly, it covers how Cisco manages and monitors their MongoDB sharded cluster using tools like Ops Manager and AppDynamics.

1. Implementation and Operation of MongoDB
Shard at Cisco Systems Inc.
Ramabathiran Arumugam
Nayeem Khaja
Mahesh Vaghela

2. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Agenda list
1
2
3
About Cisco
Use Case – Order Processing
MongoDB Sharding
4 Application Migration
6 Key takeaways
5 Architecture & Operation

3. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco – Products & Services
Network Wireless Security Collaborate Data Center Video
* Switches
* Routers
* Optical
* Interfaces
* SDN
* Controls
* Access points
* 5G
* Cloud
* Fire walls
* Malware
* Email
* Web
* Cloud
* VPN
* Webex
* Conference
* Telephony
* Endpoints
* Cloud calling
* Contacts
* UCS-Servers
* DC Security
* DC Storage
* DC Automate
* Virtual N/W
* Hyperconverge
* End points
* Collaboration
* Intelligence
* Transformation

4. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Global Data Centers
Tier-III
(Redundant)
Tier-II
(Less Redundant)
Globally Centralized:
Business Apps
Continental Hub:
Order Processing, Comms
Continental Hub:
Communications
Cloud Services available
Private Cloud, self Service capabilities:
IaaS / PaaS
Latency-Sensitive
Software Development
2x
Texas
1x Amsterdam
1x Singapore
Global DC Footprint
26 Data Centers
35k server IT + ENG
92% servers virtualized
80+ PB raw storage
35+ MW Power (Supply)
~334,000 Sq Ft.

5. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Why Mongo DB?

6. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
High Availability & fault tolerance architecture to keep application “Always-ON”Availability
Data aggregation, Secondary indexes and right sizing of working set improves the
application performance in multi-fold
Performance
Sharding enables DB for horizontal scaling for fast growing databaseScalability
Flexible schema makes application more flexible for new business modelFlexibility
Why MongoDB ?

7. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MongoDB – Private Cloud
Apps Platform
Cloud DB
Cisco Private Cloud Platform
App A App B App C
DBaaS
Provisioning
Infrastructure
ü Choice of Sizes
ü Choice of Storage
ü Choice of SLA
DB Edition
ü Community
ü Enterprise
Security
ü Encryption in Motion
ü Encryption at Rest
ü SOX Compliance
ü LDAP Authentication
Service Assurance
ü Monitoring (AppD)
ü Backup and Restore
Add On Services
ü DB Upgrades
ü User Management
ü DB Patching
ü Refresh/Cloning
High Availability
Scalability
Resiliency
Workload Isolation
ü E2E Cloud DB Provisioning Under 30 Min
ü 99% API Availability
ü 99.8% to 99.99% uptime
ü Zero Downtime maintenance
KPIs for Success
Richardson Tx Allen Tx RTP, NC

8. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
c
Compute Cluster
OpenStack Management
Nodes
Nova Compute NodesUsers
Platform 3 - OpenStack Infrastructure
ACI
Network Fabric
ACC
Storage
UCS C Series Low I/O Storage
RADOS GW
(Swift)
OSD / MON
High I/O Storage
Storage
Application driven
consumption
Building Blocks for Cloud Platform
UCS C Series
Direct API access
Multicloud Manager

9. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MongoDB at Cisco
eCommerce
Order Processing
Social
Supply Chain Security Network
Install BaseHR Management
Object Oriented Cloud Native PerformanceDevelop Fast

10. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Use Case: Order Processing System

11. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Order Processing System
Commerce B2B Partners Acquisitions Market Place
Validations BookingOrder Services Workflow
ContractsFulfillment ReportingSales CreditBilling
Order Processing
Database
Order Sources
Integration and Data
Distribution
Integrated Systems
Order Life Cycle Management
- Validate
- Book
- Fulfill
- Invoice
- Order data SOR
- Booking data SOR
- Order Orchestration
- Data distribution

12. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Rapid business change
Dynamic offers
Scalability
- Fast database growth
- Fixed order and offer structure
Performance
Monolithic architecture and
heavyweight framework
Highly normalized database
contributing to performance
degradation with data growth
Downtime
release, changes to hardware, upgrades,
patches, all result into application
downtime
Before…

13. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Model
Relational data model
to object oriented data
model
User Experience
Response time and
transaction processing
time
Data Size
Sharding considered for
Rapid growth of
transaction volume
Refactoring
Application on
lightweight, scalable,
and containerized
framework
Migration
Transparent migration
of live application on
MongoDB
Considerations

14. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Sharding

15. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Primary
Secondary
Secondary
Driver
Application
Non-Shard
Single ReplicaSet
E.g. 1 TB Data
Sharding is a method for distributing data across more than one replica set
What is sharding?
Shard
Multiple ReplicaSets
E.g. 300 GB
Data
E.g. 400 GB
Data
E.g. 100 GB
Data
E.g. 200 GB
Data
Query Router

16. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
When to Shard
• Large data sets or high throughput applications is limited by single (primary) server capacity.
• If working set size is larger than the system’s RAM, will cause slow performance due to physical I/O.
• If data size grows we have to increase capacity (RAM) by Vertical scaling
• Large size servers are quiet expensive
• To increase data size, MongoDB sharding facilitates to partition collections across multiple replica set
• Each replica set can be of small size and we can use low cost commodity servers
• Use Horizontal scaling to increase data size further. Just add new replica set and expand horizontally

17. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How Shard Works
Application
Driver
mongos mongos mongos
Query Routers
P
S
S
S
S
P
S
P
S
P
S
S
Shard 1
Replica Set
Config
Replica Set
Shard n
Replica Set
Shard 2
Replica Set
Shard:
A Replica Set contains a
subset of the sharded
data.
mongos:
A Query router, providing an interface
between client applications and the
sharded cluster.
Config Server:
config servers: Is a Replica Set.
Stores metadata and configuration
settings for the cluster.
Provides
ShardKey <>
Shard location
mapping

18. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How to use Shard?
A database can have sharded and unsharded collections.
Sharded collections are partitioned and distributed across the shards
in the cluster.
Unsharded collections are stored/distributed to a different (primary)
shard. Each database has its own primary shard.
Sharded
collection
UnSharded
collection
You must connect to a mongos router to interact with
any collection in the sharded cluster.
Clients should never connect to a single shard in order
to perform read or write operations.
Collection 3
for DB2

19. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ShardKey • MongoDB partitions the collection using the shard key and distributes.
• Shard key: An immutable field or fields in a sharded collection.
• You can choose the shard key when sharding a collection.
• A sharded collection can have only one shard key.
• Choose proper shardkey, so that data is distributed evenly.
• Shard key cannot be changed after sharding.
Application Application Application
X : 1 X : 12 X : 25
Query Router
Query Router
Query Router
Shard Key X: < 10 Shard Key X: 10 to < 20 Shard Key X: > 20
Define ShardKey when
creating collection
X is the ShardKey
Connection is routed to specific Shard where
the data exists with ShardKey (X)

20. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Zones in Sharded Clusters:
Create zones of sharded data based on the shard key.
You can associate each zone with one or more shards.
Sharding Method/Strategy
Ranged Sharding:
Shard collection based on range of the shard key values.
Hashed Sharding:
Shard collections based on hashed values of the shard key values.

21. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Sharding collection

22. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Sharding example
Tables Collections
Header
oplOrders
Lines
Discounts
Contacts
oplOrders

23. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Sharding example
Average doc
size
80KB
1M documents
per month
80GB data
size / month
40GB
storage /
month
480 GB
storage /
year
960GB data
size / year
Shard 1
2TB, 16 Core,
128GB RAM
Shard 2
2TB, 16 Core,
128B RAM
Sharding Methods:
1. Hashed
2. Ranged
oplOrders
Shard Key à

24. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Migration

25. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How application was migrated to MongoDB
Order Import
Order Data
Processing Integrations
Data
Distribution
Order Import Processing Integrations
Data
Distribution
Payload
Payload
Kafka Topics
Kafka Topics
Consumer
Consumer 1
Consumer 2
Consumer 3
Message
comparison
process
Payload
comparison
process
Data
comparison
process
Orders
Consumer 1
Consumer 2
Consumer 3
Orders
Order Data

26. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
10x faster02
Increased velocity03
Application
modernization
04
99.99% Uptime01
50%more features enable
in same time
Service Response time Cloud Native
ZERO downtime since application migrated
to MongoDB compare to 1 hour to 8 hours
of downtime every month for deployment,
patches, and infra releases.
Benefits
100ms-500ms
compare to few seconds
in old platform
4x Growth: 40K à 170K
in two quarters

27. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Mongo Shard Architecture, Operation
& Monitoring

28. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
RTPAllen
AZ2AZ2 AZ2
Replica Set
MongoDB Shard Deployment Architecture
Nodes are distributed across DCs & Availability Zone(AZ)s
Maintenance work in each AZ, will impact 1 node per replica set
MongoS and Config server co-locate on same VM
Config &
Mongos
Nodes
AZ1
Shard 1
RCDN
Shard 2
Replica Set
AZ3 AZ1
AZ3 AZ1 AZ3

29. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Deploy Mongo Shard
Automation Agent Config
Monitoring Agent Config
Backup deployment
Upgrade Automation
Manage Security
Ops Manager : Manage Shard
Ops Manager helps to
manage complex mongo
shard with multiple
replicate sets/ nodes.

30. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
API Type APIs used
Group_status GET, /api/public/v1.0/groups/byName/<group_name>
Create_group POST, /api/public/v1.0/groups?pretty=true
Check_deployment_status GET, /api/public/v1.0/groups/<group_id>/automationStatus
Get_raw_automation_config GET, /api/public/v1.0/groups/<group_id>/automationConfig
Update_raw_automation_config PUT, /api/public/v1.0/groups/<group_id>/automationConfig
Automation_agent_status GET, /api/public/v1.0/groups/<group_id>/agents/AUTOMATION
Update_monitoring_agent_config PUT, /api/public/v1.0/groups/<group_id>/automationConfig/monitoringAgentConfig
Monitoring_agent_status GET, /api/public/v1.0/groups/<group_id>/agents/MONITORING
Ops Manager APIs helps to manage complex Shard deployments
Simplifies Automation, Configuration, Monitoring & Backup
Ops Manager APIs for Automation

31. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Monitoring

32. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Application flow map Activity
AppDynamics & Ops Manager for MongoDB Monitoring
• Application flow mapping
• Drill down top wait events/ queries
• Additional monitoring features on top
of Ops Manager
• Run jobs based on alerts/events
• Deployment/ creation of mongo
replicaset/ shards
• Automation / configuration change
• Upgrade/Backup/Security
• Monitoring
• Alerts
Ops Manager AppDynamics
Drill down each
operation
Few good features in AppDynamics….

33. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Final Take Away

34. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
If you have large database
If you want to update in parallel
Multiple replicaset with smaller size.
It helps to manage, backup & recover.
Reduces H/W cost by using smaller capacity servers & horizontal
scaling
Key Take away

35. © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Q & A