Eagle6 is a product that use system artifacts to create a replica model that represents a near real-time view of system architecture. Eagle6 was built to collect system data (log files, application source code, etc.) and to link system behaviors in such a way that the user is able to quickly identify risks associated with unknown or unwanted behavioral events that may result in unknown impacts to seemingly unrelated down-stream systems. This session is designed to present the capabilities of the Eagle6 modeling product and how we are using MongoDB to support near-real-time analysis of large disparate datasets.

1. MongoDC 2014
Confidential & Proprietary

2. Adam Bell, Director, Product Management
◦ 15+ years enterprise software solutions experience
◦ 10+ years enterprise architecture
◦ Healthcare
◦ Technology
◦ MongoDB user since 2012
Confidential & Proprietary

3. Service-Disabled Veteran
Owned Small Business
Minority-Owned Business
Primary NAICS Codes:
◦ 541511
◦ 541512
Confidential & Proprietary
Established: 2002*
Subject Matter Expertise:
Enterprise Software
Development
Business Process
Reengineering (BPR)
Proprietary Software:
Eagle6 Modeling &
Simulation
*Over 30% of Rivera Group Employees Are Veterans

4. Eagle6 is a modeling tool that automatically
collects system data (application code,
database schemas, log files, etc.) and
provides an ability to continuously monitor
for unwanted system states (bugs) that may
result in system degradation and/or system
outages.
Confidential & Proprietary

5. Large sets of Multi-Dimensional Data
Heavy Read and Write
Heavy Audit Requirements
Fast Near Real-Time Analytics
Analytics are User Driven
Confidential & Proprietary

6. About Our Documents
◦ 1000(s) of leafs on documents
◦ Document Sizes (Bson):
2300 bytes (2.3Kb) - 729699 bytes (729Kb)
◦ Need to quickly add new data structures
Confidential & Proprietary

7. Confidential & Proprietary
"network" : {
"host" : [
{
"host" : {
"network" : null,
"ips" : [
"6PNfL9bV7BJO"
],
"names" : [ ],
"mx" : [ ],
"txt" : [ ],
"srv" : [ ],
"ns" : [ ]
}
},
{
"host" : {
"network" : null,
"ips" : [
"yqb6q7er3DvWf"
],
"names" : [
"tQypbmzVrEZHtWG1n"
],
"mx" : [ ],
"txt" : [ ],
"srv" : [ ],
"ns" : [ ]
}
}
],
}

8. Do not drop transactions
Capturing large volumes of real-time data
(web access logs, database transactions, etc)
Confidential & Proprietary

9. Indexing has been a challenge
MongoDB only supports 64 indexes
To many indexes defeats the purpose of
indexes
Scenarios exist where we will not know what
fields the user needs until they request them
Confidential & Proprietary

10. Key Value Approach
TODO: Need an example of KV indexes
Confidential & Proprietary

11. Confidential & Proprietary

12. Shard Key is an object hash
Goal is equal distribution of data across
shards
◦ Example
{
hash:
'00003820efcff8b669b055606813bcd360ace3f43fbf9c129845b3028992eacabca
ef8cd13796dc7a96b7a5f38b0efaceaadecfd537c72eaec8a8f9c10a00a1e’,
offset: -1}
Confidential & Proprietary

13. Needed a flexible way to Aggregate
Needed a way to represent Aggregations to
end users with out writing functions
Needed a way to cache frequently run
analytics
Confidential & Proprietary

14. Provides a Rich set of operations for
aggregating data
We have been using the Aggregation
Framework since MongoDB 2.2
Aggregation Pipelines have enabled us to do
smart caching
JSON versions of the pipeline allow end users
flexibility without writing Map/Reduce code
Confidential & Proprietary

15. Confidential & Proprietary
{
"$match": {
”servers": {
"$exists": true
}
}
},
{
"$group": {
"_id": ”$serversr",
"clients": {
"$addToSet": ”$client.names
}
}
},
{
"$project": {
"title": {
"$join": "$_id.names"
},
"_id": {
"id": {
"$join": "$_id.ips"
}
},
"clients": 1,
"server": "$_id"
}
},
{
"$group": {
"_id": "$_id",
"server": {
"$addToSet": "$server"
},
"clients": {
"$addToSet": "$clients"
},
"title": {
"$addToSet": "$title"
}
}
},
{
"$unwind": "$clients"
},
{
"$unwind": "$clients"
},
{
"$project": {
"clients": {
"$join": "$clients"
},
"server": "$server",
"title": "$title"
}
},
{
"$group": {
"_id": "$_id",
"server": {
"$addToSet": "$server"
},
"clients": {
"$addToSet": "$clients"
},
"title": {
"$addToSet": "$title"
}
}
},
{
"$unwind": "$title"
},
{
"$unwind": "$title"
}

16. Ported Aggregation framework to Javascript
◦ Custom built extensions
Accumulators
$stdDev
Expressions
$regex
$slice
Document Sources
$projectPrevious
$split
Confidential & Proprietary

17. Confidential & Proprietary