The role of NoSQL in the Next Generation of Financial Informatics

The Role of NoSQL
In Next Generation Financial Informatics
December 10, 2015

2
Proprietary & Confidential | © 2015 Aerospike Inc. All rights reserved.
■ Market Overview
■ The role of NoSQL in Financial Services
■ Customer Use Cases
■ Fraud
■ Order Management
■ How you can achieve Speed at Scale
Agenda

Steve Yatko
CEO, Oktay Technology
Financial Services
Market Overview

4
Hours - Days
Analytics Store
Operational Store
(Landing Zone – Fast Ingest)
MarkLogic
ElasticSearch
Deep Archive
Apache Storm
Apache Spark
Apache Ignite
Project Heron
Ingest/
Gateway
Tibco EMS
JMS
60 East
29 West
TCP
FTP
IBM MQ
Data
Warehouse
(Legacy)
Teradata
DB2 WH
Actian(ParAccel)
Pivotal
Astor Data/TD
Neteeza/DB2 WH
Exadata
Mark Logic
Veritca
Data Lake
(100+ Petabyte)
Tiered External Storage
Flash Arrays to
Warm/Cold/Frozen
Hadoop
MapReduce
Redis
IMDGrid
Hazelcast
Aerospike
Graph Db
Large Footprint
Front Office
Real-Time Processing
On The Wire “Fast Data’
Hours Days-Years
No SQL
HBase
MongoDB
Cassandra
Aerospike
Time Series
ETL
In Memory
Cache
Data Operating System – YARN, MESOS
Deeper Analysis
World Class Big Data Ecosystem: “Service Oriented Information”
HDFS Interface
Small Footprint
Surveillance Trading
Analytics
Metrics,
Monitoring and
Alerts
Search ComplianceRT Discovery &
Visualization
Oktay Technology LLC Confidential
RAM-first
Storage
Flash-enabled
Storage
SQL-Based
MemSQL
Volt DB
Kafka
In Memory
Fil;esystem
Tachyon
IgniteFS

Brian Bulkowski
Founder and CTO, Aerospike
The Role of NoSQl in Next
Generation Financial Informatics

6
2010
Aerospike 2.0 deploys with first at-scale customers
2011
Funding, company launch
2013
Aerospike 3.0 adds indexes, analytics integration
2014
Open Source
2015
Deployments in Financial Services, Telecom, etc
Aerospike History

7
Modern Scale Out Architecture
Research
Warehouse
Long-term cold storage
HDFS BASED
App Servers
Fast, stateless
Load Balancer
Simple, stateless
High Performance NoSQL
Operational Key Value
Session, authentication, account status, cookies, deviceID,
IP address, location, segments, trades, debits, billing,
prices...
Real-Time Decisions
Best sellers, top scores, trending tweets

8
Powering Pioneers Across the Internet
Mobile
Advertising
Omni-Channel
Marketing
Search, Video, Social,
Gaming
Web Advertising

9
1 Million Writes/Sec on Google Compute
New results: 20 nodes, and 4M reads per second
• Aerospike hits 1M writes/sec with 6x fewer servers than Cassandra
• Delivers consistent low latency with no jitter for both read and write workloads

10
Shared-Nothing System, High Availability
• Every node in a cluster is identical, handles both transactions and long running tasks
• Clients give rich semantics in multiple languages and connect over the network
• Data is replicated synchronously with within the local cluster
• Data is replicated asynchronously across data centers
• Primary key hash RIPE MD160 ( 20 byte ) for extreme collision resistance ( DHT ), Red black
tree for records within hash bucket
• Scatter-gather B+ secondary index

11
The NoSQL Pattern Applied to Inline
Fraud Intervention

12
Fraud Prevention
4% of online transactions are fraudulent
False positives cost millions in lost sales
Global criminal agents
New opportunities:
Data sources
Algorithms
Speed at Scale

13
Operational Scale Problem
LEGACY DATABASE
(Mainframe)
XDR
Decision Engine
DATA WAREHOUSE/
DATA LAKE
LEGACY RDBMS
HDFS BASED
BUSINESS
TRANSACTIONS
Request
for
Payment
( Mobile Queries ) (
Recommendation )
( And More )
500
Business Trans per sec
5000
Calculations per sec
X =
“REAL-TIME BIG DATA”
“DECISIONING”
2.5 M
Database Transactions per sec

14
Advanced Fraud Scoring
Anything can be a “feature”
POS type, item sold, price of item, time of day for given items
Browser type, screen resolution, item + browser
But you’ll need recent data
And cutting edge libraries
( your private algorithms )

15
Example IP Addresses
2B to 4B addresses
Unknown number of networks
Different risk models for different
network ranges
Recent behavior required

16
Example – IP based scoring
ID START_IP END_IP Score
1 192.168.4.1 192.168.4.26 20
2 192.168.4.7 192.168.17.0 2
SELECT id, score WHERE START_IP >= this_ip AND END_IP <= this_ip
Not so bad!
Too slow, and
too simple

17
Using SQL
ID IP TIMESTAMP BEHAVIOR
1 192.168.4.1 2015-06-02 12:00 transaction
2 192.168.4.7 2015-06-02 12:01 tweet
Add the most recent 5 minutes of traffic for each IP address and lookup by previous network
INSERT ip, time, behavior
FOR ( network )
SELECT behavior WHERE ip <= network.start_id AND
ip >= network.end_id AND
timestamp > now - 600
You can do it
in SQL!
Too slow, call
the DBAs.

18
With a NoSQL KVS
key Network list
192.168.4.0 [ net1, net2, net3 ]
192.168.5.0 null
ID START_IP END_IP Score
net1 192.168.4.1 192.168.4.26 20
net2 192.168.4.7 192.168.17.0 2
Key on Class C address
List of networks
Batch lookup networks
Filter on client
That’s messy.
Two network round
trips, it’ll go fast.

19
With a NoSQL KVS – add behavior
Already have filtered list of networks
For each network, insert into capped or time limited collection
key Network list
192.168.4.0 [ net1, net2, net3 ]
192.168.5.0 null
ID START_IP END_IP Score behavior
net1 192.168.4.1 192.168.4.26 20
{ collection
}
net2 192.168.4.7 192.168.17.0 2
{ collection
}
That’s really
messy! It’s fast and easy
to explain.

20
Use Cutting Edge Libraries in Your Favorite App Environment
If it’s in SQL, it’s too late

21
Modern Scale Out Architecture
App Servers
Fast, stateless
Load Balancer
Simple, stateless
Operational Key Value
Session, authentication, account status, cookies, deviceID,
IP address, location, segments, trades, debits, billing,
prices...
Real-Time Decisions
Best sellers, top scores, trending tweets
Scale and Compute
Data
Enabled by Fast
Networks
Least Vendor Lock-in

22
The NoSQL Pattern Applied to
Order Management

23
Financial Services – Order Management Systems
Challenge
• Must update stock prices, show balances on 300
positions, process 250M transactions, 2M updates/day
• Scale-out requirement
• Multi-asset aggregation for combined:
Trading
Risk
Surveillance
Customer
• Real-time data view and computation
Speed at Scale

24
Individual data silos
• No global, integrated data view
• No simple post-order visibility
• Often built on non-scalable technologies
• Little shared operational infrastructure
DB1 DB2 DB3
Order Management
System
Order Management
System
Order Management
System
FIXED INCOMEEQUITIES DERIVATIVES

25
Shared Data OMS
•True multi-asset aggregation
•Supports real-time risk models
•Immediate customer visibility
of order state
•Natural NoSQL data model
including parent-child and fills
•Microsecond performance with
persistence
•Scale-out architecture
Order Management
System
Order Management
System
Order Management
System

26
Hybrid order management
•Existing order management code
unchanged
•Order states placed onto message bus
•Recorded in unified NoSQL storage
•Achieve unified multi-asset aggregated
view
•Storm and Spark integration
•Beyond RAM-based processing solutions
DB1 DB2 DB3
Order Management
System
Order Management
System
Order Management
System
MESSAGE BUS (KAFKA, etc)

27
How Much Data Can I Use?

28
Multiple Engagements
10B to 50B objects
300 T before HA replication
2 M TPS
50 milliseconds per 5,000 queries
Local and remote DR

29
1 Million TPS on Flash per server
Options for storage on a database before Aerospike:
 RAM, which was fast, but allowed very limited storage
 Disk, which allowed for a lot of storage, but was limited in speed
Intel achieved 1M TPS using 4 Intel P3700 SDs with 1.6 TB
capacity on a single Aerospike server. The cost per GB is a
fraction of the cost of RAM, while still having very high
performance.

30
Aerospike Has the Experience
Flash optimized from the beginning
5 9’s reliability
( longest running cluster: 5 minutes of outage in 4.5 years )
Multi-million TPS clusters in production today
Enterprise Support of your POC & NoSQL design
Used in financial services, telecom, gaming
( and adtech of course )

31
Questions?
@aerospikedb
brian@aerospike.com
@bbulkow

32
High Performance
NoSQL Database
Powering New
Opportunities at Scale
@aerospikedb
NEXT STEPS:
See how much you can save with Aerospike:
http://www.aerospike.com/tco-calculator/
Ready to get started?
http://www.aerospike.com/quick-start/
If you have any questions or want to further
explore if Aerospike is right for you, contact
us:
info@aerospike.com

The role of NoSQL in the Next Generation of Financial Informatics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to The role of NoSQL in the Next Generation of Financial Informatics

Similar to The role of NoSQL in the Next Generation of Financial Informatics (20)

More from Aerospike, Inc.

More from Aerospike, Inc. (11)

Recently uploaded

Recently uploaded (20)

The role of NoSQL in the Next Generation of Financial Informatics

Editor's Notes