Fraud Detection - This talk will outline NRT Fraud Detection and event detection using the EDH to achieve sub-second response times. This talk will focus specifically on Fraud and Event Detection architectures and tools necessary to build the system. Kevin O’Dell has been an HBase contributor since 2012. Kevin currently works as a Field Engineer for Cloudera building Big Data applications with a specialization in HBase. He is also the co-author on the upcoming Architecting HBase Applications book being published by O'Reilly.

1. Fraud Detection
Using Cloudera EDH Fraud Detection
Kevin O’Dell/Field Engineer

2. 3© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Agenda
• Overview of Problem
• Offline Fraud
• Online Fraud
• Discussion

3. 6© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Problem Statement
• About $15B annually – Card only
• 1.3B Credit, Debit and Pre-paid cards – 5 per adult.
• Omni-channel – Debit, Credit, Online, PoS, Deposits
• Increasing Regulatory / Compliance Requirements
• How do we integrate multiple systems and sources

4. 13© 2014 Cloudera and/or its affiliates. All rights reserved.
Fraud Systems

5. 14© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Key Requirements
• No data loss is acceptable
• Stream processing must complete ASAP, <500ms
• Support approximately 400M transactions per day in aggregate
• Highest Volume Flow:
• Current – 1.8k transactions/s
• Projected – 10k transactions/s
• Each flow has at least three steps
• Adapter, Persistence, Hadoop Persistence
• Most complex with approximately seven steps
• Avoid massive code refactoring

6. 15© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Fraud System Categories
• Online
• Ingest
• Enrichment (Profiles, feature selection, etc.)
• Early warning / detection (model serving / model application)
• Persistence
• Offline (Human activities)
• Model building / discovery
• Case management
• Forensics

7. 16© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Some Numbers
• Approximately 400M events (not necessarily transactions) per day
• Analysts get data within 5 minutes (Approx 90M/day)
• Over 100 Source Systems
• Offline system rolls files every 5 minutes
• Online system processes transaction authorization flows in < 500ms.

8. 17© 2014 Cloudera and/or its affiliates. All rights reserved.
Architecture

9. 19© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
ClientClient
Incoming Events Cloudera EDH
Automated & Manual
Analytical Adjustments and
Pattern detection
Kafka Cluster
Topic A
Topic B
Topic C
ClientClient
Outgoing Events
Storage
HDFS
SolR
HBase
Event Processing Interactivity
HBase
Search
Serving Layer
Rules Engine
Model Building
Speed Layer
Batch Layer
Processing
Impala
Map/Reduce
Spark
3rd Party
Fraud Architecture
EDH:
Model Building, Automated Alerting, Profile Persistence
Layer, Forensics, Pattern Detection, Discovery
Analytics
Event Processing
Alerting, Enrichment, Business Rules
Spark Streaming
Case Management And
Alerting

10. 26© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Online Systems
• Can be incorporated to the authorization pipeline
• Rules Engine incorporation
• Application of models
• Must deliver results sub-second
• Must scale to spikes in transaction volume
• Historically outside of Hadoop
• 0 data loss tolerance and tight SLA requirements

11. 28© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
Online System Advantages
• Enriching the record in Real Time allows us to apply any number of
algorithms
• Travel Scoring
• Anomaly Detection Models (Clustering)
• Commercial ML model application
• All with sub-second latency
• Integration into EDH allows easy deployment, monitoring and
integration with offline/batch activities

12. 33© 2014 Cloudera, Inc. All rights reserved.
CONFIDENTIAL—DO NOT
DISTRIBUTE
ClientClient
Incoming Events Operational Cluster (6 months)
Automated & Manual
Analytical Adjustments and
Pattern detection
HULC (13 months)
Storage Processing
HDFS
Impala
Map/Reduce
Spark
Security / Transport
Service Activator
Kafka Cluster
Topic A
Topic B
Topic C
ClientClient
Outgoing Events
Security / Transport
Service Activator
Storage
HDFS
SolR
HBase
Event Processing Interactivity
HBase
Search
Serving Layer
Rules System
Model Building
Speed Layer
Batch Layer
Real-time Cluster
Processing
Impala
Map/Reduce
Spark
3rd Party
Multi-cluster Fraud Architecture
Operational Cluster:
Model Building, Automated Alerting, Profile Persistence
Layer
Discovery Cluster
Batch model updates, Discovery Analytics, Pattern
Detection
Real Time Cluster
Event Processing, Alerting, Enrichment, Business
Rules
Business Users
HDFS Replication

13. Thank you.