Big Data Analytics to Enhance Security
Predictive Analtycis and Data Science Conference May 27-28
Anapat Pipatkitibodee
Technical Manager
anapat.p@Stelligence.com
RABBIT: A CLI tool for identifying bots based on their GitHub events.
Big Data Analytics to Enhance Security
1. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Big Data Analytics to
Enhance Security
Anapat Pipatkitibodee
Technical Manager
anapat.p@Stelligence.com
2. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Agenda
• Big Data Analytics
• Security Trends
• Example Security Attacks
• Integrated Security Analytics with Open Source
• How to Apply ?
10. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Big Data Analytics
• The process of examining large data sets
containing a variety of data types i.e., big
data.
• Big Data analytics enables organizations to
analyze a mix of structured, semi-structured,
and unstructured data in search of valuable
information and insights.
12. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection
systems
• 2nd generation: Security information
and event management (SIEM)
13. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Limitation of Traditional SIEMs
Storing and retaining a large quantity of data was not economically feasible.
Normalization & datastore schema reduces data
Traditional tools did not leverage Big Data technologies.
Closed platform with limited customization & integration options
15. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Security Trend from Y2015 to Y2016
• Threats are hard to investigate
Fireeye M-Trends Report 2016
16. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
All Data is Security Relevant = Big Data
Servers
Storage
DesktopsEmail Web
Transaction
Records
Network
Flows
DHCP/ DNS
Hypervisor
Custom
Apps
Physical
Access
Badges
Threat
Intelligence
Mobile
CMDB
Intrusion
Detection
Firewall
Data Loss
Prevention
Anti-
Malware
Vulnerability
Scans
Traditional
Authentication
17. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Data Analytics for Intrusion Detection
• 1st generation: Intrusion detection
systems
• 2nd generation: Security information
and event management (SIEM)
• 3rd generation: Big Data analytics in
security (Next generation SIEM)
19. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Advanced Persistent Threats
• Advanced
– The attack can cope with traditional security solutions
– In many cases is based on Zero-day vulnerabilities
• Persistent
– Attack has a specific goal
– Remain on the system as long as the attack goal is not met.
• Threat
– Collect and steal information-Confidentiality.
– Make the victim's system unavailable-Availability.
– Modify the victim's system data-Integrity.
20. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Example of Advanced Threat Activities
HTTP (web) session to
command & control
server
Remote control,
Steal data,
Persist in company,
Rent as botnet
WEB
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
.pdf
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exeCalc.exe
Attacker hacks website
Steals .pdf files
Web
Portal.pdf
Attacker creates
malware, embed in .pdf,
Emails
to the target
MAIL
Read email, open attachment
Threat intelligence
Auth - User Roles
Host
Activity/Security
Network
Activity/Security
21. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Link Events Together
Threat intelligence
Auth - User Roles,
Corp Context
Host
Activity/Security
Network
Activity/Security
WEB
Conduct
Business
Create additional
environment
Gain Access
to systemTransaction
MAIL
.pdf Svchost.exeCalc.exe
Events that
contain link to file
Proxy log
C2 communication
to blacklist
How was
process started?
What created the
program/process?
Process making
C2 traffic
Web
Portal.pdf
22. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Correlated Security Log
Aug 08 06:09:13 acmesep01.acmetech.com Aug 09 06:17:24 SymantecServer acmesep01: Virus found,Computer name:
ACME-002,Source: Real Time Scan,Risk name: Hackertool.rootkit,Occurrences: 1,C:/Documents and Settings/smithe/Local
Settings/Temp/evil.tmp,"""",Actual action: Quarantined,Requested action: Cleaned, time: 2009-01-23 03:19:12,Inserted:
2009-01-23 03:20:12,End: 2009-01-23 03:19:12,Domain: Default,Group: My CompanyACME Remote,Server:
acmesep01,User: smithe,Source computer: ,Source IP: 10.11.36.20
Aug 08 08:26:54 snort.acmetech.com {TCP} 10.11.36.20:5072 -> 10.11.36.26:443 itsec snort[18774]: [1:100000:3]
[Classification: Potential Corporate Privacy Violation] Credit Card Number Detected in Clear Text [Priority: 2]:
20130806041221.000000Caption=ACME-2975EBAdministrator Description=Built-in account for administering the
computer/domainDomain=ACME-2975EB InstallDate=NULLLocalAccount = IP: 10.11.36.20 TrueName=Administrator
SID =S-1-5-21-1715567821-926492609-725345543 500SIDType=1 Status=Degradedwmi_ type=UserAccounts
Sources
All three occurring within a 24-hour period
Source IP
Data Loss
Default Admin Account
Malware Found
Time Range
Intrusion
Detection
Endpoint
Security
Windows
Authentication
Source IP
Source IP
23. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Incident Analysis & Investigation
Search historically - back in time Watch for new evidence
Related
evidence
from other
security
devices
30. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Determining Data That Can Be Collected
Threat
intelligence
Auth - User
Roles
Service
Host
Network
Network Security Through Data Analysisby Michael S
CollinsPublished by O'Reilly Media, Inc., 2014
• Third-party Threat Intel
• Open source blacklist
• Internal threat intelligence
• Firewall
• IDS / IPS
• Web Proxy
• Vulnerability scanners
• VPNs
• Netflow
• TCP Collector
• OS logs
• Patching
• File Integrity
• Endpoint (AV/IPS/FW)
• Malware detection
• Logins, Logouts log
• Active Directory
• LDAP
• AAA, SSO
• Application logs
• Audit log
• Service / Process
31. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Option 1 : Replace All Solution
• Data sent to new Big Data
Analytic Platform
• Big Data Analytic Platform
– Static Visualizations /
Reports
– Threat detection, alerts,
workflow, compliance
– Incident
investigations/forensics
– Non-security use cases
Big Data Analytic Platform
Raw data
Alerts
Static
Visualizations
Forensics / Search
Interface
32. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Option 2 : Big Data to Traditional SIEM
• Data sent to both system
• Big Data Analytic Platform
– Incident
investigations/forensics
– Non-security use cases
• Traditional SIEM
– Static Visualizations /
Reports
– Threat detection, alerts,
workflow, compliance
Big Data Analytic
Platform
Raw data
Forensics / Search
Interface
SIEM
Alerts
Static
Visualizations
Connectors
33. Copyright Stelligence Co.,Ltd. 2016 All rights reserved
Factors for evaluating
Big Data Security Analytics Platforms
Factors for Evaluating Open Source
• Scalable data ingestion HDFS
• Unified data management platform Cassandra / Accumulo
• Support for multiple data types Ready to Customized
• Real time Spark / Strom
• Security analytic tools No
• Compliance reporting No
• Easy to deploy and manage Manage many 3rd Party
• Flexible search, report and create new
correlation rule
No