Cloudera Federal Forum 2014: Hadoop-Powered Solutions for Cybersecurity
•
1 like•1,738 views
The document discusses how Apache Hadoop can be used for cybersecurity historical analysis. Hadoop allows organizations to capture and archive large amounts of security data from multiple sources at scale. This collected data can then be transformed, enriched, and sessionized within Hadoop to correlate petabytes of information and identify anomalies or security threats. Tools like Impala and Spark can also enable interactive queries and iterative machine learning on Hadoop for further analytics and rare event prediction for cybersecurity monitoring.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Cloudera Federal Forum 2014: Hadoop-Powered Solutions for Cybersecurity
Similar to Cloudera Federal Forum 2014: Hadoop-Powered Solutions for Cybersecurity (20)
More from Cloudera, Inc.
More from Cloudera, Inc. (20)
Recently uploaded
Recently uploaded (20)
Cloudera Federal Forum 2014: Hadoop-Powered Solutions for Cybersecurity
- 1. Apache Hadoop-Powered Cybersecurity Joey Echeverria, Chief Architect Cloudera Government Solutions February 2014 1 ©2014 Cloudera, Inc. All rights reserved.
- 2. What is Cybersecurity? • Applications • Security-in-depth • Local/public networks source: http://www.flickr.com/photos/shimgray/2985486716/ 2 ©2014 Cloudera, Inc. All rights reserved.
- 3. Why Cybersecurity? • Protect your data • Service availability source: http://www.flickr.com/photos/kevinmarks/5218166919/ 3 ©2014 Cloudera, Inc. All rights reserved.
- 4. WHY IS THIS A HARD PROBLEM? 4
- 5. Breadth • Firewalls • Switches • Servers • Desktops • Mobiles source: http://www.flickr.com/photos/larskflem/3100856376/ 5 ©2014 Cloudera, Inc. All rights reserved.
- 6. Depth • Deep packet inspection • Intrusion logs • Server logs • Desktop logs 6 ©2014 Cloudera, Inc. All rights reserved.
- 7. Update all the things source: http://www.flickr.com/photos/bovinity/2125620107/ 7 ©2014 Cloudera, Inc. All rights reserved.
- 8. TWO WAYS TO FIGHT BACK 8
- 9. Streaming • Intrusion Detection System (IDS) • Intrusion Prevention System (IPS) • Complex Event Processing (CEP) • Alert/react in real-time 9 source: http://www.flickr.com/photos/conifer/9535872266/ ©2014 Cloudera, Inc. All rights reserved.
- 10. Historical Analysis • Not all attacks are real- time • Hiding in plane sight Anomaly detection • Steganography • source: http://www.flickr.com/photos/bjornmeansbear/4249524324/ 10 ©2014 Cloudera, Inc. All rights reserved.
- 11. Example: Attack Timeline • When did the attacker get in? • How did the attacker get in? • What data was stolen? 11 ©2014 Cloudera, Inc. All rights reserved.
- 12. Scale • 10 Gbps • 900,000 packets per second • Thousands of servers • Tens of thousands of desktops and mobile devices • Two aspects Capture at scale • Archive • 12 ©2014 Cloudera, Inc. All rights reserved.
- 13. FOCUS ON HISTORICAL ANALYSIS 13
- 14. Apache Hadoop • Massive storage • Massive throughput • Batch processing • Correlate petabytes of data source: http://svn.apache.org/viewvc/hadoop/logos/out_rgb/elephant_rgb.pdf?view=log 14 ©2014 Cloudera, Inc. All rights reserved.
- 15. Transform source: http://www.flickr.com/photos/accardi1/3294248678/ 15 ©2014 Cloudera, Inc. All rights reserved.
- 16. Enrich source: http://www.flickr.com/photos/shookphotos/4597753813/ 16 ©2014 Cloudera, Inc. All rights reserved.
- 17. Sessionize • Combine activity into user sessions • Analyze application sessions, 17 not packets ©2014 Cloudera, Inc. All rights reserved.
- 18. Beyond Batch • NoMR • Any framework Interactive SQL • In-memory processing • Machine learning • Search • Accumulo • source: http://www.flickr.com/photos/rhinoneal/5633001128/ 18 ©2014 Cloudera, Inc. All rights reserved.
- 19. Impala • MPP analytic query engine for Hadoop • SQL • UDFs and UDAFs • Future: window functions 19 ©2014 Cloudera, Inc. All rights reserved.
- 20. Pretty Charts! source: http://www.flickr.com/photos/caseorganic/5407670731/ 20 ©2014 Cloudera, Inc. All rights reserved.
- 21. Reporting • What are the most commonly accessed blacklist sites? • What are the top destinations by country? • Summary of port scanning activity • Top IDS event types 21 ©2014 Cloudera, Inc. All rights reserved.
- 22. Spark • In-memory processing • Iterative algorithms • Simple API source: http://www.flickr.com/photos/mrzeon/4458423242/ 22 ©2014 Cloudera, Inc. All rights reserved.
- 23. Analytics source: http://www.flickr.com/photos/iconolith/253426954/ 23 source: http://www.flickr.com/photos/derletzteschrei/193503222/ ©2014 Cloudera, Inc. All rights reserved.
- 24. Finding Outliers 24 ©2014 Cloudera, Inc. All rights reserved.
- 25. Rare Event Prediction 25 ©2014 Cloudera, Inc. All rights reserved.
- 26. Questions • Contact me! • Joey Echeverria • joey@clouderagovt.com • @fwiffo 26 ©2014 Cloudera, Inc. All rights reserved.
- 27. 27 ©2014 Cloudera, Inc. All rights reserved.