Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

[ElasticStack]What happens when you visualize servers exposed to the world?

683 visualizaciones

Publicado el


Publicado en: Ingeniería
  • Sé el primero en comentar

[ElasticStack]What happens when you visualize servers exposed to the world?

  1. 1. Elastic Stack Technology Innovation Group 2017.06.20(Tue) Masamitsu Maehara What happens when you visualize servers exposed to the world?
  2. 2. Self Introduction l Masamitsu Maehara l Future Architect, Inc. l Technology Innovation Group l Messing around with AWS l yurufuwa Engineer @micci184
  3. 3. The Purpose l Get to know the wonders of Elastic Stack l Get to know the fun part of visualizing logs
  4. 4. Elastic Stack??
  5. 5. Elastic Stack Elastic CLoud LogStash Beats Elasticsearch Kibana + Security Alert Monitor Graph l Logstash/Beats:Import Logs l Elasticsearch:Store/Index/Analyze l Kibana:User Interface
  6. 6. Expose to the world?
  7. 7. HoneyPot
  8. 8. ??
  9. 9. HoneyPot l High Interactive HoneyPot l Use real OS and applications l Easy access to information l High Risk l Low Interactive Honeypot l Audit by emulating OS and applications l Limited function l Easily noticed by attackers l Safer than high interactive HoneyPot
  10. 10. Dionaea l Low Interactive HoneyPot l Gathers malware l SMB/HTTP/HTTPS/FTP/TFTP/MSSQL/SIP l Will create front-end view(just to make it look real) l Low Interactive HoneyPot l Specialized for SSH l Better than Kippo Cowrie
  11. 11. Configuration l Built on AWS l Install Beats on HoneyPot l Gather data into Elastic Stack Dionaea Region@Virginia HaneyPot VPC Public Subnet Cowrie Elastic Stack Wordpress Client ・ ・ ・ Monitoring Attack Logging
  12. 12. Install Dionaea ### Ubuntu 14.04 $ sudo apt-get update $ sudo apt-get dist-upgrade $ sudo apt-get install software-properties-common $ sudo add-apt-repository ppa:honeynet/nightly $ sudo apt-get update $ sudo apt-get install dionaea ### Start Dionaea $ sudo service dionaea start
  13. 13. Install Cowrie ### Ubuntu 16.04 $ sudo apt-get install git python-virtualenv libmpfr-dev libssl-dev libmpc-dev libffi-dev build-essential libpython-dev python2.7-minimal authbind ### adduser Cowrie $ sudo adduser --disabled-password cowrie $ sudo su - cowrie ### Setup Virtual Enviroment $ virtualenv cowrie-env $ source cowrie-env/bin/activate ### Install configuration file $ export PYTHONPATH=/home/cowrie/cowrie ### Start Cowrie $ bin/cowrie start Activating virtualenv “cowrie-env” Starting cowrie: [twistd -l log/cowrie.log --umask 0077 --pidfile var/run/ cowrie ]... $ bin/cowrie status cowrie is running (PID: 5979).
  14. 14. Beats l Data Shipper l The Beats FamBam l Filebeat:Sends log files l Metricbeat:Sends metric data(CPU/Mem..etc) l Packetbeat:Sends packet capture data l Winlogbeat:Sends Windows event logs l Heartbeat:Audits system stats
  15. 15. Data Flow l Store Apache/MySQL for WordPress directly into Elasticsearch l Store logs from HoneyPot to Elasticsearch via Logstash Dionaea Filebeat Ubuntu + Log/dionaea.log /binalies/* Amazon Linux Logstash + Input + Filter + Output Cowrie Filebeat Ubuntu + Log/cowrie.log Elastic search KibanaMetric Beat Filebeat Amazon Linux + Apache Apache WordPress Packetbeat + MySQL
  16. 16. Filebeat Modules l Install Filebeat $ curl -L -O $ sudo rpm -vi filebeat-6.0.0-alpha2-x86_64.rpm ### Configuring $ vim /etc/filebeat/filebeat.yml #------------------------------- Apache2 Module ------------------------------ - module: apache2 # Access logs access: enabled: true var.paths: ["/var/log/httpd/access_log"] error: enabled: true var.paths: ["/var/log/httpd/error_log"] #-------------------------- Elasticsearch output ------------------------------- output.elasticsearch: hosts: [“"]
  17. 17. Ingest Plugins l Install Ingest Geoip & Ingest user agent l Ingest Geoip:Maps IP addresses to maps l Ingest user agent:Deals with user agent as it thinks best l Install Ingenst Plugins on the Elastic Stack server ### Ingest Geoip $ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip ### Ingest user agent $ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
  18. 18. !!Attention #01 l Be mindful of proxy environments l Will receive a timeout error when installing Ingest Plugins l Make sure to define proxy setting on the startup script before installing $ sudo /usr/share/elasticsearch/bin//elasticsearch-plugin install ingest-user-agent -> Downloading ingest-user-agent from elastic Exception in thread "main" Connection timed out ### Setup Proxy $ export ES_JAVA_OPTS="-Dhttp.proxyHost=xxx -Dhttp.proxyPort=xxx -Dhttps.proxyHost=xxx - Dhttps.proxyPort=xxx" ### Install ingest-user-agent $ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent -> Downloading ingest-user-agent from elastic [=================================================] 100% ### Ingest Geoip
  19. 19. !!Attention #02 l Be mindful of proxy environments l Handy setting that imports Dashboards when starting Filebeat #Configure dashboard settings on filebeat.yml l However in environments with proxy settings it will not work! :( l In that case, install it manually $ sudo vim /etc/filebeat/filebeat.yml #============================== Dashboards ===================================== - #setup.dashboards.enabled: false + #setup.dashboards.enabled: enable $ sudo /usr/share/filebeat/scripts/import_dashboards -file /tmp/ -es http://xxx:9200
  20. 20. Visualization♥
  21. 21. Beautiful♥
  22. 22. Until recently… l Send logs to be visualized from Filebeat to Logstash l Normalize received logs with Logstash and store it on Elasticsearch l Create a dashboard on Kibana to make it cool
  23. 23. Sooooo,,, Who needs Logstash?
  24. 24. Wait!
  25. 25. Logstash & Dionaea
  26. 26. Malware l Are these malwares on Dionaea? l They’re all over /opt/dionaea/var/dionaea/binaries… $ ll /opt/dionaea/var/dionaea/binaries -rw------- 1 dionaea dionaea 53 Jun 6 02:59 d41d8cd98f00b204e9800998ecf8427e.gz -rw------- 1 dionaea dionaea 162168 Jun 7 22:56 dc8c32d7f26352c8484bc490b6467843.gz -rw------- 1 dionaea dionaea 153820 Jun 7 02:34 dd0400bed68d272b08d1d0272bc18462.gz -rw------- 1 dionaea dionaea 129803 Jun 5 01:01 de1e602b2452a95ba57ef53347e50094.gz -rw------- 1 dionaea dionaea 22778 Jun 6 17:38 e0ddd8bf8e3b97ad25855721dc75daae.gz -rw------- 1 dionaea dionaea 155154 Jun 7 04:33 e53ed987e82ad7bf076c23d91401cac7.gz -rw------- 1 dionaea dionaea 1189 Jun 8 15:32 ead49a9b7b0c8ad6894be45674cebf77.gz -rw------- 1 dionaea dionaea 22777 Jun 6 17:39 eb18a7d302bbc8c0b3ed2cd1612e8d59.gz … -rw------- 1 dionaea dionaea 21966 Jun 5 16:52 ee0efafc69a13cd57d714ffdc603d8fc.gz -rw------- 1 dionaea dionaea 154329 Jun 4 16:48 f09ee5028fd1b1eaaf22df1538de159b.gz -rw------- 1 dionaea dionaea 156637 Jun 9 08:51 f5f1fd0d093d81a4a769c20aca1d6232.gz -rw------- 1 dionaea dionaea 29643 Jun 8 15:34 fc9b0b8b711e44ce0d4f91b0cedb1c76.gz
  27. 27. ClamScan l What do you do when you suspect a malware? You scan it. l Malware FOUND $ clamscan /opt/dionaea/var/dionaea/binaries/ /opt/dionaea/var/dionaea/binaries/f09ee5028fd1b1eaaf22df1538de159b.gz: Win.Worm.Kido-200 FOUND /opt/dionaea/var/dionaea/binaries/621c0b356c49edc5ce4cf3ee88c30f82.gz: OK /opt/dionaea/var/dionaea/binaries/90e02a26204ade7771acf7e8521bdf09.gz: Win.Worm.Kido-297 FOUND /opt/dionaea/var/dionaea/binaries/02830b424d88664cc3576941dd9841f9.gz: Win.Worm.Kido-307 FOUND /opt/dionaea/var/dionaea/binaries/a7bc14c1bd7271a45391f1e1541afe43.gz: Win.Worm.Downadup-110 FOUND /opt/dionaea/var/dionaea/binaries/87136c488903474630369e232704fa4d.gz: Win.Worm.Kido-113 FOUND /opt/dionaea/var/dionaea/binaries/1195dfde6305980ed050a9751b157f42.gz: Win.Worm.Kido-293 FOUND /opt/dionaea/var/dionaea/binaries/1b4cd56e54d3f9030a153590fb3fa9e5.gz: Win.Worm.Kido-316 FOUND /opt/dionaea/var/dionaea/binaries/fc9b0b8b711e44ce0d4f91b0cedb1c76.gz: OK /opt/dionaea/var/dionaea/binaries/cae8a8524eeb0e7de1fb3704bd14b7ba.gz: Win.Trojan.Ramnit-1847 FOUND /opt/dionaea/var/dionaea/binaries/7bb455ea4a77b24478fba4de145115eb.gz: Win.Worm.Kido-197 FOUND /opt/dionaea/var/dionaea/binaries/eb18a7d302bbc8c0b3ed2cd1612e8d59.gz: OK /opt/dionaea/var/dionaea/binaries/smb-az4poq4s.tmp.gz: OK /opt/dionaea/var/dionaea/binaries/16acf30169d089b8a967f40d9a38d8f7.gz: Win.Trojan.Agent-129152 FOUND
  28. 28. What if we want to surveillance these malwares in realtime?
  29. 29. Data Flow l Regularly runs ClamScan to output logs l Filebeat sends the logs to the Elastic Stack server l Logstash normalizes the logs and stores them into Elasticsearch l Visualize with Kibana Dionaea Filebeat Ubuntu + Log/binalies/* /log/scan.log Amazon Linux Logstash + Input + Filter + Output Elastic search KibanaMetric Beat
  30. 30. By the way, do you use Logstash?
  31. 31. Logstash vs fluentd l Compared on Google Trend l By country l Blue:Logstash l Red:fluentd
  32. 32. Grok filter
  33. 33. Grok Filter l Below were the ClamScan results l We need to somehow normalize it to get certain Key-Value data l Data we want l OK/FOUND (Key : check) l Malware Name (Key : malware) $ clamscan /opt/dionaea/var/dionaea/binaries/ /opt/dionaea/var/dionaea/binaries/f09ee5028fd1b1eaaf22df1538de159b.gz: Win.Worm.Kido-200 FOUND /opt/dionaea/var/dionaea/binaries/621c0b356c49edc5ce4cf3ee88c30f82.gz: OK /opt/dionaea/var/dionaea/binaries/90e02a26204ade7771acf7e8521bdf09.gz: Win.Worm.Kido-297 FOUND /opt/dionaea/var/dionaea/binaries/02830b424d88664cc3576941dd9841f9.gz: Win.Worm.Kido-307 FOUND
  34. 34. Such a pain in the neck…
  35. 35. Grok Constructor
  36. 36. Grok Constructor l Let’s you test on web browsers l You can also check stdout on Logstash l Convenient if you don’t want to rewrite Logstash.conf
  37. 37. Paste log here Grok Filter Click GO! after pasting
  38. 38. Grok Constructor l Results look like this lOK/FOUND is contained in “check” l But, where is the malware name? l Work on that Grok Filter again
  39. 39. use contents detected in data Grok Filter Click GO! after pasting
  40. 40. Grok Constructor l Results look like this l Malware name is matched in malware!
  41. 41. Logstash.conf l Final product looks like this input { beats { port => 5044 } } filter { grok { match => [ "message", "/[^/]+/[^/]+/[^/]+/[^/]+/(?<field>[^/]+)/%{GREEDYDATA:data}%{WORD:check}"] remove_field => [ "host", "message" ] } grok { match => [ "data", "(?:[¥w._/%-]+)%{WORD}(?:[:]*)%{GREEDYDATA:malware}"] remove_field => [ "data" ] } } output { elasticsearch { hosts => "" } }
  42. 42. Looking good
  43. 43. Malware♥
  44. 44. Summary l Let Elastic Stack do everything from input to output l Easy visualization with Beats l Expose your server to expand your log variation l Are you excited to share your server with the public? l Don’t let Logstash beat you!
  45. 45. Thanks