Hikarabo Elasticstack Honeypot Dionaea ClamAV

1. Elastic Stack
Technology Innovation Group
2017.06.20(Tue)
Masamitsu Maehara
What happens when you visualize servers exposed to the world?

2. Self Introduction
l Masamitsu Maehara
l Future Architect, Inc.
l Technology Innovation Group
l Messing around with AWS
l yurufuwa Engineer
@micci184

3. The Purpose
l Get to know the wonders of Elastic Stack
l Get to know the fun part of visualizing logs

4. Elastic Stack??

5. Elastic Stack
Elastic CLoud
LogStash Beats
Elasticsearch
Kibana
+
Security
Alert
Monitor
Graph
l Logstash/Beats：Import Logs
l Elasticsearch：Store/Index/Analyze
l Kibana：User Interface

6. Expose to the world?

7. HoneyPot

8. ??

10. HoneyPot
l High Interactive HoneyPot
l Use real OS and applications
l Easy access to information
l High Risk
l Low Interactive Honeypot
l Audit by emulating OS and applications
l Limited function
l Easily noticed by attackers
l Safer than high interactive HoneyPot

11. Dionaea
l Low Interactive HoneyPot
l Gathers malware
l SMB/HTTP/HTTPS/FTP/TFTP/MSSQL/SIP
l Will create front-end view(just to make it look real)
l Low Interactive HoneyPot
l Specialized for SSH
l Better than Kippo
Cowrie

12. Configuration
l Built on AWS
l Install Beats on HoneyPot
l Gather data into Elastic Stack
Dionaea
Region@Virginia
HaneyPot VPC
Public Subnet
Cowrie Elastic
Stack
Wordpress
Client
・
・
・
Monitoring
Attack
Logging

13. Install Dionaea
### Ubuntu 14.04
$ sudo apt-get update
$ sudo apt-get dist-upgrade
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:honeynet/nightly
$ sudo apt-get update
$ sudo apt-get install dionaea
### Start Dionaea
$ sudo service dionaea start

14. Install Cowrie
### Ubuntu 16.04
$ sudo apt-get install git python-virtualenv libmpfr-dev libssl-dev libmpc-dev libffi-dev build-essential
libpython-dev python2.7-minimal authbind
### adduser Cowrie
$ sudo adduser --disabled-password cowrie
$ sudo su - cowrie
### Setup Virtual Enviroment
$ virtualenv cowrie-env
$ source cowrie-env/bin/activate
### Install configuration file
$ export PYTHONPATH=/home/cowrie/cowrie
### Start Cowrie
$ bin/cowrie start
Activating virtualenv “cowrie-env”
Starting cowrie: [twistd -l log/cowrie.log --umask 0077 --pidfile var/run/cowrie.pid cowrie ]...
$ bin/cowrie status
cowrie is running (PID: 5979).

15. Beats
l Data Shipper
l The Beats FamBam
l Filebeat：Sends log files
l Metricbeat：Sends metric data（CPU/Mem..etc）
l Packetbeat：Sends packet capture data
l Winlogbeat：Sends Windows event logs
l Heartbeat：Audits system stats

16. Data Flow
l Store Apache/MySQL for WordPress directly into Elasticsearch
l Store logs from HoneyPot to Elasticsearch via Logstash
Dionaea Filebeat
Ubuntu
+ Log/dionaea.log
/binalies/*
Amazon Linux
Logstash
+ Input
+ Filter
+ Output
Cowrie Filebeat
Ubuntu
+ Log/cowrie.log
Elastic
search
KibanaMetric
Beat
Filebeat
Amazon Linux
+ Apache
Apache
WordPress
Packetbeat
+ MySQL

17. Filebeat Modules
l Install Filebeat
$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-alpha2-x86_64.rpm
$ sudo rpm -vi filebeat-6.0.0-alpha2-x86_64.rpm
### Configuring
$ vim /etc/filebeat/filebeat.yml
#------------------------------- Apache2 Module ------------------------------
- module: apache2
# Access logs
access:
enabled: true
var.paths: ["/var/log/httpd/access_log"]
error:
enabled: true
var.paths: ["/var/log/httpd/error_log"]
#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
hosts: [“xxx.xxx.xxx.xxx:9200"]

18. Ingest Plugins
l Install Ingest Geoip & Ingest user agent
l Ingest Geoip：Maps IP addresses to maps
l Ingest user agent：Deals with user agent as it thinks best
l Install Ingenst Plugins on the Elastic Stack server
### Ingest Geoip
$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
### Ingest user agent
$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent

19. !!Attention #01
l Be mindful of proxy environments
l Will receive a timeout error when installing Ingest Plugins
l Make sure to define proxy setting on the startup script before installing
$ sudo /usr/share/elasticsearch/bin//elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
Exception in thread "main" java.net.ConnectException: Connection timed out
### Setup Proxy
$ export ES_JAVA_OPTS="-Dhttp.proxyHost=xxx -Dhttp.proxyPort=xxx -Dhttps.proxyHost=xxx -
Dhttps.proxyPort=xxx"
### Install ingest-user-agent
$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
-> Downloading ingest-user-agent from elastic
[=================================================] 100% ### Ingest Geoip

20. !!Attention #02
l Be mindful of proxy environments
l Handy setting that imports Dashboards when starting Filebeat
＃Configure dashboard settings on filebeat.yml
l However in environments with proxy settings it will not work! :(
l In that case, install it manually
$ sudo vim /etc/filebeat/filebeat.yml
#============================== Dashboards =====================================
- #setup.dashboards.enabled: false
+ #setup.dashboards.enabled: enable
$ sudo /usr/share/filebeat/scripts/import_dashboards -file /tmp/beats-dashboards-x.x.zip -es
http://xxx:9200

21. Visualization♥

25. Beautiful♥

26. Until recently…
l Send logs to be visualized from Filebeat to Logstash
l Normalize received logs with Logstash and store it on
Elasticsearch
l Create a dashboard on Kibana to make it cool

27. Sooooo,,,
Who needs Logstash?

28. Wait!

29. Logstash & Dionaea

30. Malware
l Are these malwares on Dionaea?
l They’re all over /opt/dionaea/var/dionaea/binaries…
$ ll /opt/dionaea/var/dionaea/binaries
-rw------- 1 dionaea dionaea 53 Jun 6 02:59 d41d8cd98f00b204e9800998ecf8427e.gz
-rw------- 1 dionaea dionaea 162168 Jun 7 22:56 dc8c32d7f26352c8484bc490b6467843.gz
-rw------- 1 dionaea dionaea 153820 Jun 7 02:34 dd0400bed68d272b08d1d0272bc18462.gz
-rw------- 1 dionaea dionaea 129803 Jun 5 01:01 de1e602b2452a95ba57ef53347e50094.gz
-rw------- 1 dionaea dionaea 22778 Jun 6 17:38 e0ddd8bf8e3b97ad25855721dc75daae.gz
-rw------- 1 dionaea dionaea 155154 Jun 7 04:33 e53ed987e82ad7bf076c23d91401cac7.gz
-rw------- 1 dionaea dionaea 1189 Jun 8 15:32 ead49a9b7b0c8ad6894be45674cebf77.gz
-rw------- 1 dionaea dionaea 22777 Jun 6 17:39 eb18a7d302bbc8c0b3ed2cd1612e8d59.gz
…
-rw------- 1 dionaea dionaea 21966 Jun 5 16:52 ee0efafc69a13cd57d714ffdc603d8fc.gz
-rw------- 1 dionaea dionaea 154329 Jun 4 16:48 f09ee5028fd1b1eaaf22df1538de159b.gz
-rw------- 1 dionaea dionaea 156637 Jun 9 08:51 f5f1fd0d093d81a4a769c20aca1d6232.gz
-rw------- 1 dionaea dionaea 29643 Jun 8 15:34 fc9b0b8b711e44ce0d4f91b0cedb1c76.gz

31. ClamScan
l What do you do when you suspect a malware? You scan it.
l Malware FOUND
$ clamscan /opt/dionaea/var/dionaea/binaries/
/opt/dionaea/var/dionaea/binaries/f09ee5028fd1b1eaaf22df1538de159b.gz: Win.Worm.Kido-200 FOUND
/opt/dionaea/var/dionaea/binaries/621c0b356c49edc5ce4cf3ee88c30f82.gz: OK
/opt/dionaea/var/dionaea/binaries/90e02a26204ade7771acf7e8521bdf09.gz: Win.Worm.Kido-297 FOUND
/opt/dionaea/var/dionaea/binaries/02830b424d88664cc3576941dd9841f9.gz: Win.Worm.Kido-307 FOUND
/opt/dionaea/var/dionaea/binaries/a7bc14c1bd7271a45391f1e1541afe43.gz: Win.Worm.Downadup-110 FOUND
/opt/dionaea/var/dionaea/binaries/87136c488903474630369e232704fa4d.gz: Win.Worm.Kido-113 FOUND
/opt/dionaea/var/dionaea/binaries/1195dfde6305980ed050a9751b157f42.gz: Win.Worm.Kido-293 FOUND
/opt/dionaea/var/dionaea/binaries/1b4cd56e54d3f9030a153590fb3fa9e5.gz: Win.Worm.Kido-316 FOUND
/opt/dionaea/var/dionaea/binaries/fc9b0b8b711e44ce0d4f91b0cedb1c76.gz: OK
/opt/dionaea/var/dionaea/binaries/cae8a8524eeb0e7de1fb3704bd14b7ba.gz: Win.Trojan.Ramnit-1847 FOUND
/opt/dionaea/var/dionaea/binaries/7bb455ea4a77b24478fba4de145115eb.gz: Win.Worm.Kido-197 FOUND
/opt/dionaea/var/dionaea/binaries/eb18a7d302bbc8c0b3ed2cd1612e8d59.gz: OK
/opt/dionaea/var/dionaea/binaries/smb-az4poq4s.tmp.gz: OK
/opt/dionaea/var/dionaea/binaries/16acf30169d089b8a967f40d9a38d8f7.gz: Win.Trojan.Agent-129152 FOUND

32. What if we want to surveillance
these malwares in realtime?

33. Data Flow
l Regularly runs ClamScan to output logs
l Filebeat sends the logs to the Elastic Stack server
l Logstash normalizes the logs and stores them into Elasticsearch
l Visualize with Kibana
Dionaea Filebeat
Ubuntu
+ Log/binalies/*
/log/scan.log
clamscan.sh
Amazon Linux
Logstash
+ Input
+ Filter
+ Output
Elastic
search
KibanaMetric
Beat

34. By the way, do you use
Logstash?

35. Logstash vs fluentd
l Compared on Google Trend
l By country
l Blue：Logstash
l Red：fluentd

36. Grok filter

37. Grok Filter
l Below were the ClamScan results
l We need to somehow normalize it to get certain Key-Value data
l Data we want
l OK/FOUND (Key : check)
l Malware Name (Key : malware)
$ clamscan /opt/dionaea/var/dionaea/binaries/
/opt/dionaea/var/dionaea/binaries/f09ee5028fd1b1eaaf22df1538de159b.gz: Win.Worm.Kido-200 FOUND
/opt/dionaea/var/dionaea/binaries/621c0b356c49edc5ce4cf3ee88c30f82.gz: OK
/opt/dionaea/var/dionaea/binaries/90e02a26204ade7771acf7e8521bdf09.gz: Win.Worm.Kido-297 FOUND
/opt/dionaea/var/dionaea/binaries/02830b424d88664cc3576941dd9841f9.gz: Win.Worm.Kido-307 FOUND

38. Such a pain in the neck…

39. Grok Constructorhttp://grokconstructor.appspot.com/do/match

40. Grok Constructor
l Let’s you test on web browsers
l You can also check stdout on Logstash
l Convenient if you don’t want to rewrite Logstash.conf

41. Paste log here
Grok Filter
Click GO! after pasting

42. Grok Constructor
l Results look like this
lOK/FOUND is contained in “check”
l But, where is the malware name?
l Work on that Grok Filter again

43. use contents detected in data
Grok Filter
Click GO! after pasting

44. Grok Constructor
l Results look like this
l Malware name is matched in malware!

45. Logstash.conf
l Final product looks like this
input {
beats {
port => 5044
}
}
filter {
grok {
match => [ "message", "/[^/]+/[^/]+/[^/]+/[^/]+/(?<field>[^/]+)/%{GREEDYDATA:data}%{WORD:check}"]
remove_field => [ "host", "message" ]
}
grok {
match => [ "data", "(?:[¥w._/%-]+)%{WORD}(?:[:]*)%{GREEDYDATA:malware}"]
remove_field => [ "data" ]
}
}
output {
elasticsearch {
hosts => "http://xxx.xxx.xxx.xxx:9200/"
}
}

46. Looking good

48. Malware♥

49. Summary
l Let Elastic Stack do everything from input to output
l Easy visualization with Beats
l Expose your server to expand your log variation
l Are you excited to share your server with the public?
l Don’t let Logstash beat you!

50. Thanks