Real World Threat Hunting Security threats have grown from network annoyances to attacks on sensitive infrastructure; penetrating network perimeters, moving laterally within networks, breaching new device types, and cloaking movements. This presentation will share techniques utilized by Cisco to detect and investigate sophisticated, embedded threats. The speaker, who has conducted monitoring and investigations on customer networks, will review recent real attacks observed on customer networks, from discovery to remediation, and provide lessons learned. These interactive case examples will highlight how to identify these threats using security intelligence, expert staff, and the Cisco OpenSOC platform. Examples of attacks and illustrations: * Sophisticated phishing attacks targeted at customer environments. * Breaches and data exfiltration resulting from the high-profile HeartBleed and Shellshock vulnerabilities. * Sophisticated malware targeting financial institutions with the goal of data theft. * Use of full packet capture to identify data exfiltration.

1. Martin G. Nystrom
Cisco Security Solutions
Real World Threat Detection

2. 2© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 222: Qadars Malware

3. Cisco Confidential 3
ATA Case 222: Qadars Malware
Malware Qadars: banking crimeware uses webinjects and coordinated attacks on Android
to defeat two-factor authentication
Exploit target Pinpoints bank users in specific regions
Detection • Detected within minutes of incident
• Escalated to within 90 minutes
• 5 cases raised since monitoring began in May 2014
Analysis • Verified callback
• Analyzed domains, file hashes, trigger packets, IP addresses, site reputations
• Verified Indicator of Compromise (IOC) appeared in traffic
• Found dozens of periodic check-ins from compromised host
• Targeted; no other customers encountering this malware
Check-ins
at one
minute
intervals
Attacker hosting c2
at St. Mary’s
University

4. Cisco Confidential 4© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Security Challenges
Changing
Business Models
Dynamic
Threat Landscape
Complexity
and Fragmentation

5. Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Changing
Business Models
Dynamic
Threat Landscape
Complexity
and Fragmentation
Security Challenges
of organizations not
“fully aware” of all
network devices
BYOD
90%
SOCIAL MEDIA
times more cloud services
are being used than
known by IT
CLOUD
5–10
of top 500 Android apps
carry security/privacy risks
APP STORES
92%
of organizations had
malware enter the corporate
network through social
media/web apps
14%

6. Cisco Confidential 6© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Changing
Business Models
Dynamic
Threat Landscape
Complexity
and Fragmentation
Security Challenges
A community that hides in plain sight avoids detection and attacks swiftly
60%
of data is
stolen in
HOURS
54%
of breaches remain
undiscovered for
MONTHS
YEARSMONTHSHOURSSTART
85%
of point-of-sale intrusions
aren’t discovered for
WEEKS
WEEKS
51%increase of companies
reporting a $10M loss
or more in the last
3 YEARS

7. Cisco Confidential 7© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco Security Hypothesis
Advisory Integration ManagedThreat-centric Platform-basedVisibility-focused
Operational
Focus
Talent
Shortage
+
Security
Challenges
+
Requires Improved Outcomes

8. Cisco Confidential 8© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Security Services Portfolio

9. Cisco Confidential 9© 2014 Cisco and/or its affiliates. All rights reserved.
Evolution of Threats
Visibility
Of Data Loss
Era 2000 2005 Present
Actor Amateur Cybercrime Cybercrime, Nations
Goal Disruption Steal money
$ + Intellectual
property
Response
Anti-Virus and
Firewalls
IDS, Behavioral
detection, Reputation
Intelligence and
Analytics
Plain IRC Tunneled
C2 Encrypted
C2
Hidden in e-­
mail and
social
networking

10. Cisco Confidential 10© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco
Active Threat Analytics

11. Cisco Confidential 11© 2014 Cisco and/or its affiliates. All rights reserved.
Active Threat Analytics Overview
§ Near real-­time analytics
§ Anomaly detection
§ Zero-­day threat focus
§ Partnership with
Hortonworks
§ Streaming analytics
§ Presentations at Hadoop
Summit and Strata
§ Access to actionable
sources of intelligence
§ Cisco intelligence
§ Customer intelligence
§ Open Source intelligence
§ Operationalization
§ Advanced expertise
§ Talos security research
§ Security talent shortage
CISCO
ATA
AnalyticsPeople Intelligence Technology

12. Cisco Confidential 12© 2014 Cisco and/or its affiliates. All rights reserved.
Active Threat Analytics Architecture
DEDICATED
CUSTOMER SEGMENT
Administrative
Consoles
PORTAL
TICKETING
COMMON SERVICES
Threat Intelligence
Dedicated Customer Portal
Alerting/Ticketing System
Investigator
Portal
Authentication
Services
24/7
ACCESS
CUSTOMER
SOC
Secure Connection
(HTTPS/SSH/IPSec)
VPN
INTERNET
VPN
CUSTOMER PREMISE CISCO DATA CENTER
FIREWALL
FIREWALL
CMSP
Advanced
Malware
Protection
Full Packet
Capture
Anomaly
Detection
Sourcefire
IDS
Collective
Security
Intelligence
Streaming
Analytics
ThreatGrid
NetFlow
Full Packet
Machine
Exhaust
Cisco
Third Party

13. Cisco Confidential 13© 2014 Cisco and/or its affiliates. All rights reserved.
Threat
Intelligence
Feeds
Enrichment
Data
OpenSOC Overview
Full packet capture
Protocol metadata
NetFlow
Machine exhaust (logs)
Unstructured telemetry
Other streaming telemetry
Parse +
Format
Enrich Alert
Log Mining and
Analytics
Big Data
Exploration,
Predictive
Modelling
Network Packet
Mining and
PCAP
Reconstruction
Applications + Analyst Tools

14. Cisco Confidential 14© 2014 Cisco and/or its affiliates. All rights reserved.
OpenSOC Framework
Sources Data Collection Messaging Broker Real-­Time Processing Storage Access
Analytic Tools
Tableau
R / Python
Power Pivot
Web Services
Search
PCAP
Reconstruction
Telemetry Sources
NetFlow
Machine Exhaust
HTTP
Other
Flume
Agent B
Agent N
Agent A
Kafka
B Topic
N Topic
PCAP Topic
DPI Topic
A Topic
Storm
B Topology
N Topology
A Topology
PCAP Topology
DPI Topology
Hive
Raw Data
ORC
Elasticsearch
Index
HBase
Packet Table
PCAP
Passive
Tap
Traffic
Replicator

15. Cisco Confidential 15© 2014 Cisco and/or its affiliates. All rights reserved.
https://github.com/OpenSOC

16. Cisco Confidential 16© 2014 Cisco and/or its affiliates. All rights reserved.
Use Case: Customer Statistics for Two-­Week Timeframe
Post-­investigation tickets71
269,808 Security Events
Unique events113,713
High fidelity events1710
207,99261,816Threat intel sourced Telemetry
generated

17. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Cisco Talos
Built on unmatched collective security telemetry that gets better every 5 minutes
101000 0110 00 0111000 111010011 101 1100001 110
1100001110001110 1001 1101 1110011 0110011 101000 0110 00
1001 1101 1110011 0110011 101000 0110 00
180,000+ File Samples per Day
FireAMP™ Community
Advanced Microsoft
and Industry Disclosures
Snort and ClamAV Open Source
Communities
Honeypots
Sourcefire AEGIS™ Program
Private and Public Threat Feeds
Dynamic Analysis
1.6 million
global sensors
100 TB
of data received per day
150 million+
deployed endpoints
600+
engineers, technicians,
and researchers
35%
worldwide email traffic
13 billion
web requests
24x7x365
operations
40+
languages
101000 0110 00 0111000 111010011 101 1100001 110
1100001110001110 1001 1101 1110011 0110011 101000 0110 00
1001 1101 1110011 0110011 101000 0110 00 Cisco®
Talos
Sourcefire
VRT®
(Vulnerability
Research Team)
Email Endpoints Web Networks IPS Devices
WWW
Cisco Collective
Security Intelligence

18. Cisco Confidential 18© 2014 Cisco and/or its affiliates. All rights reserved.
Network / Protocol Behavior Anomaly Detection
§ Anomaly detection provides
best chance to catch
unknown / 0-day malware or
advanced attackers
§ Cisco focused on anomaly
detection using predictive
techniques – not rules
§ Recent acquisition of
Cognitive Security (NetFlow /
HTTP anomalies)
§ Techniques include normalcy
models / goodness-of-fit
tests / time-series analysis /
decision trees / graph cluster
analysis

19. Cisco Confidential 19© 2014 Cisco and/or its affiliates. All rights reserved.
Global Threat Intelligence
Cisco Threat
Intelligence Platform
(Hadoop)
Cisco-Generated
Intelligence
Licensed
Intelligence
Government
Intelligence
Community
Intelligence
Individual Feed / Sources
Indicators of Compromise
DNS Names / IP Addresses / File Hashes

21. Cisco Confidential 21© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Network Hunting Cookbook Samples

22. 22© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Threats, Trends, and Incidents

23. Cisco Confidential 23© 2013-2014 Cisco and/or its affiliates. All rights reserved.
§ Plug-in Poison
§ Sandbox Evasion
§ Tor Client Malware
§ Decline of Zeus
§ Obfuscation
§ Malicious Macro Delivery
Threat Landscape
Targeting end users
Similar activity in the last 10 years, techniques are improving

24. Cisco Confidential 24© 2013-2014 Cisco and/or its affiliates. All rights reserved.
LURE
Social media
and other
sites are
sources for
targeted
information.
RECON
Social media
and other sites
are sources
for targeted
information.
REDIRECT EXPLOIT
KIT
DROPPER
KIT
CALL
HOME
DATA
THEFT
Users are
sent
unknowingly
from one site
to another.
A user’s
system is
inspected for
vulnerabilities.
Malware
infects a
vulnerable
system.
Infected
system
reaches out
to command-
and-control
servers.
Sensitive
information is
exfiltrated.
Kill Chain

25. Cisco Confidential 25© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 1011
Sandbox Evading Malware
Targeted at Corporate Users

26. Cisco Confidential 26© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Attack Stages
Trojaned WordPress servers + “Chanitor” malware
Compromise
WordPress
servers to
host exploit
Phish
corporate
users with
volume
license
agreement
email
User clicks
link
Trojan directs
user to real
real Microsoft
server and
starts
download of
trojan via
JavaScript
User opens
malicious .zip
and executes
trojan
Trojan installs
itself as
winlogin.exe
Trojan
connects to
API to get IP
of c2 server
Trojan tests if
it connect to
Tor for c2
Trojan
connects to
Tor for c2
From here, attacker remotely controls the machine,
exfiltrating data, attacking other devices, and moving
laterally within network

27. Cisco Confidential 27© 2013-2014 Cisco and/or its affiliates. All rights reserved.
1. Attacker Sent User Phishing Email
“Congratulations…to begin registration, please download…”
Real user’s email
address in both To:
field and URL, to
look more
legitimate

28. Cisco Confidential 28© 2013-2014 Cisco and/or its affiliates. All rights reserved.
2. Victim Clicked Link and Received Malware Download
Opens real,
SSL-verified
Microsoft site
Malware
downloaded
from a different
site via
JavaScript trick

29. Cisco Confidential 29© 2013-2014 Cisco and/or its affiliates. All rights reserved.
3. ATA Analyst Observed Retrospective Alert for 1.php
!

30. Cisco Confidential 30© 2013-2014 Cisco and/or its affiliates. All rights reserved.
4. ATA Analyst Researched Threat
• Virus detection 9/57
• Sandbox execution failed
• Escalated to ATA Investigator
Known, 9
Unknown, 48

31. Cisco Confidential 31© 2013-2014 Cisco and/or its affiliates. All rights reserved.
5. ATA Investigator Conducted Forensic Analysis
Discovered malware as “Chanitor”; uses sandbox evasion
• All sandboxes
timed out
• Ran file on
physical box with
network and
memory capture,
file system
monitoring
Malware
programmed
sleep function
to fool sandbox
analysis

32. Cisco Confidential 32© 2013-2014 Cisco and/or its affiliates. All rights reserved.
6. ATA Investigator Determined Malware C2 Servers
DNS Queries IP Resolution at Time of Analysis
api.ipify.org 50.16.221.126, 54.225.211.214,
54.235.186.52
o3qz25zwu4or5mak.tor2web.org 194.150.168.70, 38.229.70.4
o3qz25zwu4or5mak.tor2web.ru 166.78.144.80
Online service to learn public IP
address
Tor servers;
malware tested
for connectivity
before sending
data

33. Cisco Confidential 33© 2013-2014 Cisco and/or its affiliates. All rights reserved.
7. ATA Investigator Searched for C2 Traffic
ATA Investigator searched NetFlow traffic for confirmation that victim was
compromised and under remote control. No evidence found.

34. Cisco Confidential 34© 2013-2014 Cisco and/or its affiliates. All rights reserved.
8. ATA Investigator Advised Customer to Block Domains
No successful exfiltration; malicious sites blocked
Advised customer
to block the file by
hash on email and
web gateways,
and block 3
domains used to
serve the
malicious files

35. Cisco Confidential 35© 2014 Cisco and/or its affiliates. All rights reserved.
Key Takeaways
Observation Conclusion
Attack targeted corporate users by
phishing with corporate-licensed
software
Attackers after more than just
personal data
Malware examination required
physical forensic analysis due to
sandbox evasion techniques
Sandbox technology useful but only
part of solution
Attacker used Tor for C2 traffic Tor connections should raise
suspicion on corporate networks
Malware domains quickly discovered
and blocked
24x7 monitoring with senior security
investigators key to protect against
advanced attacks

36. Cisco Confidential 36© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 383
Insider Using TOR and VPN

37. Cisco Confidential 37© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 383
Employee Hides Traffic via VPN/TOR

38. Cisco Confidential 38© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 383
Analyst Detects TOR Exit Node Access via Threat Intel

39. Cisco Confidential 39© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 383
Analyst Checked for Related Malicious Events
Server certificate matches
that used by malware for
encrypted communication
Self-signed:
Internet Widgits Pty
Analyst escalates case 383 to investigator

40. Cisco Confidential 40© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 383
Investigator Observes VPN traffic to TOR exit node
POST /vpninfo/servers HTTP/1.1
Accept: */*
User-Agent: Ruby
Host: www[.]privateinternetaccess[.]com
Content-Length: 49
Content-Type: application/x-www-form-urlencoded
version=28&os=win&nonce=9rxmz3cz1pet2adaq46mgrlrvHTTP/1.1 200 OK
Server: Apache/2.2.22
X-UA-Compatible: IE=Edge,chrome=1
ETag: "43961e0dd0f118465a7d55b6857f9ab6"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: 8ede8eada8d97983827f260342f389d9
X-Runtime: 0.084319
X-Rack-Cache: invalidate, pass
X-Powered-By: Phusion Passenger 4.0.10
Status: 200 OK
Content-Type: text/html; charset=utf-8
Date: Tue, 16 Sep 2014 00:09:17 GMT
Content-Length: 4237
Connection: keep-alive
Set-Cookie: u=3e3lv3at7xk3q0vbb9iy9i9u; path=/
Set-Cookie:

41. Cisco Confidential 41© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case 383
Investigator Confirms VPN Traffic for Weeks
Source IP Destination IP sPortdPort
packet
s bytes Time dur
10.220.233.81 216.155.131.70 54017 8888 3 114 2014/09/12T14:59:32.000 30
10.220.233.81 96.31.87.158 54020 8888 3 114 2014/09/12T14:59:32.000 30
10.220.233.81 162.253.129.18 54027 8888 3 114 2014/09/12T14:59:32.000 30
10.220.233.81 216.155.131.70 54017 8888 1 66 2014/09/12T14:59:32.000 0
10.220.233.81 162.253.129.18 54027 8888 1 66 2014/09/12T14:59:32.000 0
10.220.233.81 46.19.139.174 54024 8888 1 66 2014/09/12T14:59:32.000 0
10.220.233.81 173.192.81.151 54019 8888 3 114 2014/09/12T14:59:32.000 30
10.220.233.81 50.23.131.245 54021 8888 1 66 2014/09/12T14:59:32.000 0
10.220.233.81 96.31.87.158 54020 8888 1 66 2014/09/12T14:59:32.000 0

42. Cisco Confidential 42© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VPN and TOR: Benign or Criminal?
OR

43. Cisco Confidential 43© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ATA Case
383
Investigator
Notified
Customer

44. ATA Case 383:
Employee Interviewed, denied malicious behavior
What would you do?

45. Cisco Confidential 45© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Case 267
APT Attack on Medical Site

46. Cisco Confidential 46© 2013-2014 Cisco and/or its affiliates. All rights reserved.
MTD Case 267
China Chopper Deploys Sophisticated Web Shell to Servers

47. Cisco Confidential 47© 2013-2014 Cisco and/or its affiliates. All rights reserved.
MTD Case 267
Analyst Detects Breach via Backdoor.Chopper Alerts

48. Cisco Confidential 48© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Cisco MTD Case 267
Analyst Verifies IoC: Accessing User Directory
Analyst escalates case 267 to investigator

49. Cisco Confidential 49© 2013-2014 Cisco and/or its affiliates. All rights reserved.
MTD Case 267
Investigator Confirms Attack via Full Packet Capture
C:RECYCLERcmd.xe
[Err] The system cannot find the file specified.
C:RECYCLERcmd.exe

50. Cisco Confidential 50© 2013-2014 Cisco and/or its affiliates. All rights reserved.
MTD Case 267
System Remediated, Eventually
1. Investigator raised
ticket to customer
advising rebuild,
phoned to alert.
2. Customer ran AV,
thought server clean.
3. Investigator reported
further evidence of
rootkit.
4. Customer rebuilt
system from scratch,
patched.
5. System still under
attack; no further
breaches detected

51. Cisco Confidential 51© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Security Operations Centers
Americas
Austin
Raleigh
EMEAR
Krakow
APJC
Sydney
Top Talent
Targeted Expertise
Custom Operations

52. Thank you.