Talk I gave at ACSAC 2011 on the paper: "Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness" which describes the 2010 international Capture the Flag (iCTF) competition.
Paper is located here:
http://cs.ucsb.edu/~adoupe/static/hit-em-where-it-hurts-acsac2011.pdf
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Hit ‘em Where it Hurts: A Live Security Exercise on Cyber Situational Awareness
1. Hit „em Where it Hurts:
A Live Security Exercise on
Cyber Situational Awareness
Adam Doupé, Manuel Egele, Benjamin Caillat,
Gianluca Stringhini, Gorkem Yakin, Ali Zand,
Ludovico Cavedon, and Giovanni Vigna
University of California, Santa Barbara
ACSAC 2011 – 7/12/11
2. What Are Live Security
Competitions?
• AKA Hacking Competitions
• Useful educational tool for teaching computer
security
• Born as a way to showcase security skills
– DefCon‟s CTF
• Various forms
– Challenge set (DefCon quals, iCTF challenges,
CMU‟s competition, DIMVA competition, RuCTF)
– Capture the flag (DefCon, iCTF 2003-2007, CIPHER)
– Other designs
• Attack-only (e.g., iCTF 2008)
• Defense-only (e.g., Cyber Defense eXercise)
3. Why Live Security Competitions?
• Real-time factor enhances understanding
• Forces teams to:
– Analyze unknown services/binaries
– Defend systems from attack
– Utilize different security skills
– Work as a team
– Create novel tools
Doupé - 7/12/11
4. Key Insight
• Security competitions can be designed to
generate datasets for research
• In the 2010 international Capture The Flag
(iCTF), we structured the competition to
create a Cyber Situational Awareness
dataset
Doupé - 7/12/11
5. Situational Awareness
• By putting perceived events into the context
of the currently executing mission, one can
improve decision making
• Mission
– Series of tasks that an organization wishes to
carry out
• Task
– Discrete step that is carried out using a service
• Service
– Provided to users to accomplish a task
Doupé - 7/12/11
6. Cyber Situational Awareness
• Situational awareness extended to the
cyber domain
• Large organizations constantly under
attack
– Which attacks are important?
– Which assets are important?
• “What if” scenarios
Doupé - 7/12/11
7. Overview
• Live Security Competitions
• Situational Awareness
• Design of the 2010 iCTF
• Cyber Situational Awareness Metrics
• Lessons Learned
• Conclusion
Doupé - 7/12/11
8. The 2010 iCTF: A Cyber SA
Competition
• Introduced the concept of cyber-mission
• “Not all attacks are created equal”
• Participants must be aware of cyber-
missions and cyber-assets
• Attackers must time their attacks to cause
the maximum amount of damage
9. The Setting
• Teams are part of a coalition to bring down the
rogue nation of Litya
• LityaLeaks site used to leak description of Litya‟s
cyber-missions
• Litya‟s network protected by a firewall and an IDS
– If an attack is detected, nation‟s access is shut off
– Nations can bribe network administrator
• Litya has a botnet in each nation, stealing their
money
– If botnet is disabled, nation‟s access shut off
• Money made by solving side challenges.
16. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot
17. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
Internal
Network
VPN server
…
18. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
Internal
Network
Firewall/IDS
Briber
Flag Submission
VPN server
…
19. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
LityaLeaks
Internal
Network Challenges
Firewall/IDS
Briber
ScoreBoard
Flag Submission
VPN server
…
20. .
. The Bank
.
.
Service 1 Service 2 … Service 10 ScoreBot Botnet C&C
LityaLeaks
Internal
Network Challenges
Firewall/IDS
Briber
ScoreBoard
Flag Submission
VPN server
…
21. Competition Overview
• December 3rd 2010 ~8 hours
• 72 teams
• ~900 participants (largest at the time)
• 7 of 10 services compromised
• 39 teams submitted 872 flags
• 69 of 72 teams solved at least 1 challenge
• 37 GB of traffic
Doupé - 7/12/11
22. Analysis of iCTF Data
• Use the data to validate models and
theories
• We introduce two Situational Awareness
metrics:
– Toxicity
• Capture the amount of damage an attacker has
caused
– Effectiveness
• Capture how effective the attacker was at causing
damage
Doupé - 7/12/11
23. Analysis – CAD - Criticality
• C(s, t): service criticality [0,1]
– Expresses the criticality of service s at time t
– Function can have any shape
• iCTF: 1 when service active, 0 otherwise
Service: MostWanted
24. Analysis – CAD - Attacker
• A(a, s, t): attacker activity [0, 1]
– Represent the attacker‟s activity with respect
to a service
– Can have any shape
• iCTF: 1 when team attacked a service, 0 if no attack
Team: PPP Service: MostWanted
25. Analysis – CAD - Damage
• D(s, t): Damage to the attacker [0, 1]
– Represents the penalty for performing an
attack against service s at time t
– Can have any shape
• iCTF: 1 when service is inactive, 0 when active
Service: MostWanted
30. Overview
• Live Security Competitions
• Situational Awareness
• Design of the 2010 iCTF
• Cyber Situational Awareness Metrics
• Lessons Learned
• Conclusion
Doupé - 7/12/11
31. Lessons Learned
• The Good
– Pre-competition information prepared teams who
took advantage
– Winning team automatically qualified for DefCon
• The Bad
– Structure of the competition was complex and
was understood by a subset of the teams
– Services too hard
• The Ugly
– Intentionally put a root backdoor into bot
– Losing points sucks
Doupé - 7/12/11
32. Conclusions
• Live security exercises great for learning
and security education
• They can be designed to create a
research dataset
• Designed the 2010 iCTF to produce the
first publically available dataset on CSA
• Presented SA metrics: toxicity and
effectiveness