The evolution chain in security testing is fundamentally broken due to a lack of understanding, reduction of scope, and a reliance on vulnerability “whack a mole.” To help break the barriers of the common security program we are going to have to divorce ourselves from the metrics of vulnerability statistics and Pavlovian risk color charts and really get to work on how our security programs perform during a REAL event. To do so, we must create an entirely new set of metrics, tests, procedures, implementations and repeatable process. It is extremely rare that a vulnerability causes a direct risk to an environment, it is usually what the attacker DOES with the access gained that matters. In this talk, we will discuss the way that Internal and external teams have been created to simulate a REAL WORLD attack and work hand in hand with the Defensive teams to measure the environments resistance to the attacks. We will demonstrate attacks, capabilities, TTP’s tracking, trending, positive metrics, hunt integration and most of all we will lay out a road map to STOP this nonsense of Red vs BLUE and realize that we are all on the same team. Sparring and training every day to be ready for the fight when it comes to us. This is an update to our 2016 Brucon talk. We plan to discuss what have we accomplished regarding the above in the last year. We plan to show how we have progressed with the automation of attacker activities and event generation using MITRE’s Cyber Analytics Repository & CAR Exploration Tool (CARET) along with pumping these results to Unfetter (https://iadgov.github.io/unfetter/) for aggregation and display in a useful format.

1. aka: Some new term to use because we keep screwing up terminology and treating people
like children
with a crayon box
Adversarial Modeling Exercises
Simulation

3. HI…I’m Chris

5. and…I’m Chris

6. Chris Gates - Sr. Security Engineer - Uber
Twitter: @carnal0wnage
Blog: carnal0wnage.attackresearch.com
Talks: slideshare.net/chrisgates

8. Trigger Warnings
● I'm 20yrs in and feel like we are going backwards
● I respect everyone in this room as a peer that can help solve
the hard problems
● My opinion is my own, but I bet a few of you share it =)
● I'm SO sick of all the b1tching
● Status quo is unacceptable
● No matter what stupid example I use, its to make it light
hearted and not INTENDED to offend
● If I do offend you… oh well
● My finger of blame points with one finger forward but 3 at
myself
● If you disagree or wanna add something in SPEAK UP! I
don’t bite.
● I don’t have answers, but I am trying to figure it out.
● I’ll work hard to not waste your time

9. http://www.pentest-standard.org/

16. Problems With Testing Today
• Limited metrics
• Increased Tech debt
• Fracturing of TEAM mentality
• Looks NOTHING like an attack
• Gives limited experience
• Is a step above Vuln Assessment
• Is NOT essential to the success of the
organization
• Is REALLY just a glorified internal pentest
team

17. https://twitter.com/B00ms1ang1357/status/921742319887568896

18. Building a successful internal
Adversarial Simulation Team

19. Tester Terminology:
Gotta get a few things straight first
We keep screwing up terms
• Vulnerability Assessment person ( U ran a Vuln scanner?)
• Penetration Tester ( U hit go go empire/msf buttons)
• Red Teamer ( Above + went laterally used bloodhound,
CME and got DA? MAYBE, got “sensitive stuff”)
• Purple Teamer (U did all of the above but charged more to
talk with the defense teams during/after the test)
• ADVERSARIAL ENGINEER (U exist to simulate real world
TTP’s, generate experience, and provide metric scoring of
corporate readiness /resistance to attack… or just another
$$$ Pentester)
or some other random
shit we will make up next to kill Adv. Eng.

21. Last Year (Jan 2016)
● There is the MITRE ATT&CK **thing**
○ It’s weird but ok, we should look into it and see if it’s
useful
● It makes sense to lay out what our security tools and
appliances actually do
● It makes sense to map your detection rules against
ATT&CK to see where you have coverage
● Let’s start thinking about automating adversarial
simulation

22. Conduct a attacker capability
Assessment

24. Conduct a defense controls inventory

25. Wins Stats (mutual)
● Tons of net new findings
● First time we could visualize things we were missing
● Visualize heatmap of coverage for controls
● Proved effectiveness of tools
● Saves Millions in M&A assessments
● Provided defenders a “map of blindspots” magic decoder ring.
● Huge social awareness in org
● Community rally to evolve testing and generating defensive
metrics
● Defense is getting sexy
○ Testing prevention/detection to attacks is way more
interesting than DA.

26. Losses stats (mutual)
● Awesome idea … shitty ability to convince others
(enrollment)
● Measurement is still a mystery / WIP
○ We need some/more defensive telemetry
● Turned into a pentest team
● So many techniques to cover (150+)
● When people leave, so does the info ( institutionalized)
● Reports are a bitch
○ Non-existent, we have to make new reporting templates
● So much data you lose things in the pile
● Too new for tools to handle/manipulate * livin on edge*
● “errybody” is “developing and aligning” meaning no one is
DOING .

27. So Manual…

28. Excel HELL

30. NOPE!
https://twitter.com/malcomvetter/status/913792656794304512

32. What to tackle next?
● Scalability
● Measurement
● Stale Data
● Data Decay
● Environments change….. A LOT!
● Detection was as good as the person ticket was
routed to.
● Consistency
● Tons of manual labor/ time vampire
● Loss of faith (takes forever)

33. We
don’t
Scale!

35. Why don’t we measure
anything?
● What does it do?
● Does it work?
● Does it do what it says it does?
● Does it fit?
● Is it consistent?
● Is it accurate?
● Is it reliable?
● Is it repeatable?

36. Time for a new term.
+ =
Trademark CG & CN

37. Trademark CG & CN
Telemetry is an automated
communications process by which
measurements and other data are
collected at remote or inaccessible
points and transmitted to receiving
equipment for monitoring.

38. This Year (2017)
So, I challenged work to let me implement/continue of
the initiatives from last year

39. Validate Detection Capabilities
Mapped our detection rules to MITRE ATT&CK
○ It was valuable to see where we had coverage and
where we didn’t with our various tools
Refactored all rules to map to their ATT&CK Phase and
Technique

40. Validate Detection Capabilities
Refactored all rules to be version controlled (yaml).
○ Making easier for all analysts to contribute.
○ Utilize various yaml fields for tool enrichment

41. (Automated) Rule QA
Detecting threats requires logging pipelines
o Activity logs from various systems
o Logs must be transported to something that aggregates and processes
§ Each step in this process is a place the pipeline could fail
§ Pipelines can go down for a variety of reasons
● Vendor schema changes
● Vendor outages
● Log processing issues (malformed logs / amount of logs)
● Firewall (host/network) changes
● OS issues
● Exceeding licenses
● Forgetting to pay the bill
o How do we ensure our rules are still current? Effective? AND working?

42. Complex Log Pipelines

43. (Automated) Rule QA
Wanted a way to programmatically validate the detection pipeline is functioning
o No/Some alerts doesn’t automatically mean all is well
o It probably means a log pipeline is down
o Daily tests of the pipeline seem like a good idea
§ Give us metrics around time of action → phantom case
Wanted a way to perform ongoing rule unit testing (QA)
o If a detection rule has zero results...it means:
§ We aren’t pwned or...
§ The rule doesn’t work correctly (when was the last time it did?)
o For every detection rule, we want to have a way to generate an event that
causes the rule to fire
§ Some rules fire all time, others almost never...do they work?
o Doing this manually is no fun/not scalable (bat file) :-(

44. (Automated) Rule QA
How
● Piggyback off the new Rule Framework and alert monitoring system
(Phantom Cyber)
● Use Virtualization (vagrant) so we can reset the host as required
o Ex. Executing malware or programs that leave host in an
undesirable state
● Use a Task Queue (Celery) to handle all the execution / requests /
responses
● Flask endpoint to receive results from Phantom Cyber

45. (Automated) Rule QA

46. (Automated) Rule QA
Use the YAML fields to hold our data

47. (Automated) Rule QA

48. (Automated) Rule QA

49. Where to get rules
● https://github.com/Neo23x0/sigma
● https://github.com/veramine/Detections/wiki
● https://car.mitre.org/wiki/Full_Analytic_List
● https://twitter.com/subTee :-)
Full Disclosure...it’s a PITA. There isn’t a great resource
for these rules, you have to make them yourself.

50. Adversarial Simulation
● What do you do when you have a couple hundred
“attacker” actions and an engine to queue them up?
● You automate them to do attacker stuff :-)
● https://github.com/uberXXX/metta
● SOON...working thru open sourcing process :-/

51. Adversarial Simulation
● Broken down by MITRE ATT&CK Phases and APT
Groups
Examples:

52. Adversarial SimulationSingle Technique

53. Adversarial SimulationScenarios

54. Adversarial SimulationScenarios

55. Adversarial SimulationScenarios + Basic Reporting

56. Adversarial SimulationOther Cool Projects
Atomic Testing by Red Canary
https://github.com/redcanaryco/atomic-red-team

57. Tools
DumpsterFire
https://github.com/TryCatchHCF/DumpsterFire

58. Tools
Blue Team Training Toolkit
https://www.bt3.no/

59. Unfetter
Using unfetter to track progress/coverage
● https://github.com/unfetter-discover/unfetter
● Mapping attack groups
● Assessments

60. Unfetter
Mapping attacker groups

61. Unfetter
Mapping attacker groups

62. Unfetter
Mapping attacker groups

63. Unfetter
Assessments

64. Unfetter
Assessments

65. Unfetter
Assessments

66. Metta Maps of the Future
Possible Plugins
● Unfetter ( grab, post, report)
● CarbonBlack
● Splunk
● Phantom
● Anything with an API is possible
C2
○ Terraform/ansible/etc to build c2 endpoints
Malicious file attachments
○ https://github.com/carnal0wnage/malicious_file_maker/
Malicious office macros
○ https://github.com/trustedsec/unicorn
○ https://github.com/curi0usJack/luckystrike
○ https://github.com/Mr-Un1k0d3r/MaliciousMacroGenerator
AWS
○ Programmatically build vuln AWS services/infra - can BT detect?

67. Additional Telemetry
● Total Coverage
● Mean Time to Detection
● Mean Time to Remediation
● % Successful Eradication
● Protection Metrics
● Automated vs Manual Protection
● Automated vs Manual Detection
● Automated vs Manual Response
● Defender proficiency & capability
● Tool/Product Reliance
● Product coverage heatmaps
● Pre-Flight verification
● Net fluctuations

68. Defensive
Coverage
Attacker/Defender
Difficulty
METTA
Defender
Capability
Self
Healing
Defense
Offensive
Inputs
Response
Auto scaling
Protection
Capability

70. Chris Gates
Twitter: @carnal0wnage
Blog: carnal0wnage.attackresearch.com
Talks: http://www.slideshare.net/chrisgates
Video: https://vimeo.com/channels/carnal0wnage/
GitHub: https://github.com/carnal0wnage
Chris Nickerson
Twitter: @indi303
Talks: https://www.slideshare.net/indigosax1
Video: https://vimeo.com/laresconsulting
Company: http://www.lares.com