With the proliferation of cheap bandwidth and vulnerable systems the DDoS attack volume has increased tremendously over the last years. The talk will cover current threat models and possible countermeasures to mitigate the attacks when they should happen

1. DDoS mitigation in the
real world
19th May 2014
Vienna DevOps/Security Meetup

2. Michael Renner
@terrorobe
My name is Michael Renner
In IT since 2001, focus on system administrations and web operations.
Freelancing since 2010 for multiple companies doing a hodge-podge of
tasks - from building data centers, DBA work, troubleshooting,
performance optimization...
If you've got questions, just ask!

3. Company I'm freelancing for.
Viennese company, founded around 2010 - our offices are on Mariahilfer
Strasse
Operations, Development, Customer Care, payment solutions, SMS
delivery, you name it ;)
And since a year ago also DDoS protection

4. DDoS?
In computing, a denial-of-service (DoS) or
distributed denial-of-service (DDoS) attack is an
attempt to make a machine or network
resource unavailable to its intended users.
Source: http://en.wikipedia.org/wiki/Denial-of-service_attack

5. Why people DDoS
...which leads us to the topic of our today's talk.
DDoS is something that has gotten more and more news coverage in the
last few years.
Chances are high that you already tried using a service while it was being
DDoSed or that you run a service on your own which was the target of an
attack
So why do people DDoS?

6. ...to extort money
This one is easy.
Was very prevalent in the gambling and sports betting area. Back at Inode
in 2003 people where already DDoSing gambling companies!
"Comes with the business"

7. ...to underline a
political agenda
Operation Payback in 2010 - done by Anonymous - targeted against
recording industry prosecuting piracy as well as finance companies
denying service to wikileaks

8. ...to escalate issues in
customer care
Has happened in the past - if you annoy the wrong customer he might
escalate things by his own means

9. ...because somebody
pissed them off on IRC
User chats from your server/network, annoys the wrong people and they
show them that they mean business

10. ...to prove a point
...or just to prove a point, whatever that might be.
The more high profile/recognized your site is the higher the chance that
somebody will attack it just for the fun of it.

11. DDoS attacks have
asymmetrical costs
The problem with DDoS attacks is that they have highly asymmetrical
costs.
The attack itself can be quite cheap whereas the lost revenue as well as
time and money to bandaid-patch an attack can be quite expensive

12. Attacks are cheap
"Prices per hour may depend on your target. Price starts 10$ per hour to
100$ per hour depending on how Huge or Protected the target is."
When we're talking money one must know that attacks are quite cheap.
This is an example from end-customer site, completely with youtube
video and everything!
Depending on how smart the attackers are and how good you're
connected attacks can be significantly cheaper.

13. Attacks happen when
people have free time
• Weekends, holidays, bridging days
• In their respective timezone!
• More often than not:
• the time when you'd like to have free
time as well
Another problem is that the attacks will happen when people have free
time.

14. Attacks are hard to
identify
• "The DB server has too many connections
open"
• "I'm seeing packet loss and a large amount
of softinterrupts on CPU0"
• "The server is down"
DDoS attacks are very disorienting
Depending on how much instrumentation you have for your
infrastructure you might not even be able to tell that you're getting
DDoSed
"Newrelic might not be enough to spot it"
Depending on attack you might need OS, application level even network
packet statistics.

15. Mitigating is hard without
proper infrastructure
• "Let's see if I can whip up an u32 classifier
to filter out this pattern I'm seeing"
And when the attack is already running you'll often have a hard time jury-
rigging yourself out of the attack.

16. Hard to find the
sources of attacks
• Most attacks use spoofed source addresses
• If it's a reflection or amplification attack
you won't even get those.
• And even if you'v got real sources, it's a
hacked server in nearly all of the cases
More often than not management will be interested on who's doing the
attack and why.
This is nigh impossible if you're not the NSA, having a big picture of all
the traffic going on.
And even if you've got "real" source addresses nearly all of them will be
hacked servers connected to an anonymous C&C servers

17. In the end...
• ...it's you and your customers who're
unhappy
so it's best to try not to get knocked off the net in the first place!

18. DDoS war stories
To give the users who haven't experienced a thorough attack yet a
glimpse of what it may look like I've collected a few things we faced in
the past.

19. So you run a website...
Most of you operate a web/online service I guess, which generates you
fame, fortune or both.
And along the way you picked up a few people who don't have the best
interests regarding your site, and so they decide to...

20. Somebody does a slow-
read attack
...open up hundreds of connections, read a few bytes at a time...
basically exhausting open workers, tcp connections, file descriptors,
memory, whatever comes first.
so you get a proxy setup like CloudFlare up and running, all HTTP
requests proxied through
They detect the botnet IPs either through pattern matching or existing
botnet IP lists and start filtering those out or presenting them with
captchas. Attack solved.

21. But they still know
your server's IP address
Then the attacker goes on netcraft.com, checks the former IP address of
your site
...and decides to send a few hundred megabits worth of UDP packets
your way.
You ask your ISP for a new subnet, renumber your servers, add the new IP
addresses to your proxy setup and hope for the best
Things are quiet for now.

22. Suddenly you get a
ERR_NAME_NOT_RESOLVED
when visiting your site
After a few hours the monitoring starts to page you. You check your site
and you get a NAME NOT RESOLVED.
...turns out the cheap registrar where you bought your domain
...who still does your DNS
...isn't quite DDoS resilient.
So you shop around, see companies like dyn.com or Amazon route 53,
move domains over, wait for cached entries to expire and after a few
hours you should be back online!

23. You notice that you're
sending mails
[..]
Received: from web01.yourcoolapp.com ([198.51.100.42])
Subject: WELCOME TO YOURCOOLAPP!
From: noreply@yourcoolapp.com
To: evilattacker@example.com
But then you notice that you're sending emails! And your attacker noticed
that the email address contain interesting information in their headers.
...and the DDoS attack on your server resumes.
...so you configure your mail gateway to not include servers IP addresses
...get new IP addresses
...hope for the best

24. ...so you think you're
good for now.
And then your ISP goes
offline
By then you think you've seen it all and you think you're good to go.
...most smaller ISPs are not equipped to sustain attacks even 10 to 50
gigabits in size.
If regular traffic is in the single-gigabit per second range it's easy to
knock them offline. Usually they then just filter the destination IPs at their
carriers and be done with it.

25. Mitigating attacks...
in Theory!
...so, this should give you a nice overview what can happen in the real
world.
So what can we do about these attacks?
In the end all of the DDoS Mitigation services out there are doing the
same thing.

26. Increase your attack
surface
• Geo-distribution is the way to go
• either via DNS or IP anycast
• -> more traffic sinks!
First part is increasing the attack surface
Most customers have just a single datacenter, amazon availability zone
and so on. Very easy to concentrate your attack on.
If you operate multiple Datacenters it's much harder to knock all of them
offline - loadbalancing done either via DNS & geoip or IP anycast setups
- same prefix announced on multiple locations
This all boils down to have more traffic sinks to filter the traffic out on

27. 150 Gbit/sec sounds
scary
• Less scary when it's:
• 40 Gbit Hongkong
• 20 Gbit Frankfurt
• 35 Gbit London
• 25 Gbit US West
• 30 Gbit US East
I'm just pulling a few numbers out of my hat to give you an example.
Suppose you've got a 150gbit attack running - which to this day is still a
very large number when it comes to DDoS attacks - this gets far more
manageable when you operate multiple sites where the traffic has to go
to
And even if they manage to knock one of the sites off the net all the
others are still online and answering requests.

28. Scrubbing the traffic
• In general: you want to filter out all
"DDoSy" or anomalous requests from your
traffic
• Statistical models, signature-based filters,
hand-written rules, etc.
• Specialized hardware with in-ASIC rule
processing
• Beyond the scope of this talk
The other big part is to scrub the traffic and remove all packets which
belong to "probably unwanted" traffic.
The further down the traffic goes through your stack the more expensive
is the processing of a malicious request. Compare the time needed to
throw away a request on the firewall vs. throwing it away in the web
application.
Big Companies use specialized hardware with network chips, being
capable of doing complex rule processing at wirespeed of the interface.
They often use statistical models to detect network anomalies, have rules
for known attack patterns (e.g. NTP reflection attacks) and can also
deploy custom rules specific to attacks currently going on.

29. Don't build your own
scrubbing setup unless
you've got good reasons
In most cases it's much cheaper to just lease scrubbing services from one
of the established players instead of building your own infrastructure.

30. Mitigating attacks...
...in Practice!
...so much for basics - what can we actually do to improve our security?

31. Don't do DDoS mitigation
as an afterthought
First and foremost...
it's much nicer to talk and discuss about these things during office hours
with sober colleagues than on a saturday night at 4 am after 8 beers

32. Decide on a scope
• Threat modeling
• Build playbooks
• Prepare & implement low-hanging
fruits right now
You need to do threat modeling - which attacks do you expect, how
you're going to manage them. Prepare contacts up front even if you don't
want to buy something right now.
Have a playbook describing how you're going to react
Implement low hanging fruits right now!

33. Have good OpSec
One example for a low-hanging fruit: Operational Security
Make sure you don't leak IP addresses you don't want to
It makes sense to have two different network segments - one for public-
facing IPs, another one for the actual servers. Actual server IPs don't even
have to be public ones if you can avoid it.
Things to check:
Mail Headers
Test systems like beta.yourcoolapp.com
MX records!

34. Choose an approach
And as with every idea - there's more than one way to do it!

35. Proxies
• The easy/cheap solution
• Protocol proxies (HTTP, SMTP, etc.)
• Geo-distributed
The cheap approach would be a proxy setup. You get a list of protocol-
level proxies where you point your DNS records to - you configure the
destination for the proxies and all the traffic hits these servers first.
Well-known company is CloudFlare, offering this service even for free
Good if you need a fast solution

36. Proxy Caveats
• Few control over proxy servers
• Can hurt troubleshooting times
• Applications don't see end-user IPs
anymore
• Fix for HTTP: X-Forward-For
• If your real IPs are leaked it's game over
again
• Not feasible for all protocols
This comes not without draw-backs though.
If you need to troubleshoot things you might not have the level of control
you'd like, since somebody else is already terminating your HTTP
requests for you.
Your applications don't see IPs anymore
If you should leak your IPs it's easy for an attacker to just go for your real
servers.
And it's not feasible for all protocols, VoIP, UDP-based gameserver
protocols, etc.

37. Routed solution
• You need a /24 network prefix
• IP tunnel to scrubbing center
• All scrubbing locations announce the prefix
• Traffic gets sent to closest location,
scrubbed and forwarded to you
Those get easier to manage if you've got a routed setup
Routed approaches are more expensive and complex to maintain
Pro: Protocol- and application agnostic

38. Routing caveats
• Takes time to implement & test properly
• More expensive to setup and maintain
• Routing & BGP can be scary
It's not without it's drawbacks though:

39. Appliances
There's still appliance vendors in the DDoS market, e.g. Arbor or
Radware.
Was a bit surprised by that, but it absolutely makes sense
Appliances targeted at end users should always come with a traffic
scrubbing offer, otherwise they're snakeoil - it's easy to DDoS a single
device running behind even a 10Gbit link.
They can make live easier though since they automate tasks like route
changes and provide additional information for the scrubbing center

40. Ask your ISP
something you shouldn't neglect is that your ISP may already have DDoS
infrastructure set up - it doesn't hurt to ask them what it is they're
running and if they can make you an offer

41. Common caveats
• Your applications latency is most likely
going to increase
• There can be false positives & overfiltering
• Your system is becoming more complex
Something to consider with any choice you're going to make is
Latency is going to increase
false positives and overfiltering - how are you going to spot this?
your system is getting more complex

42. Interested?
Talk to us!
info@ixolit.com
m.renner@ixolit.com
I hope that gave you a bit of an overview of how the current situation
looks like when dealing with DDoS
If you're interested - please talk to us
Ixolit is both a Prolexic and a Dyn Partner
Prolexic is one of the major traffic scrubbing operators in the world. Got
recently acquired by Akamai.
Dyn.com offers anycasted DNS setups which are DDoS resilient
Already have a few production setups running for _huge_ sites which
solved all our DDoS issues to this day

43. Questions?
Do you have any questions?
Thanks!