A talk about scaling web security

1. Leveling the Playing
Field
Aaron Bedra
Chief Security Officer, Eligible
@abedra
keybase.io/abedra

2. Right now, your web
applications are being
attacked

3. And it will happen again,
and again, and again

4. As you grow so will
the target on you

5. Keeping up with
security is difficult

6. Actually, it’s unfair

7. Things you have to get right Things the attacker has to get right

8. Time the attacker has to focus on you Time you have to focus on the attacker

9. It’s asymmetric
warfare

11. There’s no way to
manually keep up

12. Manual
Automated
Intelligent

13. Scaling your defenses
means strategic
automation

15. STOP!

16. Let’s talk about the
problem we are solving
for a minute

17. Problems
• We don’t know what people are doing
• We don’t know how often they are doing it
• We don’t know how effective we are
• We are don’t have enough resources to keep up

18. Goals
• Reduce noise
• Generate better signal
• Reduce operational overhead
• Build better business cases
• Spend energy on the really important stuff

19. Reducing Noise

20. It starts with really
simple stuff

21. Tie up the loose ends
with static configuration

22. Static configuration checklist
At least a B+ rating on SSL Labs*
Reject extensions that you don’t want to accept
Reject known bad user agents
Reject specific known bad actors
Custom error pages that fit your application
Basic secure headers

23. You’ll be surprised
how well this works

24. It has a fringe benefit of
creating better
awareness

25. You can feed this back
to your intelligence

26. Reducing Operational
Overhead

27. Dealing with malicious
actors has to be easy

28. It shouldn’t require
deploys, reloads, or any
potential forward impact

29. Let’s talk about how to
create something that will
help

30. Step 1
Put everything in one place!

31. Centralization of
events is critical

32. If you can’t see it, it
didn’t happen

33. There are options

34. Log aggregation and
a query engine

35. The query engine can
serve as your discovery
agent

36. A nice first step

37. But it will eventually fall over

38. That’s when you reach
for a messaging system

39. Log to topics in a
queue

40. Create processors to
understand events

41. Step 2
Process Events

42. For every event type you
will need to understand
how to process it

43. Structured logging can
help, but it doesn’t fit
everywhere

44. The goal is to accept an
event and return
consumable details

45. type logEntry struct {
Address string
Method string
Uri string
ResponseCode string
}
func processEntry(entry string) logEntry {
parts := strings.Split(entry, " ")
event := logEntry{
Address: parts[0],
Method: strings.Replace(parts[5], """, "", 1),
Uri: parts[6],
ResponseCode: parts[8],
}
return event;
}

46. You will likely have
multiple processors

47. Split topics by event
type or application

48. Once you have the data
accessible, figure out
what happened

49. Track everything!
• HTTP Method
• Time since last request/average requests per sec
• Failed responses
• Failure of intended action (e.g. login, add credit card, edit, etc)
• Anything noteworthy

50. type Actor struct {
Methods map[string]int
FailedLogins int
FailedResponses map[string]int
}
func updateEvents(event logEntry, counts *map[string]Actor) {
counts[event.Address].Methods[event.Method] += 1
if event.ResponseCode != "200" || event.ResponseCode != "302" {
counts[event.Address].FailedResponses[ResponseCode] += 1
}
if event.Method == "POST" && event.ResponseCode == "200" {
counts[event.Address].FailedLogins += 1
}
}

51. Once you have things in one
place, it’s all about counting

52. Simple counts with
thresholds go a long way

53. Step 3
Thresholds, Patterns, and Deviations

54. Exceeding a count is a
signal that something
needs to be done

55. There are a lot of signals
that could be malicious

56. You can start with simple
thresholds
• Too many failed logins
• Too many bad response codes (4xx, 5xx)
• Request volume too high

57. These provide a lot of
signal

58. But they don’t get you
all the way there

59. There are patterns of
behavior that signal
malicious intent

60. Example

61. 10.20.253.8 - - [23/Apr/2013:14:20:21 +0000]
"POST /login HTTP/1.1" 200 267"-" "Mozilla/
5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/
20100101 Firefox/8.0" "77.77.165.233"

62. 10.20.253.8 - - [23/Apr/2013:14:20:22 +0000]
"POST /users/king-roland/credit_cards HTTP/
1.1" 302 2085 "-" "Mozilla/5.0 (Windows NT
6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/
8.0" "77.77.165.233"

63. 10.20.253.8 - - [23/Apr/2013:14:20:23 +0000]
"POST /users/king-roland/credit_cards HTTP/
1.1" 302 2083 "-" "Mozilla/5.0 (Windows NT
6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/
8.0" "77.77.165.233"

64. 10.20.253.8 - - [23/Apr/2013:14:20:24 +0000]
"POST /users/king-roland/credit_cards HTTP/
1.1" 302 2085 "-" "Mozilla/5.0 (Windows NT
6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/
8.0" "77.77.165.233"

65. That was a carding
attack

66. As you dig in, you will
find many patterns like
these

67. But again it doesn’t
cover everything

68. There will also be
interesting deviations

69. 5%
5%
4%
27%
59%
GET POST HEAD PUT DELETE

70. Deviations in normal flow
are interesting but not
necessarily malicious

71. You will have to build more
intelligent processing to
understand them

72. Example

73. A password reset request
comes from a new
location

74. Is it a harmless request
or an account takeover?

75. Your processors will have to
make complicated choices
based on lots of information

76. Nailing deviation requires
the largest amount of
effort

77. Step 4
Act

78. Once you have enough
information to make a
decision, you must act

79. There are multiple ways to act
• Blacklist
• Whitelist
• Mark
• Do nothing

80. Blacklist and whitelist are
pretty straight forward

81. Blacklist when thresholds
are exceeded or
patterns/deviation fit

82. Whiltelist things you
never want to be
blacklisted

83. Marking is more
interesting

84. Marking allows you to tag
actors as potentially
malicious

85. This allows you to
dynamically modify your
responses

86. And choose how you
react

87. “Of course machines can't think as
people do. A machine is different from
a person. Hence, they think differently.”
-- Alan Turing, The Imitation Game

88. You can often render bots
useless with small
changes

89. Which exposes them
as bots

90. And gives you the
confidence you need to
blacklist them

91. Marking also helps you
lower the rate of false
positives

92. Step 5
Visualize

94. Visualization is
incredibly helpful

95. You need a window into your
automation

96. Spending a few minutes
a day looking at what
happened is vital

97. You can pretty easily
catch bugs this way

98. Architecture &
Peformance

100. There are three main ideas
• The thing that acts on actors
• The shared cache
• The event processors

101. Acting on actors
should be fast

102. Fast in a web request is
single digit milliseconds

103. You can choose to embed
this in your applications
or your web servers

104. Data locality is
important

105. It usually involves
replicating the global cache
to each decision point

107. The cache should hold
everything needed to act
on actors

108. The web server asks
the cache what to do

109. The event processors
work out of band

110. Their sole purpose is
to populate the cache

111. Processors tend to be
more custom

112. But the cache and the
acting logic is common

113. github.com/repsheet

114. Pitfalls

115. Things to consider
• False positives
• Decision latency
• Incorrect modeling
• Bad data
• Monitoring

116. There’s a good chance
you will block incorrectly

117. Make use of
whitelisting

118. Mobile carriers will be
a problem

119. So will NATed IP
addresses

120. Time to decision
should be monitored

121. Create a solid
regression suite

122. Run all your models
through it when you make
even a single change

123. Understand where bad
data can impact you

124. Build tolerance of bad
data so you don’t make
incorrect decisions

125. Monitor everything!

126. This type of automation
deserves every monitor
and metric you can get

127. Questions?