You got DevOpsed! Your sysadmin team got renamed as the DevOps team. Developers got prod access. Code deploys to prod happen multiple times a day now. In the eyes of the business, things are great. Yet, the security team continues to be left out and really nothing seems to be better. In fact it feels worse. Time to learn how to hack some devops for great good. This talk will equip you with advice and tools to join in on the devops. You will also leave with a sample continuous delivery pipeline that is armed to dangerous and ready to identify security issues in a typical web application stack. We'll use a range of open source technology including OWASP ZAP, gauntlt, brakeman, nmap, sqlmap, arachni and more.

1. DevOps for the

2. James Wickett
james@wickett.me
Austin, TX
Work at Signal Sciences
Rugged Dev Podcast
Gauntlt Core Team
LASCON Founder and Organizer
DevOps Days Austin and Global Organizer

3. We’re making AppSec effective and practical
signalsciences.com

4. in honor of SkyMall

5. If you find
yourself in
Austin, stop by!
Austin OWASP (last
Tuesday of the month)
LASCON Oct 22-23

6. Conclusions
It is easy to get discouraged in our industry, but
there is hope!
Agile, DevOps and Continuous Delivery practices
have an impact for AppSec / InfoSec
InfoSec is behind but has a unique opportunity to
add value

7. Conclusions continued
Integrating into the build pipeline and operational
tooling wins
Unit and Integration tests are not enough, we need
testing that focuses on attack tooling

8. you ==
“person you
once knew at
a previous
company”

9. Do you feel unable to
cause positive change in
your organization?

10. Is security ever left out of
important engineering
decisions?

11. Have you ever reported a serious
internal vulnerability only to see
it open 6 months or more later?

12. Why we are here

13. Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony,Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony,
Sony, Sony, Sony, Sony, Sony, Sony, Sony, Sony

14. Humans optimize for
the probable

15. We optimize for the
probable

16. Unit Testing

17. Integration Testing

18. Happy Path
Engineering

19. We optimize for the
possible

20. Over Engineering

21. Stress and load testing

22. We optimize for the
perceived probable

23. How do we perceive
what is probable?

24. Epistemological Problem
of Software Development

25. We attempt to solve it
by gathering data or
rhetoric

26. 3 approaches to solve the
Epistemological Problem
of Software Development

27. Arc 1:
Agile

28. Agile side-steps the
problem

29. Agile says we don’t know
what we are building

30. Solution: release features
to customers rapidly

31. Just Ship It!

32. Behavior Driven Dev

33. Behavior Driven Development is a second-
generation, outside–in, pull-based,
multiple-stakeholder, multiple-scale, high-
automation, agile methodology. It
describes a cycle of interactions with well-
defined outputs, resulting in the delivery
of working, tested software that matters.
Dan North , 2009

34. Amplify
the
feedback
loop

35. TLDR;
Rapid Iterations Win

36. Agile is our
guiding
light

38. We don't sell software like we
used to

39. Software as a Service

40. The last 15 years have
brought a complete
change in our delivery
cadence, distribution, and
revenue models

41. DevOps is the
application of Agile
methodology to system
administration
- The Practice of Cloud System
Administration Book

42. Arc 2: DevOps

44. Agile Infrastructure
@littleidea @patrickdebois at Velocity 2009
http://itrevolution.com/the-history-of-devops/

45. http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

46. First DevOps Days,
Ghent 2009
@patrickdebois

47. The opposite of DevOps is despair
- Gene Kim

48. DevOps is the
operationalism of the world
- Theo Schlossnagle

49. DevOps is a
community movement

50. http://dev2ops.org/blog/2010/2/22/what-is-devops.html

52. DevOps realized that
Ops doesn't know what
Devs know and vice
versa

53. DevOps is an
epistemological
breakthrough joining
disparate people around
a common problem

54. DevOps is an inclusive
movement that
codifies a culture
- Adam Jacobs

55. Dev : Ops
10 : 1
Traditional Dev to
Ops Ratio

56. “That the word #devops gets
reduced to technology is a
manifestation of how badly
we need a cultural shift”
- @patrickdebois
http://www.slideshare.net/cm6051/london-devops-31-5-years-of-devops

57. Culture is the most
important aspect to DevOps
succeeding in the enterprise

58. What we
value
determines
our culture

60. Mutual Understanding
Shared Language
Openness
Visualization
Tooling

61. DevOps is the inevitable
result of needing to do
efficient operations in a
[distributed computing and
cloud] environment.
- Tom Limoncelli

62. DevOps is not a
technological problem.
DevOps is a business
problem.
- Damon Edwards

63. http://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf

64. the first scientific study of the
relationship between organizational
performance, IT performance and
DevOps practices

65. DevOps practices
improve IT performance

66. Culture
Automation
Measurement
Sharing
@botchagalupe
@damonedwards

67. Culture Influencers
Decrease time from development to release
Blameless post-mortems
Reward failure and have a high emphasis on
testing
Unite different disciplines (like dev + ops) to solve
problems
http://www.slideshare.net/wickett/the-devops-way-of-delivering-results-in-the-enterprise

68. Antipattern:
Rebrand
your ops
team to
devops team

69. Automation

70. Seek automation to
increase repeatability

71. Chef, Puppet, Ansible, CfEngine
Rundeck, Mcollective
Jenkins, Travis, Kitchen
Cucumber, Gauntlt, ServerSpec
Vagrant, Docker
A Sample of the
Automation toolspace

72. Antipattern:
Manual
config of
production
environment

73. Beware of the
DevOps
Software
Solution

74. Measurement

76. Business Metrics
Event Correlation
Usage based monitoring

77. Sharing

78. Dashboards for all
Deploy Bot

80. Arc 3:
Continuous
Delivery

81. Continuous Delivery is
not merely how often
you deliver but how little
you can deliver at a time

83. Batch size of 1

84. Old Way
Changes break stuff, so
limit them and batch
them all together

85. New Way
Delivery of one change at
a time reduces outages,
increases performance,
and limits technical debt

86. You must deploy your stuff

87. Never Pass Defects to
the Next Step
The Practice of Cloud System Administration

88. Increase the Flow of
Work
The Practice of Cloud System Administration

89. Let the bots troll the
users for the lolz.

90. Allocate time to enhance
the build, test and deploy
system
The Practice of Cloud System Administration

91. Reduce code latency and
increase code velocity

92. The Next Arc: Security
Rugged

93. “… those stupid
developers”
- Security person

94. “Security prefers a system
powered off and unplugged”
- Developer

95. Cultural unrest with
security in an organization

96. Compliance Driven
Culture: PCI, SOX, …

97. “[risk assessment] introduces a
dangerous fallacy: that
structured inadequacy is almost
as good as adequacy and that
underfunded security efforts
plus risk management are about
as good as properly funded
security work”

98. Ratio Problem
Devs : Ops : Security
100 : 10 : 1

100. Security Tools
are run out-of-band

101. Security tools are
confusing

103. and when they are done they
give you this lovely gem

104. The tide is changing

105. Resiliency
Engineering

106. Netflix
famously
released
chaos
monkey

107. Rugged

110. The Rugged Manifesto
(excerpts)

111. I am rugged and, more importantly,
my code is rugged.
I recognize that software has become
a foundation of our modern world.
I recognize the awesome
responsibility that comes with this
foundational role.

112. I am rugged because my code
can face these challenges and
persist in spite of them.

113. http://www.slideshare.net/wickett/putting-rugged-into-your-devops-toolchain

114. ruggeddev.org

115. Rugged Journey
Quality
Transparency
Value Creation
Culture infusion

116. http://videos.2012.appsecusa.org/video/54250716

117. http://www.youtube.com/watch?v=jQblKuMuS0Y

118. https://speakerdeck.com/garethr/security-monitoring-penetration-testing-meets-monitoring

119. https://speakerdeck.com/mkonda/appsecusa-2013-insecure-expectations
http://vimeo.com/75930344

120. Security Tooling to
Delivery Pipeline

121. …to influence Culture,
Automation, Measurement
and Sharing

122. Security Testing
Static Code Analysis
Dynamic Testing
Virus Scanning
Code Signing Checks
Business logic/flow testing

123. Wouldn’t it be great if we
could automate our
security tests

124. http://static.hothdwallpaper.net/51b8e4ee5a5ae19808.jpg

125. You could say things
like…

126. The login page on my
application should not be
vulnerable to SQL injection

127. The search page should
not be vulnerable to XSS

128. Our app should not
have a backdoor in it

129. Enter Gauntlt

130. gauntlt.org

131. Integration tests meet
Dynamic App Security
Testing

132. Gauntlt Philosophy
Gauntlt comes with pre-canned steps that hook
security testing and attack tooling
Gauntlt functions as part of the CI/CD pipeline
Gauntlt is a good citizen of exit status and stdout/
stderr
Gauntlt does not install tools
MIT Open Source License

133. Security + Cucumber =
Gauntlt

134. Attack Logic
GIVEN
WHEN
THEN

136. Who uses Gauntlt?
CabForward

137. arachni
nmap
sqlmap
sslyze
dirb
garmr
generic

138. sqli
xss
fuzzing
forceful browsing
info leaks
heartbleed
…

139. TLDR;
Gauntlt automates
attack tooling

140. TLDR;
Gauntlt facilitates
collaboration

141. more on gauntlt
• Google Group > https://groups.google.com/d/
forum/gauntlt
• Wiki > https://github.com/gauntlt/gauntlt/wiki
• Twitter > @gauntlt
• IRC > #gauntlt on freenode
• Issue tracking > http://github.com/gauntlt/gauntlt

142. Try
this at
home

143. Fully functioning
attacking pipeline

144. Fork this repo

145. https://github.com/secure-pipeline/rails-travis-example

146. labs in ./velocity

147. ./velocity/lab_2/.travis.yml

148. ./Gemfile

149. ./velocity/lab_2/.travis.yml

150. ./Rakefile

151. ./test/attacks/xss.attack

152. ./test/attacks/xss.attack

153. ./test/attacks/backdoors.attack

154. ./test/attacks/sql_injection.attack

155. Gauntlt Next steps
Gauntlt currently doesn't install attack tooling, we
are working on a gauntlt docker container to
change that
Integrate into Kali distro

156. but wait,
there is
more…

157. github.com/secure-pipeline/
jenkins-example

162. +----------------------------------+--------+-----------------------------------------+
| Alert | Risk | URL |
+----------------------------------+--------+-----------------------------------------+
| Cross Site Scripting (Reflected) | High | http://localhost:3000/forgot_password |
+----------------------------------+--------+-----------------------------------------+

163. Conclusions
It is easy to get discouraged in our industry, but
there is hope!
Agile, DevOps and Continuous Delivery practices
have an impact for AppSec / InfoSec
InfoSec is behind but has a unique opportunity to
add value

164. Conclusions continued
Integrating into the build pipeline and operational
tooling wins
Unit and Integration tests are not enough, we need
testing that focuses on attack tooling