Jo Rhett's talk from http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/129859602/

1. Where fans buy & sell tickets™
Security in the Deployment Stack
SF Bay Area Large Scale Production Engineering
October 17th, 2013
Jo Rhett
Senior Operations Architect

2. Disclaimer:
No StubHub! solutions are being discussed in
this presentation, because:
1. I’m drawing from 20+ years of experience.
2. I’m an FNG at StubHub! and I haven’t
learned all the great things we do yet.
3. We’ve got even more entertaining ways to
break things.

3. Common Security Paradigms:
1) Analyze every decision for security benefits.
2) Hard exterior, “soft and gooey interior”
3) Strict (or not) host level security
4) Strict (or not) application level security

4. These Paradigms are incomplete:
1) Only small sites can be understood completely.
2) Applications require differing levels of network
access, which creates “gooey spots” in supposedly
crunchy exterior.
3) Doesn’t account for insecure by design:
- Wikipedia, Craigslist
4) Doesn’t account for center-less design:
- BitTorrent
5) Insecure by ignorance: “You could do that?”

5. Insecure by Protecting One Layer:
Hardened network
prevents external logins
§ Application hacked
to create outbound
console
reverse shell
§ Application hacked
to change data
without login
SQL injection,
remote file inclusion
Hardened application
enforces security paradigm
– versus –
§ Network attacks disable
service
slowloris, slowpost
§ Network attack
bypasses authentication
session hijacking, replay
attacks, authorization
sniffer

6. Insecure by Ignorance:
Easy to laugh about, much harder to solve.
1) Network security is aimed at protecting protocols.
2) Most things are done on the same protocol today.
3) Decisions made in different teams create an
interwoven mesh of security expectations.
- Network only allows HTTPS access
- Web API allows DB access via HTTPS

7. Insecure by Design:
1) Each web service provides one or more “features”
2) Security has to identify every approach vector, and
the intersections between each feature.
3) An attacker only has to find one feature that provides
data or hook which makes it possible to meet the
expectations of a different feature.
4) Complete security review requires N^N-1 analysis.

8. Insecure by Politics:
1) Draconian security controls frustrate teams and
produce “outside” or “bypass” implementations
2) Failure to understand the value of cracking your
application. If the attacker can make/save money,
they will be very persistent.

9. Access versus Authorization:
“If we all have root, anyone can fix a problem...”
1) Most security instruments deal with
authentication, not authorization.
2) Static files, DB Lookups, LDAP, Java Keystore…
3) Many implementations use “if can access” as the
authorization scheme.
4) Projects require more access to enable Mobile
and batch processing clients.

10. “If Can Access” Authorization:
Real e-mail:
Hey guys, this is #####. So I built out our application
stack in the office and took it home on my laptop.
Turns out, I can test the entire thing from home if
you’ll give me direct database access…
This was possible because every dependency was a
Tomcat application available on an exposed role.
But wait… It gets better!
Followup e-mail:
Nevermind, I realized I can point at our production
gateway for the DB API calls and it works fine…

11. You can’t lock them out:
1) Application service providers often must expose
APIs to customers
2) Mobile customers don’t use “trusted networks”
3) The consumer provides valuable content
- Wikipedia
- Craigslist
- StubHub! to some extent

12. What the Business Wants: Security
1) Lots of eyeballs
2) Careful analysis
3) Stability is paramount
== Slow

13. What the Business Wants: Agility
1) Lean teams
2) Respond to market quickly
3) “Let’s break things!”
== Fast

14. Every person in this room stands
between these points every day:
1) Keep it up no matter what.
2) Move faster! Let’s break things!
Feels real comfortable, doesn’t it?

15. Security in the Deployment Stack
We’ve discussed the problem.
Now it is time to talk about solutions.

16. If every component must be secure,
How can you do it better and faster?
1) You make it smaller.
2) You make it simpler.
3) You do it more often.

17. Smaller Security Evaluation
1) Security lead from each team evaluates proposal
2) Evaluate each intersection -- the small points
3) Build shared knowledge
This collaboration allows each person to focus on the
components they know best.

18. Simpler Design Theory
1) Make each component as small as possible
2) Use components that can be reused.
--avoids “what the heck is this again?”
3) Less moving parts with common dependencies

19. Test More Often
1) Test for new potentials identified in review
2) Test previous release concerns, every time
3) Test production too
4) Automated testing required
humans get bored

20. Deployment -- devOps
1) Application security needs and deployment needs
are not the same
2) Automated is better: manual deployments lead to
inconsistent deployment
human mind: didn’t I do that already?
3) Operations Developer writes code/policy to deploy
application stack
a. Kickstart / Cobbler / Foreman / Razor
b. Cfengine / Puppet / Chef
c. Mcollective / Capistrano
d. …custom built
4) Commit the deployment code with the release
branch

21. Security in the Deployment Stack
Keystores
Common Security Keystores
1. Static files
2. Database lookups
3. LDAP
4. Java Keystores
5. SSL Certificates
All of these can be installed during your application
installation, or outside of it (system build or side load)

22. Security in the Deployment Stack
Install during System Image
Install security keystores during system build
Advantages:
1. only needs to be tested as a unit
Disadvantages:
1. no updates to an existing role without rebuild

23. Security in the Deployment Stack
Install with System Packages
Install security keystores with system packages
Advantages
1. only needs to be tested as a unit
2. simple redeploy with system packages
Disadvantages:
1. requires package rebuild for simple changes
2. limited scripting language for install/uninstall

24. Security in the Deployment Stack
Install with Config Management
Implement security keystore as policy implemented by
configuration management.
Advantages:
1. Configuration change can be pushed
2. Incremental upgrades are possible
3. Complex upgrade procedure can be
implemented
4. Failover, failback process can be
employed
Disadvantages:
1. must test the deployment as well as the unit

25. SUMMARY
1) Security models don’t match modern expectations.
2) Security failures are not usually Apps or APIs, but
the intersection of access between several of them.
3) evaluate Smaller, build Simpler, test More Often.
4) Develop deployment code/policy as carefully as you
do the application. Commit it with the release branch
of the application.
5) Automate your tests.
Automate your deployment.
Automate failure analysis.

26. Where fans buy & sell tickets™
We are hiring!
Operations Architects
Tools & Automation
devOps
Site Operations
Network Operations