As the world of system and application deployment continues to change, the sys admin and security community needs to change with it. With agile development, continuous deployment, the pace of change in IT has only increased. Add in Dev/Ops and the traditional sys admin and security processes just don’t work. How can you rapidly deliver servers and applications while making sure they are built reliably and securely. Rackspace has been developing a tool to help them design, deploy and security assess complex configurations for customers called Checkmate. This talk will cover the concepts behind and the architecture of Checkmate and how it helps minimize the time to deploy systems and verify they have been created to spec and in a secure state. A discussion of how Checkmate has inspired the concept of Test Driven Security based on the Test Driven Development model familiar to the development world.

1. DEV/OPS, CONTINUOUS DEPLOYMENT
& APIS, OH MY!
Matt Tesauro, Texas Linux Fest
– San Antonio, TX, August 2012

2. 2
WHO AM I?
Matt Tesauro – Cloud Application Security Guy + OWASP
Racker since October 2011
Rackspace’s Cloud Product Group
Work with developers and QE
matt.tesauro@rackspace.com
OWASP International Foundation Board
Member and Treasurer
Project Leader of OWASP Live CD &
OWASP WTE projects
matt.tesauro@owasp.org
RACKSPACE® HOSTING | WWW.RACKSPACE.COM

3. 3
RACKSPACE® HOSTING
The Service Leader in Cloud Computing
4,000+
RACKERS WE SERVE
172,000+
CUSTOMERS
40% OF
THE
FORTUNE® 100
120 +
COUNTRIES
9 GLOBAL DATA
CENTERS
LEADER IN
GARTNER'S MAGIC
RAX QUADRANT FOR
MANAGED HOSTING
2008, 2010, 2011
& 2012
RACKSPACE® HOSTING | WWW.RACKSPACE.COM

4. OUR VISION
To be recognized as one of the
World’s greatest service companies.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
4

5. RAX CLOUD APPROACH
Open source orchestration, management & provisioning
cloud platform
RACKSPACE® HOSTING | WWW.RACKSPACE.COM

6. 6
THE FUTURE: FANATICAL SUPPORT
ANYWHERE
Rackspace Provides
The Fanatical Support
DEDICATED PUBLIC CLOUD PRIVATE CLOUD PRIVATE CLOUD PUBLIC CLOUD
RACKSPACE LOCATIONS CUSTOMER SITE PROVIDER DC
• One Control Panel across OpenStack connected clouds
• One Fanatical Support Team
• Our Cloud, Your Cloud, Partner Hosted OpenStack Cloud
• Global Reach
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
6

7. SECURING APPS IN A
DevOps WORLD
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
7

8. A quick Overview of DevOps
• The combination of traditional development activities with operations and
testing (QA/QE)
• Collaboration, communication and integration is key
• Agile development model (sprints, scrum, …)
• Release coordination and automation
"DevOps" is an emerging set of principles, methods and practices for
communication, collaboration and integration between software development
(application/software engineering) and IT operations (systems
administration/infrastructure) professionals.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
8

9. CI, CD, CD, TDD and API
CI == Continuous Integration
CD == Continuous Deployment
CD == Continuous Delivery
TDD == Test Driven Development
API == Application Programming Interface
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
9

10. 10
THE PROBLEM
• Cycle time for software is getting
shorter
• Continuous delivery is a goal
• Scanning windows are not viable
• First mover / first to market
advantage
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
10

11. THE PROBLEM – or at least more
• Traditional software development left little time to test
• DevOps, Agile and Continuous Delivery squeeze those windows
even more
• New languages and programming methods aren’t making
this better
• Growth of interpreted languages with loose typing
hurts static analysis efforts
• Few automated tools to test APIs especially
RESTful APIs
• Little time for any testing, manual testing is doomed
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
11

12. 12
THE SOLUTION
• Automated software
testing
• Automated operational
infrastructure
• Automated security
testing
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
12

13. Think like a developer
Sprints break software into little pieces…
• Break your testing into little pieces
• Use your threat model to know the crucial bits to test
Long and short running tests
• Testing time drives testing frequency
• Code for tests needs to be optimized
Smoke test versus full regression test
• Smoke test early and often
• Full regression tests on regular intervals
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
13

14. Maximize what you’ve got
Make the most of your frameworks
• Embrace, understand and fill gaps where necessary
Make the best use of your time…
• Make tests easily repeatable
• Make tests easy to understand
• Make tests abstract and combine-able
• Ala carte tests for mixing and matching
• Think about the Unix pipe | and its power
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
14

15. Test Driven Development Security
Under the constraints of DevOps, Continuous Deployment
Your testing has to be nimble
Dare I say…Agile
In TDD, you know your code works
when the tests pass
In TD(S), you know your app has met
the baseline when the tests pass
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
15

16. A snail on fire!
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
16

17. 17
AUTOMATING
• Declarative configuration language
• Plain-text configuration in source control
• Fully programmatic, no manual interactions
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
17

18. 18
CHEF
Server / Hosted / Private
1. Solo
Racker
2. Server
3. Hosted
4. Private Hosted Node
Node
Node
Node
Node
Node
Node Node Node
Node Node Node
Node Node Node
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
18

19. 19
COOKBOOKS
• Most major software
packages have cookbooks
• You will have to write your
own / customize
• Good place to spend
security cycles
- Merge patches upstream for
extra points.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
19

20. 20
GROUPING & TAGGING
Node • Tagging your
Node
Apache Node servers applies
Node
Web the required set of
recipes
Node • A base set of
Node
MySql Node
Node
recipes is
DB
common
• Each server will
Node
Node
have multiple tags
Memcache Node
Node set at bootstrap
Cache
time
Monitoring 20
RACKSPACE® HOSTING | WWW.RACKSPACE.COM

21. 21
LIMITATIONS
• Focus on single machines Cloud Load Balancer
• A multi-box configuration
is based on copying
Web Web Web Web
existing configurations
• No support for implicit
application or environment Memcached
Database as a
Service
configuration
• Applications include more
Cloud Files CDN
than just servers
• Images have security
issues
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
21

22. 22
CHECKMATE
Inspector
• Verification
Contractor • Due Diligence
• Decomposition
Architect • Orchestration
• Templates
• Questions
A system to build generic application configurations
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
22

23. 23
ARCHITECTURE • Components communicate
through a common queue
Architect • Each provisioning
component is independent
Checkmate Message Contractor Compute
Web Queue
Caching Storage
Message
Inspector
Queue
Load
Hadoop
Balancer
Database
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
23

24. 24
base:
ARCHITECT name: wordpress large
environment-name: {tenantId}-
wordpress-large
Template
providers:
- rackspace:
- compute: &rax-cloud-servers
Generic Provider Definitions
endpoint: https://...
- loadbalancer: &rax-lbaas
Architecture Questions
endpoint: https://...
- database: &rax-dbaas
Scaling Factors endpoint: https://...
- common:
vendor: rackspace
credentials:
- token: {token}
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
24

25. 25
ARCHITECT
Template • Requests per hour?
• Budget
Generic Provider Definitions • High availability
• Disaster resistant
Architecture Questions
• SSL
Scaling Factors • Backup
• CDN
…
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
25

26. 26
ARCHITECT tiers:
- name: web
resource: &loadbalancer
min-occur: 1
Template type: loadbalancer
connection: public
port: [80, 443]
allow: all
Generic Provider Definitions isolation: none
resource: &webheads
min-occur: 2
Architecture Questions type: compute
os: Ubuntu 11.10
memory-min: 2Gb
memory-max: 4Gb
Scaling Factors
configs:
- wordpress-mp
attributes:
- role: web
connection: *database
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
26

27. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
27

28. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
28

29. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
29

30. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
30

31. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
31

32. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
32

33. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
33

34. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
34

35. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
35

36. 37
CONTRACTOR
• Takes Architect’s plan and
builds it
• Task Decomposition
- Uses standard workflow
patterns
• Orchestration / Ordering
• Status Reporting
• Farms out tasks to sub-
Our current implementation uses an open source
contractors Python workflow engine, SpiffWorkflow.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
37

37. 38
INSPECTOR
• Takes Architect’s plan &
contractor’s output
• Focuses on checking for
code compliance
- Not perfection, bare minimums
• Can include multiple facets
- Security
- Scalability
Our current implementation includes WP Scan for
- Compliance WordPress and the Nikto vulnerability scanner.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
38

38. 39
INSPECTOR
+ Server: Apache/2.2.12 (Ubuntu)
+ No CGI Directories found (use '-C all' to force check all
possible dirs)
+ Apache/2.2.12 appears to be outdated (current is at least
Apache/2.2.17). Apache 1.3.42 (final release) and 2.0.64 are also
current.
+ ETag header found on server, inode: 12534048, size: 317, mtime:
0x4b9436dbea280
+ Allowed HTTP Methods: GET, HEAD, POST, OPTIONS
+ OSVDB-3268: /icons/: Directory indexing found.
+ OSVDB-3233: /icons/README: Apache default file found.
+ 6448 items checked: 0 error(s) and 5 item(s) reported
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
39

39. 40
INSPECTOR
[!] The WordPress "http://---.com/readme.html" file exists.
[!] WordPress version 3.1 identified from meta generator.
[+] Enumerating installed plugins...Checking for 2394 total
plugins
[+] We found 2 plugins:
Name: disqus-comment-systemLocation:
Name: wordpress-popular-postsLocation:
[+] There were 1 vulnerabilities identified from the plugin
names:
[!] ["WordPress Plugin Disqus Comment System <= 2.68 Reflected
Cross-Site Scripting (XSS)"]*
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
40

40. 41
FUTURE WORK
Monitor Architect
• Trending • Templates
• Thresholding • Questions
Inspector Contractor
• Verification • Decomposition
• Due Diligence • Orchestration
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
41

41. So I was talking with a friend…
He was bemoaning the pace of change and the speed at which software
was being pushed to production…
In essence, management has made the decision that
getting their app out the door with possible bugs is
more valuable to the business then having strong
assurance that the software has few or no significant
bugs.
You’ve got to up your game,
get automated, agile and
get on pace with your developers.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
42

42. ANY QUESTIONS?
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM