Woot, an Amazon subsidiary, specializes in offering great new product deals every day. Woot's deeply discounted deals; and signature events like the 'Woot Off 'and 'Bag of Crap' sales launch at specific times throughout the day, and the resulting spiky traffic patterns are highly correlated to revenue.
In this session, we offer an unvarnished perspective into how Woot uses services such as Amazon DynamoDB, EC2, ELB, CloudSearch, CloudFront, and SES. Learn how to architect for security and PCI for a retail website running on AWS. Dig into the technical details of a data-store comparison between DynamoDB, Mongo, Oracle, and SQLServer, to find the right solution for unique workloads. Join us as we share our musings and real-lessons learned from using a cocktail of AWS services. We encourage you to attend even if none of this makes sense or is interesting. Don't miss the opportunity to hang out with Mortimer the Woot monkey and his crew and to walk away with one of our legendary flying monkeys.
3. Woot Trivia (Question 1)
Q: What is Woot?
A. A rabid monkey only found in the Amazon jungle
B. A daily deal site, flash site and vibrant deal community
C. The name of a dance performed by hungry SDEs
D. A retail website selling toys
11/15/2013
Amazon Confidential
3
4. Woot Trivia (Question 2)
Q: Name the event on the Woot website where we sell a
random collection of leftover inventory and “surprises”?
A. Bunch of Carrots
No Canadians were
harmed in the
making of this
presentation
B. Bitter Old Canadians
C. Baldness on Curly
D. Bag of Crap
11/15/2013
Amazon Confidential
4
5. Woot Trivia (Question 3)
Q: When was Woot acquired by Amazon?
A. 2004
B. 2008
C. 2010
D. None of the above because Woot is only a figment of
a six year old’s imagination
11/15/2013
Amazon Confidential
5
10. Woot (Pre-Acquisition)
S3
EC2
• Traditional co-location (since 2003)
– 8 servers
– Monolithic application
• Amazon S3 for images from 2006
• Woot Deals launch in 2009
– Amazon EC2 hosted (public)
– SimpleDB MongoDB + SOLR
– Amazon SQS to relay to Woot Retail
ELB
SQS
11. Woot (Post-Acquisition)
S3
EC2
• Migrated to Amazon VPC (2011)
–
–
–
–
–
2 months to prep VPC
VPN to colo (for secure data migration)
P2V for SQL servers
Migrate MongoDB replicas
18 hours to move production services
ELB
SQS
Route 53
CloudFront
SES
VPC
RDS
• Went from 8 servers to 20
12. Application Rewrite in 2012
• Business Goals
– Support multiple deal types (Daily, Flash, VIP)
– Changes must be visible quickly
• Technology Goals
–
–
–
–
Services-oriented architecture
Improve site reliability
Optimize for cloud deployments
Language and platform neutral
17. Monitoring
• System Center + AWS Management Pack
– .Net aware application monitoring
– Ties to external monitors
– Correlates security and system events
18. Monitoring
• System Center + AWS Management Pack
• OpenNMS
– Service discovery
– SNMP integration
– Multi-AZ perspective
19. Monitoring
• System Center + AWS Management Pack
• OpenNMS
• StatsD + Graphite + Skyline
– Great dashboards – Measure everything
– High resolution
– Flexible graphing options
20. Monitoring
•
•
•
•
System Center + AWS Management Pack
OpenNMS
StatsD + Graphite + Skyline
Amazon CloudWatch
– Most direct source of truth
– Systems performance and usage metrics
25. Woot Throwdown Tenets
•
•
•
•
Don’t make assumptions; all options on the table
AWS is not the default choice
Simulate real-world behavior
Compare as many facets as possible
- Driver support
- Resiliency
- Performance
- Support
- Scalability
- Cost
26. Throw Down 1: Load Balancers
Contenders
Pros
Cons
Varnish
Flexibility
No SSL
NginX
Lightweight and fast
No HTTP1/1 to origin
Apache
HTTP1/1 in/out
Hard to tune
Elastic Load Balancing
Cloud Aware
Limited logging
27. Throw Down 2: Cart Persistence Storage
Woot “Cart Service”
– Simple
– Host a bag of crap
– Fast
– Scalable to follow growth
– Resilient
– Lots of small data facts
– Secure
patterns
31. Cart Throw Down Results
- MongoDB and DynamoDB were our finalists
- DynamoDB outperformed MongoDB in our
performance tests
- We could launch and scale DynamoDB with a
button push
- We could host a bag of crap
34. DynamoDB – Rough Edges
• Only basic index support (primary, secondary)
– Use searching technologies such as Amazon CloudSearch to
index data
• Ad-hoc query support not so simple
– Export data we want to query into a data warehouse
37. Triage of an outage: July 2, 2012
• Massive Storm + Maintenance Problem =
No Power in an Availability Zone
• Availability Zone hosted Woot’s primary
SQL and AD role masters
• Amazon EBS failures prevented recovery
• AD DNS collapsed
38. Lessons Learnt
• Data integrity means more Availability Zones
– Mirror/replicate to other Availability Zones
– Lots of hosts for MongoDB
– Dedicated hosts for irregular patterns
• Backups
• Long-running queries
• Specialized indexes
39. Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything
–
–
–
–
Decouple systems
Save state information
Inspection of service operation
Recoverable
40. Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything
• Hosts are disposable
–
–
–
–
Don’t store unique data
Push-button launches
Centralize configuration
Dynamic role assignment
41. Lessons Learnt
•
•
•
•
Data integrity means more Availability Zones
Queue everything
Hosts are disposable
Unique solutions
– Different failure modes than traditional data center
– Combine off-the shelf items for resiliency
43. Security
• Same problems as anywhere else
– Centralize logging
– Centralize accounts
• Application-level logging
– Gain visibility
– Follow state
• Use application tiers
– SOA separation
44. Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
45. Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
• EC2+EBS Best Practices
– Deploy EBS-optimized hosts for required bandwidth
– Provisioned IOPs to ensure lower latency for disks
46. Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls.
– Perform updates in batches
• EC2+EBS Best Practices
– Deploy EBS Optimized hosts for required bandwidth
– Provisioned IOPs to ensure lower latency for disks
47. Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
• EC2+EBS Best Practices
– EBS-optimized hosts for more bandwidth
– Provisioned IOPs to ensure lower latency
• Scaling Best Practices
– No reward for using too much or too little
48. Woot’s AWS Experience
• Tools continue to improve
- AWS Dashboard
- Expanded CloudWatch
- AWS Trusted Advisor - Premium Support
• AWS advocates for users
• Capacity really is elastic
• The more we see, the better it looks
49. Please give us your feedback on this
presentation
ARC212
As a thank you, we will select prize
winners daily for completed surveys!