The Amazon Simple Workflow (Amazon SWF) service is a building block for highly scalable applications. Where Amazon EC2 helps developers scale compute and Amazon S3 helps developers scale storage, Amazon SWF helps developers scale their business logic. Customers use Amazon SWF to coordinate, operate, and audit work across multiple machines—across the cloud or their own data centers. In this power-packed session, we demonstrate the power of workflows through 7 customer stories and 7 use cases, in 7 minutes each. We show how you can use Amazon SWF for curating social media streams, processing user-generated video, managing CRM workflows, and more. We show how customers are using Amazon SWF to automate virtually any script, library, job, or workflow and scale their application pipeline cost-effectively.

1. Amazon Simple Workflow
7 use cases : 7 minutes each
Sunjay Pandey, Amazon Simple Workflow
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. What is the Amazon SWF service?
Coordinate, track, and audit
programs
With …
Multiple steps on multiple machines

3. A multi-step process:
Booking travel to Vegas…

4. Update seat
inventory (Batch)
Send confirmation
email
( Web, mobile, SMS …)
Search for
flights
Update
inventory
Confirm
inventory
Select
seats
Provide
credit card
Confirm
Trip

5. Reliable, Scalable,
Auditable?

6. Update seat
inventory (Batch)
Send confirmation
email
( Web, mobile, SMS …)
Search for
flights
Update
inventory
Confirm
inventory
Select
seats
Provide
credit card
Confirm
trip

10. How to handle these challenges

11. Amazon SWF provides …
Programming framework
Convenient libraries
APIs
HTTP, task processing
instructions
Control engine
Manage state, route
tasks, distribute load

12. Key Amazon SWF concepts
Workflow (deciders)
• Control logic for workflows:
execution order, retry policies,
timer logic, etc. – Decision tasks
• Discrete steps in your
application, processors
Activity workers

13. Update seat
inventory (Batch)
Send confirmation
email
( Web, mobile, SMS …)
Search for
flights
Update
inventory
Confirm
inventory
Select
seats
Provide
credit card
Confirm
trip

14. Workflow (decider)
Update seat
inventory (Batch)
Send confirmation
email
( Web, mobile, SMS …)
Search for
flights
Update
inventory
Confirm
inventory
Select
seats
Activity
Provide
credit card
Confirm
trip

15. SWF customers are diverse
Private, venture-backed, publicly traded
Java, Ruby, C#, PHP, Python, on premises,
fully AWS
Media, e-biz, transportation, entertainment,
space exploration, life sciences, non-profit,
gaming, …

16. Many use cases

17. What they have in common…
Multiple steps
Multiple machines
Higher potential for failures
Need resilience, scale, auditability

18. Customers

19. Engineering offices
Austin
San Francisco
New York
Content moderation as a service
Thousands of clients
100s of millions of pieces of content
100s of millions of unique visitors per month
10s of billions of pageviews per month
Java-based workflows

20. Content Moderation System (CMS1)
Humans read every piece of content, twice !
Content coded with tags: product flaw,
shipping issues …
Step 1
Step 1
Customer
review
submission
Step 1
Step 1
Review
Step 1
Step 1
Approve/Rej
ect
Step 1
Step 1
Publish &
Email
customer

21. Content Moderation System (CMS1)
Design challenges
•
•
•
•
•
Latent
Rigid state machine
Multi-cluster environment
Balancing, scheduling, admin was hard.
Acquiring new company to integrate didn’t help
things

22. Content Moderation System (CMS2)
Moderation as a service
•
•
•
•
One service, many types of content moderated
Fully de-coupled from other systems
Real-time, rich prioritization of tasks
Control logic flexible, built with SWF Flow
framework (AWS Java SDK)
• Execution managed by SWF

23. Moderation Workflow
50k-150k Parallel executions
3-5 activities
2-3 days duration

24. SWF, AWS architecture brought
Benefits
• Efficiency
Moderation time (submission – publish) down
35% since launching CMS2.
• Flexibility
Started with few activities, was able to scale
logic as we grow

25. Engineering offices
Austin
San Francisco
New York
User generated video processing
4 Million+ Lessons Taught in 12 years
In 2012 > 25% of all golf lessons in US
Next Competitor < 1%
Strategic Partners: Sports Illustrated, GolfSmith
PHP-based workflows

27. My Pro to Go
Video
capture
(phone)
A Golf Pro in Your Pocket ..
• Partnership with Sports Illustrated
• Mobile App = virtual golf lessons
Upload
Preprocess
Telestrate
(coach)
Account
updated
Web
lesson
Post
processing

28. Telestrate
(coach)
Preprocess
Account
updated

29. Before Simple Workflow & AWS
Design challenges
•
•
•
•
•
Traditional datacenter
Limitations: scale, dynamic Growth, capital costs
Single workflow & web server
Patchwork of PHP scripts
No logging, monitoring or Reporting

30. •
Lesson processing pre- AWS (2011)
FT
P
TWO UBUNTU
FFMPEG
TRANSCODERS
UBUNTU
APACHE
WEBSERVER
EMAIL
CORPORATE
EXCHANGE
SERVER
UBUNTU
MYSQL
SERVER
FIBER
CHANNE
L
SAN
Environment challenged to
produce 88,740 lesson
videos
per month.
Lesson storage on local system was 1TB.
Regular purges of data had to be done due
to space limitations.

31. •
Lesson processing on AWS (2013)
SIMPLE STORAGE SYSTEM
(S3)
All uploads from centers
open a SWF workflow,
allowing all stages to be
tracked and errors to be
automatically corrected or
staff to be alerted.
PAIR OF
SWF
DECIDERS
SIMPLE WORKFLOW
SERVICE
8-20 SWF workers are
deployed based on
load. We regularly
produce 655,000
lesson videos per
month.
RELATIONAL DATABASE
As a managed
SERVICE
LOAD
BALANCED
EC2 APACHE
Easy to scaleWEBSERVERS
and manage. Developers
able to quickly and easily spin up clones of
the production servers.
CLOUD WATCH
All systems monitored
by custom alarms,
immediately notifying
techs before
emergencies happen
All lesson and
marketing resources
new stored on Amazon
Simple Storage Service
(S3). We now hold
multiple versions of
videos, store 100TB of
data with no fear of any
limits.
CLOUD FRONT CDN
Mass market videos
now distributed via
content delivery
network.
service, we don’t
worry about
patching or
maintenance. Point
in time recovery
and the ability to
quickly spawn
clones aid
development.
SIMPLE EMAIL SERVICE
Automated emails go
through Amazon
Simple Email Service
(SES), increasing
deliverability and
decreasing load on
Exchange

32. SWF, AWS architecture brought
Benefits
• Scalability
30,000+ videos per day (30, 50, 120, ++ fps)
• Flexibility
Decouple logic (decider), limited by creativity

33. Post transaction processing
Loyalty marketing as a service
Connect digital offers and in-store payments
Serves: merchants, developers, publishers
Payment network < > developer apps
Ruby-based workflows

34. Cardspring platform connects …
Connect online behavior with in-store
purchases
•
•
•
•
Developers create card-linked apps
Merchants choose to run campaigns “in app”
Customers pay through POS terminals
API receives real-time webhooks about credit card
transactions
• Post-transaction data updated for merchants,
developers, customers

35. Before Amazon SWF
Previous starting point
• Worker machines process batch file jobs +
async work outside of API requests
• Running custom job processor framework +
many ad hoc scripts
• Triggered via SNS messages, cron, & jobs
controller

36. Before Architecture Diagram
API
Jenkins
SNS
ELB
worker
worker
worker

37. After architecture diagram
API
Jenkins
SWF
worker
worker
worker
decider
decider
decider

38. After Amazon SWF
Simpler, resilient architecture
• All scripts/processors consolidated into 30
distributed workflows
• Implement priorities via different task lists
• Easy to scale
• Easy to restart

39. Amazon SWF brought
Benefits
• Resilient
Restart in the case of failure
• Efficient consolidation, Easy Management
30 automated workflows do all the processing

40. Photo Credit: flickr/hectorir
Complex order processing
CRM + ERP for non-profits, associations
Mid-to-large sized associations
100+ customers
C#/.NET based workflows

41. CRM 360° Suite
Fully featured with asynchronous needs
• Full ERP: membership, financial, fundraising, emarketing portal, API platform, etc.
• File processing/Amazon S3 uploads
• Amazon CloudSearch encoding
• Long-running processes like membership approvals

42. Before Amazon SWF
Windows, Quartz.net
• Unreliable, no replacement
• Poor performance
MSMQ (Microsoft Message Queue)
• Better throughput, but queues don’t know how to run
tasks
• Impossible to scale remote queues
• Still had to use Quartz.net for recurring tasks

43. After Amazon SWF
Scalable, manageable architecture
• Replaced old system with ≈ 24 workflows
• Encouraged good design – broke up code into
“activities”
• Replaced recurring activities with “always on”
workflows that woke up on scheduled timers

44. Post SWF architecture

45. SWF benefits
• Stability
System never crashes
• Replay, retry-ability
30 automated workflows do all the processing

46. Chicago
San Francisco
New York
High-performance image processing
Customers: Reebok, Jansport, Nine West,
Solutions: Shoppable images, Dynamic
imaging, Visual product customization
Java-based workflows

47. Before Amazon SWF
Scalability challenges
• 1 product = 1000s of images
• Generate images by rendering 3D models
• Poor performance, scalability

48. After architecture
•
SWF brings parallelism
to both the 3D rendering
+ post-processing
•
Image processing time
dropped 30 – 80%

49. SWF benefits
• Performance
Image processing times dropped 30 – 80%
• Costs optimized
Processing machines auto scaled as necessary

50. Back-office Automation: IT Services
ERP for technology service providers: cloud
and on-premise. Datacenters on 4 continents
5,500+ Customers with 80,000+ end users
Trouble ticketing, time/billing, calendar/syncing
for scheduling, deployments

51. Before SWF – customer-specific
Email
Processing (A-M)
Email
Processing (N-Z)
Calendaring (A-M)
Calendaring (N-Z)
Ticketing
Workflow
System
Installation
System
Corporate Data center
•
•
•
•
•
Services were manually partitioned amongst customers
Server failure resulted in customer downtime
Server utilization was uneven – activity varied by customer
Adding capacity was a complex process
Logs were stored on each machine

52. After SWF
100K+ workflows daily with
1.5 Million+ activities, decisions
Flow
Framework
Email Processing
Centralized
Logging to
Amazon S3
Calendaring
Installation System
Workflow System
Services
are
Workers
Amazon SWF
Corporate datacenter
•
•
Identical worker servers
Centralized logging to
Amazon S3
Amazon Web Services
•
•
•
Java and Flow Framework used for deciders
.NET services act as SWF workers
Geo-diverse workflow coordination

53. SWF benefits
• Costs/utilization
Consistent architecture/servers all customers,
server utilization more balanced
• Simplified management, reliability
Single control panel with visibility into all services.
Split complex workflows across many identical
worker servers w/o having to maintain local state.

54. Multi-step, long-running user enrollment
Ridesharing/virtual carpool service
80% of users are repeat customers
Average ride < $10
100,000+ users
Python-based workflows

55. Enrolling private citizens
Multi-step workflows
• Headshot > Car photo > Photo of license Consent to
background check > Online training, Mentoring
session (driver ridealong)
• Processes long running, can “fail” at any step –
queues, scheduled processes would have be brittle
• Built from the beginning on Amazon SWF

56. SWF Architecture
Driver enrollment
process, status
checking
Activity
Workers
Workflow Workers
(Decider)
Application updates,
Reminders via
Email and SMS ask
drivers to move
forward in their
applications
Many, many thousands of
applications per day
With 1 decider, and 1
activity worker
Amazon SWF

57. SWF benefits
• Auditability
State of user’s application consistent
Reminder emails not duplicated
• Flexibility
Application steps can be non-linear
Users complete different steps, different times

58. 7 of many use cases
Content moderation
Video processing
GolfTEC
Post transaction processing
Cardspring
Order processing
Amazon SWF
BazaarVoice
Membersuite
Image processing
Fluid
Business/IT automation
Connectwise
Partner enrollment
Lyft

59. Benefits
Scalability
Flexibility
Resilience, reliability
Performance
Amazon SWF
Simplification
Costs, efficiency

60. Please give us your feedback on this
presentation
SVC101
As a thank you, we will select prize
winners daily for completed surveys!