Ever wonder what's in the Zuora cloud? Join us and learn how Zuora has built a scalable and secure cloud based subscription billing management service. Hear from scalability, security and operations engineers and have your questions answered.
Behind the Wizard’s Curtain: Scalability and Security at Zuora (Subscribed13)
1. Behind the Wizard’s Curtain:
Scalability and Security at Zuora
Subscribed 2013
Thomas Fou
Information Security &
Compliance
Levon Stepanian
Performance Engineering
&
2. Key 9
• Key 9
• A Day in the Life of Zuora
• Behind the Curtain: The Zuora Cloud & Platform
• Zuora’s Investment in Performance
• Security & Compliance
• Q/A
AGENDA
3. Key 9 Key 9
• 9 Keys to Subscription Success
• Key 9: Mission Critical, Reliable, Scalable & Secure
“Subscription businesses want a reliable ‘enterprise-grade’ system with
services that are built on a secured, mission-critical, and scalable infrastructure.
The SaaS Subscription Infrastructure must have reliable 7x24x365 operations,
regular new feature deployments, and well-formed, predictable business
continuity”
4. Key 9 Disclaimer Pt. 1
• This
session
may
contain
product
features
that
are
currently
under
development.
• This
session/overview
of
new
technology
represents
no
commitment
from
Zuora
to
deliver
these
features
in
generally
available
product.
• Customers
who
purchase
services
should
make
the
purchase
decisions
based
upon
features
that
are
currently
available.
• Technical
feasibility
and
market
demand
will
affect
final
delivery.
• Pricing
and
packaging
for
any
new
technologies
or
features
discussed
or
presented
have
not
been
determined.
5. Key 9 Disclaimer Pt. 2
Zuora Confidential
Not for distribution beyond the intended audience at Subscribed 2013
thebillablehour.com
6. Key 9 A Day in the Life of Zuora
• Monthly Synchronous Transaction Volumes
– SOAP and REST APIs
• Subscription Management
– 550K created
– 1M amended
– Half a Billion API calls a month (15M/day)
– > 80% are queries
• Top 3 Objects: Subscriptions, Products, RatePlans
7. Key 9 A Day in the Life of Zuora
• Monthly Asynchronous Transaction Volume (Aug/2013)
• 39M
total
asynchronous
transac0ons
• ~50%
during
1st
day
&
last
4
days
of
month
• Transac0on
mix
–
small
&
large
• Monthly
varia0on
6M
1M
6K
?K
336M
15M
INVOICE
8. Key 9 Performance Data
• Benchmark Data
– 375 orders/sec for a single tenant
• 2012 Amazon Cyber Monday Peak Rate: 306 items/sec
– 150+ payment authorizations/sec for a single tenant
• Production Data
– 50% of our tenants -> 70K invoices per hour
– Tenant generating 1.7M invoices in a single bill run
9. Key 9
L
B
AMQ
DB
(M)
Behind the Curtain: The Zuora Cloud
L
B
Billing
&
Payment
Servers
UI/API
Server
Global
(S)
PDFGen
Servers
Web
Server
Zuora
for
Salesforce
Servers
Messaging
Infrastructure
File
Storage
Global
(M)
Tenant
Shard
Tenant
Shard
(M)
Tenant
Shard
(M)
Tenant
Shard
(S)
File
Storage
AMQ
DB
(S)
(M)aster/(S)lave
RO
Replicas
(Not
Shown)
F
W Security
Appliance
10. Key 9
L
B
AMQ
DB
(M)
Scaling Zuora
L
B
Billing
&
Payment
Servers
UI/API
Server
Global
(S)
PDFGen
Servers
Web
Server
Zuora
for
Salesforce
Servers
Messaging
Infrastructure
File
Storage
Global
(M)
Tenant
Shard
Tenant
Shard
(M)
Tenant
Shard
(M)
Tenant
Shard
(S)
File
Storage
AMQ
DB
(S)
F
W Security
Appliance
11. Key 9 Infrastructure Scalability
– Enterprise Ready Tier 1 Data Center
• Switch SuperNAP (Las Vegas)
• High density, state of the art infrastructure
• Super beefy hardware, storage and networking gear
• 7x24x365 resource monitoring and alerting
10
90
Avg.
Produc0on
Capacity
U0liza0on
uMlized
idle
– Plenty of standby spare capacity to
accommodate growth
• Max utilization ~ 30%
12. Key 9 Zuora’s Investment in
Performance
• Bottlenecks are everywhere!
• Zuora’s massive & continual investment in performance
– Search & Destroy philosophy adopted by all teams
– Refactoring/optimizing code
– Production-like environment profiling/analysis
– Better aligning s/w and h/w architectures
– Investment in state of the art technology
13. Key 9 Customer Facing
Performance Improvements
• Rating & Billing Engine (RBE) TurboBooster
• 1.6X to 25X Bill Run speedup in production
• Optimizing & minimizing # of queries
• More charges/subscription = Larger speedups
• Zuora for Salesforce 360 TurboSync
• Up to 50X 360 Sync speedup in production
• Exploiting parallel pipelines, Bulk Salesforce APIs
• More objects to sync = Larger speedups
16. Key 9 Customer Testimonials
• “After directly engaging with Zuora Engineering on a looming requirement to
support a large increase in scale, I was reassured by how quickly they moved to
support the stated 100tps for creation of hosted payment methods. I was impressed
that Zuora Engineering then applied the 100tps requirement to the other API calls,
in anticipation of downstream increases in volume. If a difficult requirement arises in
the future, I will have no hesitation in directly engaging Zuora Engineering again,
knowing that it will be a job well done.” – Architect
• "We're really excited about Zuora's improvements to Bill Run execution times. As
a business that bills hundreds of thousands of transactions in each bill run, it's
important for Hosting.com to generate invoices rapidly and get paid as quickly as
possible" said Rick Moore, VP Finance & Business Operations at Hosting.com.
"The latest performance improvements have significantly reduced our scheduled bill
run times by over 50%--that's a huge improvement, and attests to the fact that as
our business grows, Zuora continues to scale to accommodate that growth.”
• “We named it TurboSync because of how fast it copied 3million records into our
system. What would normally would have taken days instead took a few hours!”
said Cathy MacDonald, Executive Vice President of IT at Xplornet
Communications Inc.
Anonymous
Zuora
Customer
21. Key 9 How To Reduce PCI Scope
• Each entity responsible for how it uses data
• Limit where PCI data is stored, processed, transmitted
• Segment cardholder data network from other networks
• Use effective encryption
• Implement strong key management practices
• Limit scope of Cardholder Data Environment (CDE)
• Zuora Hosted Payment Method (HPM)
24. Key 9 Scalability Case Study
• Customer Profile
– Leading global news company
– Top UK newspaper publisher
• Subscription Launch
– Digital paper (smartphones, tablets, online)
– Access to breaking news (24x7)
– Fantasy soccer
– Apps for streaming soccer matches
25. Key 9 Scalability Case Study
• Performance Requirement
– 100 transactions per second customer acquisition rate
• 360,000 customers an hour!
• End to end testing uncovered bottlenecks
– Internal and External to Zuora
• Outcome
– Configuration tuning
– Horizontally scaled back-end servers
– Enhanced monitoring + alerting around launch dates