With an increasing number of applications being deployed in the cloud, this trend will soon touch performance testers within every organisation. This presentation will dispel the hype, tell you what you need to know to embrace this opportunity, and answer the following questions: * What are the challenges specifically related to performance testing cloud-based applications? * What are some common performance problems seen in cloud-based applications, and how can you test for them? * How will cloud-based load generators help your performance testing? Don't get left behind! A solid understanding of cloud concepts will be invaluable to your testing career. This presentation was originally given at Iqnite Australia (Melbourne) on October 16th, 2014.

1. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
Performance Testing in the Cloud
Stuart Moncrieff & Chris De Lorenzo
Iqnite Australia 2014

2. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
2
www.myloadtest.com
About Us…
Stuart Moncrieff
•Performance Testing consultant for over 10 years
•More than 10,000 people read something I have written about performance testing every month
−www.myloadtest.com
Chris De Lorenzo
•Performance & Automation Test Lead at Tabcorp
•Lives and Breaths Performance Testing

3. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
3
www.myloadtest.com
Presentation Roadmap
Part 1
•Tabcorp – Auto-scaling
•Vic Emergency – testing with a CDN
•Monitoring
•Auto-scaling
Case Studies & Examples
•Tabcorp – load testing at massive scale
•SA Health – using cloud- based load gens to test an internal app
Load Generators in the Cloud
•Testing outside the firewall
•When you need a lot of load generators quickly and cheaply
•Cloud costs
Load Testing Cloud- based applications
Case Studies & Examples
Part 2

4. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
4
www.myloadtest.com
Local Load Gens
Cloud-based Load Gens
Cloud-based Test Envt
Local Test Envt
traditional load testing
Previous Cloud Experience

5. 5
© Copyright 2014 Stuart Moncrieff, www.myloadtest.com
www.myloadtest.com
Part 1: Load Generators in the Cloud

6. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
6
www.myloadtest.com
Public Cloud (IaaS) Cost Calculation
Example costings for a 2-week test cycle
•50 Large Windows instances in the Sydney Region
•10 days of test execution
•4 hours of test execution per day
•4 GB of data transferred per hour during testing
On-Demand Instance Type (in Sydney Region)
Cost/Hour
Small (1.7GB, 1 CPU core/1 thread)
$0.12
Medium (3.75GB, 1 CPU core/2 threads)
$0.23
Large (7.5GB, 2 CPU cores/4 threads)
$0.46
Extra Large (15GB, 4 CPU cores/8 threads)
$0.92
Data transfer = $0.18/GB (charged for outbound data only)
Elastic IPs = $0.005/hour (when not in use)
Line item
Calculation
Cost
Instances
50 x 10 x 4 x $0.46
$920
Data
10 x 4 x 4 x $0.18
$29
IP Addresses
50 x 10 x (24-4) x $0.005
$50
TOTAL
$999
A Performance Testing workload suits cloud-based infrastructure perfectly. Why have servers sitting idle for 20 hours/day?

7. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
7
www.myloadtest.com
Use Case 1: Testing outside of the firewall
Your users are on the Internet, but your load generators are in the Data Centre. What potential bottlenecks are you not testing?
network devices
load balancer
firewall
system under test
end user
load generator
Data Centre
the Internet
Bottleneck: Internet link capacity
Bottleneck:
internal
firewall
Bottleneck:
Network
devices

8. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
8
www.myloadtest.com
Why aren’t you testing from the cloud?
If you are testing a public-facing app, ask yourself these questions:
•What potential bottlenecks am I not testing?
•What additional risk is the project carrying due to this test approach?
•Is this the right person to be doing my performance testing?

9. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
9
www.myloadtest.com
Use Case 2: You need a lot of load generators
It may be impractical to organise enough load generators for large tests
Virtual User Type
Vusers/Load Gen
Web (HTTP/HTML)
2000
Flex
500
TruClient
50
RDP
20
Citrix ICA
20
SAP GUI
20
Note: these are approximate numbers only.

10. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
10
www.myloadtest.com
Case Study: Load testing an internal app at SA Health
Load testing an internal app using RDP virtual users. Impossible to get 50 load generators set up in time for test execution dates.
•Set up load generators and RDS farm in AWS cloud.
•Main problem was connectivity from cloud to internal network of large government department.
•First time doing anything “cloud” for SA Health. Security team had no process for it. Escalated exemptions.
•Set up Amazon “Virtual Private Cloud”, and created a VPN connection to internal network. VPN was restricted to test environment IP addresses.
•Lots of meetings!
system under test
The Cloud
LoadRunner Controller
load gens
Remote Desktop servers
Data Centre
Install client application here
VPN

11. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
11
www.myloadtest.com
Case Study:
Melbourne Cup Race 7 Jumps @ 15:00
The race that stops a nation and hopefully not our site!
•Total transactions supported: 68,000,000
•Total bets: 11,000,000
•Total logins: 1,500,000
•Unique users per hour: 321,000
•Peak transaction rates per second
−Betting Peak: 3,046
−Paying Peak: 10,916
−Customer Transactions Peak: 1,317
−Mobile Native Apps Peak 3,846
$90.6 million in turnover on just the Melbourne Cup Race
Mobile Growth Trending Up
Large focus for performance testing
Melbourne Cup Year
# of Bets From Mobile Native Apps
% Growth
2011
186,024
2012
837,489
350%
2013
1,281,022
53%
2014
Tuesday 4th November

12. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
12
www.myloadtest.com
Drivers to Cloud Infrastructure
Driver: Large revenue growth continuing on digital distribution channels
•Cost
•Supports testing objectives
−Ease of use
−Elastic IP Capability
•Industry is Maturing
−Get On It! -> {Natural Selection}
−High Compute!
−API Integration
•Challenges
−Privacy | Terms and Conditions | Security | Accounts Payable
The driver to migrate load testing capability to the cloud?

13. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
13
www.myloadtest.com
Cloud Performance Testing - Tabcorp
Physical / Internal Cloud and External Cloud
Lesson’s learned from the TAB
•Still have internal physical and internal cloud machines
•Trial a number of products prior to defining final solution
•Amazon worked for us due to:
−Automation
•Small, Medium, Large Executions
−Firewall = No worries
−Quick Provisioning (min) / Cleanup (sec)
−Up and running in 1 week
−External location
•DDOS, Bandwidth, Data Cleansing services

14. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
14
www.myloadtest.com
Managing a large number of Load Generators
What does it take?
Final Product
•Final Product – Getting There…
−Invest in automation to take out the manual bottlenecks
−Execution Scheduled
•Bamboo and Ansible Tower
•All sized executions
−Status of execution is published
−Screens are available for everyone to see
−Large scale testing with minimal effort

15. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
15
www.myloadtest.com
So what does it look like?
This is just a sneak peak

16. 16
© Copyright 2014 Stuart Moncrieff, www.myloadtest.com
www.myloadtest.com
Part 2: Load Testing Cloud-based apps

17. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
17
www.myloadtest.com
Infrastructure Monitoring
Good instrumentation is critical for performance testing and for your Operations team
Performance testing should also validate that the system has sufficient monitoring for future trouble-shooting
•Amazon CloudWatch vs. Azure Diagnostics
−Azure gives average of last hour
−CloudWatch counters are 1 minute averages (i.e. 1-minute granularity)
•CloudWatch console access is free, but you need to pay for API access
•Can write own values to counters. This is worth doing.
Maximise your visibility into system health and application behaviour

18. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
18
www.myloadtest.com
Hardware won’t save you!
Myth 1: Adding server capacity solves all performance problems
Myth 2: Infinite cloud capacity means that you don’t need to do any performance testing
•Bottlenecks could be due to code or configuration, not just capacity
•What are the operational costs of paying for infinite hardware?

19. 19
© Copyright 2014 Stuart Moncrieff, www.myloadtest.com
www.myloadtest.com
“A backup that you haven't done a successful test recovery from is #notabackup.”
-- Some guy on Twitter
“Don’t assume your auto-scaling works unless you have actually tested it under load.”
-- Stuart Moncrieff

20. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
20
www.myloadtest.com
Auto-Scaling is not a Silver Bullet
If your system uses auto-scaling, you must test auto-scaling under load.
•What metric will trigger auto-scaling?
−Is it actually the metric that indicates a bottleneck?
•Optimise before auto-scaling.
−Tune until your workload is either CPU-bound, or I/O bound.
•How fast will auto-scaling kick in?
−Azure metrics based off last hour (moving average). A sudden spike in demand will not trigger auto-scaling.
−How long does it take an instance to spin up?
Auto-scaling never seems to work correctly the first time

21. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
21
www.myloadtest.com
War Story: Auto-scaling at Tabcorp
Considerations
•Even with a slow ramp-up solution failed due to 6 min provisioning time
−Based off CPU% > 50%
−Realized there are too many variables…
•Settled with setting “enough” capacity for “predictable peaks”
−Works for Tabcorp

22. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
22
www.myloadtest.com
Traps for new players
New (or important) test cases:
•Can you redeploy while under load?
•Does alerting/monitoring work?
−Will your Ops team have enough visibility to solve problems in Production?
•Does Autoscaling work as expected?
•Failover/HA/DR test cases
−Between availability zones, between regions
−Do instances need to be “warmed up” before putting them under load?
•How will you test something behind a CDN?
−All traffic hits single POP of CDN, means less traffic hits backend (VicEmergency example)
Watch out for:
•First time “cloud architects”
−You can always get help through your AM/SA
•How reproducible are your test results?
−“noisy neighbours”, CPU steal
−Inconsistent hardware for same instance type
•Routes for traffic from load gens
−All traffic routed over small pipe (MFB example)
•Restricting access to your test environment to load generators (using a whitelist on your firewall)
−Default limit of 5 static IP addresses (Elastic IPs)
•Monitoring in a PaaS environment
−You won’t have OS-level access to servers
It’s a trap!

23. Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo
23
www.myloadtest.com
Summary
Load generators in the Cloud
•Cost model suits servers which are not needed 24 hours/day. Low-cost, pay as you go.
•Great for testing public-facing applications from outside the firewall.
•Useful for situations where you need a large number of load generators.
Load testing Cloud-based applications
•Infrastructure monitoring is different. Invest the time to set this up properly.
•Don’t assume auto-scaling will work perfectly without testing under load. New Cloud-specific test cases!

24. 24
© Copyright 2014 Stuart Moncrieff, www.myloadtest.com
www.myloadtest.com
Thank you Please see us afterwards if you have questions
Email:
•stuart@myloadtest.com
Twitter:
•@StuartMoncrieff
LinkedIn
•http://au.linkedin.com/in/stuartmoncrieff
Website:
•http://www.myloadtest.com
Email:
•cdelorenzo@gmail.com
Twitter:
•@cjdelorenzo
LinkedIn
•http://au.linkedin.com/in/cjdelorenzo