Performance testing: respect the difference, highlighting important points of performance testing, presented at Software Quality Days 2014 in Vienna

1. Performance Testing: Respect the
Difference
Software Quality Days 2014
January 16, 2014
Alexander Podelko
apodelko@yahoo.com
http://alexanderpodelko.com/blog
Performance Testing: Respect the Difference
@apodelko

2. A bout Me
 Have specialized in performance for the last 16 years
 Currently performance testing and optimization of Hyperion
products at Oracle
 Board director at the Computer Measurement Group (CMG)
http://cmg.org,
organization of performance and capacity
professionals

Next conference November 3-6, 2014 in Atlanta, GA
Disclaimer: The views expressed here are my personal views only and do not necessarily represent those of my current or previous
employers. All brands and trademarks mentioned are the property of their owners.
Performance Testing: Respect the Difference
Slide 2

3. A genda
 Introduction
 Points to ponder
Performance Testing: Respect the Difference
Slide 3

4. Ter minology
Applying multi-user synthetic load to the system
 Load testing
 Concurrency testing
 Performance testing
 Burst testing
 Stress testing
 Endurance testing
 Scalability testing
 Longevity testing
 Volume testing
 Soak testing
 Reliability testing
 Stability testing
Performance Testing: Respect the Difference
Slide 4

5. Goal
 To contrast load and functional testing
 To highlight points to ponder in performance testing
Performance Testing: Respect the Difference
Slide 5

6. Load Testing Pr ocess
Collect Requirements
Define Load
Create Test Assets
Run Tests
Modify System
Analyze Results
Goals are not met
Goals are met
Done
Performance Testing: Respect the Difference
Slide 6

7. What Load Testing A dds
 Verification that the system handles the load
 Verification of multi-user performance
 Performance optimization

Exactly the same load
 Debugging/verification of multi-user issues
 Testing self-regulation functionality

Such as auto-scaling or changing the level of service depending on load
Performance Testing: Respect the Difference
Slide 7

8. Ster eotype
 Load / Performance Testing is:







Last moment before deployment
Protocol Level Record-and-Playback
Large corporations
Expensive tools requiring special skills
Lab environment
Scale-down environment
…
Performance Testing: Respect the Difference
Only one
option !
Slide 8

9. Perfor mance Testing
 In no way limited to pre-production:
•
•
•
•
•
•
•
Technology evaluation
Infrastructure evaluation
Prototypes / POC
Component / unit
Continuous testing
What/if scenarios
Performance optimization
Performance Testing: Respect the Difference
• Performance troubleshooting
• Benchmarking
• Production
Slide 9

10. A genda
 Introduction
 Points to ponder
Performance Testing: Respect the Difference
Slide 10

11. What to Test
 Can’t be comprehensive
 Typical scenarios
 Classes of users
 Risk
 Component coverage
Performance Testing: Respect the Difference
Slide 11

12. Performance Requir ements
 Goals vs. Requirements
 Response times

Aggregate values
 Throughput
Performance Testing: Respect the Difference
Slide 12

13. Performance Requir ements
 Concurrency
 Number of users and think time


Indirect way to define throughput
Named, active, concurrent users
 Resource utilization
Performance Testing: Respect the Difference
Slide 13

14. Wor kload Implementation
 Manual
 Load testing tool
 Recording/playback, protocol level
 Test harness
 Recoding/playback, GUI level
 Programming
Load Generator
Server
Load Testing Tool
Application
Virtual Users
Performance Testing: Respect the Difference
Network
Slide 14

15. Wor kload Ver ification
 Tools default

Minimal (such as timeouts or HTTP error codes)
 Direct

Analyzing server response
 Indirect



Logs
Data
Monitoring
Performance Testing: Respect the Difference
Slide 15

16. The Effect of Data
 Size and structure
 Caching / concurrency
 Adding new data
Performance Testing: Respect the Difference
Slide 16

17. Test Envir onment
 Sizing / Configuration
 Lab vs. Cloud vs. Service

Scope vs. Variability
 Testing Limits
vs.
 Performance Optimization
Performance Testing: Respect the Difference
Slide 17

18. Explor ing the System
 Explore a new system
 Run all scenarios manually
 Iterative approach

Run scripts as soon as they are ready
 Monitor all components
Performance Testing: Respect the Difference
Slide 18

19. Implicit Tasks
 Tuning
 Diagnostics
 Capacity planning
Performance Testing: Respect the Difference
Slide 19

20. Time Consider ations
 Steady mode
 Reliability
 Tuning and diagnostics
 Real / intensive users
 Throughput / connections

Different tuning
Performance Testing: Respect the Difference
Slide 20

21. Pr ocess
 Synchronous process may be needed




vs. Asynchronous
Problem can block further testing
Full setup can be required to reproduce
Diagnosis is a collaborative process
Performance Testing: Respect the Difference
Slide 21

22. Systematic A ppr oach to Changes
 Tuning / Diagnostics
 Systematic approach

One factor at a time
 Log all changes
Performance Testing: Respect the Difference
Slide 22

23. A nalyzing Results
 Far more than passed/failed

Challenge for Continuous Integration (CI)
 Response times, throughput, number of users
 Resource utilization
 Logs / diagnostics
 Variability between runs
Performance Testing: Respect the Difference
Slide 23

24. Summar y
 Performance testing is a complex activity involving testing,
performance analysis, administration, development, and
business processes
 Skills and processes needed are quite different, don’t apply
functional testing practices blindly
 A load testing tool is one point on one slide
 Excellence requires time
Performance Testing: Respect the Difference
Slide 24

25. Questions?
Alexander Podelko
apodelko@yahoo.com
@apodelko
To learn more check my collection of
performance-related links and documents at
www.alexanderpodelko.com
Performance Testing: Respect the Difference
Slide 25