4. Performance tests - why so hard?
Simulate production as close as possible:
● hardware
○ CPU, RAM, storage, ...
● software
○ OS, Virtualization, DBs, …
● load
● isolation
5. Performance tests - why so hard?
Necessary to have:
● infrastructure
● monitoring
● logging
6. Performance tests - why so hard?
● Your performance intuition is wrong!
1. collect the data (monitoring, logging, profiling)
2. find the bottleneck (based on data)
3. fix the bottleneck
4. collect the data and check the assumptions
5. go to 1.
7. Performance tests - why so hard?
Lies, damned lies, and statistics:
● arithmetic mean = 2.9
● median = 1
● standard deviation = 6 (only for normal distribution)
8. Performance tests - why so hard?
● Anscombe's quartet
http://bravenewgeek.com/tag/coordinated-omission/
Property Value
Mean of x 9
Sample variance of x 11
Mean of y 7.50
Sample variance of y 4.125
Correlation between x and y 0.816
Linear regression line y = 3 + 0.5x
Coefficient of determination of
the linear regression
0.67
9. Performance tests - why so hard?
Lies, damned lies, and statistics:
● arithmetic mean = 2.9
● median = 1
● standard deviation = 6 (only for normal distribution)
Use:
percentiles
50th = 1
70th = 1
90th = 2.9
95th = 11.45
99th = 18.29
Check percentiles implementation!
10. Performance tests - why so hard?
● Coordinated omission problem by Gil Tene
http://bravenewgeek.com/tag/coordinated-omission/
20. Coordinated omission problem
total time 60 s
max 30 s
99th 1 s
time in % for max 50%
expected time in % for 99th 50% - 1% = 49%
real time for 99th 60 s * 49% = 29.4 s