3. 3
Why performance testing?
2012: Research showed that Amazon would lose $1.6 billion in sales every year if
its site took one more second to load.
2013: 39% of e-retailers claimed they lost money last year due to performance or
stability problems.
2014: The web performance monitoring company Catchpoint Systems looked at
aggregate performance on Black Friday and compared it to the same timeframe in
2013.The results are notable: desktop web pages were 19.85 percent slower, while
mobile web pages were a whopping 57.21 percent slower.
2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday
traffic during 2015’s Cyber Monday peak traffic times.
5. 5
Performance only as sign off
Commit
Build
Unit
Smoke
Regression
Integration
Release deployment
Performance tests and sign off
Security sign off
UAT sign off
Functional E2E
Release Candidate
7. 7
Principles
Divide and conquer: stage your test execution
One throat to choke: target APIs for all testing goals
Keep your eyes on the prize: get your risks and KPIs right
Automate everything: one-click automation must be end-to-end
Analyze this: discover, visualize and deliver performance insights
Retain all results: value of data grows over time
8. 8
Include performance tests to CI loop
Commit
Build
Unit
Smoke
Perf SmokeRegression
Perf
Regression
Integration
Perf E2E
Release deployment
Performance sign off
Security sign off
UAT sign off
Functional E2E
Release Candidate
9. 9
CPT goals
Certify that the application meets performance targets.
Plan capacity and manage growth.
Track useful performance metrics.
Identify load-related weaknesses and track SLA break points.
11. 11
Solution parts
CPT PLATFORM
Create test loads for our SUT (System Under Test)
Collect metrics and store the results
Provide an analytics and visualization engine
Integrate with the CI/CD pipeline
TEST DATA MANAGEMENT
Generate data based on existing subsets
Always use fresh test data
Compare test results on test data and production
Mask any sensitive test data
ENVIRONMENT MANAGEMENT
Cloud
Provisioning on demand
Automated configuration
Automated deployment
12. 12
CPT framework requirement
Load
Generation different types of load (TPS,
RPS, etc.)
Distributed load
Complex user scenarios for load
Visualization
Summary of test results
Detailed view during test run
Comparison of test runs and building
trends
Metric
Standard performance metrics (response
time, throughput, etc)
System and JVM metrics (CPU, memory,
disk, network, etc)
Custom metrics (specific for SUT)
Results of validators (functional
verification of the SUT responses)
Integration with CI platforms
15. 15
Automation Decision Making
Baseline build is established as a reference point
New build results are compared with baseline
Results analysis:
Pass when value is in range (Lower Warning Threshold *ref .. Upper Warning Threshold*ref)
Warning when value is in range (Lower Error Threshold*ref .. Lower Warning Threshold*ref) OR
(Upper Warning Threshold*ref .. Upper Error
Threshold*ref)
Error when value is less than Lower Error Threshold*ref OR is greater than Upper Error Threshold*ref
19. 19
Jagger specifications
General
Performance test in Jagger – Java project
To develop test, some basic knowledge in programming is required
Simple tests can be implemented just by XML editing
How to start
Download and start test project example: Jagger Getting Started
21. 21
CPT: Search and Browse for large eCommerce site
21
Challenges for delivery from
testing perspective Solution overview
Continuous performance testing platform
Jagger based performance testing platform with load
generation and full performance and JVM metrics
collection, automated trend analysis.
Evaluated SLA and critical metrics for all part our SUT
(System Under Test) and automatically decision making
based on them
Jagger WebUI for analytics and visualization. Compare
test and session results, build trend, monitoring parameters
during test.
Integrate with the CI/CD pipeline
Continuous delivery platform
Jenkins based multi-stage testing pipelines
Cloud based deployment and configuration of all artifacts
Daily deployment into UAT environments
Bi-weekly deployment to production
Search engine has very flexible
configuration
Complex business requirements
Platform consists of many collaborating
services, including NoSQL database, SQL
database, clustered search engine, two
clustered services
Strict Non-functional and performance
requirement
Results
Reduce time for investigation of
performance issues from weeks to 1-2
days
No critical or major performance defects
in production in 3 years
1-2 performance issues/year discovered
in pre-release performance certification
22. 22
Thank you!
Mikhail Klokov: mklokov@griddynamics.com
Grid Dynamics blog: blog.griddynamics.com
Follow up on twitter: @griddynamics
Jagger: jagger.griddynamics.net
We are hiring!
griddynamics.com/careers
23. Privileged and confidential
www.griddynamics.com
About Grid Dynamics
Founded in 2006, Grid Dynamics is an engineering services company built on
the premise that cloud computing is disruptive within the enterprise technology
landscape. Since that time, we’ve had the privilege to help companies like
Microsoft, eBay, PayPal, Cisco, Macy’s, Yahoo, ING, Bank of America, Kohl's,
among others, to re-architect their core mission-critical systems, develop new
cloud services, accelerate innovation cycles, increase software quality, and
automate application management.
Grid Dynamics has multiple locations in the USA and Europe, and employs over
700 expert engineers worldwide.