Ed Curran, VP of Engineering at XBOSoft, shares some of his hands on experience in working with JMeter for load and performance testing. In the webinar, he provided explanations of different types of performance testing and how you can use Google Analytics to understand what users are really doing on your web apps and then how to leverage JMeter and analyze the results to improve your app's performance.

1. Using JMeter and Google
Analytics
How to avoid
performance
testing
yourself into a
corner...
Ed Curran
Performance Testing Considerations

2. XBOSoft, Inc. All Rights Reserved.
XBOSoft
Dedicated to software quality improvement
Founded in 2006
We speed products to market with our expertise:
• Software QA consulting
• Software testing
Global team with offices in San Francisco, Beijing
and Amsterdam
“Thorough, accurate and fast”
2

3. XBOSoft, Inc. All Rights Reserved.
House Rules
▪ Participants other than the speakers are muted
▪ Questions via the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
▪ Questions may be asked throughout the webinar -
we’ll try to answer them at the end
▪ You will receive info on recording after the
webinar
3
Webinar Hashtag: #XboPerformance

4. XBOSoft, Inc. All Rights Reserved.
Meet Our Speakers
Philip Lew
CEO and Founder, XBOSoft
Relevant specialties and
passions
• Software quality process, evaluation,
measurement and improvement
• Software quality in use / UX design
• Mobile User Experience and usability
• Cycling and travel
4
XBOSoft Team

5. XBOSoft, Inc. All Rights Reserved.
Ed Curran
VP of Engineering
5
• Over 25 years of experience: technology;
international; BSEE/MBA
• Much of career dedicated wireless industry,
as well as software development
• Developer of performance management
monitoring systems for wireless networks for
use in capacity planning
• VzW Representative for Verizon Quality Council
• QuEST Forum / TL9000 Speaker
Meet Our Speakers
XBOSoft Team

6. XBOSoft, Inc. All Rights Reserved.
But Today is All About You
… Why are you here?
6

7. XBOSoft, Inc. All Rights Reserved.
Introduction
Software Testing Challenges
• Lack of Standard Processes
• 30% Overruns in Development Time
• Time to Market Pressures
Shanthakumaran, Prasanth. "Trends in Software Testing." KPMG - Business Excellence
(2016): 5-6.
>> A Real Need to Automate
7

8. XBOSoft, Inc. All Rights Reserved.
Automated Testing Trends
Rehn, Cecilia, ed. "European Software Testing Benchmark Report 2016 - Automation." Editorial.
Test Magazine Aug. 2016: 1-2. Web.
Types of Automation Utilized
Automation Survey Respondents
8

9. XBOSoft, Inc. All Rights Reserved.
Automated Testing Trends
Automated Testing Benefits
• 90% of companies have seen ROI in year 1
• Large range of automated processes
employed
• Focus on UI Testing
Rehn, Cecilia, ed. "European Software Testing Benchmark Report 2016 - Automation." Editorial.
Test Magazine Aug. 2016: 1-2. Web.
9

10. XBOSoft, Inc. All Rights Reserved.
Automation Testing Trends
• Test Environment (configuration,
maintenance, stability)
• Many types of automated processes available
– Difficult to determine tests to perform
– Scripts need to be created and maintained
Practical Challenges
10

11. XBOSoft, Inc. All Rights Reserved.
What is Wrong with My
Performance Test?
Automation Shortcomings
• 82% Automation on UI
– Customer-facing UI
– Potential to overemphasize
• 53% Performance
– Critical Components Left Out (e.g.
Ecosystems)
11

12. XBOSoft, Inc. All Rights Reserved.
Automation Testing Trends
• Load Tests
– Identify bottlenecks, check applications ability to perform under anticipated
loads
• Stress Testing
– Find breaking point
• Endurance Testing
– Support expected load over time
• Spike Testing
– Test software reaction to sudden load spikes
• Volume Testing
– Test software reaction to sudden load spikes
• Scalability Testing
– Determine software effectiveness when scaling up to support user load
Performance Test Types
"Performance testing : Process & Important Metrics - Guru99." 3 Nov. 2016,
http://www.guru99.com/performance-testing.html.
12

13. XBOSoft, Inc. All Rights Reserved.
Automation Testing Trends
Additional Considerations
• Different Types of Tests
– Tests do not stand in a vacuum
– Can be related (e.g., Performance Testing
within the Ecosystem)
• Performance Testing is Unusual
– Time-component dimension
– Exploited through automated testing
13

14. XBOSoft, Inc. All Rights Reserved.
Automation Testing Trends
Software
• Part of a system
• Needs to be designed to specific
requirements
… Critical to consider software operation within
the larger system.
System Complexity
14

15. XBOSoft, Inc. All Rights Reserved.
What is it you Really Want to Test?
“... the totality of features and characteristics of a
product or service that bear on its ability to satisfy
stated or implied needs.”
"What is Quality | IGI Global." http://www.igi-global.com/dictionary/quality/24254. Accessed 16 Dec. 2016.
Quality in Technology is:
15

16. XBOSoft, Inc. All Rights Reserved.
What Is It You Really Want to Test
(cont.)
High Level Software Test Concepts
• Smoke Test / Happy Path
• Negative / Boundary Tests
• Timing/Performance/Load dimension
16

17. XBOSoft, Inc. All Rights Reserved.
What Is A Test?
● h1 (Blue) - A happy path, or
smoke test, set to test a
given function that exists at
t=0.
● b1a, b1b (Red) - This
outward pressure adds to the
smoke test by expanding the
test to include additional
negative tests using
boundary/equivalence value
analysis.
● t=0, t=n - used to denote a
linear time where the
Blue/Red tests can
repeatedly execute over the
course of the time.
● th1 - shows that, during
performance testing, the
blue/red tests can repeatedly
be performed, but not
necessarily at periodic
intervals.
17

18. Load Testing Example
Stepping Thread Group - Design

19. Load Testing Example
Stepping Thread Group - Response Results - Part 1

20. XBOSoft, Inc. All Rights Reserved.
What Is a Test? (cont.)
Real-World Software System
• Simple Example
– Box metaphor
– Simple
– Well organized
• Real World
– Much more complex
20

21. XBOSoft, Inc. All Rights Reserved.
What is a Test (cont.)
Multiple Performance Tests - Metaphor
21

22. XBOSoft, Inc. All Rights Reserved.
What is a Test?
… Where Do You Stand?
22

23. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
Why Drive Load?
… Determine capacity bottlenecks for System under test
• Key Parameters/Factors
– Simultaneous varying inputs
– Simultaneous varying conditions
• Simulate Production Environment
• Run Load Tests to Exceed Requirements
– Ensures known level of buffer
23

24. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
Key Prioritization Parameters/Factors
XBOSoft Performance Prioritization Matrix
24

25. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
• User
– View of Person using application
• Platform Section
– Function within application under test
• Covered in .jmx
– Currently a script covering
Key Prioritization Parameters/Factors (cont.)
25

26. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
• Usage (XBO)
– XBO review of function usage levels
• Tier 2 Customer Input (a.k.a. “Customer
Service Complaints”)
– Generated by end-users
– Holds weight as “real customers”
– Determine relevance
• related to software under evaluation?
• related to a load/performance related issue?
Key Prioritization Parameters/Factors (cont.)
26

27. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
• Google Analytics Usage %
– Most Widely Used Analytics Service
– Key insights to activity taking place on site
… GA Pageviews Report
Key Parameters/Factors (continued)
… Behavior Flow Report
27

28. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
Key Parameters/Factors (continued)
GA Pageviews Report
"How to Use Google Analytics Behavior Reports to Optimize Your ...." 27 Oct. 2014,
http://www.socialmediaexaminer.com/google-analytics-behavior-reports/. Accessed 12 Dec. 2016.
28

29. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
Key Parameters/Factors
GA Pageviews Report
• Simple representation of an application
performance
– Page Views
– Average Time on Page
• How Functions/pages are being accessed
29

30. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
Key Parameters/Factors (continued)
GA Behavior Flow Report
30

31. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
GA Behavior Flow
• Can assist in optimizing the application
• Can be directly employed into Performance/
Load Script
– Volume
– Path
• Predictor of future traffic load
Key Parameters/Factors (continued)
31

32. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
• Functional Isolation
– Knowledge of Software Architecture
• Negative Test Required
– Required for a given function?
– Level and type
• General Importance Factor
– Stakeholder view of a given function (e.g.,
monetary transaction, school grades)
Key Parameters/Factors (continued)
32

33. XBOSoft, Inc. All Rights Reserved.
Knowing Where You Stand
... What Do You Need?
33

34. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
• Type of Business
• Customer Expectations
– 20-minute outage from Amazon costs $3.75 Million
– 28% of users will not return to a slow site
– 9% of users will not return after an outage.
– Bounce rate for pages loading in 2 seconds = 26%
– Bounce rate for pages taking more than 10 seconds
resulted in 41%
Business Requirements
"2016 Holiday Retail Insights Report." SOASTA, Performance Is Everything. SOASTA, 2016. Web.
… Consumer expectations are increasing.
34

35. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
Freedom from Production Servers
• Benefits
– Limits risk to real customers
– Freedom to schedule tests any time
– Supports time-dependent tests that can go on for hour
• Challenges
– Must be properly sized
– Must be maintained (e.g., backups, database
reconfigurations)
Test Application Server
35

36. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
JMeter Master/Slave Configurations
36

37. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
"How to use distributed JMeter to test netty server performance | Code ...." 29 Mar. 2016,
http://shengwangi.blogspot.com/2016/03/how-to-use-distributed-jmeter-to-test-netty.html.
JMeter Master/Slave Configuration
37

38. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
Benefits
• Slaves Offload Master as primary load driver
• Emulate users from different areas and
locations
• Allows graphical real-time viewing on Master
of tests in progress
JMeter Master/Slave Configuration
38

39. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
Application
Server 1
JMeter M/S Configuration (Regional)
By Thesevenseas (talk) - Own work, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=7376761
Application
Server 2
JMeter Slaves
39

40. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
Benefits
• Simple method for driving load to web
application servers
• Allows test support from centralized location
JMeter M/S Configuration (Regional)
40

41. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
JMeter M/S Configuration (Multi-Regional)
Map By Thesevenseas (talk) - Own work, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=7376761
Application
Server
JMeter Slave
JMeter Slave
JMeter Slave
JMeter Slave
JMeter Slave
JMeter Slave
41

42. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
Benefits
• Emulate users from different areas and
locations in “real world” scenarios
• Allows graphical real-time viewing on Master
of tests in progress
JMeter M/S Configuration (Multi-Regional)
42

43. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
API vs. HTML
43

44. XBOSoft, Inc. All Rights Reserved.
Knowing What You Need
• Required at the Start of Performance Analysis
– Current and anticipated utilization of CPU
– memory requirements
– disk drive capacity needs
• System Profile = System View
– >>> should include
• Application Server
• Performance/load computers
• All associated networks
System Profile for JMeter
44

45. XBOSoft, Inc. All Rights Reserved.
Getting Real
Create Realistic Tests
• Mirror Production Environment
– To the extent practicable
• Focus on Critical Areas
– Utilize Performance Prioritization Matrix
• Beware of Too Test Scripts
– Avoid Non-Critical Areas
– IDEs create scripts fast (too many?)
– Can create test management challenge
45

46. XBOSoft, Inc. All Rights Reserved.
Getting Real
Standalone Performance Tests Have Merit ...
A Game of Statistics
…. You can’t test everything
46

47. XBOSoft, Inc. All Rights Reserved.
For more, please download:
Performance / Load Testing
Using JMeter and Google Analytics
http://xbosoft.com/knowledge-center/software-testing
-white-paper/
Edwin H. Curran
@ecurran
ed.curran@xbosoft.com
Thank You!
47