This document provides an agenda and overview for a NeoLoad training. It introduces load testing principles and the NeoLoad tool. The agenda covers an introduction, load testing principles including test types and methodology. It then covers designing a test in NeoLoad including recording a virtual user profile, building a population, and monitoring. It concludes with running a test, analyzing results, and an overview of advanced NeoLoad features.
2. Introduction
• The Objectives of this Training
Load-Testing Principles
Design
Runtime
Results
For Advanced Users
AGENDA
3. What You Are Going To Learn
At the end of the class, you will know how to use NeoLoad
Introduction
Many tools to design
your VU
Some methodology
How to make your
own analysis
5. Determine how a system performs in
terms of responsiveness and
stability under a particular workload
Definition
Web Load-Testing Principles
6. Why Do Load Testing?
If your website earns $100.000 a day, this year you could lose $2.5 MILLION
in sales.
Web Load-Testing Principles
16%
decrease in
customer
satisfaction
11% fewer
page views
7% loss in
conversions
A 1 second
delay in page
load time
7. Load-Testing Allows Us To
Web Load-Testing Principles
Predict
application
performance
Detect
bottlenecks
Determine
scalability
8. Several Tests
Web Load-Testing Principles
Load Test
determine the behaviour of the system under a specific expected load
Scalability / Capacity
determine the system’s maximum load capacity
Stress
subject the application to a much greater activity than it normally handles
Endurance / Soak
detect memory leaks and other resource consumption issues using long
tests and static load
Degraded
determine the behaviour of the system under non-optimal conditions
9. Load-Testing Methodology
Web Load-Testing Principles
Test plan
Load-Testing
Consultant
Project manager
Functional
specialists
• Objectives
• Test platform
• Scenarios
• Scheduling
• Contributors
Preparation
Load-Testing
Consultant
Project manager
Functional
specialists
• Testing environment
installation
• Test platform validation
• Test plan validation
Design
Load-Testing
Consultant
Functional
specialists
Developers
• Test data generation
• VU profile design
• DB update procedure
• Test data validation
• Population & monitoring
Runtime
Load-Testing
Consultant
Server
specialists
Administrators
• Single-user baseline
• Ramp-up load test
• Combination test
• Maximum load test
Tuning
Validation
Analysis
Load-Testing
Consultant
Server
specialists
Administrators
• Response time analysis
• Benchmark comparison
• System metric analysis
• Tuning if required
• Report creation
10. NeoLoad Architecture
Web Load-Testing Principles
Monitor
(optional)
NeoLoad Monitoring Engine
3G+H+ WiFi4G+
Controller
Command
Cloud Load Generators On-premise Load Generators
Load
APP
Application Server Database ServerWebServer
• NeoLoad management console
• Design, run and analyze tests
locally
• Manage LG and MA
• Load generation as defined in
the controller
• Only active when the test is
launched
• Retrieve system information
• Understand the bottlenecks
• Alerts for faster analysis
• Network emulation
• Device emulation
• System to be tested
11. The Basics of A NeoLoad Project
Web Load-Testing Principles
Runtime
Set the number of users,
start the test and look at
the real-time information
Record and design VU
profiles
Create a population and
set each VU profile rate
Design
Setup the monitors
Analyze the results and
issue a report
Results
13. How Does NeoLoad Record?
Recording a Virtual User Profile
The client
(web-browser)
Servers
Requests are
recorded
NeoLoad acts as a proxy
and listens on port 8090
Responses
are recorded
14. Starting the Recorder
• Use the button and fill-in the following pop-up.
Recording a Virtual User Profile
Name of the
VU
Record in Init
or Actions
Start and set
the browser
Choose the
protocol to record
Choose the
recording mode
Set the user-agent
15. During the Record
• Use the recording bar to modify the user.
Recording a Virtual User Profile
Stop or pause
the record
Change the
section
Create
containers
Set Rendezvous
points
Access the
documentation
16. Why Using Containers?
• Group actions together
• Organize VU’s for easier maintenance
• Extract statistics (average times, hits/s…)
Recording a Virtual User Profile
Without
With
17. Objectives
After completing this exercise, you will be able to:
• Use the Neotys training environment.
• Record a simple VU profile.
• Recognize the components of a VU profile.
Scenario
1. Open the Neotys Training Applications portal and access the Jpetstore.
2. Record a VU profile, using containers, called ‘Browser’ that only browses the shop.
3. Stop the record and click on Finish to not execute the post-recording wizard.
4. Navigate into the VU profile to see the different components.
Exercise 1
18. The VU Profile at First Sight
Recording a Virtual User Profile
Actions
container
Request
Page
Transaction
19. The VU Profile Structure
• Init
- Executed once, and only once, at the beginning of the test.
- Common uses: initialization routines, application log in, etc.
• Actions
- Repeatedly executed until the end of the test.
- Cookies and cache are preserved between iterations (default behaviour).
- Reset the user context from the runtime parameters.
• End
- Executed once at the end of the test.
- Common uses: application log out, etc.
Recording a Virtual User Profile
20. The Runtime Parameters of the VU Profile
• A VU instance keeps running when an error occurs. It may be changed to:
- Go to the next iteration.
- Stop and start a new VU.
• Behaviour on assertions may be set the same way.
• All think times may be modified:
- Use think time defined on page.
- Override the think times.
- Apply a factor for each page.
- Add a random delay.
Recording a Virtual User Profile
21. VU Profile: Pages
• NeoLoad representation of an internet page.
• Made of an HTML page and the embedded resources (GIF, PNG, CSS, JS…).
• May be renamed.
Recording a Virtual User Profile
A page
The resources
22. VU Profile: HTTP Requests
• NeoLoad representation of a request sent over the network (HTTP/S).
Recording a Virtual User Profile
Method
Server
URL
Path
Parameters
23. The End of the Record
• When the navigation is completed, finish the record either by:
- Clicking the stop button .
- Closing the web-browser.
• If the recorded application keeps sending data, this pop-up may be displayed.
Recording a Virtual User Profile
24. The End of the Record
• Go through the post-recording wizard.
• It helps taking care of the:
- Think-time.
- Dynamic parameters: framework (pre-defined rules) and dynamic (comparison based
correlation).
- Authentication.
Recording a Virtual User Profile
25. Adding Pages To A Record
• You can also start the recorder within an existing VU Profile.
• Right-click in the VU Profile and choose ‘Record here’.
Recording a Virtual User Profile
26. Objectives
After completing this exercise, you will be able to:
• Record a structured VU profile.
• Use the flags to find an information.
Scenario
1. Record a VU profile, using the containers, called ‘Searcher’ that:
a. Starts at the Jpetstore homepage
b. Enters the store
c. Enters ‘Amazon Parrot’ in the search field
d. Clicks on a product
e. Clicks on an item
2. Stop the NeoLoad recorder and do not execute the post-recording wizard. When
prompted to execute click on Finish.
3. Locate the chosen product and item.
Exercise 2
28. Why?
• Fix the issues due to dynamic parameters.
• Reach realistic behaviours.
• Define several types of VU profile.
Designing a Virtual User Profile
29. Design Methodology
This sequence may be reordered. However, do not forget any step!
Designing a Virtual User Profile
VU profile
ready
The check VU
The variable
extractor
The logical
actions
The NeoLoad
variables
The validation
You will
learn
Validating the VU
behaviour
• Play a unit test
• Check the errors
Correcting the VU
behaviour
• Identify the request
that returns an error
• Identify the parameter
• Handle the data
Changing behaviour
using logical actions
• Adapt to the simulation
requirements
Using different login
accounts and values
• Define realistic
conditions of use
Validating key pages
• Set validations to make
sure responses are the
ones expected
30. Design Methodology
Designing a Virtual User Profile
Correcting the VU
behaviour
Changing behaviour
using logical actions
Using different login
accounts and values
Validating key pages
Validating the VU
behaviour
VU profile
ready
The check VU
The variable
extractor
The logical
actions
The NeoLoad
variables
The validation
You will
learn
31. Why?
• Find any issue that may be in the recorded VU profile.
• The recorded pages may require tuning.
• Modern apps exchange a lot of parameters.
• Avoid bad surprises while the test is running.
Validating the VU Behaviour
32. How?
• Using the ‘Check Virtual User’ (NeoLoad debugger).
• A single VU test with no thinktime.
• Accessed via .
Validating the VU Behaviour
33. The Check VU in Details
• Comparisons with recording.
• Difference rate.
• Variables.
• Requests and responses.
• Failed assertions.
• HTML page rendering.
Validating the VU Behaviour
34. Objectives
After completing this exercise, you will be able to:
• Check whether a VU profile is valid.
• Dig into the details of the page source code.
Scenario
1. Check the ‘Searcher’ VU profile and make sure it runs properly.
2. Examine the information provided, including:
- The HTML rendering.
- The requests that comprise the application home page.
- The Comparison of the recorded and replayed responses.
3. Locate the chosen product and item.
Exercise 3
35. Design Methodology
Designing a Virtual User Profile
Validating the VU
behaviour
Changing behaviour
using logical actions
Using different login
accounts and values
Validating key pages
Correcting the VU
behaviour
VU profile
ready
The check VU
The variable
extractor
The logical
actions
The NeoLoad
variables
The validation
You will
learn
36. Why?
• Because a VU displays an unexpected error.
• Because HTML pages usually contain dynamic elements.
• Because NeoLoad cannot automatically handle all parameters.
Correcting the Virtual User Behaviour
37. Focus: The HTTP Parameters
• Static parameters
- Values never change, regardless of
which user is connected or the number
of times the VU profile is played.
• Functional dynamic parameters
- Entered by the end user.
- Making them dynamic makes the VU
profile more realistic.
• Technical dynamic parameters
- Generated by the server, often used by
the client in subsequent requests.
- Values change at each call.
Must be handled to have a valid
VU profile.
Correcting the Virtual User Behaviour
38. Objectives
After completing this exercise, you will be able to:
• Understand how technical parameters may influence a VU profile.
Scenario
1. Record a VU profile called ‘Buyer’ that:
a. Starts at the Jpetstore homepage and then enters the store
b. Logs into the store (j2ee/j2ee)
c. Selects a category by clicking one of the images beneath the center bird
d. Adds a pet to the shopping cart
e. Checks out and enters payment information
f. Confirms the sale
g. Logs out
2. Stop the NeoLoad recorder and do not search for the dynamic parameters in the post-
recording wizard.
3. Check the ‘Buyer’.
4. Explain why it fails.
Exercise 4
39. How?
• Using a mecanism called variable extractor which extracts content of an HTTP response.
• The content is assigned to a NeoLoad variable.
• A variable may be called using the syntax ${<variableName>} or with the variable
picker that gives a quick access to the existing variables.
• The variable extractor is accessed from the request advanced settings .
• The variables created with the variable extractor are only accessible by the VU profile in
which they are extracted.
Correcting the Virtual User Behaviour
40. The Variable Extractor
Correcting the Virtual User Behaviour
Variable name
Target
extraction
Occurrence to
extract
Encoding
Test
XPath definition
Start boundary
Groups
End boundary
41. Hypothesis: a request fails to pass the Check VU
Make the replacement
Select the failed request
Validate the variable extractor
Go to the variable extractor settings and configure the extraction
Select the response containing the data to be extracted
Procedure: the Variable Extractor
Correcting the Virtual User Behaviour
42. Objectives
After completing this exercise, you will be able to:
• Use the NeoLoad tools to properly take into account a dynamic paramater.
Scenario
1. From the observations made on the previous exercise, fix the VU profile using a variable
extractor.
The extraction must be set on a request before the one that fails.
2. Use the Check-VU function to make the VU profile works as expected.
Exercise 5
43. The Variable Extractor: Automatic Configuration
• Automation of the manual procedure.
• Avoid mistakes.
• Save time.
Correcting the Virtual User Behaviour
Select the automatic
configuration
All is automatically
setup
44. Objectives
After completing this exercise, you will be able to:
• Take advantage of the automatic correlation feature.
Scenario
1. Duplicate the ‘Buyer’ and call the new user ‘Buyer_AutoConfig’.
2. Restore the value of the id parameter in the ‘Buyer_AutoConfig’.
3. Use the automatic configuration to fix the issue.
4. Check the VU to be sure the issue has been handled.
Exercise 6
45. The Framework Parameters: Practical Work
Correcting the Virtual User Behaviour
1. Record a VU profile called ‘BuySeveralItems’
that:
a. Starts at the Jpetstore homepage and then
enters the store
b. Logs into the store (j2ee/j2ee)
c. Selects a category by clicking one of the
images beneath the center bird
d. Adds a pet to the shopping cart
e. Clicks on “Main Menu”
f. Repeats steps c. to e. 5 times
g. Checks out and enters payment information
h. Confirms the sale
i. Logs out
2. Run a check-VU and observe the result
46. How should we handle the id
parameter?
Use the search and replace?
Do all extractions manually?
The Framework Parameters
Correcting the Virtual User Behaviour
47. The Framework Parameters: Why?
• Parameters are often the same across the scenario.
• “Teach” NeoLoad new correlations rules and save time.
• Re-use your framework parameters for future records.
Correcting the Virtual User Behaviour
48. The Framework Parameters: How?
• NeoLoad provides predefined correlation rules (.NET, Oracle, GWT, Siebel, Flex,
Silverlight…).
• For your own apps, create your own framework parameters.
Correcting the Virtual User Behaviour
Move the extraction
to the framework
Move the extraction
to the framework
49. The Framework Parameters: Advanced Settings
• Only edit existing configurations. Do not create a framework parameter from scratch.
Correcting the Virtual User Behaviour
Name
Replacement
Extraction rule
50. The Framework Parameters: Tips
• Framework parameters are part of the NeoLoad installation. Exporting a project does not
include them.
• Framework parameters must be exported to be shared (advanced settings, NeoLoad
preferences).
• Search for the framework parameters via .
Correcting the Virtual User Behaviour
51. Objectives
After completing this exercise, you will be able to:
• Take advantage of the framework parameters for automatic correlation.
Scenario
1. Record a ‘SearchBooking’ user on the Hotel Booking Application:
a. Access the home page.
b. Start (click on ‘Start your Spring Travel Experience’).
c. Click on ‘Find Hotels’.
d. View the first hotel in the list.
e. Close your browser.
2. Do not execute the post recording wizard. Check the VU, and observe the results.
3. Define extractors and add them to a ‘HotelBooking’ framework for the following
parameters:
- dtid,
- execution,
- the first uuid_0 (rename it ‘uuid_Find’ when you add it to the framework)
- the second uuid_0 (rename it ‘uuid_View’ when you add it to the framework)
Exercise 7 (1/2)
52. 4. Apply the framework parameters to the ‘SearchBooking’ user. Check it and observe the
results.
5. Record a ‘Logged_SearchBooking’ user following these steps:
a. Access the home page.
b. Login (credentials are provided on the page).
c. Start (click on ‘Start your Spring Travel Experience’).
d. Click on ‘Find Hotels’.
e. View the first hotel in the list.
f. Logout.
g. Close your browser.
6. Apply the framework parameters to the ‘Logged_SearchBooking’ user. Check it, and
observe the results.
Exercise 7 (2/2)
53. Generic Dynamic Parameters vs. Framework Parameters
• Generic Dynamic Parameters
- Found ‘on the fly’ using comparison.
- Less accurate identification.
- Slower identification.
• Framework Dynamic Parameters
- Defined using regular expressions.
- More accurate identification.
- Faster identification.
Correcting the Virtual User Behaviour
54. The Variable Extractor: Multiple Occurrences
• If an extraction template has several matches, all occurrences may be returned as a set.
• The following variables are created:
- <variableName>_matchNr : number of occurrences found.
- <variableName>_n : nth occurrence (n=1, 2...).
- <variableName>_rand : random occurrence.
- <variableName> : default value.
• Unless for specific purpose, when doing multiple extractions, do not use the only name of
the variable.
Correcting the Virtual User Behaviour
55. Design Methodology
Designing a Virtual User Profile
Validating the VU
behaviour
Using different login
accounts and values
Validating key pages
Correcting the VU
behaviour
Changing behaviour
using logical actions
VU profile
ready
The check VU
The variable
extractor
The logical
actions
The NeoLoad
variables
The validation
You will
learn
56. Why?
• To simulate more realistic conditions.
• To make simpler records.
• To have new VU profiles by only modifying one already recorded.
Changing the Virtual User Behaviour
57. How?
• Using the logical actions.
• Adapt to the simulation requirements with:
- Loops.
- Conditional branching.
- Error management…
• Drag and drop the actions. No script.
Changing the Virtual User Behaviour
58. Container
• Group together certain actions and statistics.
• Two execution modes:
- Sequentially: the elements of the container are played
in the order they are declared.
- Randomly: the elements of the container are played
randomly.
• The advanced mode allows to configure each element to
be repeated a number of times (repetition or probability).
• Pacing may be set on each container.
It is the minimum time for the container to be executed.
Changing the Virtual User Behaviour
59. Delay
• Suspends the execution for a specified duration.
• Expressed in milliseconds.
• May be a variable.
Loop
• Repeat elements.
• The number of iterations may be constant or a variable.
• The variable <LoopName>_counter is automatically created. It gives the current
iteration.
Changing the Virtual User Behaviour
60. While
• Execution of elements while a condition remains true.
• Several conditions may be applied.
• The variable <WhileName>_counter is automatically created. It gives the current
iteration.
If…Then…Else
• Execution of conditional actions.
- If condition is true, Then folder is executed
- If condition is false, Else folder is executed
• Several conditions may be applied to an If…Then…Else action.
Changing the Virtual User Behaviour
61. Objectives
After completing this exercise, you will be able to:
• Modify the sequence of a VU profile using the GUI.
• Change the VU profile to fullfill specific conditions.
• Use the ‘Update Recorded Content’ function.
Scenario
1. Duplicate the ‘Buyer’ and call it ‘LoopBuyer’.
2. Modify the ‘LoopBuyer’ in order to purchase a pet from each category (several pets in
the shopping cart and only one order):
- Set up multiple extractions.
- Use a Loop.
- Variables may be concatenated following the syntax ${var1${var2}}.
- Automatic variable extractions cannot be used.
3. Update the recorded content of the ‘LoopBuyer’ (advanced settings of the Check-VU).
4. Duplicate the previous VU profile and call it ‘WhileBuyer’.
5. Modify the ‘WhileBuyer’ using a While to include a condition on the total price (for
example, less than 500 dollars).
Exercise 8
62. Try…Catch
• Adapt the test execution behaviour in function of events (error management…).
• Actions included in the Try are executed sequentially.
• If an assertion or an error is detected then the Catch is executed.
Changing the Virtual User Behaviour
63. Go to Next Iteration
• Interrupt the VU runtime and go to the next iteration.
• The behaviour depends on where the action is:
- Init: the VU stops its initialization and goes to the Actions container.
- Actions: the VU stops its current iteration and run a new Actions container or goes to the End
container (depends on the load policy).
- End: no interruption, the VU keeps executing the End and an error message is logged.
• Often combined with If...Then...Else or Try...Catch.
Changing the Virtual User Behaviour
64. Shared Containers
• Share elements to re-use them in several VU profiles (login or logout for instance).
• Statistics are given across several VUs.
• Shareable elements are:
- Container
- Loop
- While
- Fork
• An element is shared with a drag-and-drop or a right-click interaction.
Changing the Virtual User Behaviour
65. Design Methodology
Designing a Virtual User Profile
Validating the VU
behaviour
Validating key pages
Correcting the VU
behaviour
Changing behaviour
using logical actions
Using different login
accounts and values
VU profile
ready
The check VU
The variable
extractor
The logical
actions
The NeoLoad
variables
The validation
You will
learn
66. Why?
• Test all branches of the application.
• Test how the servers handle the cache.
• Make things more realistic.
Using Different Values
67. How: the Variable Manager
• Numerous variable types (lists, strings, integers, counters…).
• Change policy.
• Distribution policy.
The variables created through the
variable manager are accessible by
all the VU profiles.
Using Different Values
68. Focus: the SQL Variable
• The SQL variable embeds a guided mode for MySQL, Oracle, DB2, PostgreSQL and MS
SQL.
• When requesting other DB, the driver and the URL must be manually defined.
• The SQL variable is populated and sent to the LGs before the test is running. It is not
calculated in real-time.
• Use the syntax ${<variableName>.<columnName>} to access each column of the
table.
Using Different Values
69. Objectives
After completing this exercise, you will be able to:
• Create and use the NeoLoad variables.
Scenario
1. Make the authentication of the ‘Buyer’ dynamic with a File variable (accounts are
provided on the Neotys Application Portal). Call this variable ‘Accounts’.
2. Create a List variable called ‘Address’ with three columns called ‘City’, ‘State’ and ‘ZIP’
that contains four data sets (rows):
- Boston, MA, 02134
- Random Lake, WI, 53075
- Beverly Hills, CA, 90210
- Echo, Oregon, 97826
3. Data values for the variable are selected in a random order.
4. In the ‘Buyer’, find the request submitting the payment details and replace the hard-
coded values with the variable ‘Address’.
5. Create (without using it) an SQL variable for the authentication called ‘SqlAuthent’ (DB:
jpetstore, table: SIGNON).
Exercise 9
70. Design Methodology
Designing a Virtual User Profile
Validating the VU
behaviour
Correcting the VU
behaviour
Changing behaviour
using logical actions
Using different login
accounts and values
VU profile
ready
The check VU
The variable
extractor
The logical
actions
The NeoLoad
variables
The validation
Validating key pages
You will
learn
71. Validation
• Check the server responses in real-time to ensure that the pages delivered by the server
are what you expect.
• Three criteria:
- Delay.
- Page size.
- Page content.
Validating Key Pages
72. Global Validation
• Add content assertions to a container or the entire VU profile.
• Only responses with a text/html or text/xhtml content-type are checked (default
behaviour).
Validating Key Pages
73. Objectives
After completing this exercise, you will be able to:
• Prepare the VU profile for the test by implementing error management.
Scenario
1. In the ‘Searcher’, change the search criteria using a List variable containing:
- Goldfish
- Amazon Parrot
- Iguana
- Manx
2. Set a validation to check that the ‘Add to Cart’ button is in the page.
3. Use a Try…Catch to stop the VU iteration if an assertion fails.
4. Set random think times between 5 and 10 seconds.
5. Add a Loop to include several searches (between 1 and 4). Change the search criteria
value on each loop iteration.
6. Stop the VU iteration if the stock of an item falls below 9500.
Exercise 10
75. Why?
• Test using different business behaviours.
• Test with specific network conditions.
• For example, test a web store by simulating 75% of users browsing the catalog (visitors),
20% carrying out a purchase through to its end (buyers) and 5% being administrators.
Building a Population
78. WAN Emulation
• All networks do not share the same characteristics.
• Typical characteristics are:
- Signal strength (for wireless networks)
- Bandwidth
- Latency
- Packets dropped.
• WAN emulation allows realistic network conditions.
Building a Population
79. WAN Emulation
• Settings available in the populations tab.
• Not configured by default (unlimited bandwidth, no packet loss, no latency).
• Pre-configured profiles carry real-world values.
• A LG may simulate several network conditions.
Building a Population
WAN emulation
profiles
WAN emulation
parameters
80. Objectives
After completing this exercise, you will be able to:
• Create a complex population.
Scenario
The population ‘PetstorePopulation’ is made of:
1. 60% of ‘Searcher’.
2. 40% of ‘Buyer’.
3. The ‘Searcher’ users do not use a cache; the other users have a full cache at the
beginning of the iteration.
4. 50% of the ’Buyer’ users have an iPad and a 3G connection with a poor signal.
5. The rest of the ‘Buyer’ users have Chrome and a Wi-Fi connection with a medium
signal.
6. 10 % of the ‘Searcher’ users have a custom browser (called ‘Proprio’) that uses 6
simultaneous HTTP connections.
Exercise 11
82. Why?
• Without monitor, NeoLoad only retrieves the response times, the errors, the number of
hits and the throughput.
• Information from the server side is fundamental to perform an efficient analysis.
• Each layer of the architecture may be monitored (webservers, application servers,
database, network…).
Monitoring the Infrastructure
83. How?
• No installation on the server side.
• All actions are performed from the NeoLoad controller.
• Specific ports need to be open on the firewall.
• NeoLoad provides preconfigured counters.
Monitoring the Infrastructure
85. Monitoring Agent
• Simplified controller to agent communication.
• Single port (TCP 7200) to open in the controller to agent direction.
• Installed and started as separate process on the server side.
• MA and LG share the same color
code:
- Green: running
- Orange: different version
- Red: halted
Monitoring the Infrastructure
86. Objectives
After completing this exercise, you will be able to:
• Use the NeoLoad monitoring capabilities.
• Look at the architecture behaviour in real-time.
Scenario
1. Add a Tomcat 6 monitor (no authentication needed).
2. Add a MySQL monitor (authentication: monitor / no password).
3. Add a Windows monitor (localhost).
Exercise 12
87. The Thresholds: Why?
• Trigger real-time alerts during test runtime on performance counters.
- If CPU usage exceeds 80%
- If available memory falls below 15%
- …
• Highlight components that suffer a deterioration in performance during the load test.
• Alerts make understanding the causes of any bottleneck much easier.
Monitoring the Infrastructure
88. The Thresholds: How?
• Alerts are set on some counters by default.
• Alerts are identified by .
• Alert thresholds may be manually added.
• An alert may be of two sorts: warning or critical.
Monitoring the Infrastructure
89. Objectives
After completing this exercise, you will be able to:
• Set thresholds and alerts.
Scenario
1. Based on the previous exercise, set an alert on the Windows monitor that follows the
condition: if the percentage of memory used is greater than 98%, raise up a critical alert.
2. Set an alert on the Tomcat monitor that follows the condition: if the
currentThreadCount(HTTP) exceeds 20, raise up a warning alert.
Exercise 13
91. Setup and Watch
Running the Test
Test
completed
The runtime
settings
The real-time
analysis
You will
learn
Configuring the test
scenario
● The duration
● The load policy
● The LGs
● Advanced settings
Looking at the real-
time information
● Plot your graphs
● Correlate the data
● Check the errors
● Follow the VUs
92. Setup and Watch
Running the Test
Looking at the real-
time information
Test
completed
The runtime
settings
The real-time
analysis
Configuring the test
scenario
You will
learn
93. Definition
• The test configuration.
• Several test scenarios may be created (short ramp-up test, long peak test…).
• All settings required are defined and applied to a population:
- Test duration.
- Load policy.
- Load generators.
- Runtime policy.
Configuring the Test Scenario
94. Duration Policy
• Unlimited.
• Iterations.
• Time.
• Iteration mode:
- Each user is run for a predetermined number of iterations with no break between iterations.
- Towards the end of the test, some users are finishing their iterations and some others are still
active. As a consequence the load decreases until all the users are finished.
- For ramp-up or peak loads, adding users for a new phase takes place once all existing users
have finished their iterations (the load may drop to 0).
Configuring the Test Scenario
95. Load Policy
• Constant: generate a fixed number of VU.
• Ramp-up: generate a number of VU that increases throughout the test.
• Peak: generate a fixed number of VU with periodic phases of high load.
• Custom: set the load to be applied by plotting the VU variation curve.
Configuring the Test Scenario
96. Load Generators
• LGs may be manually added by entering:
- The machine's name.
- The IP address.
• LGs may be automatically discovered.
• Define from the advanced settings:
- The network interface card.
- The IP spoofing.
- The load factor.
- The upgrade.
Configuring the Test Scenario
97. The Cloud Load Generators: Why?
• Perform heavy load tests without hosting the necessary infrastructure.
• Load test applications from several geos.
• Test the entire delivery chain.
• Benefits of the hybrid approach: cloud LGs / on premise LGs.
Configuring the Test Scenario
98. The Cloud Load Generators: How?
Configuring the Test Scenario
Open the Cloud console
and choose an on-demand
or reserved session
99. The Cloud Load Generators: On-Demand Session
Configuring the Test Scenario
Choose the duration and the
geos for an immediate use
100. The Cloud Load Generators: Reserved Session
Configuring the Test Scenario
Book your session
online and use it later
101. The Cloud Load Generators: Managing the Session
Configuring the Test Scenario
Cloud and hosted LGs
are used the same way
The session is managed
from the console
103. Scenario Settings
Configuring the Test Scenario
URL filtering
Debug mode
settings
Runtime users and
monitoring parameters
Apply SLA
104. Scheduling a Test
• NeoLoad -project <file> -launch <scenario> [-options].
• Export SLA results in JUnit XML format.
• Generate reports (XML, HTML, JUnit XML).
• Publish results to the Neotys Team Server.
• Specify monitored hosts.
Configuring the Test Scenario
105. Setup and Watch
Running the Test
Configuring the test
scenario
Test
completed
The runtime
settings
The real-time
analysis
Looking at the real-
time information
You will
learn
106. Runtime Overview
• Time is absolute or relative (Preferences>General Settings>Advanced).
The Real-Time Information
Main statistics
LGs status
Runtime
information
107. Runtime Graphs
• Time is absolute or relative (Preferences>General Settings>Advanced).
The Real-Time Information
111. Objectives
After completing this exercise, you will be able to:
• Set a load-test scenario and launch a test.
Scenario
1. Create a load test scenario called PetstoreScenario.
2. Make sure the population PetstorePopulation is active.
3. Define an increasing workload that starts with a single VU and adds an additional VU
every 15 seconds. Indicate the maximum number of VUs allowed by the training license.
4. Start the test with a duration of 5 minutes and explore the real-time data displayed
including:
- Overview data.
- Performance data for containers.
- Server-side metrics.
- Runtime errors, if any.
Exercise 14
120. Memo: Graphs
• Zoom in with left mouse button selection.
• Export data with (CSV file or picture).
• Time, user load or monitor for horizontal axis.
• Plot percentile graphs.
• Do comparisons with the drop-down list.
• Use and create your own templates .
Analyzing the Results
121. Errors
Analyzing the Results
20.000 errors
displayed max
Request, response
or assertions
Error list
Detailed error
information
Detail of the
previous request
122. Alerts
Analyzing the Results
Red zones for critical
alerts, yellow zones for
warning alerts
Alerts triggered on
performance counters
during the test
123. Logs
• Check the log files of the LGs and MAs.
Analyzing the Results
Check the details
Select MAs or LGs
The log level
126. Objectives
After completing this exercise, you will be able to:
• Create charts showing performance indicators.
• Drilldown through a series of measurements to highlight the cause of an issue.
Scenario
Based on the previous exercise:
1. Determine the average duration for the container where the category is selected.
2. Determine the page with the slowest average response time.
3. With a graph template determine the ‘health’ of the LGs used in the test.
4. Examine the errors encountered during the test.
5. Check if the overall page response time changed when the user load increases.
6. Explore the impact the user load had on the number of Tomcat threads.
7. Determine whether the number of current Tomcat threads affects the amount of OS
memory available.
Exercise 15
127. For Advanced Users
Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
128. Using a Firewall
Advanced Installation
TCP 7100 CTRLLG
TCP 7200 CTRLMA
UDP 1359 LGCTRL
TCP 4569 LGCTRL
for the automatic discovery
function
Monitor
(optional)
NeoLoad Monitoring Engine
3G+H+ WiFi4G+
Controller
Command
Cloud Load Generators On-premise Load Generators
Load
APP
Application Server Database ServerWebServer
TCP 7100 CTRLLG
TCP 443 CTRLLG
129. For Advanced Users
• Advanced Installation
Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
130. The Graphical User Interface
Servers
• Servers are automatically created during the
record.
• The target server may be quickly changed.
• Variables may be used (load-balancing).
• Server side authentication.
• Filled in during the record.
• Accounts may be dynamic (variables).
• Automatic session tracking when cookies
are used.
• When detected during recording, the
argument is automatically filled in.
131. Authentication Mechanisms
• NeoLoad supports Basic, Digest, NTLM and Negotiate mechanisms.
• On standard environments, the default protocol for Negotiate is Kerberos. If Kerberos
fails, Negotiate tries NTLM. However, for performance reasons NeoLoad uses NTLM in
place of Kerberos.
• If Kerberos is absolutely required, NeoLoad configuration file needs to be modified.
• NeoLoad prioritizes schemes in the order Negotiate, NTLM, Digest and Basic.
Servers
132. URL Rewriting and Browser Profile
• Deactivation forces the server to use URL rewriting to keep the session alive.
Servers
Cookie activation
133. For Advanced Users
• Advanced Installation
• Servers
Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
135. Page Settings
• Request playback: NeoLoad plays the requests sequentially (by default requests are
played back in parallel).
• Dynamically execute the page resources: NeoLoad analyzes the HTML response of the
main request to execute dynamically the page resources.
• NeoLoad retrieves resources from the HTML code. Resources referenced by CSS style
sheets and JavaScript scripts are ignored. Also, this setting has no effect on the module
requests.
Advanced Page Settings
137. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
138. Why?
• Hard to define what ‘good performance’ is.
• User standpoint evolution:
- 2006: 75% of users would not return to a website that take longer than 4 seconds (Akamai
research)
- 2009: a website needs to load up in 2 seconds (Akamai research)
• As a consequence, assessment of the service quality must be defined with end users
and/or functional experts.
• These thresholds are called Service Level Agreement (SLA).
Service Level Agreement
139. Two Types of SLAs in NeoLoad
• Per run: evaluated at the end of the test.
• Per time interval: evaluated each time a result is received during test runtime (real-time
alerts).
Service Level Agreement
141. Statistics on SLAs
Service Level Agreement
Check the status of
each SLA per profile
Check the status of
each SLA per VU
142. Edit SLA Profiles After a Test
• If an SLA profile is not enough accurate, it may be modified from the result manager.
• Transfer an SLA profile from Design to Results with to move it into several test results
and re-compute the result summary using the new configuration.
• Transfer an SLA profile from Results to Design with to copy it into the Design.
Service Level Agreement
Same options as
the Design
143. Objectives
After completing this exercise, you will be able to:
• Apply service level agreements that meet your needs.
Scenario
1. Open the project called ‘Training_Project’ (available on the resources page) and look at
the result called ‘Reference Test’.
2. Create a new SLA profile called ‘Reference’ following the criteria:
- Average response time per time interval must be less than 0.5 second.
- If the average response time per time interval is greater than 1 second the transaction must be
failed.
- The SLA profile must be applied to all containers of all VU profiles.
3. Once done, apply your changes and look at the results.
Exercise 16
144. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
Logical Actions
• Variables
• Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
145. Variable Modifier
• Used to modify the value of a variable before the value change policy is applied.
• Re-initialize or modify value (change policy is applied).
• Manage the Shared Queue actions.
Logical Actions
146. Fork
• Used to play actions in a different thread to the current one.
• When the VU main thread stops, all the threads created using Fork action are immediately
halted.
• Example of use, an action that plays every x seconds on the server to refresh a
component (Ajax, Flash plug-in…).
• Existing variable values may be copied locally. When a value is modified in another
thread, the variable value remains unchanged in the Fork action.
Logical Actions
Unchecked by
default
147. Wait Until
• Pauses the current execution thread until certain conditions have been verified.
• A condition is composed of two operands and an operator (except with Exists and
Doesn’t exist operators).
• A timeout may be defined. Once the delay is timed out, the thread is restarted. The
following action is then executed, even if the conditions have not been verified.
Logical Actions
148. Objectives
After completing this exercise, you will be able to:
• Use the Fork logical action.
Scenario
1. Create manually a VU profile called ‘SearcherAndBrowser’ that only contains a Fork
action in the Actions container.
2. Copy the content of the ‘Searcher’ in the main thread and the ‘Browser’ in the second
thread.
3. Check the ‘SearcherAndBrowser’. Observe the concurrency of both paths.
4. Create a variable called ‘isComplete’ that has two values, TRUE and FALSE.
5. Use a Wait Until action to have the main thread waiting for the end of second thread
before its execution.
6. Check the ‘SearcherAndBrowser’. Verify that it works as expected.
Exercise 17
149. JavaScript
• Execute JavaScript in a VU profile.
• The NeoLoad API provides functions to manipulate variables and set up cookies.
• When using the Logger API, information is written to the LG log file
(loadGenerator.log.00x in the Logs directory).
• Handle strings, dates, calculation...
Logical Actions
151. Java and JavaScript
• Use Java classes and functions in JavaScript actions.
• Copy the JAR files into <NeoLoad-project>/lib/jslib . The file is taken into
account automatically.
• The class names must be fully qualified within the JavaScript (even for the Java classes
like java.util.List) and preceded by Packages. For example var obj = new
Packages.com.company.MyClass();.
Logical Actions
152. JavaScript Libraries
• Create functions used in all the VU profiles.
• It helps reduce memory usage on the LGs, since the code stored in the libraries is shared
by the users.
Logical Actions
153. Objectives
After completing this exercise, you will be able to:
• Insert a JavaScript action into the VU profile to perform specific actions.
Scenario
1. Add a JavaScript action to the ‘Buyer’ to log an entry every time an item is added to
the shopping cart.
2. Include a time-stamp for each log entry.
3. Add a second JavaScript action to retrieve the name of the LG to the log.
Exercise 18
154. Rendezvous
• Synchronize VUs to perform a specific task simultaneously.
• Create a precise load peak in the VU execution.
• Policies are global, regardless the number of LGs.
• Rendezvous may be controlled from a JavaScript action.
Logical Actions
Timeout
Rendezvous
policies
155. Custom Action
• Take advantage of predefined ‘high-level’ actions embedded in NeoLoad
• Full GUI integration
• More to come on the Neotys Community
Logical Actions
The command line
action
156. Objectives
After completing this exercise, you will be able to:
• Use the command line custom action.
Scenario
1. At the end of the ‘Buyer’, use a custom action to log the total price of the order into a file
(use the ‘Echo’ function).
2. The title of the file must contain the instance number of the VU being executed.
Exercise 19
157. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
Variables
• Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
158. The Variable Manager
• Variables are created before the test is running.
• Depending on the policies, values are provided either by the controller or the LGs.
• With the Unique policy, a NoUniqueValueAvailableException exception is logged
when there is no more available value.
Variables
Global
• Sequential
• Random
Unique
• Sequential
• Random
Local
• Sequential
• Random
• Any
Global
• Any
Values provided by
the LGs
Values provided by
the controller
159. Shared Queue: Why?
• Share dynamic content amongst VU profiles.
• A FIFO pile with the following constraints:
- When full, all new arrivals are lost.
- When empty, the requesting VU has to wait for a new arrival to be added.
• A swap file may be used to pre-populate the queue at the start of the test or to save it at
the end.
• The Check VU does not have any effect on the swap file.
Variables
161. • Values are provided to the LGs by the controller when the test is running.
Shared Queue: How?
• Populating the Shared Queue
Variables
– Consuming the
information
Select where the
value from the
queue is copied
Select the Shared
Queue that contains
the information
Select the variable
to add
Select the Shared
Queue to populate
162. Shared Queue and JavaScript
• A Shared Queue may be controlled by a JavaScript action.
• NeoLoad provides a comprehensive set of APIs.
• Main functions are:
- addSharedValue.
- createSharedQueue.
- getSharedQueueSize.
- peekSharedValue.
- pollSharedValue.
Variables
163. Objectives
After completing this exercise, you will be able to:
• Share data accross two different VU profiles.
Scenario
1. Duplicate the “Buyer” and call it “SQ_Buyer”.
2. In the ‘SQ_Buyer’, populate a Shared Queue with the order number generated at the
end of the scenario.
3. Set the ‘SQ_Buyer’ authentication to j2ee / j2ee.
4. Record a new VU profile called ‘BuyOneItem’ that:
a. Logs in the Jpetstore.
b. Open the last order referred by the order number stored in the Shared Queue (manually alter
the URL in the browser using jpetstore/shop/viewOrder.shtml?orderId=1000 for
the record).
c. Buys the first item that appears in the shopping cart.
d. Logs out.
5. Launch a test combining both ‘SQ_Buyer’ and ‘BuyOneItem’ to have the ‘SQ_Buyer’
creating a list of orders and the ‘BuyOneItem’ using that list.
Exercise 20
164. JS Variable
• A variable computed by a piece of Javascript.
• A new function evaluate returns the variable value:
function evaluate() {
return new function() {
this.firstField = "a value";
this.secondField = myLibraryFunction();
}
}
• The field values are accessed by ${<variable name>.col_<field
number>} or ${<variable name>.<field name>}.
• The JavaScript variable is always computed on the LG side during the test.
Variables
166. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
Data Exchange API
• Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
167. The Data Exchange API Module makes it easy
to integrate data from a third party tool
Why?
Data Exchange API
168. Why?
• RESTful data service based on the Open Data Protocol
• OData is a standardized protocol for creating and reading data APIs
• NeoLoad receives external data to generate the results displayed in the GUI
• More information here
Data Exchange API
169. How?
• NeoLoad Open API accepts JSON format
• Create your own scripts to send data to the NeoLoad Server from any application
Data Exchange API
170. Objectives
After completing this exercise, you will be able to:
• Create your script to plot any information from any application.
Scenario
1. Duplicate the buyer and call it “DEA_Buyer”.
2. In the ‘DEA_Buyer’, create a script that uses the Data Exchange API to plot the total
price of what the “DEA_buyer” buys.
3. Launch a test with a few virtual users to see the results.
Exercise 21
171. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
Mobile Support
• Advanced Results
• The NeoLoad License
AGENDA
172. Mobile App Load Testing (Web-based and Native)
• Record native and web-based apps directly from the mobile device.
• Record web-based apps from the desktop browser with the Identify as feature.
• Generate realistic traffic with the WAN emulation.
Mobile Support
173. Recording Native and Mobile Web from Device
Mobile Support
Navigate as usual, the
traffic is recorded
Do not launch any
browser
Set the NeoLoad IP
and the recording port
Connect the device
(and NeoLoad) to a
WIFI network
174. Recording Native and Mobile Web from Device
Mobile Support
Parameters sent by
the mobile app
Server used by the
mobile app
175. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
• Mobile Support
Advanced Results
• The NeoLoad License
AGENDA
176. Debug Mode
• Look at how behave the VU profiles in detail once the test is completed.
• Start the debug mode with .
• Two modes available (scenario advanced settings):
- Only the iterations containing errors are displayed (default behaviour).
- All VUs iterations are displayed.
• Launching a scenario in debug mode may severely hamper performances.
Advanced Results
177. Debug Mode
Advanced Results
Open the Check
VU window
The VU iterations that
contain errors
The instance of the
VU containing errors
178. Filters
• Filter any test result by time, population, zone and error.
• When a filter is applied, a new set of results is created.
• Filtered results may be viewed and used as any other test runs.
• Test reports may be generated following the usual procedure.
Advanced Results
180. Analysis: a Multi-Tiers Architecture
Advanced Results
Web browser Web server App server DB server
Potential issues:
• Thread saturation
• Cache usage
Potential issues:
• Thread saturation
• Data pool saturation
• Internal application issue
(code, memory leak,
garbage collection…)
Potential issues:
• Cache usage
• Internal database issue
(no index, costly
requests, memory…)• Apache
• IIS
• Tomcat
• JBoss
• webLogic
• webSphere
• OAS…
• MySQL
• DB2
• Oracle
• MS SQL
• PostgreSQL
• Internet Explorer
• Firefox
• Chrome
• Opera
• Safari…
181. Analysis: HTTP Server Contention
• Adding workers (depending on server resources, especially CPU) or an additional Apache
server may improve the situation.
Advanced Results
Limit of the
busyworker is
reached
Response times
are impacted
182. Analysis: App Server Contention
• Garbage collection causes the threads to be suspended (not the case with parallel
garbage collection). If it takes too long, response times are impacted.
Advanced Results
JVM overload
Response times
are impacted
Errors raise up
183. Analysis: Database Contention
• NeoLoad performance indicators give a detailed view of the most resource-hungry SQL
queries (Oracle and DB2 only).
• Such information helps optimizing SQL queries and validating the use of cache, memory,
sorts, indexes…
Advanced Results
184. Analysis: Network Contention
• In case of a network contention, if the time gap between response times and TTFB:
- Increases, that often indicates a network problem.
- Remains the same, that often indicates a server problem.
Advanced Results
185. Objectives
After completing this exercise, you will be able to:
• Use the NeoLoad features to analyze a test result.
Scenario
1. Open the project called ‘Training_Project’ and select the test results (‘Charge_Prod_1’
and ‘Charge_Prod_2’).
2. Analyze both results by looking at the response times and each layer of the JpetStore
architecture.
Exercise 22
186. For Advanced Users
• Advanced Installation
• Servers
• Request & Page Settings
• Service Level Agreement
• Logical Actions
• Variables
• Data Exchange API
• Mobile Support
• Advanced Results
The NeoLoad License
AGENDA
187. A Combination Of
The NeoLoad License
Number of
VUs
Protocol
Modules
Monitoring
Modules
188. FREE STANDARD PROFESSIONAL ENTERPRISE ULTIMATE
Suggested Usage
Developers and
testers running
small tests
One tester testing
one application at a
time
Teams testing one
application at a time
Organizations
testing multiple
applications
concurrently
Corporate CoEs and
teams needing to
run very large tests
Virtual Users 50 50 – 1,000,000 50 – 1,000,000 50 – 1,000,000 1,000,000+
Protocols All Included
HTTP/S Included,
Others Optional
HTTP/S Included,
Others Optional
HTTP/S Included,
Others Optional
All Included
Command Line
Access
Continuous
Integration Plugin
Monitoring Optional
Collaboration
Integration &
Customization
• APM
• Data Exchange
API
• Custom Action
Extensions
Optional
Shared License
The NeoLoad License
189. Available Modules
The NeoLoad License
O/S Web Server DB Server App Server Network Protocol Integration
Windows IIS MS SQL Server .NET SNMP SOAP
CA / dynaTrace /
AppDynamics
Linux Apache Oracle JBoss Flex
Solaris Nginx My SQL Tomcat Silverlight
IBM AIX DB2 WebLogic Oracle Forms
HP-UX PostgreSQL WebSphere GWT
Data Exchange API
VMWare MongoDB Oracle AS Java
RSTAT JOnAS Oracle Siebel
Adobe LCDS Push
GlassFish RTMP
Custom Actions
SAP NetWeaver
Generic JMX
Kaazing
190. Standard
• To be used on a single machine
• May be rent (a minimum of 8 days)
• Testing days must be consumed after
the license activation within ‘10 x days’
of the license term
Shared
• Installed on the Neotys Team Server.
• Minimum split is 5VUs.
• Modules available for each split.
• Cannot be rent.
The NeoLoad License
191. The No Key Mode
• Features
- Record, design and check new or
existing VUs.
- Set up already existing monitors.
- Analyze existing test results.
- Perfect to design before running a test.
• Limitations
- Tests cannot be run.
- No collaboration.
- No ‘Monitoring Data Import’.
The NeoLoad License
192. Thank you for attending this training
Other courses are available at www.neotys.com