Exactpro Systems - "Trading Systems: Testing at the Confluence of FT & NFT" Alexey Zverev, Managing Director, Exactpro Systems Alyona Bulda, QA Project Lead, Exactpro Systems Ivan Bobrov, HFT Analyst, Exactpro Systems

1. Trading Systems:
Testing at the Confluence of FT & NFT
Alexey Zverev, Managing Director
Alyona Bulda, QA Project Lead
Ivan Bobrov, HFT Analyst

2. Introduction
1. Introduction
2. Our topic - how to find / reproduce "toxic issues" which can happen under
the load or in real live environment conditions
3. We will talk about:
a. A classical approach where FT and NFT are separate streams
b. What is missed here
c. Some approaches we use or may apply to fill this gap between FT and
NFT (applying to trading systems)
4. Our presenters

3. FT NFT
focus on: focus on:
• Test scenarios are based on functional • Testing is based on non-functional
requirements requirements
• Test tools which allow to run simplistic tests • Load scenarios are driven by the
such as to send an order -> to verify a response technical characteristics in PROD.
• Prepare a functional library starting from • The main target of the tests is to assess
simple scenarios and adding more complex ones non-functional requirements: latency,
step by step capacity and fault tolerance
• Test scenarios are automated
• A lot of efforts are invested in correct
• A test Library is enhanced as we proceed latency measurements and determining
with its testing and production clients scenarios conditions when the system breaks
• An automated library will be reused as a • Further regressions are done to
regression test library which is executed compare system performance indicators
against each release after go live and can be with the previous releases
taken for consequent projects

4. What do we miss when we keep FT
and NFT separated ?
- Any complex system must sustain a certain level of concurrency. Some issues
appear when several events happen simultaneously (Race conditions).
Example 1: Clients send a CxlReplace request and approx. at the same time
the ME sends a fill back.
These 2 events meet at
the Order Management
server, and the state of
the order can get corrupted.

5. What do we miss when we keep FT
and NFT separated ?
- Production order books and various system object states are more complex
than in our FT scripts, and no amount of test cases can cover all possible
scenarios
Example 2: A User tries to
hit two different implied
orders coming from the
same parent order
which is being restated
in the middle of the process.

6. What do we miss when we keep FT
and NFT separated ?
- Components may behave differently under the load
Example 3: The System must support various
throttling limitations and the requirements of
the system behaviour. The correctness
of functional system behavior can be
tested only under the load.
- Integer memory overflow
Example 4: Turnover (VWAP) fields in the statistics messages and incremental
OrderID overflow;
* the picture was taken from https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcS9nlDdLAmuvmyRqqZsanbyfwL601nJlOvtHGEfJBS4DKnmSTANUA

7. What do we miss when we keep FT
and NFT separated ?
- Dynamic Mass events which happen during DLC
Example 5: Market Open,
Market Close, Auctions,..
The state of the system
at exact instance
when the system goes
from one session to another
should be verified with the load
applied against the system.

8. Detailed Verification Of The System Behavior Under Random Load
 We need to make sure that we
provide extensive coverage
- basic consistency check
 We need to make sure that the load test mimics - market data must be internally consistent
what happens in the production system - order book should be available and not be
 We need to verify that the system different
works as expected
+ Recreation of exact steps of real production issues
- It is difficult to record multiple network streams

9. Performing Functional Testing Against The System Under The Load
to leave a part of the system intact by the load
to adjust functional tests to cope with the
activity of the load
to make scripts in case when we
perform
tests actively interacting with the load
* scenarios must be flexible
* sustain interaction with the script

10. Testing With The Baseline
The example of the cycle of a Clearing System
A detailed, step by step analysis

11. The “Mini Robots” Solution

12. The “Mini Robots” Solution

13. Questions & Answers-tO
Thank You-tO