2. Who am I?
• Paweł Pikuła pawel.pikula@erlang-solutions.com
• Developer @ Erlang Solutions Ltd. Cracow office
• I work on MongooseIM XMPP server and some
IoT related project.
• github.com/ppikula
• twitter: @pawelpikula
3. Agenda
• What is load testing
• Introducing AMOC - our simple load testing tool
• Overview of our test infrastructure
• Breaking a single node
• Distributed scenario (scaling MIM and AMOC
horizontally)
• What to do next ?
4. Prepare the environment
• Install VirtualBox and Vagrant
• git clone https://github.com/ppikula/euc2015/
• Follow instructions from the README file (Setting
up the environment from a usb stick)
5. What is load testing ?
“In software engineering, performance testing is in
general testing performed to determine how a
system performs in terms of responsiveness and
stability under a particular workload. It can also
serve to investigate, measure, validate or verify
other quality attributes of the system, such as
scalability, reliability and resource usage.”
wikipedia
6. Why do we do load test ?
• Measure capacity of the whole system.
• Measure latencies, time to deliver and other quality
attributes.
• Find the bottlenecks in our system.
• Test a bad case scenarios under heavy load (ex. net splits,
overloaded DBs).
• Discover possible race conditions .
• Stress 3rd party software.
9. Types of traffic - spike
!
• Check if the system
can recover after the
spike
• How long does it take
to go back to the
normal state
10. TSUNG
• We‘ve been using it for a long time
• It does the job
• Tsung is dumb - doesn’t understand XMPP
• XML DSL …
• Manual setup
• It is not designed for bidirectional protocols
• HTTP semantics (request, transaction, page time)
• some strange race conditions? (I was never able to login 10K users every
time I tried I had ~9997 of them )
12. OUR USE CASE
• stream management enabled (XEP 198) auto
respond to <r> with <a>
• authentication process requires acquiring a token
from http service
• carbon copies - multiple jabber resources
13. ESCALUS
• esl/ejabberd_tests use it for black box testing
• supports many transports (ws, bosh, tcp, tls)
• stanza generators
• built-in predicates (is_message, is_iq_result)
16. AMOC design goals
• Use the esl/escalus library
• Don’t reinvent the wheel - cooperate with graphite,
ansible, graphite notifications, graylog.
• Simple and practical
• No DSL - plain erlang
• build everything on one host
17. AMOC features
• Automatic deployment and environment setup via
ansible
• Exometer for collecting metrics - very easy to add a
new metric
• Uses modified erlang slave module
• Dynamically increase/decrease the amount of users
• Sends notifications about events mentioned above
18. Environment checklist
• File descriptors limit
• Check firewall rules
• TCP buffers and other options
• Increase local port range
• Create more network interfaces if you want to
generate more than 60K users
19. Why there is a limit of 60K
per interface?
• TCP port is a16 bit integer - we have only 65536
outgoing ports - reserved ones
20. AMOC Inside
!
• AMOC scenario - define init and start function
• AMOC controller - “user interface” start/stop
scenarios.
• The controller spawns users on AMOC slaves in
round robin fashion
22. MongooseIM
• Fork of ejabberd 2.1.8
• Removed “non-scalable” extensions
• Added unit & integration tests (black box testing)
• Added many metrics
• Want to hear more? go to “MongooseIM - The Right
Tool for Scalable Messaging” by Michał Piotrowski
25. Task3
Add custom metric to AMOC
• Message rate (spiral)
• Connected users count (counter)
• Time to deliver (histogram)
26. Future of AMOC
• automatic tests
• allow to pass inter arrival per scenario
• support for multiple scenarios
• ansible recipes for ubuntu/debian family
• dockerized version of AMOC?
• integration with elixir - that will allow us to create DSLs
