Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Functional and scale performance tests
using zopkio
{‘Event’: ‘PyCON HK 2015’,
‘Name’: ‘Marcelo Araujo’,
‘Email’: ‘marcelo...
Agenda
● About me.
● Why we test software?
● Zopkio test framework.
● Q&A.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
About me
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Name: Marcelo Araujo.
● Bachelor in Computer System Networking...
Why we test software?
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Assure what we created does what it supposed to d...
A typical application infra.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
App Server Database
Your application starts ...
Then, becomes like this.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
App Server Master DB
Slave DB
Cache
Storage
Bala...
A question!
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
As a DevOps, It is not only about test software anymore!
How ...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Made by Linkedin.
● It is a test framework built...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio provides the ability to write tests that ...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● What kind of tests can we perform with zopkio?
○...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● What we need from a functional and performance t...
My PYHK test lab.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
bare-metal machine
FreeBSD OS
Hypervisor Type 2 (bhyve)...
What tests I will perform in this lab?
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● VM1 and VM2: Check how many file...
The basic of Zopkio.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
.
|-- deployment.py
|-- naarad.cfg
|-- perf.py
|-- r...
The Zopkio framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio has a main script that takes several optio...
Main run.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 2 lines: import os---------------------------------------...
The config file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 {
2 "comment": "This is the first test",
3 "tip": "How...
The dynamic config.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 4 lines: !/usr/bin/env python------------------...
The deployment file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● It is in charge of provisioning machines.
● A depl...
The deployer.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 3 lines: !/usr/bin/env python------------------------...
An test file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 6 lines: !/usr/bin/env python------------------------...
Plot graphs.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 [machine1-icmp]
2 infile=/tmp/pyhk/collected_logs/machine1...
The Zopkio GUI.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
Let's take a look on my machine
The conclusion is:
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
We need to be able to test the infrastructure where I
...
References.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio:
○ https://github.com/linkedin/Zopkio
● Naarad:
○ ht...
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
Thank you!
marcelo@gandi.net
araujo@FreeBSD.org
Próxima SlideShare
Cargando en…5
×

Functional and scale performance tests using zopkio

987 visualizaciones

Publicado el

PyCon Hong Kong 2015.

Publicado en: Internet
  • Sé el primero en comentar

Functional and scale performance tests using zopkio

  1. 1. Functional and scale performance tests using zopkio {‘Event’: ‘PyCON HK 2015’, ‘Name’: ‘Marcelo Araujo’, ‘Email’: ‘marcelo@gandi.net’
  2. 2. Agenda ● About me. ● Why we test software? ● Zopkio test framework. ● Q&A. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
  3. 3. About me PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Name: Marcelo Araujo. ● Bachelor in Computer System Networking and Telecommunications. ● Post-Degree in Quality on Software Engineering. ● Python consumer since 2007. ● FreeBSD Developer since 2007 (ports and *kernel). ● DevOps at Gandi since 2015. From To
  4. 4. Why we test software? PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Assure what we created does what it supposed to do. ● The behavior with one user could be different with thousand users. ● Users could do something unexpected or not planned. ● Many different devices, browsers, operating systems and so forth. ● We want assure a good software quality. ● Validation: Are we doing the right job? ● Verification: Are we doing the job right?
  5. 5. A typical application infra. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> App Server Database Your application starts like this!
  6. 6. Then, becomes like this. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> App Server Master DB Slave DB Cache Storage Balancer/Proxy API Servers Storage Replication VMs Services
  7. 7. A question! PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> As a DevOps, It is not only about test software anymore! How can I test the infrastructure?
  8. 8. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Made by Linkedin. ● It is a test framework built to support at scale performance and functional testing. ● It can be installed via pip. ● Latest code at: https://github.com/linkedin/Zopkio ● There is documentation. ● Enough pydoc for every each class. ● They have some examples of code. ● Under Apache 2.0 License. ● They are open for pull requests. ● Active development.
  9. 9. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Zopkio provides the ability to write tests that combine performance and functional testing across a distributed system. ● Writing tests using Zopkio should be nearly as simple as writing tests in xUnit or Nose. ● Zopkio strongly depends on: ○ Naarad: A system analysis tool that parses and plots time series data. ○ Paramiko: Python implementation of SSHv2 protocol.
  10. 10. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● What kind of tests can we perform with zopkio? ○ Functional. ■ Ensure every function produces its expected outcome. ○ Load. ■ Understand the behaviour under a specific and expected load. ○ Stress. ■ What is the behaviour beyond the normal expected load. ○ Performance. ■ Determine the speed or effectiveness of an software or service.
  11. 11. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● What we need from a functional and performance test framework? ○ ASSERT: Must be able to test a condition! ○ Get data vs time and compare with what we expected. ○ Plot graphs! ○ Provisioning before run any test. ○ Parallelize tests. ○ Control the sequence of tests. ○ Collect logs. ○ Parse logs. ○ Cleanup environment after tests. ○ An interface with test result. ○ Be able to debug! Our test is a software too.
  12. 12. My PYHK test lab. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> bare-metal machine FreeBSD OS Hypervisor Type 2 (bhyve) VM1 (BSD) VM2 (BSD) VM3 (Linux) VM4 (BSD) Zopkio Switch Server
  13. 13. What tests I will perform in this lab? PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● VM1 and VM2: Check how many files inside /etc/ and apply an ASSERT. ● VM1 and VM2: ICMP to example.com and check the lantency. ● VM3 and VM4: TCP Server/Client and make a connection between them.
  14. 14. The basic of Zopkio. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> . |-- deployment.py |-- naarad.cfg |-- perf.py |-- run.py |-- test_suites | |-- connection.py | |-- machine1_etc.py | |-- machine1_ping.py | |-- machine2_etc.py | `-- machine2_ping.py |-- configs | `-- pyhk.json |-- scripts | |-- ping-csv.sh | |-- run.sh | |-- server.py | |-- client.py | `-- scripts.tar ● A test suite consists of: ○ A deployment file. ○ A dynamic configuration file. ○ One or more test files. ○ A config directory.
  15. 15. The Zopkio framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Zopkio has a main script that takes several optional arguments: ○ Run only named tests to help debug broken tests. ○ Configs overrides at execution time. ○ Log level. ○ Console log level. ● To run my test suite: araujo@coxinha:/z/hk# zopkio run.py --nopassword --log-level DEBUG
  16. 16. Main run. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 2 lines: import os------------------------------------------------------------------ 3 test = { 4 "deployment_code": os.path.join( 5 os.path.dirname( 6 os.path.abspath(__file__)), "deployment.py"), 7 "test_code": [ 8 os.path.join( 9 os.path.dirname( 10 os.path.abspath(__file__)), "test_suites/machine1_etc.py"), 11 os.path.join( 12 os.path.dirname( 13 os.path.abspath(__file__)), "test_suites/machine1_ping.py"), 14 os.path.join( 15 os.path.dirname( 16 os.path.abspath(__file__)), "test_suites/connection.py"), 17 os.path.join( 18 os.path.dirname( 19 os.path.abspath(__file__)), "test_suites/machine2_ping.py"), 20 os.path.join( 21 os.path.dirname( 22 os.path.abspath(__file__)), "test_suites/machine2_etc.py")], . |-- run.py 23 "perf_code": os.path.join( 24 os.path.dirname(os.path.abspath(__file__)), "perf.py"), 25 "configs_directory": os.path.join( 26 os.path.dirname(os.path.abspath(__file__)), "configs/") 27 }
  17. 17. The config file. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 { 2 "comment": "This is the first test", 3 "tip": "How many files on my /etc", 4 "pyhk_install": "/tmp/pyhk/", 5 "pyhk_exec": "scripts/scripts.tar", 6 "pyhk_cmd": "sh run.sh", 7 "ping_cmd": "bash ping-csv.sh --add-timestamp example.com >/tmp/pyhk/ping-output.csv", 8 "tcp_server_cmd": "python server.py &", 9 "tcp_client_cmd": "python client.py >connection.log", 10 "show_all_interations":true, 11 "verify_after_each_test":true, 12 "pyhk_hostname1": "10.0.1.22", 13 "pyhk_hostname2": "10.0.1.12" 14 } |-- configs | `-- pyhk.json
  18. 18. The dynamic config. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 4 lines: !/usr/bin/env python----------------------------------------------- 5 LOGS_DIRECTORY = "/tmp/pyhk/collected_logs/" 6 OUTPUT_DIRECTORY = "/tmp/pyhk/results/" 7 +-- 2 lines: -------------------------------------------------------------------------- 9 def machine_logs(): 10 return { 11 "client1": [os.path.join("/tmp/pyhk/", "connection.log")], 12 "machine1": [os.path.join("/tmp/pyhk/", "run.log")], 13 "machine2": [os.path.join("/tmp/pyhk/", "run.log")], 14 } 15 +-- 2 lines: -------------------------------------------------------------------------- 17 def naarad_logs(): 18 return { 19 "machine1": [os.path.join("/tmp/pyhk/", "run.csv")], 20 "machine2": [os.path.join("/tmp/pyhk/", "run.csv")], 21 "machine1": [os.path.join("/tmp/pyhk/", "ping-output.csv")], 22 "machine2": [os.path.join("/tmp/pyhk/", "ping-output.csv")], 23 } 24 +-- 2 lines: ----------------------------------------------------------------------------- 26 def naarad_config(): 27 return os.path.join( 28 os.path.dirname(os.path.abspath(__file__)), "naarad.cfg") . |-- perf.py
  19. 19. The deployment file. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● It is in charge of provisioning machines. ● A deployment file can contain four functions: ○ setup_suite() - Will run before any of tests. ○ setup() - Will run before each test. ○ teardown() - Will run if setup() ran successfully. ○ teardown_suite() - Will run if setup_suite() ran successfully.
  20. 20. The deployer. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 3 lines: !/usr/bin/env python------------------------------------------------------------------------------- 4 import zopkio.adhoc_deployer as adhoc_deployer 5 import zopkio.runtime as runtime 6 +-- 6 lines: pyhk_deployer = None-------------------------------------------------------------------------- 12 def setup_suite(): 13 +-- 2 lines: print "==> Starting tests for PYHK."--------------------------------------------------------- 15 runtime.set_user('root', '') 16 +-- 4 lines: global pyhk_deployer--------------------------------------------------------------------------- 20 tcp_server = adhoc_deployer.SSHDeployer( 21 "server", 22 {'executable': runtime.get_active_config('pyhk_exec'), 23 'extract': True, 24 'start_command': runtime.get_active_config('tcp_server_cmd'), 25 'stop_command': "ps ax | grep '[p]ython server' | awk '{print $1}' | xargs kill -9"}) 26 runtime.set_deployer("server", tcp_server) 27 28 tcp_server.install("server1", 29 {"hostname": "10.0.1.23", 30 "install_path": runtime.get_active_config('pyhk_install')}) 31 32 tcp_client = adhoc_deployer.SSHDeployer( 33 "client", 34 {'executable': runtime.get_active_config('pyhk_exec'), 35 'extract': True, 36 'start_command': runtime.get_active_config('tcp_client_cmd')}) 37 runtime.set_deployer("client", tcp_client) 38 39 tcp_client.install("client1", 40 {"hostname": "10.0.1.24", 41 "install_path": runtime.get_active_config('pyhk_install')}) 42 +-- 20 lines: pyhk_deployer = adhoc_deployer.SSHDeployer(-------- 62 def setup(): 63 for process in tcp_server.get_processes(): 64 tcp_server.start(process.unique_id) 65 66 def teardown_suite(): 67 for process in tcp_server.get_processes(): 68 tcp_server.undeploy(process.unique_id) . |-- deployment.py
  21. 21. An test file. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 6 lines: !/usr/bin/env python----------------------------------------------- 7 import zopkio.runtime as runtime 8 import zopkio.test_utils as testutilities 9 10 test_phase = 3 11 +-- 2 lines: ------------------------------------------------------------------------ 13 def test_ping_host1(): 14 print "==> ping example.com (machine1)" 15 pyhk_deployer = runtime.get_deployer("pyhk") 16 17 pyhk_deployer.start( 18 "machine1", 19 configs={ 20 "start_command": runtime.get_active_config('ping_cmd'), 21 "sync": True 22 }) 23 +-- 2 lines: ------------------------------------------------------------------------ 25 def validate_ping_host1(): 26 ''' 27 Send 10 icmp to example.com . 28 ''' 29 hostname1_log_file = os.path.join( 30 perf.LOGS_DIRECTORY, "machine1-ping-output.csv") 31 hostname1_logs = testutilities.get_log_for_test( 32 "test_ping_host1", hostname1_log_file, "12:00:00") 33 34 # Number of icmp packages 35 size_p = len(hostname1_logs.split(',')) / 2 36 37 assert size_p == 10, "Less than 10 replies on host1" |-- test_suites | |-- machine1_ping.py
  22. 22. Plot graphs. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 [machine1-icmp] 2 infile=/tmp/pyhk/collected_logs/machine1-ping-output.csv 3 columns=sequence latency 4 sep=, 5 qps.sla=mean<500 6 latency.sla=mean<400 p50<300 7 8 [machine2-icmp] 9 infile=/tmp/pyhk/collected_logs/machine2-ping-output.csv 10 columns=sequence latency 11 sep=, 12 13 [GRAPH] 14 graphs=machine1-icmp.sequence,machine1-icmp.latency machine2-icmp.sequence,machine2-icmp.latency . |-- naarad.cfg
  23. 23. The Zopkio GUI. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> Let's take a look on my machine
  24. 24. The conclusion is: PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> We need to be able to test the infrastructure where I will run my application.
  25. 25. References. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Zopkio: ○ https://github.com/linkedin/Zopkio ● Naarad: ○ https://github.com/linkedin/naarad/ ● Code for PYHK Test Lab: ○ https://github.com/araujobsd/pyhk2015 ● Slides at: ○ TBD
  26. 26. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> Thank you! marcelo@gandi.net araujo@FreeBSD.org

×