Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Benchmarking NGINX for Accuracy and Results
Similar a Benchmarking NGINX for Accuracy and Results (20)
Más de NGINX, Inc.
Más de NGINX, Inc. (20)
Último
Último (20)
Benchmarking NGINX for Accuracy and Results
- 1. Benchmarking with NGINX Introduced by Owen Garrett Presented by Rick Nelson Nginx, Inc.
- 2. About this webinar When you want to know how many resources to allocate for your NGINX servers or what the capacity of your current NGINX servers is, you need to be able to perform proper benchmark testing, but this can be complicated. In this webinar, you'll learn the considerations that need to go into planning, configuring and running benchmarks.
- 3. Agenda • Introduction • Common Pitfalls • Tips and Techniques • Demonstration • Questions
- 4. INTRODUCTION
- 5. What is NGINX? Internet Proxy Caching, Load Balancing… HTTP traffic N Web Server Serve content from disk Application Server FastCGI, uWSGI, Passenger… Application Acceleration SSL and SPDY termination Performance Monitoring High Availability Advanced Features: Bandwidth Management Content-based Routing Request Manipulation Response Rewriting Authentication Video Delivery Mail Proxy GeoLocation
- 6. Why do Benchmarking? • Stress test • Capacity planning • Comparison testing (bake off)
- 7. Benchmarking Considerations • It’s complicated • What is the goal? • What kind of test environment do you have? • What testing tools do you have? • How well can you simulate production traffic?
- 8. What areas are you testing? • Web server • Application Server • Reverse Proxy • All of the above
- 9. What are you testing? • Can you simulate production traffic? • If not, what are you concerned about: – Connections – Request rate – Bandwidth – SSL – All of the Above
- 10. What are you testing? • Can you do a full production test? • If not, do a smaller scale test and extrapolate • Know your traffic – Get vs. Post, request/response sizes, etc. • What you need to test vs. what you are actually testing
- 11. What are you testing? • You may want to test a single variable – Connections – Request rate – Bandwidth – New SSL handshakes
- 12. Test Environment • You are always testing the whole environment – Testing Tools • Load generators • Web Servers – Systems under Test (SUT) • Reverse proxies • Web servers
- 13. Test Environment • Good rule of thumb for cores needed: – Load Generators: 2N – Reverse Proxies: N – Web Servers: 2N
- 14. COMMON PITFALLS
- 15. Not Knowing What You Are Testing • Know your testing tools • What question does a test answer? – For example: • How many requests per second can a SUT handle? • Can a SUT handle a certain number of requests per second
- 16. Real Clients versus Synthetic Clients Real Clients Synthetic Clients Latency Low-High Low Packet Loss Low-High Low Bandwidth Low-High High Time between requests Long Short Idle connections Yes No
- 17. Unrealistic Synthetic Clients • Misleading results – System looks good during benchmark – System has problems with real clients • Why is this? – Synthetic clients are ideal for the server • Low latency, low packet loss, busy connections – Real clients are not • High latency, packet loss, idle connections
- 18. Synthetic Clients • Unrealistic synthetic clients – Apache Bench (ab) • More realistic synthetic clients – Open Source • Siege, wrk – Commercial • Spirent, Ixia, Cloudtest, other cloud services
- 19. Misconfiguration • Many configuration settings can impact tests – Some Linux kernel settings may be too low for heavy loads – Keepalives – SSL key sizes make a big difference – Compression – Benchmark clients or servers
- 20. Tips and Techniques • Use multiple approaches – Real world simulations for real world results – Simple tests for baselines and debugging
- 21. Tips and Techniques • If you have found the real limit of a SUT then: – At least one system resource should be exhausted • CPU • Memory • Bandwidth • Disk I/O – If not, then a bottleneck exists elsewhere
- 22. Tips and Techniques • Start with the NGINX defaults • NGINX directives can impact performance – accept_mutex, worker_processes, worker_connections, keepalive_timeout, lingering_close, sendfile, keepalive, aio, open_file_cache
- 23. Tips and Techniques • Some 10G drivers don’t use cores effectively – Some cores at 100%, others have low usage – Solution: driver dependent • Scripting • New driver version • New Card
- 24. Tips and Techniques • Monitor error files and errors from testing tools – Errors can return faster – You may be hitting system limits or errors • Don’t have load generation on a SUT • Double check the result figures
- 25. Tips and Techniques • Run all tests multiple times • If using virtualization, be aware of other loads on the host • If you don’t understand the results - simplify
- 26. Demonstration Load Generator Load Generator Datacenter ab siege Apache NGINX
- 28. Closing thoughts • 38% of the busiest websites use NGINX • Check out the previous webinar on tuning at nginx.com • Future webinars: nginx.com/webinars Try NGINX F/OSS (nginx.org) or NGINX Plus (nginx.com)
Notas del editor
- Thank you Owen. Doing benchmarking can be a very important component to any project. You may be trying to determine how many servers and what size of servers you will need or you may want to discover the limit of a particular system. Benchmarking can be complicated and confusing, and to be successful you need to do proper planning and you need to understand the environment, the tools and the tests you will be using. In this webinar we will discuss many of the things you need to consider when doing benchmarking.
- I we will start with an introduction and go over some of general things to consider when doing benchmarking, then we will move on to some of the common pitfalls to watch out for during your testing and then we will cover some tips and techniques and we fill finish with a demonstration.
- So let’s get to it.
- This webinar will be focusing on benchmarking in general, but NGINX can often be the answer when your benchmark tests show that you need to increase the performance of your system. NGINX is a high performance web server of static content, including video and also a full reverse proxy or Application Delivery Controller with many advanced features. For dynamic content it can connect to application servers over HTTP, FastCGI, uWSGI, Passenger and other methods.
- There are many reasons you might be doing benchmarking. You may want to do a stress test to find the limit of a particular system or systems. Or you may want to do capacity planning where you have a particular performance level in mind and you want to find the environment that will satisfy that need. These two goals can be related in that in order to do capacity planning you may stress test a particular configuration and then extrapolate the results of that test to figure out what system would be required to meet you capacity goal. You may also be testing to do a comparison between different solutions, for example different reverse proxy servers. And a key questions is, are you trying to simulate real world traffic from real clients. When trying to test a real application it is normally desirable to find out how it will perform in the real world, but there can also be value in doing more controlled testing. With your goals in mind you can design your tests to meet these goals.
- When it comes to doing benchmarking, I’m not going to sugar coat it, doing proper benchmarking is complicated. It takes planning and you need to know what you are doing. There are many things that are important to consider. What is the goal of the testing? It is not good enough to say that you want to test the performance of you system. Are you trying to find the limit of a particular configuration? Are you trying to find a configuration that will meet a certain level or performance? Are you interested in the number of simultaneous users the system can support, or the amount of bandwidth it can send to clients, etc. What test environment do you have? How close is it to production? What tools do you have for doing the testing? The tools used can make a huge difference. Can you simulate production traffic? The closer you can get to testing with production like traffic the closer your results will be to matching the real world.
- What areas of you system do you want to test? For this discussion we will limit ourselves to web servers, application servers and reverse proxies. The more layers of you system you want to test, the more complicated the testing will be. If you are having issues with testing multiple layers, you can test each layer individually.
- What are you actually going to test? You aren’t going to want to test on an actual production system, so what can you test? Are you able to simulate production traffic? If so that is great because then your test will be more real world, but I have dealt with many organizations that are unable to simulate production traffic. In that case it is often useful to be able to pinpoint what particular area of performance you are concerned about based on your application. For example, are concurrent connections what you are worried about, or is it the request rate, or the amount of bandwidth the system can handle, etc. or a combination of factors.
- Do you have an environment that can match production? If you are trying to find out what size and quantity of systems you will need for production, can you scale your tests until you find the answer? Often times this is not possible, so you can do smaller scale testing and then extrapolate these results. One of the nice things about NGINX is that it scales in a linear fashion allowing you to extrapolate test results. If you don’t have the ability to capture live traffic and use it as input to your benchmarking tests and are going to try to simulate this input, then it is important to understand the nature of your live traffic. For example, what is the percentage of GET’s to PUT’s? What are the minimum, maximum and average request and response sizes. Are you using SSL? And be sure that you are testing what you think you are testing. If tests are not designed correctly or understood, then you may find that the test isn’t doing what you thought it was.
- If you are unable to simulate production traffic or if you want get a basic set of metrics, you can do simplified testing where your are looking for individual limits. For example if you want to know how many connections per second a system can handle, in isolation from other factors, you can do a test where you request a zero byte file with keepalives disabled. That way all the system is doing is connection handling. For the request rate you can request files of different sizes and see what the request rate is for each size file. For bandwidth you can request a large file (1 meg is usually enough).
- Earlier I said that you need to understand what areas of your system you are going to test, but in reality you are always testing the entire environment. So in a test where you have load generators to a reverse proxy to web servers, even if the system you are really looking to test is the reverse proxy, you are also testing the load generators and the web servers, as well as switches and anything else in the network path. So this means that you must know which systems are the focus of your testing but you must also realize that the entire test environment is part of the test. If we limit ourselves to load generators, reverse proxies and web servers, then we can always look at the load generators as testing tools, and if we are testing a reverse proxy, then the web servers are also testing tools in this case.
- When testing a reverse proxy, a good rule of thumb is that you will need twice as much computing power for the load generators and web servers as you need for the reverse proxy. This means that benchmarking can be resource intensive, but if you don’t allocate enough resources for the load generators or web servers, then they will become the bottleneck and you will not be able to adequately stress the reverse proxy.
- Now let’s start to talk about some common issues you may run into while doing your testing and how to avoid them.
- It is very important that you know how your testing tools work. The more full featured and better a tool can simulate real clients, the more complex it is. Some of the commercial products are very sophisticated but also hard to learn and understand. I have dealt with customers using these tools who did not really understand how the tests worked and so didn’t really understand what they were testing. For example I worked with a customer who was trying to find the maximum number of requests per second a system could handle, but they set up the test so that the tool had a goal of a certain number of requests per second. When they set this goal so that it was greater then the maximum the SUT could handle, the SUT would start to slow down as it reached its maximum, but since the tool was told to drive a certain number of requests per second, as the SUT slowed down, it sent even more requests to it thus overloading it further and in the end the SUT showed a requests per second rate far lower then its actual maximum. This is just one example, but the bottom line is that if you don’t really understand how a test is being run, you can’t interpret the results.
- One of the biggest issues when testing a reverse proxy or a web server, is the type of clients used in the test. By there nature, during a benchmarking test the clients will be synthetic in that the client load will be generated by a tool rather then a real user. Synthetic clients, in their simplest and least sophisticated form, are very unlike real clients and this table highlights some of these differences. By a simple synthetic client I am referring to one that is co-located with the SUT and doesn’t do anything to simulate real client behavior, so it sends requests as fast as it can. Real clients, if they are remote will have higher latency, increased packet loss and lower bandwidth compared to a simple synthetic client. And real clients can have long gaps between requests. Think of a user on a browser who downloads a web page which causes a number of requests, but then the user stops to read the page for several seconds before clicking on a link causing a new set of requests. With a simple synthetic client, it will send requests continuously. This means that the connections from real clients will often be idle while the connections from simple synthetic clients will never be idle.
- So these simple synthetic clients are unrealistic if you are trying to see how your system will perform against real world traffic. And what you can see when using such tools is that during the benchmark, the system shows one level of performance but when rolled into production and faced with real clients, the performance is much lower. The reason for this is that these synthetic clients act in a way that is ideal for servers. From the servers perspective they are very well behaved. They have very low latency and packet loss and don’t leave connections idle, just the opposite of of real clients. That is not to say that simple tools don’t have value. The good thing about doing more real world like testing is that it will give you results that should better match what you will see when you roll a system into production, but by their nature these are more complex and add more variables. So, it is often valuable to use both types of tools. The simple tools can be good for getting initial baselines and also for trying to debug your tests when you get results that you don’t understand. They are also good if your goal is not to simulate real traffic, but rather to find a set of limits for a system in isolation. This is common for vendor testing, where you want to find individual limits, such as how many connections, requests or bandwidth a system can handle. This is the single variable testing I discussed earlier.
- There are many load generation tools available, both open source and commercial. Here are a just a few examples. One of the most well known, simple testing tools, is Apache Bench (ab). Some open source tools that do have some level of real client simulation are Siege and wrk. There are many commercial products available such as Spirent, Ixia, Cloudtest and there are other cloud services. Other things you can do to help simulate real clients is to add latency and packet loss to your test. Latency can be added simply using the Linux tc command and latency and packet loss can be added using a WAN simulator. It is important to note, that properly simulating real clients is essential for real world results, but when doing the simplified testing I mentioned early, this is not important since for those tests we are trying to remove variable, such as the complexity of real clients.
- Another thing that can lead to inaccurate results, is the misconfiguration of one or more of the machines involved in the testing. For example you may need to do some Linux kernel tuning on the load generators and SUT to avoid hitting an OS limit. An example of some of the settings that may need to be increases are the number of file descriptors, the ephemeral port range, the number of time wait buckets, and the size of the input queue. Keepalive connections can have a large impact on performance. Using keepalives removes processing workload from both clients are servers so it is important to know where they can be used. If you are testing a reverse proxy then it will have both client and server side connections and keepalives can be set independently. SSL key sizes have a large impact on performance. 2048 bit keys require five times the processing power of 1024 bit keys when handling new SSL handshakes. If you are doing real world testing, make sure to test with the size of key you will use in production. If you are trying to maximize the results, use a 1024 bit key. Also, to get real world results you need to have an idea of how many SSL connections will require new handshakes. Compression also impacts performance. Turning compression on increases the demand of the CPU to do the compression/decompression but reduces the bandwidth. So it is important to test with the compression you will use in production. It is vital that the machines used to create the load and serve the content, in the case where you are testing a reverse proxy, are configured correctly and have enough resources allocated to be able to generate and server the load. If not, your testing tools will be the bottleneck and you will not be able to find the limit of the SUT.
- You may think from my discussion of realistic versus unrealistic synthetic clients that I am recommending using the more complex test to simulate real world results except where you don’t care about real world results. But in fact, even if in the end you want your tests to mimic a real world use case, you will usually want to also so some more simplistic testing. This is because the more complex tests add a lot of variables and this can make interpreting the results more difficult. It can be helpful to begin by doing some simple testing to make sure you don’t have any obvious bottlenecks, because these bottlenecks may not be at all obvious when doing more complex testing. During simple testing allows you make sure that get the CPU to 100%, and saturate the available bandwidth, etc. For example, I was involved in a test where we had an issue on a machine with 2x20G NICs, but when doing a simple bandwidth test we were barely able to get 5G. Since we were doing a very simple test we were more easily able to pinpoint the problem as a setting on a switch. Once this was fixed, the same test was able to saturate the 2 NICS. One these initial simple test show that the system seems to be working as expected you can move to more complex, real world testing. You may then find it useful to use these simple tests again to help while troubleshooting these more complex tests. For doing these simple test you can use simple tools, like apache bench or you can use the more complex tools.
- Let’s talk a bit about interpreting results. When you run a tests to try and find the limit of a SUT, the fact that your tests seems to have found some sort of maximum value doesn’t necessarily mean that you have found the limit of the SUT. If you have truly found the limit of the SUT, then you should have maxed out a system resource, such as CPU, memory, bandwidth or disk I/O. If not, then you have hit a bottleneck somewhere else. It could be in the software you are testing, or it could be a driver, or the load generators or the web servers, if you are testing a reverse proxy, or it could be somewhere else in the infrastructure. Use the simple testing I talked about in the previous slide can help to troubleshoot these issues.
- The NGINX default setting have been designed to be optimal for most environments—so don’t change them if you don’t have too. If you do need to adjust the NGINX configuration, then certain directives can impact performance. Some of the directives to look into are accept_mutex, worker_processes, worker_connections, keepalive_timeout, lingering_close, sendfile, keepalive, aio and open_file_cache. Please refer to the NGINX documentation for details.
- I have found that some 10G NIC drivers do not properly distribute interrupts across all the cores of a machine. If this happens, you will see some of the cores at 100% utilization while other cores show very low utilization, so you are effectively only getting the processing power of the number of cores the driver is spreading interrupts across. The solution to this problem depends on the driver. You may be able to work around it with some scripting, or you may need a new driver, or in the worst case you may want to get a card from a different vendor.
- It is very important that you monitor error files. This includes error files from the testing tools, SUT and OS. Errors can return faster then actual requests so if you are unwittingly getting errors from a web server you may see a deceptively high request rate. You may also be hitting some system limits that are impacting the tests. You should never run a load generator on the same machine as SUT if you want any real performance results because they will be competing for resources. You should double check the numbers you get from the tests to be sure they make sense. For example, in the demonstration I will show you shortly, I am requesting a 10K sized file. After a test is run, the amount of bandwidth received by the client should match the number of requests times 10K. If instead of getting the 10K file I get a 404, I will not only see a deceptively high request rate but if I check the bandwidth I will see that it is less then it should be.
- All tests should be run multiple times. Depending on the amount of variation in the results you may want to average all the results or you may want discard outliers. If you are running in a virtualized environment, be aware of what else is running on the host, if you can. With regards to the previous point, if you are testing in the cloud, then you will have little control or visibilty into the infrastructure used for you testing. You may therefore see very inconsistent results, depending on what else is going on in the infrastructure. This makes it even more important to run tests multiple times, ideally at different times of the day. If you are getting results that you don’t understand, then simplfy your test to remove variables, as I discussed earlier with regards to testing for single system limits.
- Now I will do some demonstrations to show a few of the things I’ve been talking about. For these tests I have a very simple setup with three Ubuntu instances in a cloud. Two of the instances are being used as load generators and they each have two load generation tools installed, apache bench and siege. The third instance has Apache and NGINX installed. Apache has a default configuration using the worker multi-processing module, and NGINX has a default configuration. In addition, I’ve added 2ms of delay to each machine to better simulate a LAN connection versus a host-to-host virtual network. We will start with a very simple test, using apache bench in the upper left window to generate traffic directly to Apache. [Show: ab –t 10 –s 10 –c 5 http://webserver/10k.html] Apache bench has a number of options, but for this test we will only use the –t option to specify that the test will run for 10 seconds, -s to specify that we want apache bench to timeout after 10 seconds rather then the default of 30 seconds (this will come in handy in a later test), and the –c option to tell apache bench to open 5 concurrent connections to Apache. By default Apache Bench does not use keepalive connections, so it will open and close each connection for each request and response. This should give us a conservative requests per second number, because as I mentioned earlier, using keepalive connections reduces the amount of CPU needed by the client and server because the machines don’t have the overhead of setting up and tearing down connections after each request and response. So by not using keepalives we are making the machines do some extra work. We will address keepalives again during another test. The requests will be for a 10K file. While I run this test in the upper left window, we will also monitor the CPU utilization for the webserver in the bottom window. [run: vmstat 3] We will be looking at the idle CPU %, which the third column from the right. [run ab] While the test is running we see that the CPU idle % drops a bit, but the CPU is still pretty idle. And we see that we get around 550 requests per second. Since the CPU was still very idle, this tells us that we haven’t hit the maximum for this test (with this size of a direct HTTP requests we won’t be stressing memory, bandwidth or disk I/O). Now lets run the same test, but this time with keepalives enabled. So now apache bench will open 5 connections but keep them open for the duration of the test. [Run: ab –t 10 –s 10 –c 5 –k http://webserver/10k.html] And we see that the request rate about doubled. Keep this in mind as we do some additional test. We will return to doing our testing without keepalives, but now we want to try and find out how many connections Apache can handle before the CPU becomes 100% busy. This should gives an idea of the maximum requests rate for this test scenario. We don’t have time to run a whole series of tests to see what number of concurrent connections causes the CPU to get to 100% busy, but from my previous testing, I know that this occurs around 40 concurrent connections. [Run: ab –t 10 –s 10 –c 40 http://webserver/10k.html] If we run a test with –c 40 we see that the idle % goes to zero and the request rate increases to around 2500 per second. We can increase this to 400 concurrent connections. [Run: ab –t 10 –s 10 –c 400 http://webserver/10k.html] And we see that the request rate stays about the same and we could keep increasing this value, but for our tests we will stop here. Let’s say that these results are within our performance requirements and so based on these results we are confident that the system will perform well in production. One note, you may remember that previously I talked about having to sometimes increase system limits for your testing. Well, in this case I had to increase the number of open files the OS would allow, otherwise apache bench errored when trying to do 400 connections. As I said earlier, Apache Bench does not do a good job of simulating actual clients, so let’s do a test with another tool, siege, in the upper right window, that does a better job of simulating actual clients, although it is by no means complete in this area. [Run: siege –t 20s –c 150 –d 10 http://webserver/10k.html] Two of the options for siege, -t and –c are the same as with Apache Bench, except for this test will let the test run longer and we will try 150 concurrent connections. The key differences with this test are that we can specify the –d option that tells siege to add a delay between requests - It will add a random number of seconds between zero and the number we specify, in this case 10 seconds – and siege will be using keepalive connections, as real clients usually do. This means that siege will open 150 connections and keep them open, unlike the apache bench tests I have been running, where apache bench was opening and closing connections for each request/response. So since siege is using keepalives and adding a delay between requests, it is adding less load on Apache both by removing some connection handling overhead and also by making fewer requests. This test takes about 15 seconds to warm up so I’ve let it run for 20 seconds. During this test we see that the CPU remains mostly idle and we get a requests per second rate of only about a hundredth of what we saw with our apache bench test. This makes since because siege is making far fewer requests, but this doesn’t appear to tell us anything about the limits of this machine. Now let’s run the siege test and the apache bench test, with just 5 concurrent connections, at the same time. [Run: siege –t 60s –c 150 –d 10 http://webserver/10k.html] [Show: ab –t 10 –s 10 –c 5 http://webserver/10k.html] Remember that for the last Apache Bench test we were opening 400 concurrent connections and now we have siege opening 150 and apache bench opening 5, so let’s see what happens. We will wait about 15 seconds for siege to warmup and then kick off the apache bench test and we will again watch the CPU utilization. [Run: ab] We see that we get an error from apache bench. This error means that it was unable to get a response from Apache, even though we saw that the CPU utilization remained mostly idle. This is why I’m using the I said earlier that if you have reached the maximum of a machine then you should see that the system has exhausted some resource, in this case CPU, or you have hit a bottleneck. Here the bottleneck is Apache’s connection handling. By having siege open 150 keepalive connections, and sending a request only occasionally, it takes up all the connections that Apache can handle, so when the apache bench test is run, it is unable to get any requests processed. You would think that by using keepalive connections to help reduce the processing needed, you would see better performance but here we see worse performance. This is because of how Apace handles connections and this is a case where a more real world tests shows a far lower level of performance then a simpler test. So now a plug for NGINX. On the machine we are running Apache on, we also have NGINX running and setup to listen on port 8080 and proxy requests to Apache. So lets run the same test again, but this time we will have both siege and apache bench send traffic to NGINX instead of Apache, and NGINX will do nothing but proxy all the requests to Apache, so it will take over the job of handling the client connections. [Run: siege –t 60s –c 150 –d 10 http://webserver:8080/10k.html] [Show: ab –t 10 –s 10 –c 5 http://webserver:8080/10k.html] We will again give siege about 15 seconds to warmup. [Run ab] This time when apache bench is run we see the CPU become busier and we get a request rate similar to what we saw in our original apache bench test, without siege running. So NGINX allows you to get around this bottleneck because of its better connection handling and better overall efficiency. You might find it interesting to know how NGINX would perform in these tests if it was serving the file rather then Apache. I have configured NGINX to also listen on port 8000 and serve the same 10K file. Remember that when we did our direct test against Apache, we saw the CPU go to 100% busy around 40 concurrent connections. Let’s run the same test against NGINX. [Run: ab –t 10 –s 10 –c 40 http://webserver:8000/10k.html] Here we see that the CPU is not 100% busy, but the requests rate has gone from around 2500 to around 3800. [Run: ab –t 10 –s 10 –c 80 http://webserver:8000/10k.html] In my testing in this environment I see that it takes about 80 connections to consistently get the CPU to 100% busy. Finally let me demonstrate something else I mentioned earlier and that is that errors can return more quickly then successful requests. If we rerun the previous test [Run: ab –t 10 –s 10 –c 80 http://webserver:8000/11k.html] But request a file that doesn’t exists so that we get a 404, file not found error, w see that our request rate as gone from just under 4000 to just under 5000.
- Now we will open the floor for questions.
- Thank you for taking the time to attend this session. I hope it was of use to you and I hope you attend some of our future webinars. You will recordings of previous webinars at nginx.com and this includes one on tuning. And it is easy to test out either the open source version of NGINX or NGINX Plus.