QA&test 2016 (Bilbao) Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it
•Descargar como PPTX, PDF•
2 recomendaciones•717 vistas
“You better start testing at the beginning,” is what everyone in testing says these days as new methodologies such as agile and test driven development are becoming increasingly popular. What is the best way to go when it comes to performance testing? Should we start it right away (taking a lean approach) or at the end (following a more waterfall style approach)? The speaker, Federico Toledo, will compare two different real world performance testing projects that he took part in over the past 10 years. The first project employed a more waterfall like approach, meaning, the whole load testing project took place at the end of development. In the second one, development and load testing projects started on the same day (planned by two different teams but as part of the same team) in a more lean approach. He will go into detail about the pros and cons of each project. After the presentation, you will end up coming to your own conclusions about which one is right for you. Additionally, Federico will reveal insights about monitoring and problem solving in production, mistakes to avoid in each, and the skills one should develop to be a successful performance tester. http://www.qatest.org/es/programme/speaker_info.php?id_speaker=247&day=1
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (10)
Similar a QA&test 2016 (Bilbao) Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it
Similar a QA&test 2016 (Bilbao) Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it (20)
Más de Federico Toledo
Más de Federico Toledo (20)
Último
Último (20)
QA&test 2016 (Bilbao) Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it
- 1. Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it PhD. Federico Toledo @fltoledo
- 2. • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion Agenda
- 3. • Co-founder and Director at Abstracta (www.abstracta.us) • PhD and author • +10 years experience in performance testing • +80 projects worldwide Who we are
- 4. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 6. USEFUL WORK TIME & RESOURCES USED VS
- 9. Measurement
- 11. • At the end • Waterfall • Acceptance testing • Simulation of the expected load scenario • Along with development • Agile • Continuous integration • Unit and component • Early feedback • Performance engineering Performance Testing
- 12. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 13. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 14. What you always have to take into account
- 15. What are the main hard-skills a performance tester needs? • Scripting and programming • Communication protocols • Architectures • Monitoring tools and metrics • Network and Operating Systems administration • Business knowledge People: hard-skills
- 16. What are the main soft-skills a performance tester needs? • Abstraction • Communication • Analysis • Perception People: soft-skills
- 17. As Scott Barber says: A good performance tester should be a “mid- level” in everything. http://www.perftestplus.com/resources/pt_for_managers_ppt.pdf People
- 18. What you always have to take into account
- 19. Technology
- 20. • Load simulation • JMeter (jmeter.apache.org) • Taurus (gettaurus.org) • BlazeMeter (blazemeter.com) • Gatling (gatling.io) • Measurement • Profilers and similar (JVisualVM, JProfiler, dotTrace) • Operating system (NMON, perfmon, Netdata) • APMs (AppDynamics, NewRelic, DataDog) Technology – Server Side
- 21. Technology
- 22. • Web • Page Speed • Yslow • monkeytest.it • APMs • Desktop • Perfmon or NMON • Profilers Java or .NET • APM • Mobile • Google Test Cloud / Testdroid / Perfecto • Xcode - Instruments / XCTest • APMs • Monkop Technology – Client Side
- 26. Ebay ESPN Fantasy Instagram Rio 2016 VivaVideo Monkop - Explorations
- 29. What you always have to take into account
- 30. • Test design • Test automation • Test execution and measurement Processes
- 31. • Execution plan • Baseline (used as a reference) • Then, iterative and incremental approach Processes Server Server
- 32. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 33. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 34. PROS 1. It’s easier to plan 2. Focus in a specific context 3. Environment similar to production CONS 1. Infrastructure availability 2. Costs of architectural changes Performance Testing at the End
- 35. PROS 1. Minimize risk 2. Early and constant feedback 3. Learning • Monitoring tools and Analysis • Good and bad practices CONS 1. More automation effort • Pay attention to which level you should automate 2. Fallacy: if you test the components then the entire system will work properly. Performance Testing along with development
- 36. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 37. Agenda • Introduction • Performance, testing, different approaches • What you always have to take into account • People, tools and methodology • Comparison: Waterfall vs. Agile • Pros and cons based on real world experiences • Conclusion
- 38. Which approach is better? You should ask a different question Conclusion
- 39. • Do you need to verify that your system supports a specific load? • Do you need evidence for your customers? Go with a LOAD SIMULATION. WATERFALL APPROACH. Which approach is better?
- 40. • Do you want to optimize costs and reduce risks pertaining to the system’s performance? • Do you want to increase your team’s knowledge in terms of monitoring and performance engineering? • Do you want to follow a real continuous integration scheme? Go with PERFORMANCE ENGINEERING along with development. AGILE APPROACH. Which approach is better?
- 41. • We need both approaches in different stages of our development cycle. • We should start early by doing performance engineering and we also need to simulate load for acceptance testing. Which approach is better?
- 42. THANK YOU! Pros and Cons of Doing Performance Testing Along with Development Versus at the End of it
- 43. • Performance testing fallacies https://www.stickyminds.com/article/7-performance-testing-fallacies-undermining-your-test- strategy • What’s more important, the tool or the tester? http://www.abstracta.us/2015/05/24/whats-more-important-the-tool-or-the-tester/ • Introduction to Taurus, an alternative to JMeter http://www.abstracta.us/2015/09/07/introduction-to-taurus-an-alternative-to-jmeter/ • Gatling http://blog.abstracta.com.uy/search/label/Gatling References
- 44. • Blog and resources (in English) • http://www.abstracta.us/knowledge-center/ • Blog (in Spanish) • http://blog.abstracta.com.uy • eBook “A complete Introduction to Functional Test Automation” (in English) • http://www.abstracta.us/2016/05/31/ebook-complete-introduction-to- functional-test-automation/ Free eBook and materials
- 45. If you like it Certification in Performance Testing with JMeter and BlazeMeter • Online course (one month) • www.abstracta.us/training Cupon code: 25% OFF this month QATEST2016 http://bit.do/librodetesting
- 46. PhD. Federico Toledo federico@abstracta.us @fltoledo Silicon Valley office 425 Broadway Street Redwood City, CA Headquarters Jose Ellauri 1126 Montevideo, Uruguay www.abstracta.us
Notas del editor
- I will start presenting general notions about performance testing and the two approaches that we are considering: performance testing at the end of development, as a load simulation for acceptance testing, and an agile approach where performance tests are considered since the beginning of the project. Then, I will start by looking at what both approaches have in common: for the two approaches we need similar things, and this is also important to know what we need in terms of people, tools and methodology. After that, we will focus on the comparison between both, talking about pros and cons of each one. At the end, I will give some conclusion about the topic based on our experience.
- Before we continue with the presentation, I’d like to present myself. I’ve been working in performance testing the last 10 years. I’m one of the co-founders of Abstracta, which is a company dedicated to software testing, providing services and tools, for performance testing, automation, mobile testing and functional testing as well. I had the chance to do a PhD here in Spain, so I’m happy to be here again because I consider this country as my second home. I feel like home here, so thanks again to the organization to give me the opportunity to be here participating in this amazing event. In this talk I want to share some lessons learned in the different projects that I’ve been involved, mainly in the last one I’ve been working the last months, for a huge company in the States, based in Syllicon Valley. They have a Continuous integration approach including performance testing, with very demanding non-functional requirements, an important team working exclusively to improve the performance of the whole system. I’ve learned a lot working with them, and I want to share here some of the those learnings.
- Let’s start for the beginning
- Let’s start with a very basic question (I hope that everyone know the answer, but just in case…) What is performance? What’s the first word that comes to your mind when you think about performance? It’s velocity, isn’t it? Well, let’s imagine a website which responds very fast, but it uses 100% of the CPU, does it have good performance? It doesn’t, right?
- Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used. We cannot only see how fast it is, because a system that is very fast but uses 100% of CPU is not performant. Then, we need to check both sides, the user experience (the time I perceive, the velocity) and the servers’ feelings (how stressed the servers are). Also, if we only pay attention to response times, we could only see the symptoms, but what we want to find are the root causes in order to identify bottlenecks and then ways to improve.
- So with performance testing we want to analyze the computer’s performance through a simulation. Don’t wait until production to know how the performance is.
- Simulation and measurement. Analyze the system’s behavior in terms of response times and resource usage. Sometimes we want to report how good or bad is the system behaving, and sometimes is a proactive approach in order to improve the obtained results. Investigate and analyze looking for bottlenecks, in order to make experiments, tuning or altering the system, seeking for improvements. Of course, it is this kind of testing where your focus is on the performance aspects: response times and resource usage. For this, it’s important to simulate the load that it is expected for production: the corresponding amount of concurrent users, with the “intensity” that will characterize them. There is a preliminary result that you can obtain that it is very useful too, that is to simulate certain stress on different components, not only on the entire system. This could be done with the aim of finding bottlenecks, concurrency issues and improvement opportunities.
- And it’s not only about simulating, but we also need to measure. “What we don’t know is what gets us killed”.
- We need to measure during that simulation to be able to detect: - bottlenecks (what’s the “thinner” part of the system that doesn’t allow it to be fast, or faster) what’s the breaking point (after what amount of load does the system start degradation) So, in a “simple” way: performance tests consist of simulation & measurement to detect these type of problems.
- So, when should I start doing performance testing? And here is the main motivation of our talk: the comparison between doing performance testing at the end of development and doing performance testing along with it. Waterfall approach At the end of the development. Used for acceptance testing: if we reach the acceptance criteria the system goes to production, if it does not: we don’t buy it / we don’t go-live / etc. Simulation of the expected load scenario. We need to understand the expected load in order to simulate it. Agile approach Performance testing along with development. Think about performance testing since the beginning. Continuous integration environment with performance tests included. Unit and component performance testing. Early feedback. Performance engineering. When you have one story implemented you can start executing it with multiple users at the same time, you can validate your architecture or any aspect that you consider risky, but the only way to do that is thinking about “what could be risky in terms of performance and how can I mitigate asap those risks?”
- We will start by talking about different aspects that are common to both approaches. For this, we determined three pillars of the continuous improvement process: people, technology and processes (or methodology)
- Talking about what your people should know, we can distinguish between hard-skills and soft-skills. Starting with hard-skills, you don’t have to learn to use different tools, we think that you have to learn how to manage different kind of tools. What I mean is useful at least for our context: in each project we face different architectures, different operating systems, components, web servers, DBMS, protocols, etc. You need to know how to program in most cases, because even though you use jmeter which scripts are shown in a graphical way, as a tree inside the tool, you will always need to automate certain tasks, prepare a bash script, or something similar. The automation is done always at a protocol level. Differently from the functional testing tools like Selenium, where the automation is done over the UI. Therefore, it’s necessary to understand the protocol. The most typical is HTTP, so you need to understand how the stateless is resolved, cookies, gets and posts, requests and responses, connections, primary requests and embedded resources, etc… !! When you have to execute a test and start analyzing, you have to understand the architectures, different layers, etc., and how to monitor them, which indicators and with which tools. The same for the operating systems. And also, it’s important to know a bit about the business, you have to understand what’s important, when your application makes you lose money or whatever risk it could bring.
- If we think of soft-skills, there is another interesting group of things to consider: Abstraction, communication (with different kind of people who speak different “languages”) Abstraction You cannot (or you won’t have enough time to) understand every little aspect of the whole system under test and you have to live with that. You will need to understand it as a whole, and go with a top-down approach as it gets necessary. Communication You will need to understand the business needs from analysts, and you will need to discuss about very technological aspects with DevOps and developers. They are people with very different languages. You will need to present your results (oral and written) to different (very different) stakeholders. Analysis You will need to read data, a lot. You have to process it, mix it, cross it with more data, see it from different perspectives, do whatever you need with that in order to get useful information, to get evidence and clues to find the bottlenecks. Perception You have to be very careful with risks.
- To describe this , Scott Barber says
- Two aspects to consider when talking about performance: server side and client side. Both affect the user experience. We will have to use different tools in each of them, different technology.
- When talking about server side: Load simulation tools Jmeter is the most popular open-source tool for this Then, we have tools like BlazeMeter that allow us to scale in the number of users by executing tests from the cloud. Taurus allows us to easily integrate Jmeter tests into a CI environment Gatling it’s my recently new favorite tool :D It uses SCALA as a scripting language, what’s interesting is that they apply Page Objects pattern to performance. It also allows us to execute more concurrent users that Jmeter from the same machine. It can also be scaled up to Blazemeter. What tools do we have available to measure from the server side? Some are specific tools related to the technologies we are using APM tools that show all the information from each component in one place
- Web Some tools that are well-known, to check for performance of a website (not under concurrence). monkeytest.it – monkey testing over a website, analysis over each page Desktop Profiles or Operative System tools to check for CPU, Memory, Network, etc. Mobile Many tools available & Monkop, that we’ll tell you more about it now. This is one of the tools we developed at Abstracta.
- Monkop is a cloud-base solution that allows you to execute compatibility tests over several real devices and check for performance issues, crashes and more.
- Monkop can be used in different stages of the development cycle. To use the Saas version, you only have to upload the app to monkop.com, and in 1 hour you will receive a report in your inbox with the results from the tests.
- During this hour, Monkop will install the app in different devices form our in-house lab, and run tests in each of them.
- These are some examples of Monkop’s explorations
- Monkop collects detailed performance information regarding CPU, Memory and Battery usage, network, as well as rendering time and other layout metrics
- And generates a report with the areas of the app that need improvement and also provides recommendations based on Android best practices The report includes videos of the explorations and screenshots for you to be able to reproduce the test. You can try it for free at monkop.com.
- 3 main activities are involved in the performance testing process. The way we distribute the effort we put in these tasks will determined if we’re being more “agile” or following a waterfall approach. In the following section we will talk about this.
- If you execute 100% of the load directly, probably you will see many errors at different levels, it will be difficult to know where to start. Best approach: execute small loads and make them grow progressively. In that way, you will face the biggest problems first, and later you can refine your analysis.
- Advantages of doing the testing at the end: It’s easy to plan: you build the team and you know that they will be assigned 3 months to that activity This can be an advantage and also a disadvantage: have a separate environment similar to production to run the tests, and then that environment can become the production one (after the tests are run) Focus on some specific characteristics to test (X amount of functionalities) Some cons of the waterfall approach: One month prior to going live, everyone wants to use the environment we need to run the tests (the one that is similar to production) This is the worst one: executing performance tests ONLY at the end is highly risky – find out at the end that the is and architecture/design problem, which requires high costs to change/adapt. ***Tell about a project: It happened with a client that one month before going live, they had to change the data base scheme because it was not possible to optimize the SQLs and the response times were too high.
- Some of the pros of the Agile approach: Avoid large problems that may require major changes in the architecture or in big structures. Imagine that after one year of development, you run a test and it tells you that your architecture is not good enough to support the expected amount of concurrent transactions. Then, you have to go in and backtrack, making large changes after working so hard on it for a whole year. With the Agile approach, you run the first load test on all the different components at the beginning of the project, reducing and managing this risk. For example, in one project two years ago, we ran a load simulation (waterfall approach) and discovered that the system could not support the concurrency properly. As it was so hard to change the architecture, they had to use several application servers for some months, while the development team was making the required adjustments. If they had executed a simple test at the beginning, they would have realized the problem existed before. You can execute frequent load tests and identify which changes make the system go slower, then it’s easier and cheaper to correct those issues. Another good practice that we tend to do in all our continuous integration projects: even if you do not have performance tests, measure your functional tests, or even your unit tests, and set an alert if the execution time gets higher than a specific value (or for example, if it’s 20% higher than the average of all the previous executions) ***Tell about a project: Fortune 500 company in SV, execute 300 scripts (unit performance tests built by developers ) , 3 times per week (in Jenkins), in a dedicated environment. Our job is to consolidate these scripts, reduce the total test time and optimize them. For example, part of our job is finding the right assertions in response times and transactions. We have a test over a REST service. The developer runs the tests with 100 users. Our goal as Performance Engineers is to find what is the optimal number of users that should execute the test once it is integrated into the CI. So, for this we ask ourselves: What’s the max number of users that can run the services without degradation? What’s the breaking point? And also, what’s the response time of the transaction before it increases? For example, I know that 225 users is the number of users this service can handle before getting to the breaking point and that the response time until that point is about 3 seconds. So I would add an assertion that verifies that response time do not increase in more than 20% and also, I will check that the number of transactions per second doesn’t decrease by more that 20% (it doesn’t get slower). Goal: identify the line of code that generated the performance problem as soon as possible One of the best benefits is that your team learns not only about performance testing, but also about monitoring and problem solving. That will prepare them (both the test team and the development team) to be ready for any several types of problems in production, because if they face problems during the construction of the software, for sure they will have learned how to get the information needed (which tools and which indicators, logs, etc.) and use it to get straight to the problem and its solution. It’s good to have DevOps, but it’s a very hard skill obtain in the market, isn’t it? Perhaps it could be very useful to train your developers a little about performance, and your performance testers about development. What if your developers learn a bit about using a profiler, set a new environment, use different optimization and analysis tools, etc. Doing performance testing frequently makes your team be involved in these tasks and face this kind of problems as a common task. Some cons of the agile approach: Requires more automation effort. There is no tool that allows you to “maintain” scripts. Most of the time we rebuild scripts from scratch because it takes less effort than trying to fix the ones you have from the previous sprint. Perhaps we could think about the proposal from Mike Cohn; the famous pyramid shows how we should invest our time in automation. If we translate this into performance testing we could conclude: We could have many unit tests for performance testing (as mentioned before, with automatic response time checks). These could be executed easily in their CI tool, many times every day, and it is very easy to maintain and very fast to execute (the best ROI) We could have API tests, also executing against our code, or against services (web services, REST), considering concurrency. These tests are easier to maintain and quicker to run than those prepared to simulate user behavior. Automate the most critical test cases at a GUI level, what in performance testing is translated to “http” in most cases, in the protocol between the client and the server. These are the most difficult to maintain, but on the other hand, these are the ones that provides the most information end-to-end. FALLACY: if you test the components then the entire system will work properly. That’s not necessary true. You need to test the components individually and then, test them working together.
- We should ask ourselves these questions instead. Do you need evidence for your customers? ***Tell example about client that needed to validate if the software/hardware was able to handle the expected load prior to buying it.