Meetup TestingAR 2016 - Performance testing durante y después
•Descargar como PPTX, PDF•
1 recomendación•386 vistas
Este documento presenta diferentes enfoques para las pruebas de rendimiento, comparando los enfoques de "agua corriente" y ágiles. Discute las ventajas de realizar pruebas de rendimiento tanto durante el desarrollo como al final, antes de la implementación, y concluye que se necesitan ambos enfoques en diferentes momentos del ciclo de desarrollo de software.
Recomendados
Recomendados
Más contenido relacionado
Destacado
Destacado (8)
Similar a Meetup TestingAR 2016 - Performance testing durante y después
Similar a Meetup TestingAR 2016 - Performance testing durante y después (20)
Más de Federico Toledo
Más de Federico Toledo (20)
Último
Último (20)
Meetup TestingAR 2016 - Performance testing durante y después
- 1. Performance Testing Durante y Después PhD. Federico Toledo @fltoledo
- 2. ¿Cómo evitar este trabajo? • Bomberos solucionando problemas en producción.
- 3. • Introducción • Performance, testing, distintos enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones Agenda
- 4. Agenda • Introducción • Performance, testing, distintos enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 6. TRABAJO ÚTIL TIEMPO & RECURSOS VS
- 9. Medir
- 10. Bottlenecks Punto de Quiebre
- 11. • Usuarios concurrentes • Proceso Tipos de Pruebas de Performance
- 12. • Durante • Agile • Continuous integration • Unit, component • Feedback temprano • Ingeniería de Performance • Después • Waterfall • Acceptance testing • Simulación de escenario de carga Performance testing…
- 13. Agenda • Introducción • Performance, testing, distintos enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 14. Agenda • Introducción • Performance, testing, diferentes enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 15. A tener siempre en cuenta
- 16. • Scripting y programación • Protocolos de comunicación • Arquitecturas • Herramientas de monitorización y métricas • Administración de Redes y Sistemas Operativos • Conocimiento del negocio y sus riesgos Personas: hard-skills
- 17. • Abstracción • Comunicación • Análisis • Percepción Personas: soft-skills
- 18. Como dice Scott Barber: A good performance tester should be a “mid- level” in everything. http://www.perftestplus.com/resources/pt_for_managers_ppt.pdf Personas
- 19. A tener siempre en cuenta
- 20. Technología
- 21. • Simulación • JMeter (jmeter.apache.org) • Taurus (gettaurus.org) • BlazeMeter (blazemeter.com) • Gatling (gatling.io) • Usábamos OpenSTA (opensta.org) pero quedó sin mantenimiento • Medición • Profilers y similar (JVisualVM, JProfiler, dotTrace) • Herramientas a nivel de Sistema Operativo (NMON, perfmon, Netdata) • APMs (AppDynamics, NewRelic, DataDog) Technología – Server Side
- 22. Technología
- 23. • Web • Page Speed • Yslow • monkeytest.it • APMs • Desktop • Perfmon o NMON • Profilers Java o .NET • Mobile • Google Test Cloud / Testdroid / Perfecto • Xcode - Instruments / XCTest • APMs • Monkop Technología – Client Side
- 27. Encuentro GeneXus Marca Fotocasa La Liga VivaVideo Monkop - Exploraciones
- 31. A tener siempre en cuenta
- 32. • Diseño de prueba • Automatización • Ejecución y medición Proceso / metodología
- 33. • Plan de ejecución • Baseline (sin concurrencia) • Para tener de referencia • Iterativo e incremental • Problemas más graves se hacen visible primero Servidor WebServidor Web Servidor WebServidor Web Proceso / metodología
- 34. ¿Cómo analizar problemas de performance? • Encontrar forma de reproducir • Asegurarse que el problema no es la prueba • Hacer un cambio a la vez • Changelog / bitácora para tener trazabilidad entre cambios y cómo afectó Tuning / ajuste Bajan tiempos de respuesta x% Cambio en una funcionalidad Uso de CPU aumentó X% Proceso / metodología Icons by Gregor Črešnar, Jamie Wilson, Shmidt Sergey, Icon Fair. Source: https://thenounproject.com/
- 35. • Los 5 ¿por qué? Para ir de síntomas a causas • El Sistema anda lento • ¿Eso es síntoma o causa? ¿Por qué anda lento? • Porque el Sistema tarda 15 segundos en completar • ¿Por qué? • Porque el componente de acceso a datos está demorando todo ese Proceso / metodología
- 36. Agenda • Introducción • Performance, testing, diferentes enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 37. Agenda • Introducción • Performance, testing, diferentes enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 38. • La mayoría de nuestros clientes nos piden por “simulación de carga” justo antes de salir en producción. • Principales motivaciones: • Nueva versión con nuevas funcionalidades • Nueva arquitectura (win to web, web to mobile) • Campañas de Marketing • Un banco que absorbe a otro • Solo algunos piden acompañarlos durante el desarrollo (en los últimos años) • Por eso decidimos comparar los dos enfoques ¿Qué es lo que se usa?
- 39. PROS 1. Más fácil de planificar 2. Ambiente similar a producción 3. Foco en un contexto específico CONS 1. Disponibilidad de la infraestructura 2. Riesgo de altos costos por cambios arquitectónicos Performance Testing al Final
- 40. PROS 1. Minimizar riesgo 2. Feedback temprano y constante • Pruebas tempranas (unidades, componentes) • Integración continua 3. Aprendizaje continuo • Monitorización y análisis • Buenas y malas prácticas CONS 1. Mayor esfuerzo de automatización • Prestar atención en qué nivel automatizamos 2. Falacia de la Composición: • Test(A)+Test(B)=Test(A+B) • Si se prueban los componentes entonces el sistema integrado funcionará apropiadamente. Performance Testing durante el desarrollo
- 41. Agenda • Introducción • Performance, testing, diferentes enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 42. Agenda • Introducción • Performance, testing, diferentes enfoques • A tener siempre en cuenta • Personas, tecnología y metodología • Comparación: Waterfall vs. Agile • Pros & Cons basado en experiencias reales • Conclusiones
- 43. ¿Qué enfoque es mejor? Deberíamos cambiar la pregunta… Conclusiones
- 44. • ¿Necesitas verificar que el sistema soporta determinada carga? • ¿Necesitas evidencia de prueba para tus clientes? • ¿Querés analizar el impacto de un cambio en producción? SIMULACIÓN DE CARGA Al final WATERFALL ¿Qué enfoque es mejor?
- 45. • ¿Querés optimizar costo y reducir riesgos de performance? • ¿Querés incrementar el conocimiento del equipo y mejorar las prácticas y habilidades? • ¿Querés incluir performance en tu integración continua? INGENIERÍA DE PERFORMANCE Durante el desarrollo AGILE ¿Qué enfoque es major?
- 46. • Necesitamos los dos enfoques en distintos momentos. • Comenzar temprano con ingeniería de performance y preparar una prueba de aceptación al final, justo antes de poner en producción. ¿Qué enfoque es mejor?
- 47. ¡Gracias! PhD. Federico Toledo @fltoledo Performance Testing Durante y Después
- 48. • Performance testing fallacies (Stickyminds) • Blogpost: The tool or the tester? • Blogpost: Taurus • Blogposts tutorial sobre Gatling Referencias
- 49. • Blog (en Español) • http://blog.abstracta.com.uy • Blog y recursos (en Inglés) • http://www.abstracta.us/knowledge-center/ • Libro “Introducción a las pruebas de sistemas de Información” (en Español, gratuito) • http://bit.do/librodetesting • eBook “A complete introduction to functional test automation” (en Inglés) • eBook de performance… coming soon… Rerefencias
- 50. • Certificación en testing de performance usando JMeter y BlazeMeter. • Curso online de un mes de duración. • www.abstracta.us/performance-testing-certification • 25% de descuento usando este cupón TESTINGAR-MEETUP este mes. Abstracta Academy
Notas del editor
- Soy Federico, vengo de Uruguay, contento de estar aquí y encontrar antiguos amigos de la facultad y del trabajo, gente con la que he compartido y aprendido mucho. También contento que puedo, gracias a Slack y redes sociales, sentirme parte de esta comunidad a pesar que no estoy físicamente cerca. Y más contento aún que hoy puedo presenciar y compartir este rato contando de una de las pasiones que tengo por el testing, y de hecho fue la que me hizo entrar al mundo del testing: el testing de performance. Hace 10 años comencé trabajando en el CES como performance tester. Hace 8 años fundamos con dos amigos Abstracta, y hoy nos dedicamos a dar servicios y herramientas de testing, en América Latina y hace 3 años especialmente con foco en Estados Unidos. Esto que les voy a presentar hoy es una charla que voy a dar en QA&Test en Bilbao en octubre, y en CMG Impact en USA, en la Jolla cerca de San Francisco. Se trata de analizar qué es más conveniente, si invertir los esfuerzos de tesitng de performance al final del proceso de desarrollo, o durante el proceso. Lo que quiero es compartirles algunas experiencias de haber participado en las dos variantes, viendo algunos pros y contras.
- No sé si esto les resulta familiar. El trabajo de apagar incendios. If your development team plans from Monday to Friday, they go live on Friday afeternoon, then your sysadmins, Devops or someone will spend weekends solving last minute fires. We dont want to be firemans. POINT B: If you are planning to execute performance tests and are deciding upon which approach to take, this presentation will cover what it is like to do both based on real world experiences, giving you a better picture of each. KEY POINTS: Understanding of the two approaches when facing a performance testing project (waterfall and agile) Learn about the pros and cons of each approach – real life examples Skills that you have to develop in order to succeed in performance testing Tools and methodologies useful for both approaches
- We will start presenting the two approaches that we are considering: performance testing at the end of development, as a load simulation for acceptance testing, and an agile approach where performance tests are considered since the beginning of the project. Then, we will show a comparison between both, talking about pros and cons of each one. After that, we will state that both approaches are not that different, and we need similar things in order to do them: we will focus in what we need in terms of people, tools and methodology. At the end, we will give our conclusion about the topic based on our experience.
- Let’s start with a very basic question (I hope that everyone know the answer, but just in case…)
- Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used. We cannot only see how fast it is, because a system that is very fast that uses 100% of CPU is not performant. Then, we need to check both sides, the user experience (the time I perceive, the velocity) and the server feelings (how stressed the servers are). Also, if we only pay attention to response times, we only could see the symptoms, but what we want to find are the root causes in order to identify bottlenecks and then improvements.
- Ok, we agree about what “performance” is, but now, what is performance testing about? Mainly we have to tasks: simulation and measurement
- Simular, el pibe sentado sobre el Puente Simulation and measurement. Analyze the system’s behavior in terms of response times and resource usage. Sometimes we want to report how good or bad is the system behaving, and sometimes is a proactive approach in order to improve the obtained results. Investigate and analyze looking for bottlenecks, in order to make experiments, tuning or altering the system, seeking for improvements. Of course, it is this kind of testing where your focus is on the performance aspects: response times and resource usage. For this, it’s important to simulate the load that it is expected for production: the corresponding amount of concurrent users, with the “intensity” that will characterize them. There is a preliminary result that you can obtain that it is very useful too, that is to simulate certain stress on different components, not only on the entire system. This could be done with the aim of finding bottlenecks, concurrency issues and improvement opportunities.
- Quote GoT – medir – lo que no conocemos es lo que nos mata Enganche con bottle necks
- There are different approaches for performance testing simulation, different types of tests: Load test: when you try to simulate exactly the amount of users that you will have in production. Stress test: you want to go further, trying to determine the breaking point of your system (the biggest load that it can support with acceptable response times) Peak test: you want to see how fast your system can recover from a stress peak Endurance test: you want to address different problems, mainly those that appear in long execution times. Then, you want to execute a test for many ours. You can find memory leaks, or any other resource leak (sockets, connections, etc.) We had a project last year where we executed a great load testing and we tuned the system properly for the production environment. They went live and everything was fine. After 21 days the system (java) had to be rebooted because it has consumed all the memory assigned to the JVM. Then everything good again. After 22 days, again. After 21 days, again. We started to investigate the issue and yes, there was a memory leak caused for a bad use of connections in a list. How to avoid this really bad situation? (bad for your customer and bad for you as a performance testing services provider, because you can lose trust): make an endurance test with focus in memory and sockets. Take a picture of the situation at the start and at the end of a long run (it could be the all night long) and compare them, pay attention to those resources that are not freed as expected. On the other hand, not all systems work under concurrency. Sometimes the stress of a system is more related to data volume, or even a single process executing certain algorithm. In these situations it’s still valid to thing about performance testing analysis of possible improvements, in the algorithms, in the way the data is managed, etc. http://www.abstracta.us/2015/10/12/why-performance-testing-is-necessary/ http://www.abstracta.us/2015/03/30/types-of-performance-tests/
- Then there is a different question: when should I start doing performance testing? And here is the main motivation for our talk: the comparison between doing performance testing at the end of development and doing performance testing along with it. Waterfall approach At the end of the development. It is used for acceptance testing: if we reach the acceptance criteria the system goes to production, if it does not: we don’t buy it / we don’t pass it to production / etc. Simulation of the expected load scenario. We need to understand the expected load in order to simulate it. There is a bunch of articles and references to learn about how to correctly simulate the expected load. My favorite: Scott Barber’s series “user experience, not metrics”. Agile approach Performance testing along with development. Think about performance testing since the beginning. Continuous integration environment with performance tests included. Unit and component performance testing. Early feedback. Performance engineering. When you have one story implemented you can start executing it with multiple users at the same time, you can validate your architecture or any aspect that you consider risky, but the only way to do that is thinking about “what could be risky in terms of performance and how could I mitigate as soon as possible those risks?”
- Now we want to talk about different aspects that are common to both approaches. For this, we classified in the three pillars of the continuous improvement process: people, technology and processes (or methodology)
- Dentro del testing hay mucho lugar para los que no programan, eso fue una discusión que se dio en el Slack de TestingAR. En el caso de los testers de performance necesitan saber programar. Scripting and programming. Knowledge about the communication protocols (for example: HTTP) and some related technologies for the automation (for example: HTML, some JavaScript, etc.). Architectures (client-server, three layers, etc.). Monitoring tools and metrics. Network and Operating Systems administration. Business knowledge. Talking about what your people should know, we can distinguish between hard-skills and soft-skills. Starting with hard-skills, you don’t have to learn to use different tools, we think that you have to learn how to manage different kind of tools. What I mean is useful at least for our context: in each project we face different architectures, different operating systems, different components, web servers, DBMS, different protocols, etc. You need to know how to program in most cases, because even though you use jmeter which scripts are shown in a graphical way, as a tree inside the tool, you always will need to automate certain tasks, prepare a bash script, or something similar. The automation is done always at a protocol level. Differently from the functional testing tools like Selenium, where the automation is done over the User Interface. Therefore, it’s necessary to understand the protocol. The most typical is HTTP, so you need to understand how the stateless is resolved, cookies, gets and posts, requests and responses, connections, primary requests and embedded resources, etc!! When you have to execute a test and start analyzing, you have to understand the architectures, different layers, etc., and how to monitor them, which indicators and with which tools. The same for the operating systems. And also, it’s important to know a bit about the business, you have to understand what’s important, when your application makes you lose money or whatever risk could be.
- If we think in soft-skills, there is another interesting group of things to consider: Abstraction, communication (with different kind of people who speak different “languages”) Abstraction You cannot (or you wont have enough time to) understand every little aspect of the whole system under test and you have to live with that. You will need to understand it as a whole, and go with a top-down approach as it gets necessary. Communication You will need to understand the business needs from analysts, and you will need to discuss about very technological aspects with DevOps and developers. They are people with very different languages. You will need to present your results (oral and written) to different (very different) stakeholders. Analysis You will need to read data, a lot. You have to process it, mix it, cross it with more data, see it from different perspectives, do whatever you need with that in order to get useful information, to get evidence and clues to find the bottlenecks. Perception You have to be very careful with risks. http://www.abstracta.us/2015/05/18/how-importantnecessary-is-it-for-performance-testers-to-understand-the-architecture-of-systems-and-the-details-on-each-component/ … analysis (for what you will find useful to manage some statistics, maths and Microsoft Excel), and perception: very important! http://www.abstracta.us/2015/03/30/system-performance-problems/
- Now we want to talk about different aspects that are common to both approaches. For this, we classified in the three pillars of the continuous improvement process: people, technology and processes (or methodology)
- http://www.abstracta.us/2015/09/07/introduction-to-taurus-an-alternative-to-jmeter/ Blazemeter.com jmeter.apache.org
- http://www.abstracta.us/2015/11/02/how-to-optimize-e-commerce-website-performance-for-black-friday/
- Esa herramienta es Monkop. Es un monito que ejecuta pruebas en celulares. Top benefits monkop - no necesito Código. No necesito automatizar - en menos de una hora probé en varios devices incluso en las últimas versiones. Si instaló o no y como se comportó. Ver vídeos de la prueba etc.
- Now we want to talk about different aspects that are common to both approaches. For this, we classified in the three pillars of the continuous improvement process: people, technology and processes (or methodology)
- Cómo distribuimos nuestro esfuerzo alrededor de estas tareas va a determinar si nosotros estamos siendo más ágiles o más waterfall En la siguiente sección vemos eso
- If you execute 100% of the load directly, probably you will see many errors at different levels, it will be difficult to know where to start. Best approach: execute small loads and make them grow progressively. In that way, you will face the biggest problems first, and later you can refine your analysis.
- And perhaps there is another “why” that helps you find why it has not been created.
- Most of our customers ask for a load simulation before they go live. Main motivations: New version with new features New architecture (mainly when the migrated to web, to micro services, or when they added mobile support) Marketing campaigns that make them think that they will double the visits in their site One bank acquires another, multiplying the amount of accounts in their database and their customers accessing their home banking. Only some asked to intervene since the beginning That’s how we decided to compare both approaches http://www.abstracta.us/2015/07/06/how-to-execute-a-performance-test-with-fewer-problems/ http://www.abstracta.us/2015/06/30/how-soon-is-too-soon-to-carry-out-performance-tests/ In this section, we will show the pros and cons between two different performance testing approaches: (1) following a waterfall process (with load testing taking place at the end of development); (2) starting development and load testing activities at the same time (planned together by two different teams). You’ll come to your own conclusions about which approach is better based on our real world experience.
- Some of the cons of the Agile approach: You have fluctuations in the demand of the performance testing team. In a one-year development project, perhaps you need 3 people for one month every 3 months, for specific intermediate load tests, and only one person during the rest of the project in order to run, analyze and maintain the performance testing environment and tests. That makes it difficult from a management point of view. Perhaps we could think about the proposal from Mike Cohn; the famous pyramid shows how we should invest our time in automation. If we translate this into performance testing we could conclude: We could have many unit tests for performance testing (as mentioned before, with automatic response time checks). These could be executed easily in their CI tool, many times every day, and it is very easy to maintain and very fast to execute (the best ROI) We could have API tests, also executing against our code, or against services (web services, REST), considering concurrency. These tests are easier to maintain and quicker to run than those prepared to simulate user behavior. Automate the most critical test cases at a GUI level, what in performance testing is translated to “http” in most cases, in the protocol between the client and the server. These are the most difficult to maintain, but on the other hand, these are the ones that provides the information end-to-end. When you prepare a performance test typically it’s very expensive to prepare the database with a representative volume, prepare a specific environment for the test, nobody knows what you are doing… all these limitations are divided in small pieces and solved one by one. When you finish the project and you want to execute a big load test simulation, you already prepared the environment, test data, and what you need, and everybody knows what you are doing and the benefits of it. 90% of the performance testing projects that exceed the planed end date do that because the following factors: The development team has to prepare a representative volume of the database and they do not have a clue about how could they do that There is no availability for a proper environment (it is not well configured, nobody thought about having one, etc.) More data and environment related problems
- Minimize costs of architectural changes Nos ha pasado de a un mes de salir en producción tener que cambiar el esquema de datos porque no hay forma de optimizar las sqls Constant feedback Contar de Shutterfly: quieren ejecutar su test con 300 scripts unitarios 3 veces por semana. De esta forma van a estar muy cerca de esa línea de código que empezó a embarrarla. Mencionar el tema de shutterfly de ir apretando las tuercas al máximo para identificar cuando se degrada. Es buenísimo para las pruebas de regresión de performance, porque sino podemos estar comiéndonos problemas y tener pruebas al pedo. :) Some of the pros of the Agile approach: Avoid large problems that may require major changes in the architecture or in big structures. Imagine that after one year of development, you run a test and it tells you that your architecture is not good enough to support the expected amount of concurrent transactions. Then, you have to go in and backtrack, making large changes after working so hard on it for a whole year. With the Agile approach, you run the first load test on all the different components at the beginning of the project, reducing and managing this risk. For example, in one project two years ago, we run a load simulation (waterfall approach) and discovered that the system could not support concurrency properly. As it was so hard to change the architecture, they had to use several application servers for some months, while the development team was making the adjustments required. If they have executed a simple test at the beginning, they would have realized about the problem. You can execute frequent load tests and identify which changes make the system go slower, then it’s easier and cheaper to correct those issues. Another good practice that we tend to do in all our continuous integration projects: even if you do not have performance test, measure your functional tests, or even your unit tests, and set an alert if the execution time gets higher than a specific value (or for example, if it’s 20% higher than the average of all the previous executions) One of the best benefits is that your team learns not only about performance testing, but also about monitoring and problem solving. That will prepare them (both the test team and the development team) to be ready for any several types of problems in production, because if they face problems during the construction of the software, for sure they will have learned how to get the information needed (which tools and which indicators, logs, etc.) and use it to get straight to the problem and its solution. It’s ok to have DevOps, but it’s a very hard skill to get in the market, isn’t it? Perhaps it could be very useful to train your developers a little about performance, and your performance testers about development. What if your developers learn a bit about using a profiler, set a new environment, use different optimization and analysis tools, etc. Doing performance testing frequently makes your team be involved in these tasks and face this kind of problems as a common task. Goal: identify the line of code that generated the performance problem as soon as possible Agile requires more automation effort. There is no tool that allows you to “maintain” scripts. Most of the time we rebuild scripts from scratch because it takes less effort than trying to fix the ones you have from the previous sprint.
- We need both approaches in different moments of our development. We should start early doing performance engineering and also we need to simulate our load for acceptance testing. FALACY: if you test the components then the entire system will work properly. That’s not necessary true. You need to test the components individually and then, test them working all together.
- We need both approaches in different moments of our development. We should start early doing performance engineering and also we need to simulate our load for acceptance testing. FALACY: if you test the components then the entire system will work properly. That’s not necessary true. You need to test the components individually and then, test them working all together.
- FALACY: if you test the components then the entire system will work properly. That’s not necessary true. You need to test the components individually and then, test them working all together.