Reinventing Application Performance Testing with Service Virtualization
•Download as PPTX, PDF•
2 likes•1,828 views
Traditional performance testing typically requires that all components of the application are “completed,” integrated and deployed into an appropriate environment. This results in testing not being done until late in the delivery cycle or sometimes skipped entirely. Which can then lead to a less then optimal user experience, expensive rework and potential loss of business. Learn why many organizations are adopting service virtualization to overcome the key challenges associated with performance testing.o Learn more about CA Service Virtualization please visit http://cainc.to/DMPQSE
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Reinventing Application Performance Testing with Service Virtualization
Similar to Reinventing Application Performance Testing with Service Virtualization (20)
More from CA Technologies
More from CA Technologies (20)
Recently uploaded
Recently uploaded (20)
Reinventing Application Performance Testing with Service Virtualization
- 1. Welcome! Reinventing Application Performance Testing with Service Virtualization August 2016 © 2016 CA. ALL RIGHTS RESERVED. ALL TRADEMARKS, TRADE NAMES, SERVICE MARKS AND LOGOS REFERENCED HEREIN BELONG TO THEIR RESPECTIVE COMPANIES. NO UNAUTHORIZED USE, COPYING OR DISTRIBUTION PERMITTED. PLEASE NOTE THAT THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY DOES NOT SERVE AS ANY TECHNICAL USER GUIDES OR DOCUMENTATION. THERE IS NO DIRECT OR INDIRECT WARRANTY PROVIDED PERTAINING THE PRODUCTS AND ACTUAL USE MAY VARY FROM THE ILLUSTRATIONS PROVIDED. Original presentation made by Nathan Devoll. Recorded version available at: https://www.brighttalk.com/webcast/9305/214105?=s
- 2. 2 © 2016 CA. ALL RIGHTS RESERVED. Agenda PERSPECTIVE CHALLENGES WITH TRADITIONAL APPROACHES REINVENTING PERFORMANCE TESTING WITH CA TECHNOL0GIES ADVANTAGES AND REAL WORLD EXAMPLES 1 2 3 4
- 3. 3 © 2016 CA. ALL RIGHTS RESERVED. Businesses no longer have the luxury of time. Rising customer expectations, competitive threats, and increased consumer choice have turned rapid delivery and iteration of software applications into a competitive differentiator. Optimizing the delivery value chain has become the new means to better business outcomes. TechRadar™: Continuous Software Delivery, Q2 2015, Forrester Research, May 1, 2015 “ ” 2014 Vanson Bourne study commissioned by CA UNPRECEDENTED 94%of executives face increased pressure to release apps more quickly “ The Need for SPEED
- 4. 4 © 2016 CA. ALL RIGHTS RESERVED. DevTestOps idea AGILE DEVELOPMENT AGILE OPERATIONS Establish a flawless customer experience with apps optimized for performance Agile aligned testing supported with advanced technology Eliminate constraints and dependencies across the SDLC CUSTOMER EXPERIENCE AGILE TESTING Agility across all 3 increases speed, reliability and efficiency For more read the article “Getting in between Dev and Ops” on CA Highlights.
- 5. 5 © 2016 CA. ALL RIGHTS RESERVED. Application Performance is as Vital as Function Speed This is taking to long. 40% Source: Ankit Oberoi. 5 Reasons Visitors Leave Your Website. Website Magazine 2014 3rd Party Integration Why Doesn’t this Work? mobile subscribers, including 76% of the smartphone owners, used location-based check-in. 16.7 Million Source: Comscore Complexity Where is the Problem? of developers performance test their code before it is pushed to production Source: www.apmdigest.com, 15 Top Factors That Impact Application Performance. April 2013 of users will abandon your app if it takes more than three seconds to load. < 5%
- 6. 6 © 2016 CA. ALL RIGHTS RESERVED. Why Do Apps Perform Poorly in Production? Not all Components are Tested1 Dev/Test Environments Don’t Mirror Production2 Systems and Data are not Available for Testing3
- 7. 7 © 2016 CA. ALL RIGHTS RESERVED. Reinventing Performance Testing Test Design and Coverage – Optimize from the Start!1 Test at the Component Level3 Make Test Environments as “Life-Like” as Possible4 Utilize a Test Data Management Solution5 Leverage Service Virtualization and Drive Testing at the API Layer2
- 8. 8 © 2016 CA. ALL RIGHTS RESERVED. CA Agile Requirements Designer - Align requirements and optimize test cases Import User Stories from HP ALM or CA Agile Central to Automatically Create, Visualize and Optimize Test Cases Automated Test Case Generation Auto Test Generation Using Existing Tests Test Design and Coverage – Optimize from the Start!
- 9. 9 © 2016 CA. ALL RIGHTS RESERVED. Leverage Service Virtualization and Drive Testing at the API Layer ESB Mainframe System of Record ERP SaaS/API Simulate unavailable systems across the software development lifecycle (SDLC), allowing developers, testers, integration, and performance teams to work in parallel. I have everything I need, when I need it! “
- 10. 10 © 2016 CA. ALL RIGHTS RESERVED. Decompose Performance Expectations “Into the Lifecycle” Verify 0.3s Lookup 0.5s Response 1.2s Application SLA: 2.1 Seconds
- 11. 11 © 2016 CA. ALL RIGHTS RESERVED. Decomposing an SLA into Performance Expectations at a Component Level Test 1 Component Performance Testing Against VSE Response 1.2s Verify Lookup Application 0.9s Virtual Service Environment (VSE) Virtual Service
- 12. 12 © 2016 CA. ALL RIGHTS RESERVED. Make Test Environments as “Life-Like” as Possible What’s the Difference Between Stubs, Mocks, and Service Virtualization?
- 13. 13 © 2016 CA. ALL RIGHTS RESERVED. Don’t Skimp on the Data. Surround and Capture!
- 14. 14 © 2016 CA. ALL RIGHTS RESERVED. CA Test Data Manager - Virtual data is generated on the basis of a message definition and injected into a deployed virtual service.
- 15. 15 © 2016 CA. ALL RIGHTS RESERVED. + Linking Performance Monitoring and Testing
- 16. 16 © 2016 CA. ALL RIGHTS RESERVED. Performance Testing in a Box Innovate Faster while Delivering a High Quality Experience Use the same test cases created and optimized for functional testing Simulate backend load by running test thru an API Simulate life like test environments Generate purpose built test data Control outlier conditions Adjust think times Save Millions in infrastructure costs and 3rd party fees Detect Defects earlier in the SDLC Eliminate capacity constraints for performance testing Reduce delays with high-performance masking and subsetting, and automated data discovery Application “X” Test Cases Thousands of transactions Agent
- 17. 17 © 2016 CA. ALL RIGHTS RESERVED. Deliver Innovations… Faster! TIME Week 4Week 2Week 1 x Week 6Week 3 DEV 1 DEV 2 DEV 3 DEV 1 DEV 2 DEV 3 DEV 1 DEV 2 DEV 3 DEV 1 DEV 2 DEV 3 DEV 1 DEV 2 DEV 3 DEV 1 DEV 2 DEV 3 App 3 App 2 App 1 Week 7Week 5 Sequential Parallel App 3 App 2 App 1 Accelerate Application Delivery Simulate needed systems … end-to-end Create virtual services … easily Increase developer productivity 1 2 3 Cut of Set up time for dev environments (from 42 days to 3) Reduced Time-to-Market by for key applications. http://bcove.me/4rx4wt6x 30 Days 15% http://bit.ly/SV-TelefonicaChile
- 18. 18 © 2016 CA. ALL RIGHTS RESERVED. 99% reduction in testing times ------------------- 25% reduction in software defects Business Autotrader sells advertising that allows private and trade buyers and sellers to exchange vehicles. More than 18 million people use Auto- Trader.com each month. Challenge To meet customer demand for new online features and accessible across multiple devices, Autotrader needed to release software updates on a weekly basis. Testing, however, had become complex, costly and time-consuming. Solution CA Service Virtualization enables Autotrader to virtualize and test new services and apps on a multitude of devices, while simulating both apps working normally and while experiencing performance issues. CA Services completed implementation and training in just six weeks. Autotrader Avoids $300,000 in Testing Costs with CA Service Virtualization Autotrader is improving the customer experience with faster time-to-market. “The CA Technologies team blew the competition away with their knowledge and passion.”
- 19. 19 © 2016 CA. ALL RIGHTS RESERVED. Qualica Supports Rapid and High Quality Parallel Development with CA Service Virtualization CA Service Virtualization has become the nucleus of parallel development, and being able to design a process that builds in quality has improved the efficiency of test processes. “CA Service Virtualization allows us to reduce the total amount of time spent on development.” Sharing knowledge and expertise made easier ------------------------ Creates high-quality systems in a shorter timeframe Business Qualica Inc. handles information systems for Komatsu. As part of the IT Holdings Group, it provides business systems, individually developed software packages and cloud services, primarily for the manufacturing, logistics and service industries. Challenge Qualica needed to create a new development infrastructure that guarantees quality and speed. Solution CA Service Virtualization virtualizes services to simulate the partner system’s behavior and data; it automates testing, which helps automate user interfaces on mobile devices; thirdly, the load test function not only places a load on the application being tested, but also simulates loading on the applications that it is linked to, and provides a virtual response.
- 20. 20 © 2016 CA. ALL RIGHTS RESERVED. CA Service Virtualization: Real Value Forrester Total Economic Impact (TEI) Study for CA Service Virtualization Based on three-year, risk-adjusted financial model for the composite organization “So without Service Virtualization, as highly integrated as our systems are, there is no way we could have speeded things up and moved to an Agile methodology.” — Senior manager, software engineering “In order to do performance testing required, we were able to virtualize a lot of back-end systems and hardware simulation, avoiding $1.8 million.” — AVP, quality assurance The Total Economic Impact™ Of CA Service Virtualization, a commissioned November 2015 study conducted by Forrester Consulting on behalf of CA Technologies Service Virtualization can help improve software quality, speed up time-to-market, and reduce costs. ROI 273% NPV $4.30M PAYBACK 3.2 mos. IMPROVED RELIABILITY Identifying 150 defects earlier in QA per release IMPROVED PRODUCTIVITY 640 hours of savings in Dev/QA processes per release
- 21. 21 © 2016 CA. ALL RIGHTS RESERVED. CA Continuous Delivery Portfolio Development OperationsTestPlanning Deploy CA Agile Requirements Designer CA Test Data Manager CA Continuous Application Insight CA Application Test CA Release Automation Agile Ops Agile Central CA Service Virtualization CA Release Automation Standard + Continuous Delivery Edition APIM APM
- 22. 22 © 2016 CA. ALL RIGHTS RESERVED. Additional Resources Next Generation Performance Testing Executive Brief – http://bit.ly/2alTsrr Forrester SV TEI Study – http://bit.ly/CASV-TEI Service Virtualization Today – www.servicevirtualization.com Gartner Market Guide for API Testing and Virtualization – http://bit.ly/Gartner_SV-Guide Key Capabilities of CA Service Virtualization Whitepaper - http://bit.ly/CASV_WP Real Customers. Real Results – Service Virtualization customer success stories www.ca.com/CDResults IT Central Station – Real User Reviews and Comparison Guide https://www.itcentralstation.com/products/ca-service-virtualization CA Service Virtualization Home Page - http://www.ca.com/us/products/ca- service-virtualization.html?intcmp=headernav
- 23. 23 © 2016 CA. ALL RIGHTS RESERVED. © 2016 CA. All rights reserved. Thank you!
Editor's Notes
- In the Application Economy, every business is in the software business. Which means your success depends on delivering applications that delight your users...every time. CA has helped thousands of companies adapt and thrive in this new reality. We can do the same for you.
- One of the key reasons successful companies are able to consistently deliver quality applications is through their deployment of better testing tools. The all-important Testing function lies in the middle of Dev and Ops. Testing is really the harbinger of application success. Done right, testing can identify application shortfalls early in the software development lifecycle. Modern testing tools can even predict the outcome of testing coverage and associated costs.
- APM Digest Article – http://www.apmdigest.com/15-top-factors-that-impact-application-performance http://www.apmdigest.com/performance-testing-and-apm-why-you-need-both Insufficient testing of the application in the actual production environment and under varying conditions impacts performance. Tied to that is for developers and testers to have a clear understanding of the non-functional performance criteria. Michael Azoff, Principal Analyst, Ovum(link is external) Agile release cycles — the reality is that less than 5% of developers performance test their code before it is pushed to production. The “make it work” over “make it perform” mantra is one of the biggest factors that impacts application performance today. Most organizations don't have the time, resource or budget to replicate production environments in test for every agile release, this is why a growing trend of customers have started to test in production out of working hours. When you consider that the codebase of an application changes several times per month, you can begin to understand why performance anti-patterns and bottlenecks make their way into production. Stephen Burton Tech Evangelist, AppDynamics
- Why do apps perform bad in production? What we’ve come to realize is that not all components are even being tested. Fringe cases are also occurring more frequently with the interdependencies of third party APIs and services. How do you test what you have not even thought of occurring? You don’t always have all the access to the systems and data you need to run a authentic performance testing. What happens when you need a significant amount of data? What is the time and effort needed to get the data, secure the data and then align to the test?
- How do we help customers overcome the challenges associated with performance testing? We do this by helping clients overcome the most prevalent challenges associated with running “life-like” performance tests. Optimizing tests for coverage, virtualizing unavailable services that look and act just like the real ones. Provisioning data needed to run the tests quickly and efficiently.
- Mapping out use cases and associated test cases ensures you test all the possibilities. Then translating them into a visual representation improves communication from DevTest. Fringe cases can also be discovered earlier in the process and associated test cases can be created. With CA Agile Requirements Designer, time-consuming manual effort is drastically reduced. Testers can automatically create the smallest number of test cases needed to achieve 100 percent functional coverage and link them to the right data and expected results. Existing test cases can also be imported and optimized and executed as automated tests, reducing the average test cycle by 30 percent. The optimized test cases can be pushed out and allocated to testers in existing tools like CA Agile Central or HP ALM. They can also be executed as automated tests, eliminating manual scripting and making the exhaustive testing of applications possible on time and within budget. As requirements change, CA Agile Requirements Designer will automatically find and repair broken tests, automatically creating any new tests needed to maintain 100 percent coverage. This eliminates the bottlenecks created by manual test maintenance, allowing you to deliver valuable software which reflects changing user needs earlier and at less cost.
- Leverage Service Virtualization and Drive Testing at the API Layer. Once you know you have all the possibilities of test cases you need to run. Then you need to figure out how to get access to all the systems you need to run those tests. This is where the concept of service virtualization comes in. Service virtualization gives you the ability to replicate the systems you don’t have access to or to replicate systems that are not performing up to the standard you need them to run a realistic load and performance test. And test fringe cases (ie traffic spike on black Friday, or a slow responding 3rd party API you don’t control). Service virtualization listens to the conversation two systems are having. This conversation is translated into a recording that is used to replace the live service. Take for example testing a mainframe service. Access maybe limited to a couple of hours a week with service virtualization you can make a copy of that service and then use it whenever you want. This also comes into play when working with 3rd parties. Many times the testing environment provided by the 3rd party is not up to snuff. For example we had a client that was experiences significant delays in regression testing because the response times from the 3rd party by using service virtualization they were able to cut their testing times from 6 hours to 45 minutes.
- Decompose Performance Expectations “Into the Lifecycle” – With service virtualization you can really understand how long is everything taking.
- This gives us the ability to replace a piece of your infrastructure with a virtual service and make it respond exactly like the real service.
- This is when a lot of people say well we can use mocks and stubs to do this type of thing. Mocks and stubs are limited to the same types of pre-canned requests and response every time you run the test. How will a mock and stub respond when you start replicating production level loads and pushing thru thousands of transactions per second. Mocks and stubs can’t scale up to the appropriate load levels needed to replicate the production environment. With service virtualization developers don’t need to create mocks and stubs and get access to the environments they need to test very early in the development time which further enhances productivity.
- Some of the testing teams we talk to only refresh test data every 6 months and when they do it is a 3+ week process. This is where CAs Test Data Manager solution comes into play. It enables you to provision the test data you need to run production level tests.
- Virtual data is generated on the basis of a message definition and injected into a deployed virtual service. Many providers have a solution for sub-setting and masking and CA does that also. What’s different about how we approach test data is by generating synthetic data. Synthetic data is purpose built data that is based on the structure of your database. It eliminated the need to mask data and security concerns. And enables you to create data for systems that don’t exist by using the structure of the data required for the test and then synthetically generating it.
- + Linking Performance Monitoring and Testing – What good is load and performance testing if you don’t know how it will actually impact the end user. This is where CA Application Performance Monitoring comes into play. During the test the system can be monitored and performance can be measured. This enables you to proactively improve end user performance before the app goes into production.
- CA Technologies can provide organizations with a comprehensive solution designed to address the issues around isolation and intelligence. CA Service Virtualization addresses the problem of providing intelligent stateful stubs through its ability to observe then simulate the behavior of dependent systems. The use of service virtualization can reduce or eliminate a large amount of manual effort and produces stubs which can be used in many lifecycle phases, including performance testing.
- Leverage virtual services and test assets to automate testing at earlier stages of the SDLC— where defects can be fixed more quickly and at lower cost—and improve the quality and stability of your production applications.
- Autotrader Avoids $300,000 in Testing Costs with CA Service Virtualization
- Qualica Supports Rapid and High Quality Parallel Development with CA Service Virtualization
- CA commissioned Forrester Consulting to conduct a Total Economic Impact™ (TEI) study and examine the potential return on investment (ROI) enterprises may realize by deploying CA Service Virtualization. Forrester interviewed five organizations who had implemented this solution in their enterprise. Taken as a whole, this composite company reported: Early defect detection in QA, leading to $1 million in average annual savings Development and testing efficiencies equating to $306,000 per year in savings Testing environment infrastructure cost avoidance of $2,747,971
- CA Technologies is ready to help your teams deliver high-quality applications faster than ever before. Using our integrated solutions and open framework, you can easily track, automate and manage all application releases—from planning and requirements to development, testing and deployment. CA Agile Requirements Designer is an end-to-end requirements gathering, test automation and test case design tool which drastically reduces manual testing effort and enables organizations to deliver quality software to market earlier and at less cost. CA Test Data Manager uniquely combines elements of data subsetting, masking and synthetic, on-demand data generation to enable testing teams to meet the agile needs of the organization. CA Application Test enables teams to start API, system and response testing prior to the development of UIs. CA Service Virtualization simulates systems across the SDLC, allowing developers, testers, integration, and performance teams to work in parallel for faster delivery and higher application quality and reliability. CA Release Automation is a proven, enterprise-ready solution that can help build an agile, resilient and scalable continuous delivery pipeline—from development to test to production.
- Thank you for taking the time to meet with us today. With that, I’d like to….. (take questions, turn the time back over to, etc)