Eradicate Flaky Tests - AppiumConf 2021
•
0 recomendaciones•244 vistas
Slides from my talk on how to Eradicate Flaky Tests from AppiumConf 2021 https://confengine.com/conferences/appium-conf-2021/proposal/15581/eradicate-flaky-tests
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Eradicate Flaky Tests - AppiumConf 2021
Similar a Eradicate Flaky Tests - AppiumConf 2021 (20)
Más de Anand Bagmar
Más de Anand Bagmar (14)
Último
Último (20)
Eradicate Flaky Tests - AppiumConf 2021
- 1. Eradicate Flaky Tests @BagmarAnand Anand Bagmar Software Quality Evangelist Essence Of Testing
- 4. @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 5. Approach for API Workflow & E2E Test Execution @BagmarAnand
- 6. API workflow Tests E2E Functional UI Tests @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 7. • Long running scenarios • Slow feedback Challenges of E2E Test Automation @BagmarAnand
- 8. • Limitations of automatable scenarios • Error case / Edge case handling from dependent systems • Triggers from external systems Challenges of E2E Test Automation @BagmarAnand
- 9. • Pain of Cross Browser execution • Slow, special handling for browsers, infrastructure cost / maintenance Challenges of E2E Test Automation @BagmarAnand
- 10. • Flaky Tests Challenges of E2E Test Automation @BagmarAnand
- 11. Why are Tests Flaky? @BagmarAnand
- 12. • Brittle / Flaky tests • UI changes • Downstream dependencies • Data dependencies • Network (speed) fluctuations Challenges of E2E Test Automation @BagmarAnand
- 13. Solutions / Way forward @BagmarAnand
- 14. • Reduce number of UI tests • Remove external dependencies via Intelligent Virtualization • Use Visual Assertions instead of Functional Assertions How to Reduce Flakiness in Tests? @BagmarAnand
- 15. #1 Reduce the number of UI Tests @BagmarAnand
- 17. #2 Intelligent Virtualization to Eradicate Dependencies @BagmarAnand
- 19. @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 20. Approach for API Workflow & E2E Test Automation @BagmarAnand
- 21. API workflow Tests E2E Functional UI Tests @BagmarAnand Internet Internal B2C App Product DB DBMS B2B App Caching System Elastic Search Authentication Gateway Service 3 Service 4 Service 7 Service 5 Service 6 Admin User DB Outgoing Notification Onboarding Event Consumer Order DB Reporting System Admin Portal Support Msg Q Direct API Service 2 Notification Service 1 Event Processor Service 8 Payment Gateway Credit System Master Product Warehouse Royalty system Offers …. …. ….
- 23. @BagmarAnand Step #1: • Test calls Service 2 Step #2: • Service 2 calls the external service, which may (unexpectedly) take long time to respond Step #3: • Service 2 gets an error back from external service • Returns an error to the test Step #4: • Test fails because of external service • We may get incorrect error as well! Test Scenario
- 24. Challenge! Scenarios having external dependencies are flaky @BagmarAnand
- 25. Solution @BagmarAnand Stub it out using Specmatic!
- 27. • Inculcates Contract Driven Development • Supports OpenAPI Spec • Executable Contract spec • Defines http method, path, header format, payload format, query parameter format, form field payload format • Supports JSON, XML data formats • Supports Kafka • Dynamically generates and executes contract tests • Apt for service virtualization • Compares contracts to ascertain backward compatibility • Manages contracts through versioning • Integrates with CI/CD Specmatic – Contract Driven Development @BagmarAnand • Decouples the ”producers” and “consumers”, with confidence!
- 28. Use Specmatic as part of Test execution @BagmarAnand Step #3: • Test calls Service 2 Step #4: • Service 2 calls the stubbed external service Step #2: • Test sets expectations on Specmatic for specific stubbed endpoint • Http POST call on Specmatic server with exact request & response as json payload Step #5: • If request matches set expectation, corresponding set response is returned in response • Service 2 processes response and responds to test appropriately Step #6: • Test asserts on the received response • Test passes Step #1. Specmatic as a Stub Server in your Test Environment • Specmatic Server is started in the Stub mode in the Environment • External services are stubbed out • Internal services point to Specmatic stubbed- endpoints instead of real external endpoints
- 29. Use Specmatic as part of Test execution @BagmarAnand Step #3: • Test calls Service 2 Step #4: • Service 2 calls the stubbed external service Step #2: • Test sets expectations on Specmatic for specific stubbed endpoint • Http POST call on Specmatic server with exact request & response as json payload Step #5: • If request matches set expectation, corresponding set response is returned in response • Service 2 processes response and responds to test appropriately Step #6: • Test asserts on the received response • Test passes Step #1. Specmatic as a Stub Server • Specmatic Server is started in the Stub mode in the Environment • External services are stubbed out • Internal services point to Specmatic stubbed- endpoints instead of real external endpoints Allows to simulate positive, negative, edge-case responses – allowing you to test your implementation logic
- 30. #3 Visual Assertions instead of Functional Assertions @BagmarAnand
- 32. Bugs still escape @BagmarAnand Traditional automated testing frameworks are not built for modern apps
- 37. @BagmarAnand
- 38. Bugs escape because our approach to testing is incorrect @BagmarAnand
- 39. @BagmarAnand Mundane Error prone Tedious Running against Time Approach to Testing is Incorrect
- 42. But, we are testing real software products @BagmarAnand
- 43. 43 @BagmarAnand Text, Images, Responsive-pages, Form-factors, User-experience Product has Context!
- 47. @BagmarAnand Computer Vision that replicates the human eye and brain
- 49. @BagmarAnand Visual AI detects bugs, not browser render diffs
- 50. @BagmarAnand Pixel comparisons waste time with false positives
- 51. Visual AI should work across all Platforms PDF, Web, Native Mobile @BagmarAnand
- 52. Perfection Across All Screens, Browsers and Viewports Tests run local - Applitools SDKs available for all major testing frameworks DOM & CSS sent to Applitools Ultrafast Grid Applitools Eyes Visual AI analysis Ultrafast Grid renders DOM snapshots in parallel Login to Eyes to view test results 53 . Firewall
- 54. @BagmarAnand Application under test Login Get list of users Edit User Test Scenario Login 1 Get list of users 2 Internet Return dynamic list of users 3 4 Edit User 5 Verify User Details 6 Return dynamic list of users Test Scenario Example Scenario
- 55. Website with rest services in backend hosted on local machine Login Get list of users Edit User Test Scenario Functional & Visual Assertions using Applitools Login 1 Get list of users 3 Verify User Details 7 Edit User 6 4 5 Test Environment Return dynamic list of users Stubbed using Specmatic Static Random Dynamic Expectations @BagmarAnand 2. Test sets “dynamic expectation” in context of the scenario Example Scenario using Specmatic I n t e r n e t
- 56. @BagmarAnand Example Scenario using Specmatic Website with rest services in backend hosted on local machine Login Get list of users Edit User Test Scenario Functional & Visual Assertions using Applitools Login 1 Get list of users 3 Verify User Details 7 Edit User 6 4 5 Test Environment Return dynamic list of users Stubbed using Specmatic Static Random Dynamic Expectations 2. Test sets “dynamic expectation” in context of the scenario I n t e r n e t Return dynamic list of users Test Scenario
- 59. • Reduce number of UI tests – Test Pyramid • Use Visual Assertions instead of Functional Assertions • Remove external dependencies via Intelligent Virtualization Make your Automation Intelligent! @BagmarAnand
- 60. • Move granular tests to lower layers of pyramid • Get faster feedback • Reduce flakiness / brittleness due to UI changes Reduce UI tests @BagmarAnand
- 61. • A single assertion for complete functional coverage • Bonus: validates the UI • Does not break when the UI changes • No coding skills required to maintain baselines • Validate UX at scale for all supported browsers Use Visual Assertions @BagmarAnand
- 62. Run once, validate everywhere @BagmarAnand
- 63. Ultrafast Grid @BagmarAnand Corporate Firewall Ultrafast Grid new page resources checkpoint images visual assertions Application under test Test / Build machine Lab VM / containers
- 64. • Covers all environments at the speed of running a single local test • Much more robust and stable • Full functional and visual coverage • No security vulnerabilities • Reduce test data preparation and consumption • Does not load the corporate network • Costs much less Run once, validate everywhere @BagmarAnand
- 65. • Test environment independent of external dependencies • Tests set dynamic expectations based on context of the test execution • Stubbed service should be able to give static responses / random responses Virtualize Dependencies at Runtime @BagmarAnand
- 66. • Visual AI using Applitools • Getting Started with Visual Testing - https://testautomationu.applitools.com/automated- visual-testing-a-fast-path-to-test-automation-success/ • Modern Functional Test Automation Through Visual AI - https://testautomationu.applitools.com/modern-functional-testing/ • Intelligent Virtualization using Specmatic • https://specmatic.in/ • https://github.com/znsio/specmatic • Test Automation University - https://testautomationu.applitools.com/ • Test Pyramid - https://martinfowler.com/articles/practical-test-pyramid.html Resources @BagmarAnand