With most organizations now using agile software development methodologies, the software development focus has changed to deliver faster releases—and this affects the way we test within the sprint. We largely develop applications using cloud and mobile technologies with short release cycles. Our challenges include frequent changes in requirements, the addition of incremental features to the product, and release at any point of time. Ganesh Iyer has found that continuous testing can seamlessly address most of these challenges. Continuous testing is the ability to run tests continuously in a particular environment, irrespective of product upgrades and dependent third-party systems. Ganesh highlights some continuous test approaches in practice including 24/7 reliability testing and continuous integration. Key takeaways include understanding the importance of endurance testing, practical considerations when we perform such durability testing, framework design elements for running tests continuously, and finally—what to look for in the results.

1.
T22
Cloud
Testing
5/11/17
15:00
Continuous
Testing
of
Cloud
Applications
Presented
by:
Ganesh
Iyer
Progress
Software
Development
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Ganesh
Iyer
Ganesh
Iyer
-­‐
QA
architect,
technology
evangelist,
researcher,
professor,
blogger,
photographer,
and
dancer
-­‐
is
a
QA
architect
with
Progress
Software
Development.
He
is
involved
in
several
key
QA
initiatives
including
cloud
test
architecture,
24/7
test
automation
framework,
web
automation
framework
using
Selenium
and
TypeScript,
and
a
high-­‐availability
test
framework
with
tests
in
long
running
mode.
A
visiting
faculty
member
at
IIIT-­‐Hyderabad,
Ganesh
has
authored
book
chapters
and
journal
articles,
and
presented
talks
and
workshops
on
various
technologies
in
international
settings.

3. Continuous Testing
Dr Ganesh Neelakanta Iyer
QA Architect
GANESHNIYER
gaiyer@progress.com
http://ganeshniyer.com

4. About Me
§ 12+ years of Research and Development experience
§ Masters and Ph.D. from National University of Singapore
§ Several talks/workshops in India, USA, Europe and Australia
§ Personal Interests: Kathakali, Teaching, Traveling
§ Website: http://ganeshniyer.com
GANESHNIYER
§ Current:
• Technical Support Manager (Asia Pacific) at Progress Software, India
• Visiting Faculty at IIIT-H (A premier academic institution in India)
• Technology/Product Evangelist
§ Past:
• Software Developer
• QA Architect

5. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.3
Quick Poll

6. Are you using Version Control?
Safe?

7. Can you release new version of
your software in one day?
Quick

8. Can you release new, well-
tested version of your software
in one day?
Quick & Safe?

9. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.7
Agenda
§ Software Development today
§ Continuous Integration and Continuous Testing
§ 24x7 Testing

10. What are top changes in
Software development
that were disruptive for
existing test
methodologies?

11. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.9

12. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.10
Agile
Credit: Blazemeter; Continuous Testing in Practice

13. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.11

14. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.12
Test Challenges with Agile Methodology
Frequently changing requirements
• Thanks to Agile à Changes can happen at any point of time during release cycle
• Requirements and Test Cases developed in parallel or test first strategy
Frequent releases
• Progress Rollbase released new features every 9 days in 2016
• Testing needs to happen Early and Often, Everyone needs to Test
Understanding and testing real world behavior
• How customers see the product behavior after a release?
• No SRS-level waterfall documents to drive testing plan
Multiple environments & differences for test and production systems
• 3rd party systems integrated to the product
• Raises a requirement for GoLive Testing à But in minimal time

15. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.13
Software Test Challenges With Cloud
§ Web-driven SaaS applications
§ Everything in a few mouse clicks
§ Frequent releases / Frequent changes
§ No SRS documents
§ Publically shared resources
§ Multi-tenant penetration testing
Paradigm Shift
§ High Availability and Resiliency are key
§ Seamless upgrades
Everything is Agile
Availability Everything is shared

16. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.14
Continuous Integration & Automation are Key in Agile
Automation
• Nightly / Weekly
Build
Integration
• Combination of
unit tests and
regression tests
Dev-
environment
• Unit-tested

17. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.15
Continuous Integration and
Continuous Testing

18. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.16
Dev/Test à Expectations continue to increase
Better – Cheaper - Faster
Past
• Balance between speed, cost
and risk
Today
• Demand for releasing high
quality software faster
Credit: IBM article on Continuous Testing

19. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.17
Applications are becoming more complex to test
§ Setting up complex environments
§ Delays caused by lack of system
availability
§ Third party services can be costly
§ Testers spend more time setting up
tests than testing
Testing teams
spend on setting
up test
environments,
instead of testing
30-50%
average
time
Credit: IBM article on Continuous Testing

20. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.18
Continuous Integration and continuous testing
Accelerate software
delivery – for faster
time to value
Balance speed, cost,
quality and risk – for
increased capacity to
innovate
Reduce time to
customer feedback –
for improved
customer experience
PC Credit: https://https://goo.gl/A2AaQw

21. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.19
Shift left, test early, test often

22. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.20
The Testing
Process – From
Development to
Operations
Credit: Blazemeter; Continuous Testing in Practice

23. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.21
Continuous Testing in Practice – Requirements Checklist
Tools, Scripting and
Overall Test Creation
• The provision of libraries and APIs to create “homegrown” tests
• The ability to create browser-based tests
• Ability for load and scalability testing
Provisioning and
Resources
• The option to run the same sets of tests with different provisioning
• The ability to easily adopt a new provisioning scheme when a test
configuration is executed in a different environment (e.g. Dev, CI,
Pre-Prod, Post-Prod)
• The ability to start testing as part of the CI process
Automation
• The ability to run any test configuration, a test, or a set of tests
using simple API calls
• The option to run as many tests as required in parallel
On-Demand Testing
and Re-Running Failed
Use Cases
• The ability to run any type of test on-demand. This is critical for
test development, debugging, troubleshooting and identifying the
root cause of a failure
• The option to automate test executions and run the same test on-
demand
Credit: Blazemeter; Continuous Testing in Practice

24. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.22
Continuous Testing in Practice – Requirements Checklist
Modules, Builds,
Release Candidates,
Releases and
Production
• The ability to combine different configuration fragments into one test
configuration in order to comprehensively test build, release, and
production snapshots.
Version Control Friendly
Incremental Testing
• The ability to support version control, incremental testing and
associate test configurations, sets of tests, and tests with versions (e.g
code, build, RC, releases).
Automatic ‘Failure’
Alerts
• Automatic indications of failures, with an alerting scheme per
developer, module and project
• The ability to gather all test artifacts and immediately send to relevant
people
• The option to run the failed tests again to identify the root cause
Reporting
• A seamless integration with existing reporting solutions (e.g. Jenkins
Performance Trend)
• The option to group tests by builds
• The ability to give jobs a ‘pass’ or ‘fail’ status and group accordingly
• Pass/fail trend reports
Credit: Blazemeter; Continuous Testing in Practice

25. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.23
24x7 Testing; A Case study of
what we do at Progress

26. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.24
24 x 7 Testing
Understanding and testing real world behavior
What is 24x7?
• Running a set of test
cases continuously 24 x
7
• Simulates real time
customer behavior
How 24x7?
• A few parallel threads
accessing the product
• Tests consisting of
regression (or it’s subset)
tests
Why 24 x 7?
• Many things can’t be
caught by regular
functional testing
• Error handling and retries
• Product can be made
more resilient and robust

27. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.25
24 x 7 Testing
VMs / Build Integration Machines
ANT ScriptTests
Reports using
Kendo UI controls
Decides
parallelism
HTTP POST reqBatch
Script
Decides
frequency
Logs
SUT
• Progress
products such
as Rollbase
• Uses any test
environments
• Existing test
cases can be
leveraged
Batch Script
• Decides the
frequency
• E.g. run on an
hourly basis
• Run in an
infinite loop
24x7
ANT Script
• We configure
#Threads,
#Tests to run in
parallel
• Tests can run
on VMs / Build
integration
machines
Reports
• Shown using
Keno UI
controls /
widgets
• Driver scripts
sends results
to the portal
• Machine stores
the logs

28. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.26
Case Study - Example
§ Cloud platform that let you build SaaS-
based business apps in days/hours
§ Responsive UI with built-in themes
§ Typical apps customers build
• CRM
• Employee Management
• Bug Tracking System
What could be some typical tests for this platform?

29. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.27
Case study - Identifying tests; Example
§ 100 parallel treads
• 50 of them access “Bug Tracking System” application
• 50 of them access “CRM Application”
§ Use cases
1. Create a bug and assign it to a user which will run an email trigger and related record trigger
– 10 users
2. Update the workflow status of some existing bug which will run a trigger – 20 users
3. Filter operation on bug object – 20 users
4. Create a lead from CRM portal and attach the lead to an account – 20 users
5. Look for all leads whose workflow status is “Open - Not Contacted” – 10 users
6. Create a new opportunity, assign it to an owner and attach it to an account – 10 users
7. Change the workflow status of an existing opportunity to “Closed-won” – 10 users
Tests should not
be self-cleaning
Why?

30. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.28
Layout of the infrastructure
BATCH script
• Checkout latest test scripts
• Compile and run them
ANT script
• Properties file containing the details of
test agents
• Invoke scripts in the agent machine
Central
Machine
Test
Agent 1
Test
Agent 2
…
Test
Agent n

31. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.29
Central Server

32. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.30
Central Server – Build.properties

33. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.31
Central Server – <Product name>.properties
testtype=Telerik
testlist="Psexec.exe 172.29.36.205 -u Administrator -p Progress!2009 -i 2 -w c:Telerik
c:Teleriktelerik.bat";"Psexec.exe 172.29.36.207 -u Administrator -p Progress!2009 -i 2
-w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.208 -u Administrator -p
Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.209 -u
Administrator -p Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe
172.29.36.215 -u Administrator -p Progress!2009 -i 2 -w c:Telerik c:Telerik
telerik.bat";"Psexec.exe 172.29.36.90 -u Administrator -p Progress!2009 -i 2 -w c:
Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.227 -u Administrator -p Progress!
2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.231 -u
Administrator -p Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe
172.29.36.229 -u Administrator -p Progress!2009 -i 2 -w c:Telerik c:Telerik
telerik.bat";"Psexec.exe 172.29.36.233 -u Administrator -p Progress!2009 -i 2 -w c:
Telerik c:Teleriktelerik.bat"

34. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.32
24 x 7 Testing – Sample report

35. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.33
Key Advantages of 24x7 testing
§ Catch issues which are difficult to catch using normal functional
testing
§ Make products resilient and robust
§ Catch unexpected integration issues
§ Test real time customer behavior

36. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.34
Catch issues which are difficult to catch using normal functional testing
https://cdn.tutsplus.com/wp/uploads/2013/12/session-diagram.jpg
http://biobeasts.artix.com/content/images/2015/07/memory-leak.jpg

37. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.35
Make products resilient and robust

38. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.36
Catch unexpected integration issues

39. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.37
Test real time customer behavior
http://blog.perfectomobile.com/wp-content/uploads/2015/10/WINDTUNNELGRAPHIC.jpg

40. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.38
Advantages of continuous Testing
Helping to make software development faster
• Continuous testing can make it possible to making software available sooner without
compromising quality
Reduced Risk
• Every time a change is made to code or new code is written, there is the potential for error to
occur. Identifying the problem is much easier when continuous testing is performed and the
issue is caught immediately so problems can be corrected.
Making consistency simpler
• Automation of testing makes it easier to achieve consistency by providing a carefully configured
environment and maintaining the same correct configuration for all relevant tests
Faster releases
• The process of regularly using continuous testing means that cleaner code can be released to
end users in a timely manner
https://www.etesting.com/2014/02/25/benefits-of-continuous-testing/

41. Are you ready to implement
Continuous Testing in your
organization?

42. gaiyer@progress.com
http://ganeshniyer.com
GANESHNIYER