An alternate approach towards building Test Automations for the Agile development environments. Traditional Test Automation approach isn't a great fit in fast pace dynamic ecosystem, hence there exists a dire need for an alternate approach. This is very well suited to an e-commerce platform, and can be easily extended to similar environments.

1. Bringing Agility in Test Automations
Hemant Kumar Jaggi
Senior Manager, MakeMyTrip.com

2. LET‟S
BE
AGILE !
Heard this before?

4. Thoughts on test automations the “Agile” way
To keep up the pace with agile
application development
Traditional Test Automation
approach isn‟t working
Best practices must be
incorporated everywhere
Ideally suited to an Ecommerce
(or similar) environment

5. An Ecommerce looks like

6. Only thing that doesn’t change is “change”

7. Building Test Automations the “Agile” way

8. WHY?

9. BECAUSE
TRADITIONAL
AUTOMATION APPROACH
JUST DIDN‟T
WORK
FOR US

10. WHY? Challenges we face @ MakeMyTrip
Fast Pace
Releases in a week?
Monthly unique visitors?
Monthly page views?
High Maintenance Cost & Test
cases not always in “shape”

11. Challenges (contd.)
Complex Business Rules
Hotel promotion combinations?
Cheapest Flights search?
Multiple GDS connections
Hotel Search engine?
~10,500 Domestic Hotels
~250,000 hotels across world
It‟s impossible to reverse-engineer!

12. Challenges (contd.)
Execution issues:
Automation execution may take 2-3
days to complete
Golden data set takes ~7 days to build
Automation teams don‟t know domain
Finally ..

13. Life of a QA Engineer is like

14. WHAT?

15. WHAT it should be?
Faster time to develop Automation
LOW maintenance cost
Agnostic to business logic
“QA team” friendly
Lesser complexity
Faster execution time
“It just works ! ”

16. HOW?

18. HOW? In lines with Agile Manifesto..
Break into smaller stories
Prioritize them basis
Functional Importance
Frequency of executions
Dependencies
Work “with” QA team
Develop a story and deliver

19. Create Buckets
Sanity tests
End to end user flows
Regression tests*
Miscellaneous flows

20. Sanity Test
Breadth wise test coverage.
MATS – Minimal Acceptance Test
Suite
Release checklist
Determines whether build is
stable.

21. User Flows
Covers end to end user flows
Detailed breadth wise coverage
E.g. different combinations of
booking flows.

22. But how to simplify regression tests?
Break them further ..
Into smaller flows, e.g. page wise
test suites
Comparative Testing
Interactive Testing

23. Comparative Testing

24. Comparative Testing
Works on two builds:
“QA” build (i.e. build under test)
“Production” build
Deploy them in same
environment.
Read all „to-be-tested‟ values from
front end

25. Comparative Testing
Execute automation script on
both builds
Dump them into XML (or excel)
Excel template has formulae to
compare values from two builds
Or use an XML comparator (in
case of XML output format)

26. Comparative Testing
IF (Actual == Expected)
THEN PASS
ELSE FAIL
* Actual Value = value(QA build)
* Expected Value = value(live build)
Take XML(or excel) as input to
reporting framework, generate
HTML reports

27. Let’s discuss
Pros & Cons
of
Comparative Testing

28. Advantages of Comparative Testing
Faster turn around to develop
Automation script does just what it is
supposed to do
Automation becomes a “dumb”
execution
Very LOW maintenance cost
Keep it simple

29. Advantages of Comparative Testing
Agnostic to business layers
Changes in business rules don‟t
matter
High level of accuracy
Makes it “thorough”
Enhances confidence in QA team

30. Advantages of Comparative Testing
Overall progress is faster
More dependency on Test Scenarios
than Test Cases
Less/No domain knowledge required
for Automation team
“Win-Win” for Automation and QA
team

31. Disadvantages of Comparative Testing
Any “live” issues won‟t be caught
Execution requires
Either double the time (if sequential)
Or double the resources (if parallel)

32. Interactive Testing

33. Interactive Testing
Complements comparative testing
Features that require user
interaction
E.g. Filters, Sorting etc.
Comparative + Interactive
equals “Regression”

34. Automation Coverage Tracking
Assign weights to each bucket,
say
Automation Score =
sum([(Tn/An)*100]*[Wn/(W1+W2+W3+W4+W5)]
Tn=Total Automated TC’s for a bucket
An=Total Automation feasible TC’s for a bucket
Wn=Weighted score of a bucket

35. Automation Coverage Tracking (Sample)

36. Computing efficiency gains (Sample)

37. Take Away
The “Agile” way of building Test
Automations
We resolved “challenges” of
traditional automation approach
Simplifying regression
Regression=Comparative + Interactive
Its working @ MakeMyTrip

38. Thank You
Hemant Jaggi
Hemant.jaggi@makemytrip.com
Hemant.jaggi@gmail.com