2. Article submission guidelines –
● Subject of article can be based on any area of Software Testing. If you want to publish your article on
theme based subject please read our announcement of monthly theme published in our site. Article
can be submitted without any theme based subject.
● There is no minimum and maximum length of article. If you feel the article is lengthy, please divide
the article into logically separated parts so that we can print them in a monthly series.
● Give a meaningful title to the article. If you want a sub-title as well , then add that in a different line.
● Add images/pictures if necessary. If you are using any image/picture which is not yours own work,
please include the source. Take care of copyrighted materials. Images need to sent separately with the
article.
● Send us the article in MS word (doc/docx) format only. Pdf files are not accepted.
● Write a short write up on the author(s). Usually 7/8 liners in 3rd person descriptive language.
● Include photograph of author(s). Preferred in high resolution .jpeg/.png format. Ideal size would be
50mmX 50mm.
● You can send your article any time. Since we publish every month, your article can be included in any
month. There is no such thing called as a dead line.
● Send in your article to editor@testingcircus.com with a subject line “Article for Testing Circus –
Author Name – Title of the article”
● If you think you can write a column in Testing Circus for at least 6 months, please submit 3 articles in
advance. We are open to any idea that may improve the user experience of Testing Circus.
● We will publish the articles in our website a week after the pdf magazine is published.
Article
Submission
GuidelinesDo you have something to share with the
testing world? We can make your voice
heard to testers.
http://www.testingcircus.com/article-submission-guidelines
www.TestingCircus.com March 2014 - 02 -
4. Write to us at editor@testingcircus.com
www.TestingCircus.com March 2014 - 04 -
Testing Circus is three and half years old. This magazine has been running on
volunteer effort, assisted by few passionate testers from time to time. It has been an
amazing journey so far. But, like all other good things in life, it is coming to an end.
We are announcing discontinuation of Testing Circus after this edition.
It is a sad moment for all of us. Running a regular magazine purely based on
volunteer effort is hard task, specially when all volunteers including me have full
time jobs to look after. We were always supported by many testers by contributing
some awesome articles but let me also admit that it is always a challenge to gather
good articles when you don’t pay the writers and busy testers do not reply to your
emails and direct messages. There was too much effort for our volunteers.
Testing Circus site will remain as it is. We will continue to publish our old articles
in the site. We will continue to promote good testing stuffs in Facebook/Twitter. But
our regular monthly magazine will stop.
I would like to thank all you of you who supported us in this wonderful journey. We
will cross the roads again.
Till then, stay hungry, stay foolish and test passionately.
- Ajoy Kumar Singha
@TestingCircus // @AjoySingha
From the Keyboard of Editor
5. SMITA MISHRA
Smita Mishra is the founder and chief consultant at QAzone
Infosystems, which is a pure-play software testing organization. She is
a first generation Entrepreneur and is a Test professional who has
spent over 12 years practicing testing and leading test efforts of
varying sizes, cutting across all key domains and technologies. In the
past, she has worked with multiple organizations, likes of - HCL
Technologies Ltd, Fidelity Investments, Nucleus Software Exports Ltd,
Churchill Insurance (Now RBS).
In her current role, she is involved in creating test teams, managing
testing for software companies, leading the overall test strategy for
them. She is also engaging constantly with different forums to assist
growth for women in her field and women in general too.
Organization: QAZone Infosystems Pvt Ltd
Current Role/Designation: Chief Test Consultant & CEO
Location: New Delhi, India
Interview with Testers
www.TestingCircus.com March 2014 - 05 -
1. Tell us about your journey to becoming a software
tester. How did it start and how this has been so far?
Was it planned or by accident?
My journey to become a good tester still continues.
However, it began in 2001, with my first job at
Nucleus Software Exports Limited, where I was
campus placed after my engineering. A batch of
about 20 engineers were picked from across the
nation and put together as the founding
(independent) test team. Honestly, I had no
introduction to “Testing” until then other than the
few chapters I read on SDLC, Verification and
validation in a book of software engineering by
“Roger Pressman”. It was planned by my employers
not me but wasn’t really an accident. Once in testing,
I quickly began to learn it and found it interesting.
I will agree that though I really enjoyed my work,
the initial years were nothing exciting. But over the
years, I realized that I enjoy testing more than
anything else. There have been tremendous learning
through the 13 yrs spent in the field. The more I
learn the more I see I have yet to learn.
Hence, I can easily say that the journey so far has
been terrific. 2014 seems to be extraordinarily good
year for me and I am loving every moment of it, as a
test professional.
2. When did you realize your passion was software
testing?
I would be wrong if I said that it was love at first
sight. It was not. But as they say – good music grows
on you even if you don’t like it much in the first go.
It was similar in my case. I realized I enjoyed testing
early in my work life. But it wasn’t until 2006 that I
began to love it. I attended the first conference in my
life – it was a QAI conference where I was presenting
a practice paper on estimations. Met many testers
there and I loved the feeling of belonging to a
community too. Then in 2007 I went to work in GE
Healthcare account under the most dynamic
technology leader and sharp as razor – Gazanfar
Hakeem and an excellent Test Manager Smita Sethi
(who also happens to be a very sharp professional). I
got exposed to very technical testing in DW-BI (ETL
Testing) and performance testing of these ETLs and
implementing lean methodology. I would term that
as the turning point in my career on how I began to
look at my testing career. I realized I was cut out to
do this.
3. Do you regret being associated with software
testing today? Given a chance would you move from
testing to any other field in IT?
* Interviewed by Jay Philips
6. www.TestingCircus.com March 2014 - 06 -
No, clearly no. Quietly no. Loudly no. Simply no.
I am proud to be a part of an ever evolving field and
that has a huge legacy of achievers. I can easily say I
never had second thoughts about my work. Also, I
would find it very difficult to work on a single
technology or domain all my life – and to look
conversely at it - which other field can give me so
much exposure in understanding “what really
matters” in IT field, while keeping me hands on with
all latest technology.
4. What does QAZone do? How are its services
different from the services offered by other similar
organizations?
QAZone is a pure play software testing organization
that offers Testing solutions and services, Test
consulting, Test training and test support services
(Test data management, test environment setup). We
offer Business as usual testing as well as specialized
testing. We have built a client base of over 20 clients
in last 3+ years.
Our business focus is our key differentiator. Our
testing solutions are designed to work for your
business. We look at technology as a platform to
make business happen. All our testing has a
component of domain focus which becomes a very
serious need of our clients when we are dealing with
regulated environments like Aerospace and
Healthcare.
Our next differentiator is that against the trend - we
believe in hiring locally, even if it means training
potential resources at our expense. For example –
anytime we need a resource in US, our first approach
is to find a resource locally. This has so far proven to
be a very fruitful approach.
5. Last year you organized a conference called
ThinkTest in India. Was that the first year of the
conference? What were some key sessions and
feedback?
We had planned for a conference ThinkTest 2013,
which would have been the first of our initiative in
this direction. We were very keen to bring James
Bach to this part of India (and we still want this to
happen). Unfortunately, due to unforeseen
circumstances at my personal end and not having
suitable alternatives, we couldn’t take this forward.
We will keep the test community updated of future
plans related to this.
6. In February 2014 you held the first meetup for Test
Practitioner’s Club. What made you want to create
this meetup?
It always feels easier to work with people whom you
have met and can relate to them as a face and as the
vibes you share with them. All leading test
conferences are still physical conferences not
webinars. That’s because learning happens in many
forms today and one of them is networking post
conferences. With the thought of building a network
of co-learners, we created a LinkedIn group – Test
Practitioner’s Club and planned for its meet-ups that
would help local testers to learn more from each
other and get introduced to global platforms too. If
we could – we would have such test meet- ups all
over the country. We are planning to do one meet
every month at the very least. We will keep posting
the updates from our meets at my blog, for all to
read.
7. I noticed that you are certified in ISTQB and QAI
(CSTE). What made you decide to go for both
certifications?
Yes, that’s true – I have indeed done both the
certifications. I went for ISTQB (in 2005) because
until then I had no certifications. And under our
training and development program, my organization
paid for this certification. However, later on I went
for QAI (CSTE), as it was not widely done so far due
to its cost and students failing in it and this made it
SMITA MISHRA
7. www.TestingCircus.com March 2014 - 07 -
appear more respected and challenging. I paid for
this one myself.
8. Do you recommend that all testers get both
certifications? Are there any other certifications you
would recommend to other testers?
I would let testers decide this for themselves. But I
can say it with reasonable confidence that these
certifications do not prepare testers to handle serious
testing problems. I enjoy being part of miagi-do and
discussing actual test issues with real testers there
and how to resolve them. I have heard very good
things about BBST and most of the leaders I follow
are associated with it. I would suggest all testers to
atleast go through the content and format and decide
for themselves.
If learning is the objective and certification is not
really the goal, I would suggest testers to go through
James Bach’s site for RST classes and RTI (online)
classes.
9. According to you, what is lacking in today’s
commercialized training industry, especially in
testing?
With all due respect to the 2 certifications I did, I
believe an ideal test certification would be one which
would have more practical problems to deal with, as
tests for their students and would involve real life
situations of testing than focusing on glossary of
testing and terminologies. Also, I would invest in
domain led testing certifications for my team. So far,
there are no such certifications available which
would certify testers for regulated environments like
those of aerospace and healthcare. I have heard of a
new certification that Cem Kaner has come up with. I
need to look into that one too, if it has domain based
testing certifications.
10. What qualities will you look for in a candidate
when you want to recruit someone for software
testing job?
I like to work with enthusiastic people who enjoy
learning new things and working on new areas. This
is true not just for software testers but front office
executive, accountant, HR and Admin folks - all of
them.
Many today do not feel the need to look at the past
experience and hire only for attitude. However,
when I am hiring testers particularly – I like to see
the kind of work they have done before and like to
hear their stories of projects on exactly what was that
they were testing and what were the key defects
found. What was their approach and the key
challenges faced and how did they bring up the
challenges to the notice of relevant stakeholders and
how did they do the contingency or mitigation as
applicable.
Formally speaking, we have 2 set of patterns – one
for freshers and other for lateral hiring.
For laterals particularly - We have a process that
helps us map the required technical skills with the
available resumes. So, anytime we have a need we
look into our database of testers who have applied
with us in past and from the matching resumes, we
shortlist based on availability / joining time and
expected compensation. Post this, we share certain
project links with these potential candidates and
expect them to submit us their Bug Reports in a
given time frame. Finally, after a candidate is found
technically suitable, we setup face to face
interactions to evaluate more HR aspects and general
communication.
For freshers, it’s really about what they have to offer
beyond their formal education. No firm expectations,
as long as they are able to convince us of them being
fast learners and having aspirational attitude, they
might.
SMITA MISHRA
8. www.TestingCircus.com March 2014 - 08 -
11. What will you suggest to people who want to join
IT industry as software testers?
I am very happy to be part of the software testing
community. And I can only suggest the new folks
that learn what is testing and then get into it. I find
people to be doing testing for all the wrong reasons
like – its easier that Development, it doesn’t need
coding, timelines are easier and hence the work
pressure, etc. None of this is true. Testing is
becoming as challenging as any other part is with the
ever evolving IT and
When I look back, I think the biggest gap that I see
during my initial work tenure was how closed and
we were in our approach to testing. working in silos
as a closed group and not being exposed to the
whole world full of knowledge and awesome
teachers and trainers willing to invest time in you.
12. Name few people you would like to thank,
people who helped you directly or indirectly in your
career as a software testing professional.
I would like to thank my family for always standing
by me – ALWAYS. But a special mention of my son
here for being the world’s most loving and caring
son and being very understanding when his mommy
needs to work.
At work front there are too many names and I would
like to thank each good and bad interaction, because
it has helped me grow in one way or other.
Everyone I know – Thanks for being part of my life.
13. One last question – Do you read Testing Circus
Magazine? If yes, what is your feedback to improve
this magazine?
Yes, I do read Testing Circus Magazine. I first heard
about it in 2010 I guess when my manager Akash
had been invited to be interviewed in it. But I never
followed it then. I began to read Testing Circus since
sometime in 2013 onwards.
I liked the original format of the magazine. But the
new format is truly amazing. I really enjoyed the
editions coming out in 2014. And I think they are
doing all the right things to get noticed and to put in
right content. My only words will be – continue the
good work.
And maybe you can cover small meetups and group
test events happening across the globe as a regular
column. This could help other meetups to learn what
more can be done and how to be effective.
Blog/Site: http://smitamishrablog.wordpress.com
Twitter ID: @smita_qazone
SMITA MISHRA
10. The simple thing is this - write your requirement as a
test. With defined inputs and outputs. Expected results
and expected (and unexpected) data.
If you do this, your life will become bright, shiny, and
you will live happily ever after. Or at least your soft-
ware will become much much, much more reliable.
And you’ll probably find out a lot of things about your
ideas before you’ve invested in building unnecessary
features and details.
You can do this with high level requirements, such as
business goals and overall objectives, as well as with
low level isolated features, and everything in between.
I think that the “everything in between” part is where
we (software industry people) lacks the most care and
insight about the importance of concrete, testable re-
quirements.
When it comes to high level requirements, we may have
business people who do follow up business cases &
objectives, i.e. test the results of the investment, at least
I have seen it done once or twice.
When it comes to very low level requirements, or mi-
cro-requirement as my friend @spindelmanne call
them, TDD do take care of it to some extent. Such as
“When renaming item x the list will keep the same sort
order”. It’s hard to separate micro-requirements from
real business requirements sometimes. “What is really
a valid input string here?” “How should we present the
date format” etc but good developers generally can
make some good micro-requirements decisions.
For the “everything in between” requirements, we have
a lot of work to do to make them testable. From what I
know it seems as there are mainly 3 ways of communi-
cating requirements today.
Either you’re “agile” and have a loosely defined prod-
uct backlog, filled with short user stories and then not
so much more information.
Or you have a heavy regulated requirements process,
with hundreds of pages of use cases or “shall”-require-
ments. Often with abstract statements such as (from
real example): “Purchase has generated a receipt”
Or the ad-hoc requirements: “Let’s send an email to the
developer telling what I need to have”. Example: “We
need to update the purchasing order receipt page. Right
now it doesn’t show the total. The total need to be there.
When can this be done?”.
As a requirements analysts / project manager I have
seen and practiced a way out of these three abstract,
ambiguous, non-informative ways of communicating
requirements. Much thanks to developers who serious-
ly cared about taking TDD to the next level, and by
having the chance to work with testers close by who
taught me how to express what I want as test scenarios.
I’ll share some examples from a previous project.
www.TestingCircus.com March 2014 - 10 -
- Ulrika Park
Requirements People Need
Your Help!
Examples of Testable Requirements
11. A testable business requirement
I was asked by the business owner to implement a fea-
ture: “Cardholders should be able to edit the rights for a
whole household to use the money on their bonus card”
Since money and banking was involved, it was a bit
complicated to implement.
My first question to the business owner was: “why?” and
how will you know it works?”
After quite a lengthy conversation, he said that what he
really cared about was that the money on the bonus
account was spent. He didn’t want the money to stay on
their bonus cards.
“How can we verify that this target is achieved?” I asked.
“Well.. he said. If the money is spent, then the feature
works.”
“So.. when in time is realistic that we can check this..?”
“Well.. within 6 months we should have a better rate of
spending the bonus money than now” he said.
“Ok. So what do you mean by ‘better’?”
“Hm…” he said. “I’d be content for now if 50% of the
total money paid out to customers bonus accounts
would be spent”.
“Thanks for clarifying! Our feature could help out with
achieving that goal. But to achieve this, other things are
involved. Customers need to know about how to share
bonus money between people in their family. How will
they know? Marketing, customer service.. a lot of factors
might affect if this feature is used by the customer.”
“Yes, of course. But this is what I really care about. And
when you have a feature households can use, we should
do an effort to inform customers”.
Good. Now we had a high level business goal, a testable
business requirement. Even though our feature
wouldn’t be the sole solution to make the business
achieve this goal, knowing the target for sure helped us
a lot in developing the feature.
What to do when you don’t have access to the business
owner? When maybe you just get a bunch of use cases
from somewhere to implement?
Well, in these cases I try to define my own hypothesis
about the main goal and result. “This is how I / we have
interpreted the target since we don’t know” and then
show for those stakeholders I do have access to. Often I
do get some feedback on my hypothetical business goal
statement. Even “You’re totally wrong in your assump-
tion!!” is good to know before developing anything. And
you have a reason to ask for answers.
Before testing or developing any feature, we have to
know or make a clear defined assumption about the
expected result for business.
A testable middle level, user requirement
So now we knew the business goal of the feature. The
feature could be implemented in many ways, with op-
tions from everything from printing and scanning paper
forms to digital authorization functionality.
As a requirements analysts, turning into a tests-before-
development tester, I defined some user stories. (We did
a lot of other things too to understand what solution
might fit, but that’s another story).
The main “middle level” user story:
As main cardholder I want to authorize other card-holders in
my family in order for anyone to use the money on the bonus
account.
Before communicating this to the development team, I
start to think about.. how to test this? What would I test?
I brought in a tester for a chat. The tester was busy with
other assignments, but he did have a few minutes to help
me out. And I asked him “How would you test this
story?”
“Identify scenarios” he told me. And with some coach-
ing I made up some scenarios. (here is just a snapshot to
keep the article short)
Scenario 1: Give authorization to other cardholder in a
household with only 2 cardholders.
Scenario 2: Give authorization to other cardholder in a
household with several cardholders.
Scenario 3: Authorization process is actively canceled
by cardholder
Scenario 4: Authorization process is canceled by un-
planned interruption
etc.
Then, exemplify these scenarios with Gherkin inspired
syntax:
www.TestingCircus.com March 2014 - 11 -
12. Scenario 1: Give authorization to other cardholder in a
household with only 2 cardholders.
Given that:
Household has 2 and only 2 cardholders
The 2nd cardholder doesn’t currently have the right to
use bonus money
Main cardholder has actively selected the 2nd cardholder
The 2nd cardholder is >= 12 years old
Expected result:
Information is shown: “You have now given authoriza-
tion to <2nd cardholders full name> with SSN: <2nd
cardholders SSN>.
The 2nd cardholder now has authority to use bonus.
Scenario 3: Authorization process is actively canceled by
cardholder
Given that:
Household has 2 and only 2 cardholders
The 2nd cardholder doesn’t currently have the right to
use bonus money
Main cardholder has actively selected the 2nd cardholder
and
has entered external digital identification application to
authorize
and
shuts down the digital identification application
Expected results:
The 2nd cardholder doesn’t have authority to use bonus
Message to main cardholder: The change has been can-
celed.
These were just two examples. Doing this I had now:
1. A specification that can be used for testing
2. Got forced to find out the exact business rules regard-
ing who actually could be authorized. The age limit, for
instance, was discovered by defining the tests. By defin-
ing test scenarios, I got a foundation to ask the right
questions to domain experts. “Can anyone be autho-
rized?”
3. A way to at any time go back and show for who it
might concern (developers, stakeholders, customer ser-
vice) what exact requirements and rules we built for.
These are just small examples from a big domain, so I
expect you have a lot of critique “Where are scenario x??
And those examples could for sure cover much more!
And aren’t that statement a bit vague? What does ‘ac-
tively’ mean really..?”
The point here is not to give the full picture. For that I’d
need to write a book :-) The point is to show that by
defining tests while working with the requirement, the
requirement got much more explicit. We saved time for
our tester, who could focus on exploratory testing when
time came. We saved a lot of time for developers who
actually got the information ahead development on what
would be tested, and what rules should be applied, so
they saved a lot of rework.
Also when having conversation with developers about
the examples we modified them a bit, and removed
unnecessary ones, or added missing. Some of the scenar-
ios could even be quite easily automated while develop-
ing, which also saved us a lot of time.
So, what to do as a tester then? If you’re just not involved
in creating the requirement definitions? Maybe the re-
quirements are just handed off to you in one way or
another, and when you get them they’re not testable at
all?
One thing you can do as a tester, is to make an effort to
be included when other people are working with re-
quirements. Offer your help!
Only once I have been approached by a tester with this
offer!
It’s always me who’ve approached testers to help me
with making testable requirements.
Have you ever as a tester tried to offer your help – and
seriously tried? To the guys working with requirements?
And there are much more efficient ways than to say “If
you need me you know where I am”.
A tester could say something like this: “I know I’m
going to work on testing for this project a couple of
months from now. Is it possible for me to see some of the
requirements already now? It will help me get ramped
up quicker when I’m in the project. I’m in another project
right now, but still I should be able to take one or two
hours to look at what you have. And yeah, incomplete
use cases or draft user stories will work fine too!”
When you do get hold of some kind of requirements
early in the process, take a quick look. Try to define some
simple test cases or scenarios, and maybe you will get an
opportunity to discuss these with the requirement ana-
www.TestingCircus.com March 2014 - 12 -
13. lysts (or whatever role who works with the require-
ments).. Then why not inviting her for a lunch or a
virtual coffee break (or other social excuse) over Skype in
case of distributed teams?
Requirements people need your help! They just don’t
know it yet. Who, if not you, will seriously invite to
make their work testable?
Ulrika Park is a
requirements geek with
a passion for testing,
methods, learning & the
development of prod-
ucts & services within
organizations and
teams.
With 15 years of
experience in software
development, management & business
she now works at SmartBear. She believes in the
synergy of people, software and quality thinking
to change the world.
www.TestingCircus.com March 2014 - 13 -
14. Look around in various IT companies, flip pages of some IT magazine, read blogs, see forums and you will be
sure to find the today’s buzz word, the Cloud. You cannot find a word that has created so much of ripples than
the Cloud. Very dramatically, the thinking process on providing computing resources has been shifted to Cloud
computing. With this shift of focus, a lot of confusion has accumulated along the way in people’s mind.
Companies are taking this concept very seriously. Many have moved to cloud, and those who are still on the
“ground” are planning to move to cloud in next couple of years.
People are still confused. They still are not comfortable with the Cloud thought. You ask anyone what a cloud
service is and you still won’t get a very confident answer. Even if you get an answer, ask what benefits they offer
and what are the disadvantages at stake, and you can see the person will be uncomfortable answering this. Make
him skip the discussion by asking how secure they are, and it will be end of discussion. When it comes to testing
applications, we are not fully sure whether to adopt the cloud approach of testing or not. What are the benefits
we get and what are the hurdles on the way?
Let’s take a closer look at the Cloud and how it can benefit out day today testing activities.
So what is a Cloud?
Cloud computing is a business and economic model. This model has been successfully deployed and executed
for various material commodities since its inception, but in the recent years it has been formalized for IT products
and services’.
Let me try and explain the basic difference between a Cloud service and a Non-Cloud service.
Consider that you have to move from one place to another. You can use your car OR use the services of a Taxi.
Both of these vehicles have several similarities
· Both are automobiles, having very similar structure and machinery.
· Both provide basic functionality of transferring people / goods from one place to other.
So where lies the difference?
The difference is in the business model for the service provided by them.
· When the car is owners, the owner has to pay for the fuel, regular maintenance and even possibly a
garage. In turn, the car provides the service solely to the owner - you.
www.TestingCircus.com March 2014 - 14 -
- Vipin Jain
- Anubha Jain
Taking Testing to
The “Cloud”
15. · When the car is a Taxi, the service provided by the taxi cab can be described as ‘Travel as a service’. Who
owns the car and who pays the maintenance is not your concern. As a customer, you just have to pay to
travel from your place to another desired place. No maintenance fees and no parking fees need to be paid
by you. This responsibility lies with the cab driver.
The Cloud is synonymous with
the phrase ‘On Demand’. You
pay only on demand (when you
require it.)
Cloud gained momentum when
IT industry got associated with
it. Today, we find that there is a
huge range of products and serv-
ices available on demand. All are
“As a service” e.g. ‘Games as a
service’, ‘Java as a service’, ‘Stor-
age as a service’ and the list is
endless.
Fig 1: TaaS or ‘Travel as a service’ – A Metro Cab service
How Testing and Cloud work together?
Cloud-based testing offers a remarkable combination of low costs, pay-per use model and elimination of initial
capital expenditures. The benefits, however, are more than just cost effectiveness. The non-cost factors include
on-demand flexibility, a respite from holding various infrastructure assets, enhanced collaboration, higher levels of
efficiency and, most importantly, reduced time to-market for key business applications. Economically the vendor
and end user gets huge benefits from the Cloud. This comes directly from reduced subscription prices of any
product or service.
Other benefits are:
· Cloud Apps are scalable: The elastic business model, as it is popularly known as, can be customized based
on the requirement
· Auto-Provisioning: Depending on needs of end users, various Cloud vendors provide and withhold the
offerings in a manner which is in an automatic and self-serving format.
· Unlimited resources: At least the end user thinks and feels this. The services / products are available as and
when they are demanded and in the required quantity
The Cloud computing model should be coined as ‘Green Model’ as it maximizes usage of resources and minimizes
wastage making it environment friendly.
Current state of testing in cloud
Many companies are still taking a cautious approach with cloud computing. This is not the case with testing
however. They are largely ready for testing in cloud and following reasons will account for that readiness:
· As we all know, testing is not a one time, but a periodic exercise and each project will require a new
environment to be set up. If a company creates a Test lab, it typically sit unused for longer periods, resulting
in a waste of cost, electricity and space. Lots of published reports indicate that more than 50% of the
www.TestingCircus.com March 2014 - 15 -
16. technical infrastructure for testing remains underutilized.
· Since testing still being considered as a must but non-critical activity for business, taking testing in cloud
premises is pretty safe as it doesn’t include important company data and has minimum impact on the
organizational business activities.
· Today’s applications are increasingly getting dynamic, more complex, distributed across continents and
more component-based. Testing them is getting more challenging. For instance, with mobile and Web
applications testing, testing needs to be done for multiple operating systems and regular critical updates,
various browsers and their versions, variety of hardware and a large pool of concurrent users to understand
real time performance. It is pretty difficult to follow the age-old approach of creating so many in-house
testing environments. These will become very complex and will need huge capital and resources.
Fujitsu in a 2010 research suggested that testing ranked second (57%) as the most likely workload to be put into the
cloud after Web sites (61%). The on-demand provisioning by Cloud addresses all the above explained issues with
one click. On top of it, the effort and resources saved by using cloud can be redeployed for core business functions.
The Cost Factor
Economic benefits are the main factor influencing companies to take testing to cloud. Another 2010 survey by IDC
hinted the same. As the global economy recesses, companies continue to find ways to regulate costs and improve
ROIs. Cloud testing reduces the unit cost of computing.
www.TestingCircus.com March 2014 - 16 -
17. Small and medium-size businesses that cannot afford high costs find cloud-based testing as a new lifeline. The
companies no longer need to invest in infrastructure and software licenses. They also do not need to worry about
various configuration issues and maintenance of test environments, and pay only for what they use.
Beyond cost Benefits?
Cloud is not all about cost saving. There are lot more benefits companies can extract using it.
- A standard infrastructure and pre-configured software is available, reducing efforts in getting servers and
licenses.
- On-demand provisioning helps companies to think forward instead of spending time to set up test labs. All
testing resources required for testing exist within the system and can be called upon instantaneously.
- Better analysis and control are offered to test teams to build and execute their tests and identify the
bottlenecks. This helps in identifying possible runtime bugs a lot before they are actually found.
- It’s a great concept in motion between geographical distributed teams. Once a tester logs in and runs a test,
the results are available over the cloud. The developer can then assess it and fix over the cloud itself. This
eliminates back-and-forth communication between teams.
Limitations
- Lack of standards: Absence of universal/ standard solutions to real world problems is a big issue. Each
cloud provider can have its own hardware, operating models and prices and may or may not offer any
interoperability. This poses a huge challenge for companies when they plan to switch to a new vendors
- Security issues: Security in the public cloud is still a huge concern because the data may be stored in a
location which is outside a company’s legal and functioning jurisdiction.
- Usage: The everyday usage costs increases very rapidly if testing is done without a proper usage of
cloud-based test. Though pay-as-you-go clouds are used, it can be expensive if the testing is out of sync with
requirements.
www.TestingCircus.com March 2014 - 17-
18. - Performance: It is always an issue since public
clouds are used by many users in parallel. Situa-
tions got created a company had to wait for the
required bandwidth to execute their tests.
Conclusion
This paper tries to explain the benefits and limitations
associated with taking testing to Cloud. We have tried to
explain that why companies should start small and gain
confidence slowly to capture maximum benefits of
cloud-based testing. Once they believe that it has led
them in speeding time to market, lowering of costs and
ensuring standards compliance, they can go big. Using
of pay-as-you-go or on-demand services intelligently
and efficiently, companies can reduce cost of operation
and ownership. Companies should pilot cloud-based
testing as early as they feel comfortable before going to
mainstream testing.
Bibliography
- “Determine if a Cloud is Usable,” blog post,
Bloomberg Businessweek, Jan. 31, 2011.
- “Solving the Challenges of Enterprise Mobile Appli-
cation Development With Cloud-Based Testing,”
blog post,
- CIO, Feb. 17, 2011.
- Rajagopal Sattaluri, “Testing Considerations for Ap-
plication Migration to Cloud Computing,”
- Cloud Computing Journal, Feb. 8, 2011.
- “Cloud Computing: The Good, Bad and the Ugly,”
blog post, Dynamic Data Inc., Feb. 1, 2011.
- “Cloud Testing, A Growing Trend,” blog post, So-
nata Software, April 4, 2010.
- Nivedan Prakash, “Cloud Testing: Attracting De-
mand,” Express Computer, Feb. 1, 2010.
www.TestingCircus.com March 2014 - 18 -
Vipin has dedicated last 10 years of his professional career to the software
quality and testing. Currently working with Metacube Software, India, he
developed his key skills in developing automation frameworks and automat-
ing applications. With a proven record of implementing and refining test
processes for various clients across the globe. he is the author of several articles
and seven well sold books in India. He is pretty active within the software
testing community by speaking at International and national conferences,
writing articles and contributing to various blogs and forums.
Anubha is working as Associate Professor & Head, IT department, Internation-
al College for Girls (ICG), located in Jaipur, India. An academician for last 11
years, she has been involved in teaching and mentoring several students in the
field of Computer Science. Knowing that teaching is the best form of giving
knowledge back to society, she worked as a lecturer in Subodh college, Jaipur
before settling in her current role at ICG. She is currently pursuing her PhD, in
field of Information architecture. Anubha is the author of 9 books and a regular
contributor in various forums.
19. To Subscribe
Click Here
Still relying on
reading
Testing Circus
from tweets
& facebook
updates?
Subscribe
just with your
email id and
get the
magazine
delivered to
your email
every month,
free!
Testing Circus
www.testingcircus.com
www.TestingCircus.com March 2014 - 19 -
20. I recently gave a talk at a conference. I learnt many
things from my interactions with the attendees at my
session. One specific thing that stood out for me was
how – testers no matter how intelligent, smart, critical
thinking they are, still fall into the common pitfall
of using test cases for communicating testing progress and
test coverage.
If anyone reading this article, is still under this mindset,
I would ask the following questions-
· Do the test cases cover each and every scenario, each
and every part of the system under test?
· Can you give accurate information of testing effort in
terms of test cases? Does it really make sense?
· Are we trying to hide from reality and give some sort
of number to make us and our project managers feel
good when we report the number of test cases covered
to communicate testing progress?
· What do you actually mean by test coverage? Can
any other information about the system influence
this?
Here is the common pitfall of completely relying on test
cases.
Firstly, 100% test coverage of the system is impossible.
“Exhaustive Testing” is a misnomer. This being said, if
anyone says “I have 10000 test cases, I executed all of
them and thus all my testing is completed”, it is flawed
because it is impossible to cover every scenario of a
system by just executing all the test cases
· What if there are some hidden scenarios that the
tester still hasn’t uncovered?
· What if another system influences the system under
test in a different way than expected?
· What if the system behaves differently under differ-
ent situations and our test cases did not cover this?
….and so on. There are so many “What if” questions to
ask ourselves. It’s an endless list.
So the point is, NO; we cannot possibility write every
possible test case to cover each and every scenario of a
system. This is where as testers, we think about testing
approaches complementary to just executing test cases
like exploratory testing, automated testing, combinato-
rial testing and other approaches based on the context
and scope of what we intend to cover. This may help to
uncover other weird scenarios or trigger unexpected
outcomes. With this being the case, it may not be valid
to say, “I have 100% test coverage because I executed all
the test cases”.
Secondly, you may then ask, how should I give up-
dates on testing progress to my stakeholders? In my
experience, I usually do not have a problem in explain-
ing testing progress to my stakeholders even if they are
old school and believe “test cases are the solution to all
the problems”. I would of course have some test cases
that I execute, but also do lot of complementary testing
to test case execution. Finally, I give the approximate
percentage of modules covered/tested in the system as
my testing progress.
So for example- If there is a system A, I generally split
it into different modules M1, M2…….Mn (test areas). I
use different testing approaches to test these modules
www.TestingCircus.com March 2014 - 20 -
- Raj Subramanian
A Common Pitfall
Of
Test Cases
21. and give an update in terms of the percentage of mod-
ules I have covered in the system. (This is a general
approximation as there is always a possibility that I
haven’t thought about some other module in the system
which may cause problems. To err is human.)
So, say if I had covered M1, M2 out of the 5 modules I
have identified i.e M1, M2……M5, then I may have cov-
ered about 40% of the system ([2 modules/5 modules] *
100, in case you were wondering how I came up with
this percentage ) . This is how I give my updates on
testing progress. This by itself helps to give the stake-
holders some level of visibility into testing progress and
helps to make decisions accordingly.
To summarize, I think as testers we know it is ethically
wrong to make false claims. That being said, just using
test cases to estimate the amount of testing progress and
test coverage falls under this ethical problem as we are
giving wrong impression to our stakeholders who trust
us. Doing this tarnishes tester credibility which is one of
the most important factors that influence a tester’s work
and helps to establish trust among his peers.
Raj Subramanian is a pas-
sionate tester, who has
previous experience work-
ing as a developer and
moved to testing to focus
on his passion. He gradu-
ated from Rochester Insti-
tute of Technology with a
Masters in Software Engi-
neering, worked as a developer for a payroll
processing company and currently works as a test
engineer for a major insurance company with
specific focus on mobile testing. He has been pret-
ty active in terms of learning and contributing to
the testing society by speaking at conferences,
writing articles, blogging and being directly in-
volved in various testing related activities. He
currently lives in Cleveland, Ohio.
Raj blogs at http://www.rajsubra.com/blog/
And tweets with @epsilon11
www.TestingCircus.com March 2014 - 21 -
22. moolya sucks
we test fast and don’t know to make
more money from our customers.
we are like this only
sales@moolya.com
23. You have been asked to test the latest Mobile based
application, created painstakingly by your organization
over the last few months. All you have with you is your
exclusive Smartphone, which you saved for, and
bought, over the last few months / years. The choices
are now with you, either beg/borrow/steal from col-
leagues or do the right thing. Fire up that web browser
on your Smartphone, open up ‘Google’ and have a go
at searching for ‘Mobile Test Partners on the Cloud’. I
would generally as a rule go for the second option and
not put ‘my precious’ phone to the test on un-tested
applications.
What you achieve when you go through the search, is a
wide variety of web sites and commercial vendors of-
fering this facility to you, at a fraction of the cost that
you would spend in setting up your own facilities and
test lab. You need to now make an informed decision by
researching and figuring out which one of these would
serve your purpose the best and also give you value for
the money spent.
Mobile testing has come a long way. From the initial
fragmented scenario of having to check on each kind of
screen resolution and phone type and screen size, add
to that the variety of mobile browsers being offered by
the various vendors, to the current situation of having
apps created using HTML5 versus native apps. Adding
to the general confusion is the non-app area, where
companies wish to check the overall ‘responsiveness’ of
their “web pages” across the same wide variety of
devices, which includes [and not restricted to] iOS
(iPhones and iPads), Android (Tablets and Phones –
low end and high end), and recently the Windows 8
(WP8 devices and Windows RT devices and desktop’s
with Windows 7 & 8 included) and Chrome OS (mainly
low end laptops & devices). This vast array of devices
also brings up with them an equally confusing array of
browsers along with them (at the last count it was
something like 9 browsers and growing).
Faced with a similar dilemma, we take refuge in the
well-trodden path of checking on the search engines for
a Web Accessible Mobile Testing Tool (applications and
browser based systems), that would be able to serve our
purpose and not cause the management to jump from
their warm seats when they finally receive the bill for
the services used to test their precious new mobile
application or website.
To get through all this, I listed down a few of the up and
coming offerings which provide a good cross-section of
the devices and are reasonable in costs. Although these
cloud platforms do provide a service which is extreme-
ly useful, keep in mind, you might need to make use of
physically handling the device to run certain tests,
which cannot be checked with automation (but there
are ways and means to handle this, so don’t be disheart-
ened). A hybrid cloud installation in these situations
can be one of the simpler solutions which come handy.
A hybrid cloud provides a small subset of devices at
your physical location and the wider variety at a remote
location. Of the Could Mobile Test providers, the most
important thing to look into is if they provide you with
storage for your tests and a way to run the automated
tests that you have so painstakingly created for your
application from within the cloud infrastructure. This is
along with the use of making sure that these tests run
www.TestingCircus.com March 2014 - 23 -
- Gagneet Singh
Mobile Application Testing
Using the Cloud Infrastructure
24. over a wide variety of network speeds simulated to
provide the 2G, 3G, 4G and Wireless speeds prevalent in
the World Wide Networks around the world.
Practically a Cloud Based system should be able to cater
for most of the requirements outlined above; and the
good news is that most of them do exactly that. A few of
the prominent ones which I have used recently and the
ones which come to my mind are: pCloudy.com,
appurify.com, perfectomobile.com, HP mobile testing
lab, to name just a few of the upcoming entrants in this
field, where by Gartner estimates, there are 5.6 billion
handsets present. pCloudy.com has stood out as an ex-
cellent upcoming product, being run by engineers who
have previously worked with Nokia and other top mo-
bile hardware development companies. They have built
the framework for testing on mobile devices from the
ground up and have got some really cool features which
go with it. (Disclaimer: I know one of the Co-Founders as
a colleague from my previous companies). I loved the
way they have handled the storage of tests and results on
the site itself, along with providing features such as
mock location maps, allowing the users to experience the
different states of the application in separate geographi-
cal locations. They have recently launched a feature of
having multiple browsers and simplified their launching
from within the cloud interface, making things easier for
users again. And have launched the latest Android
‘KitKat’ version with the Nexus5. All in all, an excellent
package to go for, with reasonable rates.
All said, the main purpose of having a cloud based
mobile test experience is important for any company
wanting to launch its web presence these days. With the
advent of HTML5 and other technologies like Founda-
tion (more on this later), the web has become a place
where people love a responsive site (or application
across iOS, Chrome OS and Windows [Phone] 8) that
caters for whatever device they are working on, and does
not have the staid look and feel when they change from
a Desktop -> Laptop -> Handheld Tablet -> Smartphone.
They want to get the feel that the web site developer /
organization has done their homework and provided
them with a site where they do not have to pinch-in and
out, just to read content. With the expansion of Smart-
phone markets in the developing nations and organiza-
tions wanting to tap into the ‘billions’ of people there to
advance their products, it has become imperative for
these organizations to go through the process of Mobile
Testing their web sites and mobile based applications
across various low end and high end devices to be useful.
Sites like pCloudy.com, appurify.com, etc. are making it
easier and faster to send out mobile products into the
market by providing the required platform at a fraction
of the cost of actually acquiring a complete mobile test
lab. That said, you would still need to work out a small
subset of your tests at a physical location, but I am sure
that this would also become possible over the cloud.
With the advances these start-ups are making with tech-
nology and improvements on their own feature sets. I
certainly would be looking forward to features which
provide a friendly interface and let me run across multi-
ple devices; and for that the Mobile Test on Cloud pro-
viders are definitely a better option than trying to keep
up with the influx of the hardware been thrown into the
market by all the top end and low end hardware manu-
facturers around the world.
Gagneet Singh has been
working in the Quality
Assurance/Test field for
the past 8 years (with an
additional 4 years in Sys-
tem Tools development)
and has been involved
with companies such as
Toshiba, Adobe (Macro-
media) , McAfee, Oracle, Yahoo! and recently Mi-
crosoft.
He likes to blog and to write about the experiences
he faced in the various organizations and situa-
tions. His work has mostly been with Automation
Testing, along with Performance QA. Also, Secu-
rity testing over the much hyped “Cloud Comput-
ing” (using Hadoop and Azure) has figured in his
work area.
Currently working out of this place they call the
“Down Under“, where he lives in Sydney, New
South Wales!
www.TestingCircus.com March 2014 - 24 -
25. I want to define what I mean by ‘late’ and ‘early’ and
then go on to give some context from a firm I once
worked for – let’s give them the fictitious name “Lin-
man Manufacturing” for convenience. If I don’t define
my terms, we will all get lost. I will use ‘early’ and ‘late’
in an almost intuitive way, although perhaps not as
everyone uses them. They are not of the Humpty
Dumpty variety (Alice Through the Looking Glass by
Lewis Carroll: 'When I use a word,’ Humpty Dumpty
said in rather a scornful tone, ‘it means just what I
choose it to mean — neither more nor less.’).
So, for me ‘early’ is before we were ready, and ‘late’ is
after the planned date. These are pragmatic terms – let’s
not get argue about them now. My hope is that you will
see the usefulness for yourself. To start with, I need to
say something about project deadlines. First of all, it is
not testers (or test managers) who determine when a
work product is implemented (into PROD, obviously).
Test resources can speak into, or provide information,
that helps others make that kind of decision, but the
final decision lies outside “testing”. Secondly, we per-
haps all recognize that if the decision were left to tes-
ters, work products would not get implemented. We
testers are a pessimistic breed, and tend to focus on
things that don’t work. It is not always the 390 require-
ments that work that get our attention, but the 1 that
does not. We concentrate on that.
Lastly on deadlines, some deadlines are more impor-
tant than others. Take the day I am writing this article.
‘Tomorrow’ I am preaching at a local church, and some-
time before standing up, I need to have prepared what
I am going to say. That is an immoveable deadline.
Similarly, the deadline to ‘post’ this article is midnight
tomorrow. If the editor gets it by then, it will be consid-
ered for the next edition. Otherwise, it will be a candi-
date for the edition that follows. This is another
deadline that has significant consequences if it is
missed. Some software projects have hard-wired dead-
lines that cannot be missed. Perhaps there are legisla-
tion changes that need to be complied with. The
introduction of the Euro is a good example. However,
even when there is a hard (i.e. cannot be moved) dead-
line, there is sometimes wriggle-room around the edg-
es. Some parts of company accounting in Euro-zone
countries needed immediate changes when the Euro
was introduced. Other parts did not – activities around
the end of the company financial year may not have
been needed to be correct on Euro day-1 – only at the
end of the company financial year. So even where there
is a hard deadline, some parts may not need to be ready
on the first day, whilst other parts must be available,
and be seen to be working.
Right. The pre-amble is over. I want to tell you about
my time at “Linman Manufacturing”. In my many
months working for them, there were 8 major imple-
mentations, where significant new functionality was
introduced. Although each implementation was sepa-
rate, there were both implicit and explicit points of
interaction; data was loaded into and extracted from
the same database, with some common database tables
used. As time progressed, later functionality relied up-
on the data that had been introduced in earlier phases,
and was loaded month-on-month. I want to concentrate
on the last 4 major deliverables, giving the targeted
delivery date into PROD and the actual date.
www.TestingCircus.com March 2014 - 25 -
- Peter Morgan
Better to Implement Late
Than Early
How Quicker can mean Slower
26. I mentioned that there 8 major implementations. Of
course, there were other times that software was promot-
ed into the PROD environment. User-requested changes,
software platform upgrades, and system tweaking oc-
curred throughout this time frame. But if the project
team as a whole had been asked before each phase tabled
above, for phases F and G the answer would have been
‘yes’, and for phases E and H ‘no’. In the time that was
given, the best testing, based upon risk, had been carried
out. But there were huge swathes where the answer at
best was “we don’t know”, and some parts were clearly
not working. The architecture was basically to take mul-
tiple-format files on a regular basis (daily / weekly /
monthly) and to load this into a standard star schema
data warehouse, and then extract the summary informa-
tion. For phase H, data had been successfully loaded
many times, but the reporting layer only developed in
outline. Now it is not until data is output that it is possi-
ble to tell whether it was loaded successfully (i.e. it is fit
for purpose). And so it proved to be.
For phase E, there were immediately amendments re-
quired, both to data loading routines and to the report-
ing layer. Seven weeks later (and after seven
implementations), the solution was stabilizing. Let me
tell you, those seven weeks were interesting. Some of the
input processes could not be run for part of that time,
meaning that reported information was only partially
correct, and it was not until nine weeks after the date that
users could gain any confidence in the output. January
2009 was the first time that month-end reconciliation (for
December 2008) could be attempted. That is quite a long
time after the actual implementation date.
If anything, the situation was worse with phase H, and
interestingly enough from the table above, this phase
was both ‘late’ (after the targeted delivery date) and
‘early’ (implemented before it was ready). The first
weeks were spent loading a backlog of data, with 10
months of data to load immediately. The reporting layer
quickly highlighted ‘problems’ with 7 of the 16 data
feeds – problems that could only be rectified by remov-
ing any data loaded to date, and either re-engineering
the data creation process (before it was made available to
load), or changing the data load procedures. There were
times that it seemed data was being backed out faster
than it was loaded – even though once a data feed was
identified as ‘wrong’, nothing else was loaded for that
feed until the problem was resolved. A big score-board
showed what data had been loaded for which feeds on a
month-by-month basis. March 2010 (when data for Feb-
ruary 2010 was loaded) was the first month-end with
stability.
There are business reasons for ensuring that “something
is delivered”, and that progress is seen to be made.
However, for “Linman Manufacturing” the information
loaded into the Data Warehouse was for long-term stra-
tegic planning, not for the day-to-day business opera-
tion. There were, in one sense, no unmovable project
deadlines.
If you have a choice, never implement a software solu-
tion ‘early’. Doing so makes for a frenetic period of
activity just after the implementation date, and some-
times aborted product launches. If you take longer, it can
mean that the software is available earlier. When that
happens, staff can be truly released to go onto the next
project without constant drag-back from activities that
should have been put to bed!
To do otherwise means that quicker is slower.
www.TestingCircus.com March 2014 - 26 -
Peter Morgan is a testing
professional who has been
involved in the ICT indus-
try for more than 30 years,
and worked in the free-
lance marketplace for
much of that time. His time
has sometimes moved from testing to ‘develop-
ment’, but he would add “always using the mind-
set of a tester”. He is passionate about testing and
a firm advocate of testing qualifications. An en-
thusiastic speaker and author, Peter tries to base
his output on hands-on experience, attempting to
relate fine sounding ideas back to how it will
affect Joe or Jane Tester in their everyday working
lives in the war of attrition that we call software
testing. He is a regular at EuroSTAR conferences,
and is speaking at Belgium Testing Days in 2014 –
the forth year in a row. At this time of life, Peter
offers experience, and can sometimes say when
offered a tricky problem: “Been there, done that,
and here are 2 or 3 options that may, just may,
work in this situation”. He continues to learn,
adding technical skills to his impressive range or
hardware / software / business sector portfolio.
27. I don’t know if anyone has noticed but the world we
live in now is different from the world we used to live
in just 5 years ago. For me, the change that occurred is
far rapid than anticipated, because if we refer this to a
20 or even 10 year past leap then most of the skeptics
would say “Why not!, it should have changed!” but the
shortness and the speed of this period leaves no room
for being skeptic or naïve – We simply need to
understand and adapt!
Mobility has broken free from just being a device to
communicate into an instinct.
We are using our hand held devices in so many different
dimensions other than it was meant to be that it is
changing the complete concept of computing and
networking. Hand held devices are now our nodes to
the clouds; we are now the new form of terminals; “The
Human Terminals”; the value to the word “Touch” has
enhanced itself from Touch screen, and then within a
very short time it again reboots itself to “Human
Senses”! It feels like a history in making where the
systems which are categorized as something purely
mechanical and electrical combination of components
with mix of complex logics and design are re-emerging
to sapiens, like an event in space and time; As I see it,
the systems are reaching to our senses, our beliefs, daily
lives and simply becoming an extension to our self
being, like an implanted new body part. Gadgets are
now not need to be held in hands; they are wearable,
and soon they will represent a mere extension to its
users.
The human mind is now adapting these effects; see for
example our children; how they have adapted the use
of touch screen, games and the use of smart phones.
How people are getting aware of the communication
and sharing bounds and exploits over social networks;
Where to “say and share” what is now dependent on
“where and how” you are registered as a user; Twitters,
Facebook, LinkedIn, Instagram, Four Square, Google,
pint it, flicker and so on. We are not structured anymore;
we are living multiple lives under multiple scenarios.
Somewhat, this is not based entirely on our choices, this
is how the environment around is “turning out to be”
– and we have to live it!
We are now thinking! Or in other more adaptable
senses we are now becoming Context Driven!
These factors are effecting what we used to know as the
solutions and systems; these factors are now playing a
major role in provisions of the right solutions to the core
requirements and needs of the users. And… the same
factors are affecting how we “Test” these systems;
In a way, we are now moving to an age where due to
the magnitude of the contextual effect on any scenario,
generic approaches will simply fall apart and fail! – In
so many words, we need specialists! We need
empowered human beings and their expertise to
address the right issues at the right time; we need to
identify the right coverage to find the important bugs,
and mind you – We need to identify ALL OF THEM!
Under several scenarios and contextual situations and
due to certain trendy and cultural bonds we have not
yet succeeded in creating our own identities as Testers.
We are still based on the “Types” and “Approaches”
scenario. Where, Testers when tend to define what they
do, start explaining the testing types and the approaches
they follow. They tend to list down several technical
tools and programming languages as skills, and not as
“Tools” to support testing. In this stream of self-
www.TestingCircus.com March 2014 - 27 -
- Arslan Ali
7 Types of Testers
What is Your Identity?
28. discovery they forget the very scarlet thread they cross
from being a “Tester” to “being” a Coder.
Well don’t worry we have worked that out as well;
James Bach has already published twice about the types of
testers there are after receiving several queries regarding the
latter; in his recent blog post, he mentioned the following as
the testers’ types:
(I am quoting him here, rather putting in my own words –
it is better that way)
(SEVEN KINDS OF TESTERS)
· Automated? Manual? There is no such thing as
manual or automated testing. It’s all just testing.
Testing is often supported by tools that attempt
to simulate user interaction with the system.
This is what people call “test automation” even
though it is only automating a crude
approximation of one aspect of testing. If you
have the ambition to be a one-man test team, it
is extremely valuable to learn how to make your
own tools.
· Exploratory? Scripted? There is no such thing as
an exploratory or scripted tester. All good
testing is exploratory to some degree and
scripted to some degree.
· Tester. This is a testing generalist who can
contribute to any test team. Sometimes called a
QA analyst, QA engineer, or test engineer. I
prefer the simplicity of “tester.”
· Omega Tester. The omega tester (which I
sometimes call a test jumper, after the analogy
of a paratrooper) is one who can do anything.
An omega tester is equipped to be the only tester
in a project team, if necessary. Omega testers can
lead testing, or work with a team of other testers.
I am an omega tester. I aspire to be a good one.
· Performance Tester. The performance tester
understands the mathematics and dynamics of
the performance of large-scale systems. They use
tools that create high loads and measure the
performance envelope of systems as they scale
up. Performance testers often don’t think of
themselves as testers.
· Usability Tester. The usability tester is a bit
mythical. I have met only two dedicated
usability testers in my entire career, but I have
seen more of them from a distance. A usability
tester specializes in studying how users feel
about using and learning a product.
· Security Tester. Security testers also often don’t
think of themselves as testers. Security is an
exciting, specialized form of testing that requires
the mastery of a great many facts about a great
many technologies.
· Testing Toolsmith. A testing Toolsmith is a
programmer dedicated to writing and
maintaining tools that help testers. This is what
a lot of people would call an “automated tester”
but you better not use that term around me.
· “SDET” Software Development Engineer in
Test. This means a full on programmer who does
testing using his programming skills.
On the other hand there are several characteristics as a
tester which can stood up and create an exclusive tester
identity; For example, boundary Testing heuristics; now
I know that several of us have used this term in our CVs
and have somehow read or taken part in forums to talk
about this type testing; but how many of us have
actually bifurcated this into “Galumphing”, or
“Steeplechase” or even “Leap and Creep” - I don’t think
any of us!
OR
While discussing about product coverage we have used
the term “San Francisco” depot instead of discussing
the missing out requirements, and blaming the
developers of not providing the right specs or cursing
the Implementation team of not discussing the right
requirements; that is what common professional does!
Why not stand out and use your own language?
I have seen professional testers asking questions and
worrying about “Testing without Requirements” or
“Finding the right bugs” within a short period of time,
but I have not seen any of them discussing about using
or creating their own Heuristics? Why not?
There is also no denial in discussing Quality and its
criteria, but have we ever tried to use and educate
ourselves with the use of the “Quality Criteria Heuristic
called “FEW-HICCUPPS”? Try and do that!
www.TestingCircus.com March 2014 - 28 -
29. Let us understand now.
Why can’t we move beyond being a technological freak
to a Tester in a meaningful term; Does the mention of
the very word “Heuristic” is not a mention of “Tool” in
your sense. Technology is a wonderful thing, being
technical is in the very genes of a computer professional;
but as testers we need to identify ourselves as someone
with a niche of being a tester and being technical at the
same time; Being technical comes with the package; So
why not mention who you really are? “A Tester”!
Empower yourself, your language, attitude and
reputation as tester. Learn Un-Learn and the re-learn!
Try and improve your language as testers, differentiate
yourself from the peers of Developers and
Implementers. Yet sewed in and work for their support
Or I can only say this then; you are in the wrong game
baby!
Arslan Ali has more than
14 years of Experience
related to IT, Industry and
Training Institutions with
exclusive experience of 5
years in teaching various
disciplines and projects in
IT Institution. He has worked in various roles in
capacity of Software Engineering, Software
Tester, Trainer and Quality Assurance Roles. The
Major focus of his expertise lies in Coordination,
Implementation and Testing of ERPs and
Customized Applications. He is also a trainer for
Context Driven Testing for various companies
and individuals.
Arslan is currently working at Sidat Hyder
Morshed Associates (www.sidathyder.com.pk)
as a “Sr. Consultant – Information Solutions; but
beside that he is also an active founding member
of TestersTestified (www.testerstestified.com)
(@testtified), Outtabox! (www.outtabox.co)
(@OuttaBoxPk) and OISOL – Open Integrated
Solutions (www.oisol.com) as a training
consultant for Software Testing and Context
Driven Testing Workshops.
You can follow him on twitter @arslan0644 and
on LinkedIn at
pk.linkedin.com/in/thegoodchanges/
www.TestingCircus.com March 2014 - 29 -
30. www.stepinforum.org www.qsitglobal.com
Interaction » Insights » Opportunities
11th
International Software Testing Conference
Today’s tester is a unique being, driven, not by deadlines and mundane tasks, but by a relentless curiosity and
a fascination with possibilities. Awaken the tester in you.This is where you want to be; this is NOW. Walk away
with exciting ideas and quality insights from other industries. Make your mark by challenging the existing
paradigm.
Most Popular Conference among Start-Ups to Fortune 1000 Companies
ExcitedtoparticipateasaSPEAKER/ DELEGATE/SPONSOR?
writetonazreen@stepinforum.org
orcall+91-80-40818888
Over 900+ delegates from India and abroad
attended in STeP-IN SUMMIT in 2013
Testing NOW
STeP-IN
SUMMIT2014
Asia Pacific’s Largest
SoftwareTesting Conference
Produced by: Hosted by:
JUNE 25-27, 2014 AT BANGALORE &
JUNE 18 & 19, 2014 AT PUNE & HYDERABAD
CONFERENCEPROGRAM
View Past Conferences View Past Speakers’Gallery
www.stepinsummit.stepinforum.org
Pre-conferencetutorialworkshops(twoparalleltracks)@Pune,HyderabadandBangalore
TargetAudience:TestLeads,TestArchitects,TestManagers,TestAnalysistsandConsultants
Conference@Bangalore
TargetAudience:CXOs,VicePresidents,Directors,BusinessHeads;Customersandend-usersofITproductsandservices;
TestManagementprofessionals–Managers,TestLeads,Software,Test Engineers;Quality/SEPGManagement
Professionals–Managers,QALeads,SoftwareQAEngineers; ProductDevelopmentManagers;ChangeAgents
18,19,25
June,2014
26-27
June,2014
31. BOOK
WORM’S
CORNER
March is the beginning of the appraisal season; that’s why 50% of the
organization actually wonder if they should be continuing in the or-
ganization or not. That’s why I decided to crawl through “Don’t Hire
the Best”.
This book is written by Abhijit Bhaduri. In case you are wondering if
it is a misguide to hiring the right team, then you are partially incor-
rect. This book preaches and lets you come to your own conclusions.
The book tries to sell itself by lecturing about good hires and how to
make good hires, however, it tries to draw the right line between
“hiring right” and “hiring the best”. Unlike many other interviewing
books that are filled with choc-a-bloc aptitude and problem solving
questions, this book also tries to examine the organization culture and
why culture fitment is important for the company. The case studies
with the standard disclaimer of having been altered to protect identi-
ty is very useful and causes the reader to ponder as to how he could
have done things better. Reasons as to why it is important to create a
fitment between the interviewee personality and the management ex-
pectations, individual traits that need measurement, why making the
wrong hire call can result in financial losses are stories talked about in
depth in this book.
What did it teach me? After reading the book, I ensure that I pass on
a set of good and bad candidates to my interviewing team; that varia-
tion causes them to be on their toes and interview without bias. If I
send only good candidates their way, a form of bias sets in over the time which would result in all of us overlooking
the most obvious reason for not hiring the candidate.
Why should you buy this book? Because it’s not free; in addition to that, you will learn valuable lessons on how to
create a team, why the culture fitment is important for your teams and why you need to ask questions beyond the
role competencies while forming teams. When is it important to look into a resume to know what he’s capable of and
when should you look beyond the resume? You will ask yourselves these questions if you were to read this book.
I hope, after reading the book, you don’t hire the best anymore; you will hire “better than the best”.
Love,
WoBo
www.TestingCircus.com March 2014 - 31 -
32. Request a free demo by sending us an email at support@sahi.co.in
http://www.sahi.co.in
Become our fan -
https://twitter.com/_sahi
http://www.facebook.com/sahi.software
www.TestingCircus.com March 2014 - 32 -
33. Part 39
www.TestingCircus.com March 2014 - 33 -
http://www.testingcircus.com/category/a-fake-testers-diary/
Random Musings
My 39th month with this magazine; that's when I real-
ized that I had inadvertently become a career counsel-
lor for most of my family members who were in their
final year of college. Is there any scope for "Java"?
Should I learn ".NET"? Is the industry scope for "SQL"
or "Oracle"? With so many questions throw at me from
family members, I've stopped visiting any family func-
tions or any event where I get to meet my extended
family. Quoting client-meetings and a workaholic
manager asking for meetings at mid-night, I manage to
escape attending most of these functions. And then
there is that irritating cousin of mine who keeps brag-
ging of his topper credentials in schools and constantly
asks me if there's more scope for "development" or
"testing"? I have a good mind to tell him that "If people
like you join Development teams, then testing career
will have a lot of scope", but I dare not.
Why do I write this column? --- "When a tester logs a
bug and engages the developer in a sadistic discussion
asking him to fix the bug, the tester is also torturing
himself on some levels; the tester's life is such that he
needs such discussions to survive his job" --- This is
what a famous manager had to say about corporate
culture for testers. Though there has been some
amount of hue and cry against this column, there are a
few instances that agree that this column unearths
what lies beneath. The real thought of documenting
this diary is to try and unmask not just the bad guys,
but also the horrid practices that have created fake
testers all over the place. I also fell prey to fake testing
sometime back and believed in the concept of sanity
testing, smoke testing, BVT testing and that all of them
were different testing types; I also thought that I could
become the greatest tester if I cleared all those exams.
And now, I stand before you, as the FAKE TESTER.
That's what I stand today. And I hope my writings will
stop creating more fake testers.
So far, in most columns, I have (unsuccessfully) tried to
provide an insight as to what's wrong with today's
testers. Most of them can be categorized as
· the inability to find good and authentic testing
teachers
· how there are some people sitting in manage-
ment who actually decide and state what test-
ing is all about and the entire company thinks
that's what testing is all about
· the low pay of the testing engineer -- how many
of them do you think have equal opportunities
at growth and how many of them have 40 hour
working weeks?
· And the inability of the testing team in any
company to function as an independent body,
but function as a sub-body of another manage-
ment function (development or support or mar-
keting or sales or .....) there is no independent
person or body leading the testing vision for the
companies.
And at the end of a year, many testers realize the
disdain that others have for him; the disdain becomes
reality. When this is pointed out to the management,
the management don't try to make good the wrong
things; they do some actions that makes the tester's life
from bad to worse.
For a long time, the sole objective of most testing com-
panies seems to be a single agenda --- bill the client
against the number of test cases executed; to showcase
34. smartness, they record and replay some of the testing
and quote savings in the name of automation. The
clients cannot back out due to the legal agreements
signed at the beginning. An impact of this is that the
test results for periodic testing that's being sent to
clients cease to matter; and regardless of the frequency
of app usage by customers, testing companies continue
this trend. 1 look at the testing economy would tell you
that 2.5 Billion Dollars was the money given to off-
shoring companies for testing alone in the 3rd quarter
of 2013. With new testing companies sprouting all
over, this is set double in the next few years. And the
testing companies have also started to exploit this.
With substandard test cases and haphazard testing,
they make hefty profits without providing any value
by testing.
An indicator of the lack of quality can be seen by the
most unimportant projects being handed to the
off-shoring testing companies; this can be seen by the
very fact that in most of the offshore company cases,
not much of business revenue would be lost if the
offshore projects are closed down. Secondly, according
to the reports of our special correspondent, in the last 5
months, there has been ZERO new product launches in
non-Asian markets due to testing done from here.
I am unsure if others realize this, but looks like India's
testing industry has started to face its worst crisis; my
personal thought is that the rest of this year will be
gone in people realizing their mistakes and the indus-
try auto-correcting itself. What do you think will hap-
pen? As I mentioned last month, only time will tell.
www.TestingCircus.com March 2014 - 34 -
35. www.TestingCircus.com March 2014 - 35 -
Anna Royzman
Context-Driven Scholar
https://twitter.com/QA_nna
Sharath Byregowda
A passionate software test professional, Principal Consultant at
SQS, London in their agile group.
https://twitter.com/sharathb
Mike Lyles
International speaker, writer & Sr QA Manager in Performance
Testing, Test Automation & Service Virtualization (20+ yrs IT).
https://twitter.com/mikelyles
Joris Meerts
Publishing the latest news from the software testing weblogs.
Keeper of lists. Testing historian, critic, thinker, member of
#DEWT. #Capgemini
https://twitter.com/testingref
http://Twitter.com/TestingCircus
#Testers2Follow
36. www.TestingCircus.com March 2014 - 36 -
#FollowUsAtTwitter
http://Twitter.com/TestingCircus
Testing Circus
Testing Circus is a world’s leading English language magazine for software testers
and test enthusiasts. Monthly editions since September 2010. #testing
Follow us at https://twitter.com/TestingCircus
37. Part 15
- Santhosh Tuppad
Santhosh Tuppad specializes in exploratory testing approach and his core
interests are security, usability and accessibility amidst other quality criteria.
Santhosh loves writing and he has a blog http://tuppad.com/blog. He has also
authored several articles and crash courses in the past. He attends conferences
and confers with testers he meets. Santhosh is known for his skills in testing
and has won the uTest Top Tester of the Year 2010 apart from winning several
testing competitions from uTest and Zappers.
www.TestingCircus.com March 2014 - 37 -
Security Testing Tips
One of the challenges in security testing is, setting up
the test environment. If you are a small scale organiza-
tion and there are no lots of processes, then it could be
easy for you to go ahead and setup a test environment
like you want, but in the bigger organizations when it
has lots of processes it could be hard. There could be
network related blockers that you may want to clear
(Example: You want to download a software which
you want to use for security testing purpose, and that
source of download is blocked by the network of the
organization). If your test environment is a blocker for
you, then I would better recommend to not performing
security testing and thereby, you can at least save costs.
Isolated network of computers
It is important to have a separate network dedicated for
security testing. This is because, you do not want to
affect the other computers on the network if you down-
load some software and it is infected with malware /
adware or any malicious thing.
No website blockers
The network shouldn’t be blocking any website that
you want to browse for learning about some hacks or
downloading any software to aid your hacking activi-
ty. Let your network policy doesn’t end up blocking
your learning.
Administrator rights for computers & other devices
In my experience, I have faced lot of blockers when the
computer that was given to me did not provide admin-
istrative rights. For changing some of the settings in
order test for security, I had to e-mail the infrastructure
team to change the settings and that consumed time.
Before starting, it is important to ask for a computer
with all the rights to change / modify any setting.
Installations of software before commencement
Make sure that you are ready with installations of all
required software before you commence security test-
ing activity. This is because, it will save you time if
something doesn’t get installed or doesn’t work prop-
erly while you are testing for security. So, installing the
required software like proxy, burp suite, Wireshark,
backtrack, kali linux, mantra browser, nmap and lot of
many other tools happens without any hassles.
No code changes to be done
When testers are testing for security, no code changes
should be done. It is important that you have separate
environment and not the same which developer uses
check in his / her code. And also, security testing needs
to be done once functional testing is done and all the
bugs reported are fixed.
DIY: Test Environment for Security Testing
39. Since this is my first article for Testing Circus I’d like to thank all the wonderful contributors for sharing info,
because there’s no other way to evolve in the IT industry!
Why would we need to run tests on a Remote VM?
You don’t actually need to do this, but from my own experience I’ve isolated a couple of scenarios where this can
prove vital:
a. Having a test-script (or suite) running on a JavaScript-rich page that needs to have complete focus all the
time. So, no touching the mouse or keyboard while this is running. If this takes 1hr to run, you’re going to
get bored and/or sleepy and let’s not forget you have to justify for that particular hour to your management.
b. The VM can run on your local machine (making it not that “remote”), it can run on another machine
lodged on a cloud-hosting server (e.g. RackSpace) or in your blade-center sitting neatly in your compa-
ny’s basement. Unless the VM is running from your computer’s VBox you’re hogging way less resources by
running this remotely.
c. You might be running across situations where your script runs smooth on your local MacOS / Windows
environment, but, for some weird reason you get weird failures on the CI machine running (in most cases) some
Linux distribution. How will you debug this?
Where to start? VBox setup and install
I’m going to run through the steps needed to build your local Ubuntu VM assuming you have a working Host
Machine (I’m currently running Mac OS 10.8.5, but this is 99% valid for Windows as well)
a. Download and install VBox from Oracle’s website
https://www.virtualbox.org/wiki/Downloads
b. Download Ubuntu. I went for 13.04 64bit but you can choose whatever suits you
http://www.ubuntu.com/download/desktop
c. Start VirtualBox
d. Click the NEW pictogram in the left
e. Populate the fields according to your need
www.TestingCircus.com March 2014 - 39 -
- Mihai Sarlea
Running Remote Selenium 2
WebDriver Scripts On a VM
43. f. From here on it’s a standard Ubuntu installation the likes of which you can find at
http://www.ubuntu.com/download/desktop/install-desktop-latest
Configure Ubuntu
I’m just going to run through the basics of Ubuntu configuration so you can get your VM up and running ASAP.
a. Install OpenJDK-7 and Maven
Since I’m using Java to run my Selenium tests, I’ll need to install the Java Development Kit and Maven (in
my case) and I’m willing to share this nice trick useful also when setting up a fresh CI based on Ubuntu:
This is the sequence that will get you a clean usable OpenJDK on a Deb machine:
apt-get remove openjdk*
apt-get remove java*
apt-get install openjdk-7-jdk
apt-get install openjdk-7-*
apt-get install maven
b. Test the install
java –version
mvn -v
c. Check the IP address and the “visibility” of your host computer
ping your_host_computer_ip
www.TestingCircus.com March 2014 - 43 -
44. Here’s how I’ve setup the VM in VBox and Ubuntu:
Now you should be able to ping to and from the VBox installed Ubuntu VM.
www.TestingCircus.com March 2014 - 44 -
45. Setting up SELENIUM 2 on the VM
a. Open up a Terminal window and download the
Selenium Server Standalone jar
wget
http://selenium.googlecode.com/files/selenium-
server-standalone-2.35.0.jar
b. What we’re doing is creating a small Selenium 2
GRID network and that’s why we need to start
up a hub here:
java -jar selenium-server-standalone-2.35.0.jar
-role hub
c. Open up a 2nd Termnal window and setup a
(single) basic node with default config on the
same machine:
java -jar selenium-server-standalone-2.35.0.jar
-role node -hub
http://localhost:4444/grid/register
d. Find your VM’s IP address – open up another
Terminal window:
Ifconfig -a
That’s it, all setup!
Adding the VM to your Selenium WebDriver Java
Framework
This is as easy as adding the lines bellow:
switch (useThisDriver) {
case FIREFOX:
aDriver = new
FirefoxDriver();//profile);
break;
case
FIREFOX_LOCAL_VM_UBUNTU:
DesiredCapabilities capability =
DesiredCapabilities.firefox();
aDriver = new
RemoteWebDriver(new
URL("http://192.168.10.26:4444/wd/hub"), capa-
bility);
break;
DesiredCapabilities capability =
DesiredCapabilities.firefox();
driver = new RemoteWebDriver(new
URL("http://192.168.1.138:4444/wd/hub"),
capability);
driver.findElement(…) // whatever you want to do
If you’re not running the test as part of a framework, just
go ahead and add the lines like this when starting the
test:
Conclusion
I’ve tried to go through every step of setting up a nifty
lightweight GRID 2 on a local (or remote) Ubuntu VM.
The knowledge is applicable, in much the same way to
a Windows VM or any other OS you might need, for the
matter. I hope it helps you get up and running with
your local *nix VM.
Selenium GRID 2 is a very cool feature that can save a
lot of time, but we’ll dive more into this in a later arti-
cle. Till that time, feel free to check out the documenta-
tion Google put up at:
https://code.google.com/p/selenium/wiki/Grid2
More on the framework “Driver” class I used at point 6:
http://blog.cloudtroopers.com/content/%E2%80%9Cdri
ver%E2%80%9D-class-selenium-2-webdriver
Mihai Șarlea is an SQA Auto‑
mation Engineer for
CloudTroopers.com and
sound engineering enthusi-
ast.
Telecom Engineer by forma-
tion, Mihai started flying
above radar while working
as Simlock Security EU region Specialist for Nokia
and, gradually, shifted gears towards QA-ing for
Endava Romania on financial projects. For the last
year and a half, he's been focusing on automated
website testing, continuous integration, light-
weight performance scripts and all adjacent tasks
at CloudTroopers.com.
Outside office hours, Mihai occasionally handles
Front of House Engineering for various clubs in
Cluj-Napoca, Romania and volunteers to help the
less fortunate artists record decent, career kick-
starter, materials.
Feel free to check out his LinkedIn profile:
http://ro.linkedin.com/pub/mihai-sarlea/40/924/b92
www.TestingCircus.com March 2014 - 45 -
46. Thinking what to do with your career?
We can HELP
www.TalentPlusPlus.com
www.TestingCircus.com March 2014 - 46 -
47. Tools Journal Testing Corner
Testing Circus in exclusive
partnership with Tools Journal
(http://toolsjournal.com) presents
the Tools Journal Testing Corner.
1. Test Automation: New Venues,
New Challenges
2. SmartBear TestComplete 10
Brings In Support For
Automated Mobile Testing
3. Squish Coco Brings In Support
For Code Coverage Of C# & Tcl
Code
"Thank You" to all subscribers who have joined us
and have been giving us some good feedback. Most
of all Thank you "Testing Circus" for providing us
a good platform to shout our views .
www.TestingCircus.com March 2014 - 47 -
About Us:
A start up journal and aspiring social
community with an aim to gain and
distribute knowledge on software
tools and concepts in Testing, Agile,
Cloud, Mobile and Enterprise
Integration.
http://www.toolsjournal.com
With over 500 products listed with
quality articles, product owner
interviews, we are moving swiftly to
launch product editorial/user
reviews, community module in next
2 months.
Connect With Us
@toolsjournal
www.toolsjournal.com
www.facebook.com/toolsjournal
Thank You
Testing Circus & Tools Journal
48. While most of the test automation engineers are aware
of the many pitfalls of test automation; there are some
associated to the automation planning & design
considerations. There are some widely used automation
frameworks, & hybrid framework tops them all.
While it’s a good practice to use hybrid framework with
benefits taken from all the conventional frameworks
like keyword, data driven etc…; the real trick in coming
up with a good design is to cater to all the project needs
while keeping the maintenance cost low, and the
learning curve short.
While there is no single way to implement this solution
but again, it’s no rocket science either. One may have to
consider various factors that automation framework has
to handle – technical & non-technical. On technical side
one may have to consider the level of complexity,
abstraction-levels, so that the front-end whether it’s a
GUI or an excel sheet needs to be easy to use & intuitive
at the same time.
While one should use excel file for easy readability,
overusing it may lead to slow execution – contradicting
the very reason automation is used e.g. keyword-driven
framework using excel file has its benefits, it makes
execution bit slower, each read and write to-from excel
file may add up to huge delays by the time automation
suite is completed. Extensive use of excel or any other
file format may lead to bulky automation leading to
slow execution & having its own weight to pull it down.
While it is good idea to give maximum control (if not
all) to the executor to manipulate suite content , it
requires good judgmental call on the skill set of executor
, reusability & degree of flexibility that needs to be
provided so that its benefits are maximized.
Though there have been good amount of focus given
lately on application performance, automation
performance is another area which is unexplored but
said that, doesn’t mean we can overlook the automation
execution speeds.
· Terminology like Library-architecture etc. is
something that is intrinsic to any software
project whether it is automation or
development, so it becomes really necessary,
that we form meaningful library
functions/methods, keep them light, reusable
and ensure proper error handling.
With the advent of various automation tools it becomes
really important to keep a track of its advantages &
disadvantages; before implementing the same.
· With loads of new browsers and devices
mushrooming in - Cross-browser & cross-device
testing is something which can’t be avoided. But
again each browser/device has its own
engine/mechanism to handle web elements
/objects. Taking an eye off such mechanisms can
lead to unexpected results, and automation
performance issues.
Many times failures in automation needn’t be because
of scripting or application alone, browsers also have
their own drawbacks which lead to automation script
failures. While various studies depict various statistics,
one major thing that comes up is IE seems to have many
issues when then comes to test automation, such
statistics need to be looked at before commencing
automation.
· Object identification/locator mechanisms also
play key role. Those who are well versed with
www.TestingCircus.com March 2014 - 48 -
Test Automation: New
Venues, New Challenges
49. automation testing, don’t have to be told that,
each mechanism has its own benefits, like one
may be robust mechanism, another may be easy
to use e.g. name, id as locators; thereby selecting
a mechanism which fits well in all the areas is
preferred.
· With client side technologies like Silverlight,
HTML5, JavaScript etc. dominating the web-
pages, it becomes really critical to take them into
consideration before committing to the client.
Though the pages may appear normal, the tool,
browser may have issues to work with. Thereby
it is always advisable to do a thorough POC,
before dwelling deeper into full-fledged
automation.
· Like in selenium, it is quite obvious to start
automation using Firefox and committing to
client on providing complete solution, what
further is required is the foresight into client’s
future requirements like cross-browser testing,
mobile device testing.
· It is always recommended to think of the
breadth during planning & design phase &
during implementation focus on individual
automation functionality, creating a self-
contained unit, preferably reusable. An
incremental approach is highly advised - each
automation component or test script needs to be
well tested on various browsers & devices before
moving to next level. This approach would
avoid surprises that come with big-bang
approach. (It is advisable to avoid big-bang
approach, wherein various automation
components are clubbed together to fulfill the
requirements. )
· With extensive usage of open source
technologies & tools (like selenium), it becomes
really important to create a proper test
environment, wherein various versions of the
each tool, process or platform is well
documented. Any change in one may lead to
automation failure.
· Integration with various other tools &/or
application/platform compatibility also is
critical, as slightest change in a version may
impact automation framework e.g. which
version of testlink (test management tool)
integrates with which version of selenium or
which version of UFT supports which browser
& version. Various configurations & versions –
inclusive of auto-updates are to be thought
about & it is really important that each step is
taken carefully to avoid last-minute surprises.
· Requirement to automate huge number of test
cases, and to get proper ROI, Distributed
testing, is fast catching up. Various machines
with various configuration, pose a major
challenge to distributed automation testing; a
single script to be executed for - cross-browser ,
cross-platform, cross-device each having their
own intricate setup (including java version, OS
version – service packs installed etc..) – each
system depending on the system below it,
creating a delicate balance for things to work :
eventually all this can add-up to unimaginable
permutation & combinations making it
mandatory to document the details.
Automation is no more limited to a single machine or a
single tool, but ranges from various tool integration to
its implementation on various machines & browsers,
posing new technological challenges to be faced with
advent of new versions, compatibility issues,
configuration issues making it mandatory to foresee all
the major unforeseen pitfalls to be addressed & risks to
be averted or mitigated, ensuring successful automation
solution delivery.
www.TestingCircus.com March 2014 - 49 -