Test Automation for Mobile Devices - DrupalCon Vienna 2017

Test Automa+on For Mobile Devices

Test Automa+on for Drupal 8 in
Mobile Devices and Tablets
Anastasios Daskalopoulos
@AnDasko

•  Me, Brieﬂy
– Originally trained as a Java Developer
– SoBware Tes+ng Since 2003
•  Nokia: 2003 – 2010
–  Test Manager at Nokia.com – ﬁrst experience with CMS
– Test Automa+on Experience since 2006
•  QuickTest Professional - Nokia
•  Robot Framework
•  Nightwatchjs & Phantomjs
•  Codecep+on
•  Appium

•  Brief History of Web
CERN, Geneva, early to mid 1980s: physicists
from around the world needed to share
informa+on - but they didn't have common
machines. TCP/IP protocols were installed on
UNIX machines at CERN, thus making it it's
own Internet site. This early infrastructure
became the founda+on of the early Web.

•  Unexpected Element To Web
– Mobile Devices
•  The expecta+on for the next 20 years would be that
interconnected networks of devices would func+on on
large computers and later on desktop computers.
– Late 1990s – early 2000s
•  The web starts to appear on mobile devices
•  That’s when the ﬁrst problems appear.

•  History of Mobile Devices

•  Early History
–  1910s: public-switched telephone networks
became common

–  1920s: radio communica+on became popular

–  1940s: portable radio-telephones became common
(walkie-talkies)

–  1990s: cellular telecommunica+on became popular –
developed by Mar+n Cooper

•  -Ironic True Quotes About The Development
of Technology
– "I think there is a world market for maybe ﬁve
computers." Thomas Watson, President of IBM,
1943

–  "There is no reason anyone would want a
computer in their home." Ken Olsen, Digital
Equipment Corpora+on Founder, 1977.

•  "Cellular phones will absolutely not replace
local wire systems." Mar+n Cooper, Inventor
Of The Cell Phone, 1981.
– Mar+n Cooper had to take a very cau+ous
approach to development
•  Mobile telecommunica+ons was just as much a poli+cal
challenge as a engineering challenge
•  Cooper had to convince on many occasions doubmul
Motorola execu+ves about the usefulness of mobile
phones

•  "I predict the Internet will soon go
spectacularly supernova and in 1996
catastrophically collapse."
– Robert Metcalfe, founder of 3Com, Ethernet
Inventor, Infoworld Columnist, 1995.
•  Old paradigms die hard – the old way of
thinking s+ll exists even to those at the
forefront of technology

•  "Apple is already dead."
– Nathan Myhrvold, MicrosoB CTO, 1997
•  “There’s no chance that the iPhone is going to
get any signiﬁcant market share.”
– Steve Ballmer, MicrosoB CEO, 2007.
•  Business wishful thinking is also a big
impediment to technology

•  “In ﬁve years, I don’t think there will be a
reason to have a tablet anymore…tablets
themselves are not a good business model.”
– Thorsten Heins, Blackberry CEO, 2013
– Tablet shipments fell by 12.5% last year.

•  "Why would anyone want to look at a web
site from a cell phone?" Nokia Manager, 2004
– Old paradigms die hard: even to managers at one
of the biggest technology companies in the world,
the paradigm of a mobile device primarily used as
a telephone oral communica+on instrument never
stopped

•  The point of the previous slides:
–  The point of the above quotes is that even powerful
technology chiefs have for long denigrated computer,
web and mobile technology. The best thing to do at
this point is to take the lead in mobile development
and quality assurance
•  What the above-men+oned technology chiefs got
wrong:
–  a workable, easy to use mobile web

•  The Cost of Buggy SoBware Is Much Greater
Than We Think
•  The Na+onal Ins+tute of Standards and
Technology Has Calculated That SoBware Bugs
Cost $59.5 Billion Annually Just in the U.S.!
•  Bugs Are Inevitable In Every SoBware
Development Project
– Losses From Buggy SoBware Will Be Reduced By
Con+nuous Tes+ng As Part of the Development
Process

•  The Web and the Mobile Device: When &
Where Did The Above 2 Intersect?
– A brief history of the mobile web
•  The ﬁrst widely available mobile web device: Nokia
9110

•  Problems with the early mobile web:
– 1) Custom code - developers had to write special
code speciﬁcally for mobile devices; this limited
mobile web development since most companies
did not want to invest in training/hiring mobile-
speciﬁc developers
– 2) Low-resolu+on imagery and text-based
approach to web. Also, slow loading +mes
because of low compu+ng power

•  Problems solved!
– iOS & Android oﬀered mobile browsers that could
read HTML, CSS, PHP & Java. Instead of
developing for a stripped-down web, iPhone &
Android invested in transla+ng their web
browsers to web-friendly opera+ng systems
– Large screen yet light and highly portable, easy-to-
use device with an and easy-to-read color screen

•  Why Do People Web Surf On Mobile

–  60% of digital media +me spent on mobile devices

•  KEEP THIS IN MIND – MOST ONLINE TIME IS NOW SPENT
ON MOBILE DEVICES!
•  A 2012 Google survey found that:
–  74% of visitors were more likely to return to mobile
friendly websites
–  61% were likely to leave if a site wasn’t mobile friendly
–  67% were more likely to buy at a mobile friendly website

•  Why Are People Online With Mobile Devices?
– Easy and immediate web access
– Discrete web surﬁng with small, easily-concealed
and fur+ve web browsers
– People are more mobile and therefore want to
use mobile browsing
– The habit of web mobility has started and will
keep growing
•  Reﬂexive, ins+nct ac+vity

•  Now to actual tes+ng!
–  KEEP IN MIND: test automa+on is one part of tes+ng
•  Test Automa+on is oBen just a sales device
–  Too oBen this simply results in the automa+on of buggy soBware
•  Test Automa+on alone will not solve your Quality Assurance
problems

–  KEEP IN MIND EVEN MORE: mobile test automa+on is
s+ll at an early stage: diﬃcult to use with a lot that
will go wrong

•  ALSO REMEMBER THIS ALWAYS:
– Plan Tes+ng/QA at the same +me you are
planning Development
– Stop leaving test planning, design, and even
execu+on to the end
– Bug finding is good!
•  Celebrate finding bugs throughout your system
– Learn to spot anomalies everywhere in daily life
•  Is there something unexpected here?
– Avoid Confirma+on Bias
•  We don’t want bugs in our soBware, therefore, we
don’t see them

•  2 Diﬀerent Approaches to Mobile Web Tes+ng
– Decide which applies to your project
– 1) Web-based mobile apps
– 2) Na+ve mobile apps
•  Drupal 8 can exist on both web-based and
na+ve mobile apps

•  Drupal on Mobile Devices
– How does Drupal come into mobile devices?
– Drupal 7 – fairly recent ac+vity with mobile
devices
– Drupal 8: expansion into mobile development
realm and builds on the accomplishments of
Drupal 7
– hyps://www.drupal.org/docs/8/mobile

•  Drupal Development Approach Debate
– Desktop development ﬁrst and then “adapt” the
project to mobile environment?
– Should Drupal projects have a “mobile ﬁrst”
development approach?

•  Mobile-First Development?

•  "If I were to start Drupal from scratch today,
I'd build it for mobile experiences ﬁrst, and
desktop experience second."
– Dries Buytaert, 2011
– Remember that your site Users will look at your
pages in mobile devices.

•  So, The Big Ques+ons To Ask Before Mobile
QA Begins
– Is it a mobile-specific website or a na+ve mobile
app?
– If “mobile-specific” is the answer, then ask:
•  “Is there different markup and layout than in the
desktop site?”
•  If the answer is yes, then test to validate the mobile
markup and layout first and then the rest of the site for
func+onality and content

•  If the answer is no, then the website is
basically the same that appears in desktop but
viewed inside mobile devices.
•  Check the site for:
– Responsiveness
– Layout
– Feature Func+onality (do all features do what
they are supposed to do)
– Content (Will the User-entered informa+on
appear as expected?)

•  Na+ve Mobile App Tes+ng
– According to the Drupal 8 website on na+ve
mobile applica+on development, Drupal can be
used as the backend for mobile applica+on
development
– As the website states:
•  "Drupal can contain your content, business logic, user
management, and search func+onality, and your app
can be a front end that talks to Drupal using the
Services module.”
– Robot Framework has both a Drupal library and
libraries to communicate and test a mobile
backend

•  Na+ve Mobile App Tes+ng
– So in this instance, test the backend for storage &
data valida+on

– Run numerous, repeated and varied integra+on
tests to verify the communica+on of the na+ve
mobile app features with the Drupal 8 backend
•  Test for these
–  Does the na+ve mobile app communicate with the Drupal
Backend?
–  Are the expected files there?
–  Are the files in the expected format?
–  Do the files have the expected data?

•  Tes+ng Mobile Apps
•  Star+ng the mobile test process:
– Emulators vs. Devices
– Emulators
•  Advantage
–  Emulators allow you to debug ﬂows as you test the app
»  Good for early itera+ons and ini+al test rounds
»  Bad - will not test realis+c client usage.
•  Disadvantage
–  No End User will go to your site and use your app on an
emulator!

•  Commonly Used Emulators
– iOS:
•  xCode
– Android:
•  Android SDK
•  ABer the ini+al tests on emulators, tes+ng
should con+nue on devices

•  THE CHALLENGE OF SOFTWARE TESTING
– What is a bug?
•  Any unexpected or undesired result

–  What is soMware tesOng?
– The ac+ve pursuit and repor+ng of soBware
errors that lead to unexpected results
•  This is an ac+ve and crea+ve process
– Ac+ve, formal and recorded explora+on to get
informa+on about speciﬁc soBware under test

•  Note on Tes+ng: SoBware Bugs vs. User-
Expecta+on Bugs
– SoBware Bug:
•  Error à Fault à Failure

– Error: a human ac+on that produces an incorrect
result
– Fault: a manifesta+on of an error in soBware, also
known as Defect, Bug or Issue
– Failure: a devia+on of the soBware from its
expected delivery or service

•  User-Expecta+on Bugs
–  Not caused by an error in producing the soBware
itself, but what appears in the web site that is
contrary to what the End User would expect to see
•  Examples:
–  typos
–  text ﬁelds that overlap text or other features
–  app features that run out of the frame
–  These are not minor or cosme+c problems: wrong-
appearing site features adversely aﬀect the User’s
experience and the Site Owner’s credibility
•  NOTE: 40% of customer complaints are for what
Developers would label as “Minor” issues

•  The 2 Jobs Of Quality Assurance
– Valida+on: Is this the Right System?
•  Is this what the customer wants?
– Veriﬁca+on: Does the System Work Right?
•  Does the site work correctly?

Test Automa+on for Mobile Devices
•  Bug Severity Status and Triage
•  Is this a problem?

•  This is oBen test automa+on

•  Test Automa+on is a useful tool but oBen
wrongly used
– There are beyer ways to use Test Automa+on
•  Keep in mind that Test Automa+on does not
itself find bugs
– The Tester’s crea+vity and coding skills finds the
bugs and not any specific tool

•  Is there a bug here?

•  Test Automa+on for the Mobile Web
– REALITY: Mobile Test Automa+on Is Just Beginning
•  Developing mobile tests will be slow and diﬃcult
•  Mobile Test Automa+on Does Not Diﬀer From
Any Other Test Automa+on
•  Test Automa+on Should Be Regarded As More
A Development Project Than A Tes+ng/QA
Project

•  Test AutomaOon RealiOes
1.  The hardest and most important part of test
automa+on is not the crea+on but the
maintenance of the scripts
1.  Test Automa+on requires constant supervision of
executed tests and their near constant maintenance
and revision
1.  The Best Test AutomaOon Engineer is the one who can
keep their tests execuOng and providing results and
feedback over Ome
2.  It is not possible to “just automate” manual tests
3.  No one can just “show you” in an hour how to
automate tests

•  Reitera+on of Points
– Test Automa+on is a soBware development
project whose aim is to provide results with
relevant script execu+on about the con+nuing
stability of soBware features over +me
•  No test automa+on tool will succeed without
good principles of automa+on

•  Test Automa+on Philosophy and Principles
– Test Automa+on is not a panacea and by itself will
not solve your Quality Assurance problems
•  A common bad prac+ce is to “automate” tests without
any plan or overall philosophy
–  This results in the automa+on of buggy soBware
– What you want to do is to create tests that will
ﬁnd bugs and inform you where they are and how
they occurred!
•  How do we do this?

•  First Thing –
– HAND OVER YOUR APP FOR TESTING!
•  YOUR SOFTWARE HAS BUGS!
–  This is not an insult – any crea+on that has features that
integrate with other features will at some point have some
kind of unexpected result
•  Independent Tes+ng is very important in catching bugs
–  Developers see in the soBware what they expect to see
•  This does not mean “over-the-wall” tes+ng philosophy
–  This means DevOps and Con+nuous Integra+on approaches

– Think DevOps and Con+nuous Integra+on
•  Individual features are tested in isola+on while others
features are in development
•  A regression test for already-created features is run
directly aBer the build process
– Therefore, we have a constant state of regression
tests for each already-created feature being run
while newer features are being built within the
sprint
– Contrast this with the end-of-project regression
sets

•  How Do We Begin?
– Start automa+ng the most stable features
– As tes+ng and bug ﬁxing con+nue in each
itera+on, the more stable areas of the app creep
farther and farther up the triangle
Less stable

More stable

•  Things to keep in mind always before we
con+nue:
– Find bugs ﬁrst!
•  I don’t think any tool or process will ever replace
organized, structured, recorded test explora+on
–  ABer developing the feature, someone, preferably an
independent and experienced QA analyst, will explore it and
quickly report unexpected results for ﬁxing
–  ABer this, test automa+on can begin

•  The automaOon tool does not find the bugs!
– The test automaOon tool is criOcized when the
problem lies in the test automaOon approach
– The person wri+ng the tests finds the bugs
– No test automa+on tool can be beyer than the
Tester who is crea+ng the tests
– The best test automa+on tool is the one that you
are most comfortable and confident in using

•  Test the Flow of Opera+ons
– Each test should be a contained flow of
opera+ons that reflect user ac+vity and confirm
whether each ac+on produces the expected result
everywhere on the screen
– Ask this: what is the flow supposed to do?
– Is something preven+ng the flow?
– Does the flow produce any unexpected result?

•  Know The Expected Result!
– Big Ques+on: does the actual result at the end of
the flow match the pre-defined expected result?
– If the answer is no, then this is a bug and should
be reported for fixing
– NOTE: the en+re screen is the expected result and
not just one area that contains a returned value

•  Always Keep in Mind:
– FINDING BUGS IS GOOD!
– A successful test run finds bugs!
– There is oBen in Developer-created Test
Automa+on that All Tests Passing = Good
Result
•  Use as many tests, asser+ons and verifica+ons as
possible to find anomalies/unexpected results in the
soBware
•  Develop a “Spot the anomaly” approach to life
-Design & Create Tests That Find Bugs!

•  You must communicate with your tests to
know that an unexpected results was found!
– Know the diﬀerence between a bug in the
soBware and a bug in the test script
– Know what your tests are doing and what they are
ﬁnding

•  Test Automa+on Requires Monitoring
– Think of Test Automa+on in the same way as the
Autopilot in an airplane
•  While the Autopilot is engaged, the pilots and ﬂight
oﬃcers are s+ll carefully watching and monitoring the
instruments
•  You need to pay ayen+on to your tests!

•  The Applica+ons of Test Automa+on

– SMOKE TESTS
•  Did the build succeed?
•  Is it worth con+nuing to test?
– Advantages:
•  Speed in development because few smoke tests are
required
•  Speed in execu+on – test automa+on will always be
faster than manual tes+ng

•  Regression Tests
1.  Regression tests for individual features during
each sprint aBer every build
2.  End of project End-to-End regression tests that
check and verify the stability of the en+re
project
•  With this approach, the stability of the en+re
system is always checked for each new
feature and how each func+on interacts as a
whole

•  How To Start Test Automa+on
– Priori+ze By Importance
•  What is the most important flow of the feature?
•  What is the next most important, etc.
– Keep The Tests Short & For A Single Purpose!
•  Separate different ac+ons and varia+ons in flow into
completely separate test
–  This aids in script maintenance
–  More importantly, it is easier to spot in which line the bug
occurred in a short script

•  Test Automa+on Scripts Should Have 3 Parts
– Setup
•  Start script
•  Open browser
•  Go to URL
– Single Purpose Flow
•  The ﬂow for the ac+on to test
– AsserOons and ValidaOons
•  Compare the result you expect to see against what
actual appears on the screen

•  Test-Driven Development
– The tests are created ﬁrst and then the feature is
developed so it passes the tests
•  In this approach, QA/Tes+ng is baked into the process
by being the ﬁrst step
–  This ensures communica+on between QA/Development/
Management/Client so everyone knows the expected design
and func+onality of the feature
–  Test runs will fail and everyone knows the reason for the
failure
– This can even be done at the Acceptance tes+ng
level, in which End-to-End tests

•  FINAL NOTES
– Test Early, Test OBen
– Use as many tests, asser+ons and veriﬁca+ons as
possible
– Move fast and actually, really, break things!

Test Automation for Mobile Devices - DrupalCon Vienna 2017

Recommended

Recommended

More Related Content

Similar to Test Automation for Mobile Devices - DrupalCon Vienna 2017

Similar to Test Automation for Mobile Devices - DrupalCon Vienna 2017 (20)

More from Exove

More from Exove (20)

Recently uploaded

Recently uploaded (20)

Test Automation for Mobile Devices - DrupalCon Vienna 2017