Software testing: an introduction - 2015

www.unamur.be
Software Testing: an introduction
Xavier
Devroey
<xavier.devroey@unamur.be>

INFO
M110
Ingénierie
du
logiciel

Namur,
Belgique

Plan
•  Introduc:on

–  Error,
fault,
failure,
and
incident

–  So>ware
tes:ng:
deﬁni:on

•  So>ware
tes:ng
typology

–  Source
of
test
genera:on

–  Characteris:c
being
tested

–  Lifecycle
phase

•  Agile
process
implementa:on:
the
TSSG
case
study

•  Hands-‐on

–  Maven

–  JUnit

–  Selenium

2015
-‐
2016
xavier.devroey@unamur.be
2

Error, fault, failure, and incident
Error

Fault
Exec.

Failure
(bug)

Reports

Incident

Bug
Tracker

2015
-‐
2016
4

Dev.

User

Software Testing: definition [SWEBOK v3]

“So>ware
tes:ng
consists
of
the
dynamic
verifica:on

that
a
program
provides
expected
behaviours

on
a
finite
set
of
test
cases,
suitably
selected
from
the

usually
infinite
execu:on
domain.”

2015
-‐
2016
5

Testing
2015
-‐
2016
6

Req.

Test
engineer

Test

case

deﬁnes

Pgm.

executed
on

Dynamic verification
•  Tes:ng
implies
execu6ng
the
program
under
test

•  Contrary
to
sta6c
analysis

–  Formal,
semiformal,
or
informal

–  Examples:

•  compiler
warnings
(code
analysis),

•  javadoc
warnings
(documenta:on
analysis),

•  design
inspec:on
(model
analysis),

•  theorem
proving
(e.g.,
Event-‐B),

•  code
analysis

–  Code
review

–  IDE
warnings

–  FindBugs
h`p://ﬁndbugs.sourceforge.net

2015
-‐
2016
7

Expected behaviours
•  It
must
be
possible
to
decide
whether
the
observed

outcomes
for
a
given
input
is
acceptable
or
not

•  The
oracle
problem

–  “An
oracle
is
any
human
or
mechanical
agent
that
decides

whether
a
program
behaved
correctly
in
a
given
test
and

accordingly
results
in
a
verdict
of
“pass”
or
“fail”.”
[SWEBOK
v3]

2015
-‐
2016
8

Finite set of test cases
•  Exhaus:ve
tes:ng
is
not
possible
in
prac:ce

–  Inﬁnite
number
of
test
cases

–  Select
a
subset
of
all
possible
tests

–  Trade-‐oﬀ
between
limited
resources,
schedules
and
unlimited

test
requirements

•  Priori:za:on
based
on
risk,
usage,
cost,
…

“Program
tes6ng
can
be
used
to
show
the
presence
of
bugs,

but
never
to
show
their
absence!”
Dijkstra
(1970)

2015
-‐
2016
9

Suitably selected
•  How
to
select
test-‐cases
?

•  Selec:on
criteria

–  Input/output
domains
coverage
(black
box
tes:ng)

•  Domain
par::oning
techniques

•  Based
on
speciﬁca:on
documents

•  No
access
to
the
source
code

–  Code
coverage:
statement
coverage,
decision
coverage,
path

coverage,
etc.
(white
box
tes:ng)

•  Access
to
source
code

–  Test
engineer
exper:se

2015
-‐
2016
10

Types of testing
•  Source
of
test
genera:on

–  Black
box
tes:ng

•  Based
on
speciﬁca:on
documents

•  No
access
to
the
source
code

–  White
box
tes:ng

•  Access
to
the
source
code

–  Model-‐based
tes:ng

•  Characteris:c
being
tested

–  Func:onal
tes:ng:
is
the
output
correct
for
a
given
input
?

–  Non-‐func:onal
tes:ng

•  Performance,
robustness,
security,
usability,
etc.

2015
-‐
2016
12

Types of testing
•  Lifecycle
phase

–  Coding

•  Unit
tes:ng

•  Component
tes:ng

–  Integra:on

•  Integra:on
tes:ng

–  System
integra:on

•  System
tes:ng

–  Release

•  Acceptance/Beta
tes:ng

–  Maintenance

•  Regression
tes:ng

2015
-‐
2016
13

Testing in the waterfall model
2015
-‐
2016
14

Requirements
speciﬁcation
Design
Coding and
unit testing
Integration testing
System testing
Acceptance
testing
Release
Maintenance and
regression testing

Testing in the V-model
2015
-‐
2016
15

Requirements
speciﬁcation
Design
Coding and
unit testing
Integration/
System
testing
Acceptance
testing
Release
Integration/
System
test plan
Acceptance
test plan
Maintenance and
regression testing

Testing in an Agile process
1.  Include
tes:ng
ac:vi:es
from
the
beginning

2.  Specify
requirements
in
terms
of
tests

3.  Testers
and
developers
are
not
enemies

4.  Test
o>en
and
in
small
chunks

2015
-‐
2016
16

Test Driven Development (TDD)
•  Red

–  Write
automated
tests

(examples)
from
user
stories

–  Tests
fail
(as
the
func:onality

is
not
implemented
yet)

•  Green

–  Developers
design
and
write

code
to
make
the
tests
pass

–  Tests
are
run
frequently

•  Refactor

–  Refactor
code
without

changing
interfaces
or

behaviour
(tests
must
pass)

Red

Green
Refactor

2015
-‐
2016
17

AGILE PROCESS IMPLEMENTATION
The
Telecommunica:on
So>ware
&
Systems
Group

(TSSG)
case
study

Agile process implementation: TSSG
•  From
Dowling,
P.,
&
McGrath,
K.
(2015).
Using
free
and

open
source
tools
to
manage
soGware
quality.

Communica:ons
of
the
ACM,
58(7),
51–55.
doi:
10.1145/2755503

•  Open
Source
tools

•  Support
team’s
ac:vi:es
management

2015
-‐
2016
19

2015
-‐
2016
20

Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

2015
-‐
2016
21

Project Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

2015
-‐
2016
22

Project Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

Workflow
•  Developer
forks
the
code

base

•  Implements
changes

•  Integrates
possible

changes
from
code
repo.

•  Runs
a
complete
build

–  Including
Unit
tests

•  Submits
changes
to
code

repo.

•  CI
server
performs

integra:on
tests

2015
-‐
2016
23

h`ps://www.ﬂickr.com/photos/b3c>/6713516627/

Do
NOT
break
the
build!

2015
-‐
2016
24

Project Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

2015
-‐
2016
25

Project Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

Hudson

2015
-‐
2016
26

Project Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
Production
Deployment

Maven 3: Introduction
•  Project
Lifecycle
management
tool

–  Project
descrip:on
in
Project
Object
Model
(pom.xml)

–  Build
automa:on
(including
unit
tes9ng
phase)

–  Project
modulariza:on

–  Dependency
management

–  Deployment
automa:on

–  Repor:ng
(code
quality
check,
documenta:on
genera:on,

website
genera:on,
test
coverage,
etc.)

•  Conven:on
over
conﬁgura:on
(…
like
Java
EE)

“Systems,
libraries,
and
frameworks
should
assume
reasonable

defaults.
Without
requiring
unnecessary
conﬁgura6on,
systems

should
“just
work”.”
[Maven
by
example]

2015
-‐
2016
28

Installation
•  See
h`ps://maven.apache.org/install.html

•  Use
command
mvn
–v

to
check
installa:on

•  Local
Maven
ﬁles
are
in
~/.m2

•  ~/.m2/settings.xml
contains
access
conﬁgura:ons

•  ~/.m2/repository
contains
the
local
Maven

repository

2015
-‐
2016
29

$
mvn
-‐version

Apache
Maven
3.1.1

Maven
home:
/usr/local/apache-‐maven-‐3.1.1

Java
version:
1.8.0_31,
vendor:
Oracle
Corporation

Java
home:
/Library/Java/jdk1.8.0_31.jdk/Contents/Home/jre

Default
locale:
fr_FR,
platform
encoding:
UTF-‐8

OS
name:
"mac
os
x”

First Maven project (using NetBeans)
2015
-‐
2016
30

First Maven project (using NetBeans)
2015
-‐
2016
31

Maven project structure
•  myﬁrstmavenproject/

–  pom.xml

–  src/

•  main/

–  java/

»  Here
goes
your
Java
source
code

–  resources/

»  Here
goes
other
resources
(ﬁles,
etc.)
included
in
the
.jar/.war

•  test/

–  java/

»  Here
goes
the
JUnit
tests

–  resources/

»  Here
goes
other
resources
used
by
the
tests

–  sub-‐module-‐1/

•  pom.xml

•  src/

–  sub-‐module-‐2/

•  …

2015
-‐
2016
32

Pom.xml
2015
-‐
2016
33

Pom.xml
2015
-‐
2016
34

Iden9ﬁer

•  Maven
iden:ﬁers
have
to
be
unique

•  Format:
groupId:ar6factId:version

•  GroupId
should
iden:fy
the
organisa:on
using
Java
package

naming
conven:on

•  Ar:factId
should
iden:fy
the
project

•  Version
should
be
encoded
on
3
number
XX.XX.XX

•  “-‐SNAPSHOT”
is
used
to
indicate
the
currently
under

development
version

Pom.xml
2015
-‐
2016
35

Type
of
the
artefact

•  Packaging
indicates
to
Maven
the
type
of
ar:fact
produced
during

build

•  “jar”
indicates
a
.jar
ﬁle
(executable
or
not)

•  “pom”
indicates
a
maven
project
without
compiled
code
(used
by
a

parent
project)

•  “war”
indicates
a
.war
ﬁle
that
will
be
deployed
on
a
web
server

Pom.xml
2015
-‐
2016
36

Proper9es
used
by
project
and
sub-‐projects

•  Proper:es
are
used
as
global
constants

•  Implicit
proper:es

•  ${basedir}
current
project
root
directory

•  ${project.version}
current
project
version

•  ${project.groupId}
current
project
groupId

•  Declared
proper:es

•  Should
contain
dependencies
versions

Sub-modules
•  Parent
project
(should
have
pom
packaging)

•  Sub-‐module
has
to
declare
parent
project

•  In
this
case:
sub-‐module-‐1
version
is
inherited

2015
-‐
2016
37

<project
…>

…

<packaging>pom</packaging>

<modules>

<module>sub-‐module-‐1</module>

</modules>

</project>

<project
…>

<parent>

<groupId>be.unamur.info</groupId>

<artifactId>myfirstmavenproject</artifactId>

<version>0.0.1-‐SNAPSHOT</version>

</parent>

<artifactId>sub-‐module-‐1</artifactId>

…

</project>

Dependencies
•  Maven
central
repository

–  h`p://repo.maven.apache.org/maven2

–  h`p://search.maven.org
(to
search
an
ar:fact)

•  Declare
dependcy
in
a
Maven
project

2015
-‐
2016
38

<project
…>

…

<dependencies>

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>${junit.version}</version>

<scope>test</scope>

</dependency>

</dependencies>

</project>

mvn <phase>
•  compile

–  Compile
source
files

•  test

–  Launch
JUnit
tests
in

src/test/java/

•  package

–  Create
.jar
or
.war
or
…
file
in

target/

•  install

–  Install
artefact
in
local
Maven

repository

•  deploy

–  Deploy
artefact
using

deployment
configura:on

2015
-‐
2016
39

JUnit: Introduction
•  Collec:on
of
classes
to
perform
unit
tes:ng

•  Uses
annota:ons
(e.g.,
@Test)

2015
-‐
2016
41

import
static
org.junit.Assert.assertEquals;

import
org.junit.Test;

public
class
CalculatorTest
{

@Test

public
void
evaluatesExpression()
{

Calculator
calculator
=
new
Calculator();

int
sum
=
calculator.evaluate("1+2+3");

assertEquals(6,
sum);

}

}

java
-‐cp
.;junit-‐4.12.jar;hamcrest-‐core-‐1.3.jar
org.junit.runner.JUnitCore
CalculatorTest

Anatomy of a Junit test class
import

…

public
class
MyClassTest{

private
sta:c
ﬁnal
Logger
logger
=
LoggerFactory.getLogger(JHConwayLifeCellTest.class);

@Rule

public
TestRule
watcher
=
new
TestWatcher()
{

@Override
protected
void
star:ng(Descrip:on
descrip:on)
{

logger.info(String.format("Star:ng
test:
%s()...”,
descrip:on.getMethodName()));};};

@BeforeClass

public
sta:c
void
setUpClass()
{logger.info("Se|ng
up
before
class");}

@A>erClass

public
sta:c
void
tearDownClass()
{logger.info("Tearing
down
a>er
class");}

@Before

public
void
setUp()
{logger.info("Se|ng
up
before
test");}

@A>er

public
void
tearDown()
{logger.info("Tearing
down
a>er
test");}

@Test

public
void
testMyMethod()
{assertTrue(true);}

}

2015
-‐
2016
42

Assertions
•  JUnit

–  asserTrue(c);
assertFalse(c);
assertEquals(expected,
actual);

assertNull(o);
assertNotNull(o);
fail()

–  assertThat(T
actual,
org.hamcrest.Matcher<T>
matcher)

•  Hamcrest
matchers
(org.hamcrest.Matchers)

–  assertThat(T
actual,
equalTo(T
operand))

–  assertThat(T
actual,
not(T
operand))

–  assertThat(Iterable<?
super
T>
c,
hasItem(T
item))

–  assertThat(T
actual,
equalTo(T
operand))

–  assertThat(T
actual,
nullValue())

–  assertThat(T
actual,
notNullValue())

2015
-‐
2016
43

Hamcrest common matchers
•  Core

–  anything
-‐
always
matches,
useful
if
you
don't
care
what
the
object
under
test
is

•  Logical

–  allOf
-‐
matches
if
all
matchers
match,
short
circuits
(like
Java
&&)

–  anyOf
-‐
matches
if
any
matchers
match,
short
circuits
(like
Java
||)

–  not
-‐
matches
if
the
wrapped
matcher
doesn't
match
and
vice
versa

•  Object

–  equalTo
-‐
test
object
equality
using
Object.equals

–  hasToString
-‐
test
Object.toString

–  instanceOf,
isCompa:bleType
-‐
test
type

–  notNullValue,
nullValue
-‐
test
for
null

–  sameInstance
-‐
test
object
iden:ty

•  Collec:ons

–  hasEntry,
hasKey,
hasValue
-‐
test
a
map
contains
an
entry,
key
or
value

–  hasItem,
hasItems
-‐
test
a
collec:on
contains
elements

–  hasItemInArray
-‐
test
an
array
contains
an
element

•  Number

–  closeTo
-‐
test
ﬂoa:ng
point
values
are
close
to
a
given
value

–  greaterThan,
greaterThanOrEqualTo,
lessThan,
lessThanOrEqualTo
-‐
test
ordering

•  Text

–  equalToIgnoringCase
-‐
test
string
equality
ignoring
case

–  equalToIgnoringWhiteSpace
-‐
test
string
equality
ignoring
diﬀerences
in
runs
of
whitespace

–  containsString,
endsWith,
startsWith
-‐
test
string
matching

2015
-‐
2016
44

Hands-on: Maven, JUnit and Hamcrest
•  be.unamur.info:mavencrashcourse-‐cellularautomata

–  From
“Travaux
pra:ques
de
concep:on
et
programma:on
orientée
objet
:
Automate
cellulaire”
G.
Saval,
2008

•  Classes
to
test

–  JHConwayLifeCell

–  Grid

–  CellGrid

•  Test
coverage
using
Cobertura

–  mvn
cobertura:cobertura

•  Code
analysis
using
FindBugs

–  mvn
findbugs:findbugs

–  mvn
findbugs:gui

2015
-‐
2016
45

UI
web-‐app
tes:ng
made
easy

Selenium
•  User
Interface
(UI)
tes:ng
framework
for
web
applica:on

–  Integra9on
and
acceptance

–  black-‐box

–  (mainly
used
for)
regression
tes:ng

2015
-‐
2016
47

Testing Environment
Selenium WebDriver
Selenium IDE
1. Design and/or
execute test-cases
2. Export

Installation
•  See
h`p://www.seleniumhq.org/docs/02_selenium_ide.jsp

2015
-‐
2016
48

Selenium Commands
•  open

–  opens
a
page
using
a
URL

•  click/clickAndWait

–  performs
a
click
opera:on,
and
op:onally
waits
for
a
new
page
to
load

•  verifyTitle/assertTitle

–  verifies
an
expected
page
:tle

•  verifyTextPresent

–  verifies
expected
text
is
somewhere
on
the
page

•  verifyElementPresent

–  verifies
an
expected
UI
element,
as
defined
by
its
HTML
tag,
is
present
on
the
page

•  verifyText

–  verifies
expected
text
and
its
corresponding
HTML
tag
are
present
on
the
page

•  verifyTable

–  verifies
a
table’s
expected
contents

•  waitForPageToLoad

–  pauses
execu:on
un:l
an
expected
new
page
loads.
Called
automa:cally
when
clickAndWait
is
used

•  waitForElementPresent

–  pauses
execu:on
un:l
an
expected
UI
element,
as
defined
by
its
HTML
tag,
is
present
on
the
page

2015
-‐
2016
49

Hands-on: Selenium IDE
•  Let’s
Chat
applica:on

–  h`ps://sdelements.github.io/lets-‐chat/

•  Test
server

–  h`p://rollier.info.fundp.ac.be:5000

•  Test
Cases

–  Create
chat
room,
post
some
message,
exit
room

–  Edit
proﬁle,
change
display
name,
ﬁrst
name
and
last
name

–  Create/Enter
room,
edit
room
name
and
descrip:on

2015
-‐
2016
50

References
•  Mathur,
A.
P.
(2008).
Founda6ons
of
soGware
tes6ng.
Pearson
Educa:on.

•  U|ng,
M.,
&
Legeard,
B.
(2007).
Prac6cal
Model-‐Based
Tes6ng:
A
Tools

Approach.
Morgan
Kaufmann.

•  Bourque,
P.,
&
Fairley,
R.
E.
(2014).
Guide
to
the
SoGware
Engineering
Body
of

Knowledge
(SWEBOK(R)):
Version
3.0.
IEEE
Computer
Society.

•  Dowling,
P.,
&
McGrath,
K.
(2015).
Using
free
and
open
source
tools
to
manage

soGware
quality.
Communica:ons
of
the
ACM,
58(7),
51–55.
doi:
10.1145/2755503

•  Tim
O'Brien
et
al.
Maven
by
Example.
Sonatype,

h`p://www.sonatype.com/books/mvnex-‐book/reference/public-‐book.html

•  h`p://junit.org

•  h`p://hamcrest.org

•  h`p://www.seleniumhq.org

2015
-‐
2016
51

Images
credits:
h`p://hedbonstudios.deviantart.com;
h`p://commons.wikimedia.org/;
h`p://wh1pla5h.deviantart.com;
h`p://wh1pla5h.deviantart.com;
h`p://
darkangelaw1986.deviantart.com;
h`p://phantomcameron.deviantart.com;
h`p://newyorkkid618.deviantart.com;
h`p://doloresmineè.deviantart.com;
h`ps://
www.flickr.com/photos/tbuser
;
h`p://fullmetaldevil.deviantart.com/
;
h`ps://www.flickr.com/photos/fastjack/
;
h`ps://www.flickr.com/photos/eli:stczar/

… last tricks
•  Google
guava

–  h`ps://github.com/google/guava/

•  Apache
commons

–  h`ps://commons.apache.org

–  Database
connec:on
pooling
services

•  h`ps://commons.apache.org/proper/commons-‐dbcp/

–  JDBC
helper

•  h`ps://commons.apache.org/proper/commons-‐dbu:ls/

•  Simple
Logging
Facade
for
Java
(SLF4J)

–  h`p://www.slf4j.org

2015
-‐
2016
52

Software testing: an introduction - 2015

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Software testing: an introduction - 2015

Similar a Software testing: an introduction - 2015 (20)

Más de XavierDevroey

Más de XavierDevroey (9)

Último

Último (20)

Software testing: an introduction - 2015