10. Power of automatic tests
• A mix of test – 15 000 of test cases
– 1h of Selenium tests
– Unit & integration tests
• 4.5h of testing
– Run in parallel on 30-40 machines
http://www.mmostation.com/
– Reduces execution time to 9 minutes
• All tests green? → deploy!
11. Tests written by developers
can be
EXTREMELY
POWERFUL
...if done right
12. Why should we care?
• Because we
– write more tests than production code
– use TDD
– are agile
– deploy continously
13. Why should we care?
• Because we
– write more tests than production code
– use TDD
– are agile
– deploy continously
We rely on our tests!
15. Software Quality - Tests
• External quality:
– Client perspective
– Really testing my application?
– Quality measure: bugs found
• Internal quality:
http://www.cypressmfg.com
– Developer perspective
– Is it easy to maintain them?
– Quality measure: development problems
16. A high quality test
• Ends green or red
• Fully automated
• Covers important functionality
• Is understandable
• Is simple (in terms of logic)
• Respects DRY principle
• Respects SRP principles
• Runs fast
18. Why code review tests?
• Learning http://wapedia.mobi/
– Is our API
convenient to use?
– Is our design
testable?
19. Why code review tests?
http://wapedia.mobi/
http://zuskin.com
• Learning • Verification of tests
– Is our API – Do they add value or
convenient to use? hinder
– Is our design modifications?
testable? – Are they easy to
understand?
25. Code Coverage
• Issues
http://www.ibm.com/developerworks
– Conditional statements
– When loop is covered?
– Threads ignored
• It is all about quantity,
– not quality!
• How much is good enough?
http://www.phpunit.de
– 100%
– green/red is bad!
27. Human nature... we do what we are
paid for
http://peureport.blogspot.com/
28. Human nature... we do what we are
paid for
Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Aliquam id nulla elit.
Phasellus turpis diam, dictum sed
venenatis luctus, dictum ac odio.
http://peureport.blogspot.com/
Aenean at nunc non libero posuere
aliquet.
29. Human nature... we do what we are
paid for
http://peureport.blogspot.com/
30. Human nature... we do what we are
paid for
Lorem ipsum?
- dolor sit amet consectetur
- adipiscing elit.
http://peureport.blogspot.com/
- Aliquam id nulla elit?
- Phasellus turpis diam
- dictum sed venenatis luctus?
- dictum ac odio!
Aenean at nunc non libero:
- posuere aliquet.
31. Human nature... we do what we are
paid for
http://peureport.blogspot.com/
32. Human nature... we do what we are
paid for
Lo-lo-lo-rem ipsu-su-su-su-
sum dolor sit amet
consectetur adipiscing eli-
li-li-li-lit. Aliquam id nulla
eli-li-li-li-lit. Pha-pha-pha-
http://peureport.blogspot.com/
pha-phasellus turpis diam,
dictum sed venenatis Colin Firth
luctus, dictum ac o-o-o-o-
o-o-dio. Aenean at nu-nu-
nu-nu-nu-nu-nunc non
libe-e-e-e-e-e-ro posuere
aliquet.
33. Human nature... we do what we are
paid for
If you can't measure
what you want to get,
you will only get
what you can measure.
34. Code Coverage will help, if:
• People are mature
• Code reviews are performed
• Coverage reports from all types of tests are
considered
...still, code coverage says little about the
quality of tests
36. Test parameters
• Even the simplest functions are impossible to
test exhaustively:
– public boolean whatever(int x)
• Tests for all three groups:
– Normal
– Boundary
– Strange
37. Many test parametrs – naive approach
public class StackTest {
@Test
public void pushTest() {
stack.push(1);
assertEquals(stack.peek(), 1);
stack.push(2);
assertEquals(stack.peek(), 2);
stack.push(3);
assertEquals(stack.peek(), 3);
stack.push(4);
assertEquals(stack.peek(), 4);
}
38. Many test parameters - JUnit
@RunWith(value = Parameterized.class)
public class StackTest {
Stack<Integer> stack;
private int number;
public StackTest(int number) { this.number = number; }
@Parameters
public static Collection data() {
return Arrays.asList(1, 2, 3, 4);
}
@Test
public void pushTest() {
stack.push(number);
assertEquals(stack.peek(), number);
}
39. Many test parameters - TestNG
@Test
public class StackTest {
Stack<Integer> stack;
@DataProvider
public Object[][] getNumbers() {
return new Object[][] {
{ 1 }, {2 }, {3 }, {4 }
};
}
@Test(dataProvider = „getNumbers”)
public void pushTest(int number) {
stack.push(number);
assertEquals(stack.peek(), number);
}
40. Many test parameters – outsource it :)
• Generate them randomly - JCheck
@RunWith(org.jcheck.runners.JCheckRunner.class)
class SimpleTest {
@Test public void simpleAdd(int i, int j) {
Money miCHF= new Money(i, "CHF");
Money mjCHF= new Money(j, "CHF");
Money expected= new Money(i+j, "CHF");
Money result= miCHF.add(mjCHF);
assertEquals(expected, result);
}
}
41. Many test parameters – outsource it :)
• Generate them randomly - QuickCheck
@Test
public void sortedListCreation() {
for (List<Integer> any : someLists(integers())) {
SortedList sortedList = new SortedList(any);
List<Integer> expected = ...
assertEquals(expected, sortedList.toList());
}
}
42. Combinations of test parameters
public void whatever(boolean a, boolean b, enum c)
• Number of test cases: 2 x 2 x enum.size()
• Rather cumbersome to test all combinations
43. Combinations of test parameters
• Pairwise testing (all pairs testing) to the rescue
• Observation:
– “most faults are caused by interactions of at
most two factors”
– Generation of test suite which covers all input
combinations of two and is therefore much
smaller than exhaustive testing.
47. Mutational Testing
• Mutations - examples
– Conditionals, a > b to !(a > b)
– Arithmetic operators + to -
– Switch statements
– Some language-specific e.g. public modifiers
• Issues
– Time! - creation of mutants, execution of tests
49. Make it readable. For everyone.
sel.open("/");
sel.type("Bugzilla_login", "admin");
sel.type("Bugzilla_password", "admin");
sel.click("log_in");
sel.waitForPageToLoad("30000");
http://www.eviltester.com/
sel.click("link=Reports");
sel.waitForPageToLoad("30000");
sel.click("link=My Requests");
sel.waitForPageToLoad("30000");
50. Make it readable. For everyone.
gotoHomePage();
loginAsAdmin();
gotoReportsPage();
http://www.eviltester.com/
gotoRequestsPage();
51. Make it readable. For everyone.
http://blog.spritecloud.com/
Scenario: Going to page Services
Given that I am on spriteCloud Home
When I click on link Services
Then the page title should be "spriteCloud
Services"
52. Make it readable. For everyone.
• Tools are mature & robust
– Fitnesse, Concordion, Cucumber, Twist
• People seems to be the problem!
53. External Quality of Tests
• Code coverage & mutational testing can help
• Test parameters are important
• DSLs can help
• No tools will make external quality happen
• Code review the tests!
• Reasonable people are crucial
http://www.ourdailyjourney.org/
55. Internal Quality of Tests
http://www.rightattitudes.com
• Managable
http://www.q8ieng.com
• Run fast or even faster
http://www.raysofhealinglight.com
• Helpful in bug fixing
62. Matchers – custom assertions
ServerSocketAssertion socket =
new ServerSocketAssertion(server.getSocket());
assertThat(socket).isConnectedTo(2000);
public class ServerSocketAssertion implements AssertExtension {
private final ServerSocket socket;
public ServerSocketAssertion(ServerSocket socket) {
this.socket = socket;
}
public ServerSocketAssert isConnectedTo(int port) {
assertThat(socket.isBound()).isTrue();
assertThat(socket.getLocalPort()).isEqualTo(port);
assertThat(socket.isClosed()).isFalse();
return this;
}
}
63. KISS
• No logic please
• Do nothing more than
– Create object
– Execute methods
– Verify results
http://robertstrongmarketing.org
• Avoid complex test dependencies
– Explicit and implicit
65. Assertion Messages
Before fixing the code,
make sure the
assertion message
clearly indicates the
problem.
• Assertion message should contain:
– information on what should have happened (but did not)
– arguments
67. TestNG logs output
• Don't miss the important information!
log4j.rootCategory=ALL, stdout, testLogFile
#only FATAL stuff (i.e. nothing) goes to standard output
log4j.appender.stdout.threshold=FATAL
...
#everything is logged to a file
log4j.appender.testLogFile=org.apache.log4j.FileAppender
log4j.appender.testLogFile.threshold=DEBUG
log4j.appender.testLogFile.File=target/testlogs.log
...
69. Static Code Analysis
• TestCase has no tests
• Proper spelling of setUp/tearDown
• Assertions always with message
• Use proper assertions (e.g. assertEquals instead of
assertTrue)
• Every test with assertion
• Test class must have some test cases
• Unnecessary boolean assertions
– e.g. assertFalse(true)
• The suite() method in a JUnit test needs to be both
public and static
70. Static Code Analysis
• Mostly Junit (3.8!)
• Many checks rendered obsolete by:
– Test-first coding
– Use of IDE http://zazzle.com
• Test are dead simple (in terms of language
constructs)
Can't tell you anything interesting
about real value/quality of your test
72. Make it fast
• Boost it:
– More CPU/RAM
– Create (costly) context once
– Write real unit tests
– Run tests in parallel
http://memegenerator.net
@Test(threadPoolSize = 3, invocationCount = 10,
timeOut = 10000)
public void testServer() {
73. Make it fast
• Make it fail fast
– Test-first
– First fast tests, then slow ones
– Pre-tested commit
– Use test dependencies wisely http://thumannresources.files.wordpress.com
– Use timeouts
74. Make it fast
• No Thread.sleep();
• No System.currentTimeMillis();
• Dependency injection
– Interface TimeSource http://images.icanhascheezburger.com
– Production code and tests uses different
implementations
76. Portability
• Problems:
– new File(„C:My Documentswhatever.txt”)
– DriverManager.
getConnection("jdbc:mysql://localhost:8080, ...);
• Not only Java code, but also test scripts
77. Portability
• Heuristic:
– Checkout on a clean system
– Run them
– ...and see what went wrong
• But start with
– other developers machines
– CI server
79. My main problem at the moment with unit tests is
when I change a design I get a stack of failing tests.
This means I'm either going to write less tests or make
fewer big design changes. Both of which are bad
things. Unicode Andy (cited from http://www.codinghorror.com/)
It’s overwhelmingly easy to write bad unit tests that
add very little value to a project while inflating the
cost of code changes astronomically.
Steve Sanderson
80. Why does it happen?
• Acceptance/Functional testing
– Testing via user interface
– Move to a higher abstraction level
• Unit tests
– With test doubles
– Price we pay for meddling in object's private
affairs
81. Red?! Yes!
Yes, some tests should
be failing if you change
your production code.
This is desirable.
82. Solutions (well, sort of...)
• Use „nice” mocks (or Mockito)
• Respect DRY principle
Think about what is really important. Is it really
important that this getter is called only once by
the tested method? No? Then record it as a stub. Do
I care about this passed parameter? No? Just use
the anyObject matcher. That seems silly but that’s
how you get resilient tests.
Henri Tremblay, EasyMock lead developer
84. Testing antipatterns
• http://blog.james-carr.org/2006/11/03/tdd-anti-patterns/
• The sloppy worker – A test that creates
persistent resources but doesn’t clean after
itself.
• The Mockery - [...] a unit test contains so many
mocks, stubs, and/or fakes that the system
under test isn’t even being tested at all,
instead data returned from mocks is what is
being tested.
• The Dodo – [...] tests behavior no longer
required in the application.
85. Testing antipatterns
• Many reasons to fail (SRP violation)
• Testing the same thing (DRY violation)
• Happy path testing
• Side effects (database change)
• Fixed values
– e.g. after saving user:
users = userDao.getAllUsers();
assertEquals(users.size(), 2);
87. Tautological tests
[...] instead of using the tests to
drive the domain model, the
developer ends up writing the
implementation code twice
Bill Six
88. Now it is worth testing
ClientDAO clientDAO;
void saveClient(Client client) {
if (client.something()) {
....
}
else {
clientDAO.save(client);
}
}
89. So what
should I
http://www.desicomments.com
do?
90. Quality - Same techniques
Production Code Tests
Make it readable Extract Method Extract Method
DLS DSL
Matchers
Verify it Tests Code Review
Code Review Code Coverage
Static Analysis Mutational Testing
DRY Code it once Test it once
SRP One reason to One reason to fail
change
KISS Limited responsibility, No logic, stick to
no awkward stuff common patterns
91. What should I do?
• Respect tests http://www.tefldaddy.com
• Use the right tools
• Stick to principles: DRY,
• Code review tests SRP, KISS
• Test on the right level • TDD rules!
• Choose test parameters • Examine coverage
well reports
• Make them portable • Be consistent
• Make them readable • Encourage people to
• Make them fast participate.
• Helpful for diagnostic