This presentation explains parameterized tests, theory tests, and generative testing. It also explains single mode faults and double mode faults and shows how to reduce the number of test cases when there's an combinatorial explosion. Lot's of JUnit examples.

1. JDays 2015
AlexanderTarnowski
Dealing with combinatorial explosionsand
boring tests

2.  Developer (2000→) Java, Perl, C, C++, Groovy, C#, PHP,
Visual Basic, Assembler
 Trainer – TDD, Unit testing, Clean Code, WebDriver,
Specification by Example
 Developer mentor
 Writer
 Scrum Master
 Coach in training
WhoamI?
https://www.crisp.se/konsulter/alexander-tarnowski
alexander_tar
alexander.tarnowski@crisp.se

3. ”We want our
customers to be able
to compute their car
insurance premiums
online.
Online quotes are in
our favor, since we
outprice our
competitors!”
Who is Tim?
Image: stockimages/freedigitalphotos.net
Alexander Tarnowski

4. Age Premium for males Premium for females
18-23 1.75 1.575
24-59 1.0 0.9
60+ 1.35 1.215
Business rules
Alexander Tarnowski

5. @Test
public void maleDriversAged18() {
assertEquals(1.75, new PremiumRuleEngine()
.getPremiumFactor(18, Gender.MALE), 0.0);
}
The first test
Age Premium for males Premium for females
18-23 1.75 1.575
24-59 1.0 0.9
60+ 1.35 1.215
Alexander Tarnowski

6. @Test
public void maleDriversAged23() {
assertEquals(1.75, new PremiumRuleEngine()
.getPremiumFactor(23, Gender.MALE), 0.0);
}
The second test
Age Premium for males Premium for females
18-23 1.75 1.575
24-59 1.0 0.9
60+ 1.35 1.215
Alexander Tarnowski

7. And this could go on…
Image: imagerymajestic/freedigitalphotos.net
Alexander Tarnowski

8.  Silly names
 Repetitive test structure
 Boredom
Smells and insights
Image: Mister GC/freedigitalphotos.net
Alexander Tarnowski

9.  How many equivalence classes?
 How many boundary values?
 Would we test drive all of them?
How many tests are needed?
0
0.5
1
1.5
2
18-23 24-59 60+
Male
Female
Alexander Tarnowski

10. @RunWith(Parameterized.class)
public class PremiumAgeIntervalsTest {
private double expectedPremiumFactor;
private int age;
private Gender gender;
public PremiumAgeIntervalsTest(double expectedPremiumFactor, int age, Gender gender) {
this.expectedPremiumFactor = expectedPremiumFactor;
this.age = age;
this.gender = gender;
}
@Parameters
public static Collection<Object[]> data() {
return Arrays.asList(new Object[][]{
{1.75, 18, Gender.MALE}, {1.75, 23, Gender.MALE}, {1.0, 24, Gender.MALE},
{1.0, 59, Gender.MALE}, {1.35, 60, Gender.MALE}, {1.575, 18, Gender.FEMALE},
{1.575, 23, Gender.FEMALE}, {0.9, 24, Gender.FEMALE}, {0.9, 59, Gender.FEMALE},
{1.215, 60, Gender.FEMALE}}
);
}
@Test
public void verifyPremiumFactor() {
assertEquals(expectedPremiumFactor, new PremiumRuleEngine()
.getPremiumFactor(age, gender), 0.0);
}
}
Theparameterized version
Alexander Tarnowski

11. @RunWith(Parameterized.class)
public class PremiumAgeIntervalsTest {
@Parameter(value = 0)
public double expectedPremiumFactor;
@Parameter(value = 1)
public int age;
@Parameter(value = 2)
public Gender gender;
@Parameters(name = "Case {index}: Expected {0} for {1} year old {2}s")
public static Collection<Object[]> data() {
return Arrays.asList(new Object[][]{
{1.75, 18, Gender.MALE}, {1.75, 23, Gender.MALE}, {1.0, 24, Gender.MALE},
{1.0, 59, Gender.MALE}, {1.35, 60, Gender.MALE}, {1.575, 18, Gender.FEMALE},
{1.575, 23, Gender.FEMALE}, {0.9, 24, Gender.FEMALE}, {0.9, 59, Gender.FEMALE},
{1.215, 60, Gender.FEMALE}}
);
}
@Test
public void verifyPremiumFactor() {
assertEquals(expectedPremiumFactor, new PremiumRuleEngine()
.getPremiumFactor(age, gender), 0.0);
}
}
Alternative syntax
Alexander Tarnowski

12.  Focus on data
 Allow comparing a set of predefined inputs with
some predefined output
 Make checking simple
 Turn tests with silly names into data-driven tests
Parameterized tests
Alexander Tarnowski

13.  Interface
 Happy path
 Learning
 Error handling
TDD vsTesting
TDD Testing(checking)
 Equivalence partitions
 Boundary values
 Decision tables
 State machines
Alexander Tarnowski

14. @Test
public void aHamburgerIsnHealthyFood() {
assertThat(Menu.HAMBURGER.getCalories(),
greaterThan(200));
}
Speaking offood…
Alexander Tarnowski

15. Hamburgers contain the
least amount of calories
among common fast foods
Theory
Alexander Tarnowski

16. public class CalorieComparisonTest {
public static List<FastFood> foods() {
return Arrays.asList(Menu.FISH_BURGER,
Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH,
Menu.HOTDOG);
}
@Test
public void hamburgersContainTheLeastAmountOfCaloriesAmongFastFoods() {
for (FastFood food : foods())
assertThat(Menu.HAMBURGER.getCalories(),
is(lessThan(food.getCalories())));
}
}
}
Proving the theory
Alexander Tarnowski

17. @RunWith(Theories.class)
public class CalorieComparisonTest {
@DataPoints
public static List<FastFood> foods() {
return Arrays.asList(Menu.FISH_BURGER,
Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH,
Menu.HOTDOG);
}
@Theory
public void hamburgersContainTheLeastAmountOfCaloriesAmongFastFoods(FastFood food)
{
assertThat(Menu.HAMBURGER.getCalories(),
is(lessThan(food.getCalories())));
}
}
Atheorytest
Alexander Tarnowski

18.  No fast food meal contains
less than 500 calories!
Amore interesting theory
Image: marin/freedigitalphotos.net
Alexander Tarnowski

19. @RunWith(Theories.class)
public class FastFoodMealTheoryTest {
@DataPoints
public static List<Main> mainCourses() {
return Arrays.asList(Menu.HAMBURGER, Menu.FISH_BURGER,
Menu.GIGANTIC_BURGER_WITH_BACON, Menu.CHICKEN_SANDWICH,
Menu.HOTDOG);
}
@DataPoints
public static List<SideOrder> sideOrders() {
return Arrays.asList(Menu.SMALL_FRENCH_FRIES, Menu.LARGE_FRENCH_FRIES,
Menu.APPLE_PIE, Menu.SMALL_CHOCOLATE_MILKSHAKE);
}
@DataPoints
public static List<Beverage> bevereges() {
return Arrays.asList(Menu.MEDIUM_COKE, Menu.LARGE_DIET_COKE,
Menu.MEDIUM_LATTE, Menu.LARGE_LATTE);
}
@Theory
public void noFastFoodMealContainsLessThan500calories(Main main,
SideOrder sideOrder,
Beverage beverage) {
assumeThat(beverage.isDiet(), is(false));
assertThat(main.getCalories() + sideOrder.getCalories() + beverage.getCalories(),
is(greaterThan(500)));
}
}
Alexander Tarnowski

20.  Feed the test with all main courses and all side
orders and all beverages
 Cartesian product: mains X side orders X
beverages
 assumeThat prunes some combinations
Behind the scenes
(Hamburger, Small french fries, Medium coke)
(Hamburger, Small french fries, Large diet coke)
(Hamburger, Small french fries, Medium latte)
(Hamburger, Small french fries, Large latte)
(Hamburger, Large french fries, Medium coke)
(Hamburger, Large french fries, Large diet coke)
(Hamburger, Large french fries, Medium latte)
(Hamburger, Large french fries, Large latte)
…
Alexander Tarnowski

21.  Are built around ”for all” type of reasoning
 Can’t pair specific data points with specific results
 Let you work with the Cartesian product of
multiple variables
Theories
Alexander Tarnowski

22. Let’s do the Caesarcipher…
A B C D E F G H I J K L M O P Q R S T U V X Y Z
S T U V X Y Z A B C D E F G H I J K L M O P Q R
CAESAR=USXKSJ
Alexander Tarnowski

23. Does it work?
… by borrowing an onlineimplementation
Alexander Tarnowski

24.  For a bunch of different arbitrary strings…
 ... and a bunch of different offsets...
 ... try the following:
CaesarCipher.decode(CaesarCipher.encode(string, offset), offset)
Dream scenario
Alexander Tarnowski

25. @RunWith(Theories.class)
public class CaesarCipherTest {
@Theory
public void caesarCipherRoundTrip(@RandomString(maxLength = 128) String plainText,
@TestedOn(ints = {0, 1, 2, 10, 26, 27, 1000}) int offset) {
assertEquals(plainText, CaesarCipher.decode(CaesarCipher.encode(plainText, offset),
offset));
}
}
We can do that!
Alexander Tarnowski

26. RandomString.java:
@Retention(RetentionPolicy.RUNTIME)
@ParametersSuppliedBy(RandomStringSupplier.class)
public @interface RandomString {
int maxLength();
}
RandomStringSupplier.java:
public class RandomStringSupplier extends ParameterSupplier {
@Override
public List<PotentialAssignment> getValueSources(ParameterSignature signature)
throws Throwable {
RandomString annotation = signature.getAnnotation(RandomString.class);
int length = (int) (Math.random() * annotation.maxLength());
final String s = RandomStringUtils.randomAlphanumeric(length);
return Arrays.asList(PotentialAssignment.forValue("random string", s));
}
}
@RandomString
Alexander Tarnowski

27. RandomStringsSupplier.java:
public class RandomsStringSupplier extends ParameterSupplier {
@Override
public List<PotentialAssignment> getValueSources(ParameterSignature signature) throws
Throwable {
List<PotentialAssignment> values = new ArrayList<>();
RandomStrings annotation = signature.getAnnotation(RandomStrings.class);
Generator<String> stringGenerator
= strings(integers(1, 128, Distribution.INVERTED_NORMAL), characters());
for (int i = 0; i < annotation.count(); i++) {
values.add(PotentialAssignment.forValue("random string", stringGenerator.next()));
}
return values;
}
}
QuickCheck style
Alexander Tarnowski

28. Examples of generators
 booleans()
 dates(Long low, Long high, TimeUnit precision)
 fixedValues(T... values)
 strings(Generator<Integer> length, Generator<Character> characters)
 arrays(Generator<? extends T> content, Class<T> type)
 excludeValues(Generator<T> generator, T... excluded)
 sortedLists(Generator<T> content, int low, int high)
net.java.quickcheck
public interface Generator<T> {
/**
* Generates the next instance.
*
* @return a newly created instance
*/
public T next();
}
Alexander Tarnowski

29.  Anything goes!
 May involve huge domains
 Often involves inverse functions
Generative testing
Alexander Tarnowski

30.  Now we can execute thousands of tests!
 But what if we want the opposite?
Congratulations!
Alexander TarnowskiImage: zole4/freedigitalphotos.net

31. Back to car insurance premiums
Gender
Male
Female
Age interval
18-24
25-59
60+
Yearly mileage
0
1-1000
1001-3000
3001-6000
6001+
Safety features
None
Airbag
ABS
HIP
Multiple
Brand
Nissan
Volvo
Ferrari
Toyota
Ford
Volkswagen
Driving record
Model Driver
Average Joe
Unlucky Uma
Bad Judgement Jed
Dangerous Dan
2 x 3 x 5 x 5 x 6 x 5
= 4500
Alexander Tarnowski

32.  One parameter causes the error
 Only 6 tests are needed!
Single mode faults
Brand Drivingrecord Yearlymileage Safetyfeatures Ageinterval Gender
Nissan ModelDriver 0 None 18-24 Male
Volvo AverageJoe 1-1000 Airbag 25-59 Female
Ferrari UnluckyUma 1001-3000 ABS 60+ -
Toyota BadJudgementJed 3001-6000 HIP - -
Ford DangerousDan 6001+ Multiple - -
Volkswagen - - - - -
Alexander Tarnowski

33.  A combination of two parameters causes the error
 Run through a tool that computes all pairs (or look
up in a table of orthogonal arrays)
 Only ~40 tests are needed!
Double mode faults
Alexander Tarnowski

34. Finding all pairs by hand
Row Variable 1 Variable 2 Variable 3
1 A X Q
2 A X R
3 A Y Q
4 A Y R
5 B X Q
6 B X R
7 B Y Q
8 B Y R
Alexander Tarnowski

35.  Theoretical foundation: orthogonal arrays
 Reduce the number of tests from thousands to
just a few
 Great to put into parameterized tests
Finding Single and Doublemode faults
Alexander Tarnowski

36.  Unit tests are examples
 Parameterized tests make writing many similar
tests easy
 Theory tests introduce general statements about
program elements
 Generative tests – Anything goes!
 Single mode faults & double mode faults –
Reduce the number of tests and feed
parameterized tests
Summary
Alexander Tarnowski

37.  https://leanpub.com/developer_testing
 http://web.archive.org/web/20110808084654/http://shareandenjoy.saff.net/tdd-
specifications.pdf
 https://github.com/junit-team/junit/wiki/Theories
 https://github.com/pholser/junit-quickcheck
 http://www.satisfice.com/tools.shtml
Some resources
Alexander Tarnowski