Introduction to Property-based testing

1. Property-based Testing
for Everyone
Dmitriy Morozov / Zoomdata Tech Lunch

2. Why is testing hard?

3. Example-based
Testing
• set up one input scenario
• run the code under test
• check the output
(and any other effects the code is supposed to have)

4. Imagine we needed to
implement a custom ADD
function

5. @Test
public void testSum() {
assertThat(sum(1, 2), equalTo(3));
}

6. private int sum(int a, int b) {
if (a == 1 && b == 2) {
return 3;
} else {
return 0;
}
}

7. @Test
public void testSum() {
assertThat(sum(2, 2), equalTo(4));
}

8. private int sum(int a, int b) {
if (a == 1 && b == 2) {
return 3;
} else if (a == 2 && b == 2) {
return 4;
} else {
return 0;
}
}

9. Enterprise Developer

10. @Test
public void testSum3() {
// When I add two numbers I expect their sum
for (int i = 0; i < 100; i++) {
int x = randomInt();
int y = randomInt();
assertThat(sum(x, y), equalTo(x + y));
}
}

11. @Test
public void testSum3() {
// When I add two numbers I expect their sum
for (int i = 0; i < 100; i++) {
int x = randomInt();
int y = randomInt();
assertThat(sum(x, y), equalTo(x + y));
}
}

12. @Test
public void testSumUsingCommutativityProperty() {
// Changing the order of arguments
// shouldn’t change their sum
for (int i = 0; i < 100; i++) {
int x = randomInt();
int y = randomInt();
assertThat(sum(x, y), equalTo(sum(y, x)));
}
}

13. // unfortunately that's not enough
private int sum(int a, int b) {
return a * b;
}

14. @Test
// `x + 2` is the same as `x + 1 + 1`
public void testSumWithAdditiveProperty() {
for (int i = 0; i < 100; i++) {
int x = randomInt();
int result1 = sum(sum(x , 1), 1);
int result2 = sum(x, 2);
assertEquals(result1, result2);
}
}

15. // WTF?!
private int sum(int a, int b) {
return 0;
}

16. @Test
// `x + 1` = `x`
public void testSumWithIdentityProperty() {
for (int i = 0; i < 100; i++) {
int x = randomInt();
assertEquals(x, sum(x, 0));
}
}

17. Testing Specification with properties
• Commutativity property
• Associativity property
• Identity property

18. Testing Specification with properties
By using specifications, we have understood the
requirements in a deeper way

19. QuickCheck

21. QuickCheck in a nutshell

22. John Hughes - Testing the Hard Stuff and Staying Sane

24. Benefits
• Less time spent writing test code
- One property replaces many tests
• Better testing
- Lots of combinations you’d never test by hand
• Less time spent on diagnosis
- Failures minimized automatically

25. 3,000 pages of specifications
20,000 lines of QuickCheck
1,000,000 LoC from 6 suppliers
200 bugs
100 bugs in the standard

26. junit-quickcheck

27. @RunWith(JUnitQuickcheck.class)
public class AdditionPropertyTest {
@Property
public void commutativityProperty(int x, int y) {
assertThat(sum(x, y), equalTo(sum(y, x)));
}
@Property
public void additiveProperty(int x) {
int result1 = sum(sum(x , 1), 1);
int result2 = sum(x, 2);
assertEquals(result1, result2); }
@Property
public void identityProperty(int x) {
assertEquals(x, sum(x, 0));
}
}

28. java.lang.AssertionError: Property identityProperty
falsified via shrinking: expected:<1> but was:<0>
Shrunken args: [1]
Original failure message: [expected:<98> but was:<0>]
Original args: [98]
Seeds: [7329045779044418918]

29. @RunWith(JUnitQuickcheck.class)
public class MoneyChangerTest {
@Property
public void sumOfCoinsEqualsAmount(
@InRange(min = "0", max = "500") int amountToChange,
Set<@InRange(min = "1", max = "100") Integer> denominations) {
assumeThat(denominations.size(), greaterThan(0));
denominations.add(1);
MoneyChanger moneyChanger = new MoneyChanger();
List<Integer> coins = moneyChanger.change(amountToChange, denominations);
int sum = coins.stream().mapToInt(coin -> coin).sum();
assertThat(sum, equalTo(amountToChange));
}
}

30. JSVerify
npm install jsverify

31. describe("sort", function () {
jsc.property(
"idempotent", "array nat", function (arr) {
return _.isEqual(sort(sort(arr)), sort(arr));
});
});

32. How to handle state?

33. How to handle state?
• Generate actions changing the state
• Apply each action to the state, if possible
• After each application, verify the model

34. Where can we benefit
from
property-based tests?

35. Spark Proxy on DataFrame
API
• Generate random DataReadRequest
• Execute request with RDD-based code
• Execute same request with DataFrame based code
• Compare the results

36. SDK
• Send START_VIS
• Generate random request, change metric, group-
by, filter, sort, etc
• Validate responses based on the test model
• Send STOP_VIS

37. • QuickCheck
• Testing Telecoms Software with Quviq QuickCheck
• Testing the hard stuff and staying sane
• The Mysteries of Dropbox
• Jepsen IV: Hope Springs Eternal
• Java Examples from this talk