At kaChing (www.kaching.com), we are on a 5-minute commit-to-production cycle. We have adopted continuous deployment as a way of life and as the natural next step to continuous integration. In this talk, I will present how we achieved the core of our extreme iteration cycles: test-driven development or how to automate quality assurance. We will start at a very high level and look at the two fundamental aspects of software: transformations, which are stateless data operations, and interactions, which deal with state (such as a database, or an e-mail server). With this background we will delve into practical matters and survey kaChing's testing infrastructure by motivating each category of tests with different kind of problems often encountered. Finally, we will look at software patterns that lend themselves to testing and achieving separation of concerns allowing unparalleled software composability. This talk will focus on Java and the JVM even though the discussion will be largely applicable. Check out http://eng.kaching.com/search/label/tests for the latest from our company's blog.

1. Extreme Testing at kaChing From Commit to Production in 5 minutes Pascal-Louis Perez – @pascallouis Pascal-Louis Perez, kaChing Group Inc.

2. Engineering at kaChing TDD from day one Full regression suite runs in less than 3 minutes Deploy to production 30+ times a day People have written and launched new features during interview process Dedicated engineers to build testing frameworks Pascal-Louis Perez, kaChing Group Inc.

4. Testable Code Because… It allows quick iterations It makes it easier to scale the team It is more cost effective than debugging It obsoletes the need for functional QA It facilitates continuous refactoring, allowing the code to get better with age It attracts the right kind of engineers Pascal-Louis Perez, kaChing Group Inc.

5. High Level Classification Pascal-Louis Perez, kaChing Group Inc. Correctness Availability λ π

7. Bad: correct but unavailable systemλ π

9. Interactions (π): send an e-mail, store data in a db, RPC call

10. In π implicit global mutable stateλ π

12. Input produces an output: just assert

13. (Not to be confused with implementation complexity!)π

15. Timing tests, benchmarking, …π

18. End-to-end tests, regression test, performance lab(Find yourself a great architect!)

19. Think Outside the Box Pascal-Louis Perez, kaChing Group Inc. A A A A C C C C λ λ λ λ π π π π

20. Repository (or Client) Pattern Pascal-Louis Perez, kaChing Group Inc. @ImplementedBy(DbUserRepo.class)interfaceUserRepository { User get(Id<User> id);} UserRepository repo = ...; User pascal = repo.get(Id.<User> of(8)); A A C C λ λ π π classDbUserRepoimplementsUserRepository { // implementation}

21. Components Software composability is key to test-driven development Testable software abstracts the control flow; As such is functional by nature Pascal-Louis Perez, kaChing Group Inc.

22. Examples of Simple Tests Marshalling tests Dealing with time Equivalence tester Pascal-Louis Perez, kaChing Group Inc.

23. Marshalling tests Pascal-Louis Perez, kaChing Group Inc. C A  Fail fast Json.Object o = TwoLattes.createMarshaller(Trade.class).marshall(...); assertSameJson( object( string("id"), number(4), string("portfolioId"), NULL, string("trade"), string("interest"), string("date"), string("2008-07-06"), string("toDate"), string("2008-06-06"), string("fromDate"), string("2008-07-05"), string("longInterest"), number(56.43), string("shortInterest"), number(-0.81)), o); λ π

24. Marshalling tests Pascal-Louis Perez, kaChing Group Inc. C A Json.Object o = TwoLattes.createMarshaller(Trade.class).marshall(...); assertSameJson( object( string("id"), number(4), string("portfolioId"), NULL, string("trade"), string("interest"), string("date"), string("2008-07-06"), string("toDate"), string("2008-06-06"), string("fromDate"), string("2008-07-05"), string("longInterest"), number(56.43), string("shortInterest"), number(-0.81)), o);  Dedicated assertions λ π

25. Marshalling tests Pascal-Louis Perez, kaChing Group Inc. C A  Descriptive expectation Json.Object o = TwoLattes.createMarshaller(Trade.class).marshall(...); assertSameJson( object( string("id"), number(4), string("portfolioId"), NULL, string("trade"), string("interest"), string("date"), string("2008-07-06"), string("toDate"), string("2008-06-06"), string("fromDate"), string("2008-07-05"), string("longInterest"), number(56.43), string("shortInterest"), number(-0.81)), o); λ π

26. Dealing with time Use clocks Provider<DateTime> Provider<LocalDate> Pass in now or today in business logic Pascal-Louis Perez, kaChing Group Inc. C A λ π

27. Time Logic Pascal-Louis Perez, kaChing Group Inc.  Time logic is easy to test @VisibleForTesting <T> T provideCurrentOrHistorical( DateTime date, Provider<T> current, Provider<T> historical) { DateTimelastTradeDate = getLastTradeDate(); booleanisCurrent = date == null || lastTradeDate == null || date.compareTo(lastTradeDate) > 0; returnisCurrent ? current.get() : historical.get(); }

28. Time Logic Pascal-Louis Perez, kaChing Group Inc.  Higher-order functions @VisibleForTesting <T> T provideCurrentOrHistorical( DateTime date, Provider<T> current, Provider<T> historical) { DateTimelastTradeDate = getLastTradeDate(); booleanisCurrent = date == null || lastTradeDate == null || date.compareTo(lastTradeDate) > 0; returnisCurrent ? current.get() : historical.get(); }

29. Equivalence Tester Pascal-Louis Perez, kaChing Group Inc. C A Tests Reflexivity Symmetry Transitivity  hashCode consistency compareTo consistency  proper null handling EquivalenceTester.check( asList( newIsin("abCdEFgHiJK3“), newIsin("ABCDEFGHIJK3“), newIsin(“abCDefGHIJK3")), asList( newIsin("AU0000XVGZA3"), newIsin("AU0000XVGZA3"))); λ π

30. Rationale for Database Testing Accessing the database correctly O/R properly configured and used Hand written queries Optimistic concurrency Pascal-Louis Perez, kaChing Group Inc.

31. JDBC Pascal-Louis Perez, kaChing Group Inc.  Global static state DriverManager.registerDriver(neworg.gjt.mm.mysql.Driver()); Connection conn = DriverManager.getConnection("jdbc:..."); PreparedStatementst = conn.prepareStatement("select ..."); st.setString(1, "..."); ResultSetrs = st.executeQuery(); while (rs.next()) { // work } rs.close(); st.close(); conn.close();

32. JDBC Pascal-Louis Perez, kaChing Group Inc. DriverManager.registerDriver(neworg.gjt.mm.mysql.Driver()); Connection conn = DriverManager.getConnection("jdbc:..."); PreparedStatementst = conn.prepareStatement("select ..."); st.setString(1, "..."); ResultSetrs = st.executeQuery(); while (rs.next()) { // work } rs.close(); st.close(); conn.close();  Low level, stateful API, wide scoping

33. JDBC Pascal-Louis Perez, kaChing Group Inc. DriverManager.registerDriver(neworg.gjt.mm.mysql.Driver()); Connection conn = DriverManager.getConnection("jdbc:..."); PreparedStatementst = conn.prepareStatement("select ..."); st.setString(1, "..."); ResultSetrs = st.executeQuery(); while (rs.next()) { // work } rs.close(); st.close(); conn.close();  Resources hell

34. Hibernate Pascal-Louis Perez, kaChing Group Inc.  Global state SessionFactory factory = ...; Session session = factory.openSession(); Transaction tx = session.beginTransaction();List<User> users = session .createQuery("from User where ...") .list(); for (User user : users) { // work } tx.commit(); session.close();

35. Hibernate Pascal-Louis Perez, kaChing Group Inc.  High-level API SessionFactory factory = ...; Session session = factory.openSession(); Transaction tx = session.beginTransaction();List<User> users = session .createQuery("from User where ...") .list(); for (User user : users) { // work } tx.commit(); session.close();

36. Hibernate Pascal-Louis Perez, kaChing Group Inc. SessionFactory factory = ...; Session session = factory.openSession(); Transaction tx = session.beginTransaction();List<User> users = session .createQuery("from User where ...") .list(); for (User user : users) { // work } tx.commit(); session.close();  Resources hell

37. Transacter Pascal-Louis Perez, kaChing Group Inc.  Global state Transactertransacter = ...; transacter.execute(newWithSession() { publicvoid run(Session session) { List<User> users = ...; for (User user : users) { // work } } });

38. Transacter Pascal-Louis Perez, kaChing Group Inc.  High-level API, no resources Transactertransacter = ...; transacter.execute(newWithSession() { publicvoid run(Session session) { List<User> users = ...; for (User user : users) { // work } } });

39. Transacter Pascal-Louis Perez, kaChing Group Inc. Transactertransacter = ...; transacter.execute(newWithSession() { publicvoid run(Session session) { List<User> users = ...; for (User user : users) { // work } } });  Lexical scoping

40. Database Testing is Hard Complex to set up Schema installation Requires data to be loaded Developer must have proper environment Pascal-Louis Perez, kaChing Group Inc. : PersistenceTestRunnerü : O/R Mapping ü : Fixtures, DbUnitü : in-memory DBs (HSQLDB, Connector/MXJ) ü

41. PersistenceTestRunner Pascal-Louis Perez, kaChing Group Inc. C A  Declarative setup @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) publicclassExamplePersistentTest { @Test publicvoid example1(Transactertransacter) { // work } @Test publicvoid example2() { // work } } λ π

42. PersistenceTestRunner Pascal-Louis Perez, kaChing Group Inc. C A @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) publicclassExamplePersistentTest { @Test publicvoid example1(Transactertransacter) { // work } @Test publicvoid example2() { // work } }  Lexical scoping λ π

43. PersistenceTestRunner Pascal-Louis Perez, kaChing Group Inc. C A @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) publicclassExamplePersistentTest { @Test publicvoid example1(Transactertransacter) { // work } @Test publicvoid example2() { // work } } λ  RuntimeException π

44. PersistenceTestRunner Pascal-Louis Perez, kaChing Group Inc. C A @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) publicclassExamplePersistentTest { @Test publicvoid example1(Transactertransacter) { // work } @Test publicvoid example2() { // work } } λ π  Non-persistent tests run as usual

45. Testing Interactions (I/O, …) Implicit global state Many cases HTTP GET: succeed, fail to connect, connection closed before headers are read, … Sending e-mail: succeed, connection lost after HELO, … Sequencing is crucial Pascal-Louis Perez, kaChing Group Inc. C A λ π

46. Testing the Transacter (success) Pascal-Louis Perez, kaChing Group Inc. C A  Mocking finalWithSessionwithSession = mockery.mock(WithSession.class); mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); will(returnValue(session)); inSequence(execution); one(session).connection(); will(...); inSequence(...); one(connection).setReadOnly(with(equal(false))); ... one(connection).setAutoCommit(with(equal(false))); ... one(session).beginTransaction(); inSequence(...); ... }}); transacter.execute(withSession); mockery.assertIsSatisfied(); λ π

47. Testing the Transacter (success) Pascal-Louis Perez, kaChing Group Inc. C A finalWithSessionwithSession = mockery.mock(WithSession.class); mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); will(returnValue(session)); inSequence(execution); one(session).connection(); will(...); inSequence(...); one(connection).setReadOnly(with(equal(false))); ... one(connection).setAutoCommit(with(equal(false))); ... one(session).beginTransaction(); inSequence(...); ... }}); transacter.execute(withSession); mockery.assertIsSatisfied();  Interactions are scripted λ π

48. Testing the Transacter (failure) Pascal-Louis Perez, kaChing Group Inc.  Failure is controlled C mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); ... one(connection).setReadOnly(with(equal(false))); will(throwException(exception)); inSequence(execution); one(session).close(); inSequence(execution); }}); try { transacter.execute(withSession); fail(); } catch (RuntimeException e) { assertTrue(e == exception); }mockery.assertIsSatisfied(); A λ π

49. Testing the Transacter (failure) Pascal-Louis Perez, kaChing Group Inc. C mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); ... one(connection).setReadOnly(with(equal(false))); will(throwException(exception)); inSequence(execution); one(session).close(); inSequence(execution); }}); try { transacter.execute(withSession); fail(); } catch (RuntimeException e) { assertTrue(e == exception); }mockery.assertIsSatisfied(); A  Exact same object (correct bubbling) λ π

50. Testing the Transacter (failure) Pascal-Louis Perez, kaChing Group Inc. C mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); ... one(connection).setReadOnly(with(equal(false))); will(throwException(exception)); inSequence(execution); one(session).close(); inSequence(execution); }}); try { transacter.execute(withSession); fail(); } catch (RuntimeException e) { assertTrue(e == exception); }mockery.assertIsSatisfied(); A  Unreachable program point λ π

51. Meta Tests Cross domain tests to sweep the codebase, exactly like static analysis Catch common mistakes Distracted developers Bad APIs New hires learning conventions Pascal-Louis Perez, kaChing Group Inc.

52. Dependency Test Declare software dependencies Controls layering of application Simpler and more flexible than different compilation units Can guarantee that certain APIs are not used directly Patched Jdepend to parse annotations Pascal-Louis Perez, kaChing Group Inc. C A λ π

53. Dependency Test Pascal-Louis Perez, kaChing Group Inc. dependencies.forPackages( "com.kaching.*" ). check("com.kaching.supermap").mayDependOn( "com.kaching.platform.guice", "org.apache.commons.lang", "voldemort.*", "org.kohsuke.args4j" ). assertIsVerified(jDepend.getPackages());

54. Bad Code Test Enforce conventions Gotchas, bad APIs E.g. System.out.println E.g. java.net.URL Code reviews Design patterns, custom APIs, etc. Enforce conventions: automate this! Pascal-Louis Perez, kaChing Group Inc. C A λ π

55. Bad Code Test Pascal-Louis Perez, kaChing Group Inc. @RunWith(BadCodeSnippetsRunner.class) @CodeSnippets({ @Check(paths = "src/com/kaching", snippets = { // no uses of java.net.URL @Snippet(value = "bjava.net.URLb", exceptions = { "src/com/kaching/...", "src/com/kaching/..." }), // never call default super constructor @Snippet("super()") })})class MyBadCodeSnippetsTest {

56. Key Takeaways Invest in your tests, and testing frameworks Design patterns to separate concerns Components TDD pushes towards functional techniques We’re recruiting jobs@kaching.com More http://eng.kaching.com Pascal-Louis Perez, kaChing Group Inc.