SpringOne 2021 Session Title: Prod-Like Integration Testing for Distributed Containerized Applications Speakers: Cora Iberkleid, Developer Advocate at VMware; Maria Gabriella Brodi, Staff Solutions Engineer at VMware

1. Prod-Like Integration Testing for
Distributed Containerized Applications
Maria Gabriella Brodi ⸱ @BrodiMg
Staff Solutions Engineer, VMware
⸺
Cora Iberkleid ⸱ @ciberkleid
Developer Advocate, VMware

2. Abstract
Integration testing for distributed containerized applications poses new challenges in terms of
practices, tools, and environments for developers who want to carry out more prod-like
integration testing earlier in the development lifecycle.
In-memory databases are useful but cannot provide the level of assurance needed as you test
against a real database. This option is also limited to data services, but not all services provide
an in-memory alternative.
Testcontainers is a framework for instantiating standalone containers for any number of
services to test against. The framework offers some out-of-the-box options, but you can also
provide your own image, and even your own Dockerfile to instantiate the service of your choice.
In this talk, we'll explore testcontainers and push the boundaries in order to explore how they
may be used in conjunction with Cloud Native Buildpacks. This approach has the added benefit
of ensuring that all testing is carried out on the same container stack.

3. Agenda
- Intro
- Testcontainers 101
- DB integration testing
- Network affinity testing
- Environment parity
- Takeaways

4. INTRO

5. What do we mean by “prod-like” integration testing?
● Integration vs Unit tests
● Greater fidelity to run-time conditions
● Type of system
● Network reliability
● OS environment
● Shifted left
● Local developer machine
● Iterate locally

6. Common Approaches to Integration Test
In Integration tests we are interested in verifying the behavior and interactions of
multiple modules.
For this purpose we can use:
- Shared instances
- Local installation
- In memory solutions

7. In-memory testing
App
Mock server or
in-memory service
Options:
● Mock server (Wiremock, Loki…)
● In-memory service (h2, hsql...)

8. Limitations of in-memory testing
App
Mock server or
in-memory service
Limitations:
● Requires framework support
● Behavior differences to real system
● Latency/bandwidth testing can be challenging
● Not every service has an in-memory option
● Differences in security configuration

9. New challenges with in-memory testing
App
Challenge exacerbated with explosion in microservices
and variety of service options over the last 10 years.
Mock server or
in-memory service
Limitations:
● Requires framework support
● Behavior differences to real system
● Latency/bandwidth testing can be challenging
● Not every service has an in-memory option
● Differences in security configuration

10. New challenges with in-memory testing
App
Challenge exacerbated with explosion in microservices
and variety of service options over the last 10 years.
Mock server or
in-memory service
Limitations:
● Requires framework support
● Behavior differences to real system
● Latency/bandwidth testing can be challenging
● Not every service has an in-memory option
● Differences in security configuration

11. Containerization Helps!
● Containerization per se is a big part of the solution
● Docker Compose - works for any framework
However…
● Cumbersome lifecycle management
● Additional skill set for developers to learn
However!
● There is an easier solution :)

12. Testcontainers: easy testing with external services
Service on Docker
App
Testcontainers
dependency

13. TESTCONTAINERS 101

14. Testcontainers intro/overview
● Java library that creates instances of Docker containers for testing
● Supports JUnit4, JUnit5, and Spock

15. Support for other languages
(check GitHub for more complete information)

16. Available Modules
● Provides lightweight,
throwaway instances
of common
databases, Selenium
web browsers, and
more
● Can start anything
else that can run in a
Docker container

17. Why Testcontainers? Where to use them?
● Prod-like systems:
● Easier instantiation of disposable services that lack an in-memory option
● Test against any application that can run as a container
● Integration testing:
● Data access layer integration
● Application integration
● Browser-based acceptance

18. inspiration...

19. package org.testcontainers.junit.jupiter;
import ...
class TestcontainersExtension implements BeforeEachCallback, BeforeAllCallback,
AfterEachCallback, AfterAllCallback, ExecutionCondition, TestInstancePostProcessor {
@Testcontainers (JUnit 5)
JUnit5 Extension - intercepts JUnit lifecycle
events and manages container lifecycles
@Testcontainers
public class MyIntegrationTests {
@Container // In JUnit 4, use @Rule/@ClassRule
static PostgreSQLContainer<?> db = new PostgreSQLContainer<>("postgres");
...
@ExtendWith({TestcontainersExtension.class})

20. @Container (JUnit 5) / @Rule (JUnit 4)
@Testcontainers
public class MyIntegrationTests {
@Container // In JUnit 4, use @Rule/@ClassRule
PostgreSQLContainer<?> db = new PostgreSQLContainer<>("postgres");
...
Flags TestcontainersExtension about a container to manage
● New container is started/stopped for every test
● Declare as static to re-use the same container

21. DEMO TIME
TESTCONTAINERS BASICS

22. “REAL” DATABASE
INTEGRATION TESTING

23. Close...but different!
...ish
Elephant by Anton from the Noun Project

24. DEMO TIME
DB INTEGRATION TESTING

25. NETWORK RELIABILITY

26. Which Challenges in a Real Environment?
Network can misbehave:
- Increase Latency
- Decrease Bandwidth
- Unexpected Closed Connections
- Changes in Packet Size
And Everything goes … BANANAS

27. Toxiproxy
A framework for simulating network conditions. It's
made specifically to work in testing, CI and
development environments.
https://github.com/Shopify/toxiproxy

28. Toxiproxy & Testcontainers
Toxiproxy available as a Testcontainers Module
Started as separate container on the Docker daemon
The toxiproxy container proxies all the traffics to the service container
Requires toxiproxy and service containers to be on the same network

29. Network Definition
Service on Docker
(e.g. postgres)
Toxiproxy
Network
We can facilitate connections between
containers without exposing ports on the
hosts by using a Network object
App testcontainers

30. Network Definition
Service on Docker
(e.g. postgres)
Toxiproxy
Network
App testcontainers
A maximum of one network can be shared
between containers.
Service on Docker
(e.g. redis)

31. DEMO
NETWORK RELIABILITY

32. ENVIRONMENT
AFFINITY

33. OS Base Layer
Path to Production - Environment Parity
CI Test Job CI Build Job
Build Layer 1
Build Layer N
OS = ?
Runtime= ?
OS = ?
Runtime= ?
OS = ?
Runtime= ?

34. Common Approaches
● Test and Build and Run environments often configured separately
○ Test job environment in CI toolchain
○ Build job environment in CI toolchain (separately, sometimes)
○ Run environment often configure in Dockerfile (‘FROM’ base image)
○ Very hard to keep synchronized
● Two-stage Dockerfiles help with specifying same build and run base images
○ Two ‘FROM’ statements in the same Dockerfile
○ Stil, hard to manage across applications at scale

35. A Better Way: Cloud Native Buildpacks
● Buildpacks provide a consistent way to build images at scale
● Build and run stacks guaranteed to be the same
● Distributed as a standalone “builder” image - easily shared across an
organization
● Polyglot
● Choice in user experience
○ CLI, Maven/Gradle plugin, Kubernetes operator, and more…

36. ● Default configuration of Maven Buildpack:
BP_MAVEN_BUILD_ARGUMENTS='-Dmaven.test.skip=true package'
Cloud Native Buildpacks (CNB)
CNB Run Image
CI Test Job CNB Builder
CNB BP Layer(s)
CNB App Layer(s)
OS = ✅
Runtime= ✅
OS = ?
Runtime= ?
OS = ✅
Runtime= ✅

37. ● Custom configuration:
BP_MAVEN_BUILD_ARGUMENTS='-Dmaven.test.skip=false package'
Cloud Native Buildpacks (CNB)
CNB Run Image
Test & Build!
CNB Builder CNB BP Layer(s)
CNB App Layer(s)
OS = ✅
Runtime= ✅
OS = ✅
Runtime= ✅

38. Test & Build!
CNB Builder
pack build my-springone-app
--builder paketobuildpacks/builder:base
--env BP_MAVEN_BUILD_ARGUMENTS='Dtest=Demo2_Toxiproxy_Tst test package'

39. Test & Build!
CNB Builder

40. mvn package
CNB Builder

41. testcontainers-java
CNB Builder
Hold your horses…
buildpacks can’t start
containers on Docker
Postgres
DOCKER_HOST = unix:///var/run/docker.sock

42. testcontainers-java
CNB Builder
Postgres
TCP
DOCKER_HOST = unix:///var/run/docker.sock

43. testcontainers-java
CNB Builder
Postgres
TCP
DOCKER_HOST = unix:///var/run/docker.sock
DOCKER_HOST = tcp://${DOCKER_HOST_IP}:${DOCKER_PORT}
Still respects security
guardrails of CNB ✅

44. Pack + Testcontainers Docker Host Detection
● We use a common utility (socat) to map the TCP port to the Unix socket
socat TCP-LISTEN:2375,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock &
pack build my-springone-app
--builder paketobuildpacks/builder:base
--env BP_MAVEN_BUILD_ARGUMENTS='Dtest=Demo2_Toxiproxy_Tst test package'
--env DOCKER_HOST=tcp://${DOCKER_HOST_IP}:2375
pkill socat

45. DEMO TIME
ENVIRONMENT AFFINITY

46. TAKEAWAYS

47. Get more “prod-like” integration testing using...
★ Testcontainers for improved system affinity
★ Toxiproxy for network resiliency testing
★ CNB with pack CLI and Paketo Buildpacks

48. Get more “prod-like” integration testing using...
★ Testcontainers for improved system affinity
★ Toxiproxy for network resiliency testing
★ CNB with pack CLI and Paketo Buildpacks
★ And… all from the comfort of your local machine!

49. Links
Demos:
https://github.com/springone-2021-testcontainers/testcontainers-demo
Testcontainers:
https://www.testcontainers.org
Toxiproxy:
https://github.com/Shopify/toxiproxy
Cloud Native Buildpacks:
https://buildpacks.io

50. Thank you!
Please join us on Slack
for 15 min of Q&A