# Testing at scale Genuine CI/CD can’t be accomplished without continous testing. At organizations like Walmart, where about 50k deployments happen every day, we need a robust and scalable test infrastructure to execute these tests. ## Concord We use Concord (http://concord.walmartlabs.com/) for deployments.It is an orchestration engine that connects different systems together using scenarios and plugins. ## Testing challenges The teams at Walmart excel at automated testing and with about 1M of CI jobs running 24X7 (many of them run end-to-end tests for an hour or two), we’re at risk of hoarding the resources on our CI/CD servers. We needed a flexible and scalable solution which can meet the needs of the teams who run long end-to-end tests. ## Solution We created a Concord plugin which will allow the users to leverage the Kubernetes infrastructure to run their end-to-end tests. Users can submit their Docker image for tests to Concord. It spins up a container called `Orchestra` which in turn spins up the jobs and suspends the Concord process. Once the jobs complete, it resumes the Concord process and users can continue with their flow. ## Advantages 1) Reliable and clean environment for testing 2) Scalable testing infrastructure with Kubernetes 3) Immutable and repeatable testing

1. Scalable Testing Infrastructure
with Concord and Kubernetes
Pranav Parikh

3. Our Mission
Save money. Live better.

4. Our Basic Beliefs
Strive for
Excellence
Respect the
Individual
Serve the
Customer
Act with
Integrity

5. Walmart Scale
11K
stores worldwide
265M
weekly customers
2.2M
associates

6. Concord
Process
Automation
Continuous
Deployment
Workflow
Orchestration

7. Architecture

8. Scalability
• Walmart Scale
• Mainframe, VM, Container, K8

9. Characteristics
• YAML based DSL
• Plugin architecture
• Process automation

10. Features
• Secrets Management
• Customizable forms
• Multi-tenant user interface
• LDAP integration
• PCI certified
• JSR scripting

11. Plugins
• Ansible
• Docker, Kubectl, Helm
• OneOps, CI
• LDAP, Email, Slack
• Git, JIRA

12. Supported
Workflows
• Whatever you define using
o DSL
o Scripting
o In built or custom plugins

13. Example
Workflows
• Infrastructure provisioning and
management
• CI/CD pipeline orchestration

14. Concord DSL Example

15. Testing

16. Test Pyramid
https://martinfowler.com/articles/practical-test-pyramid.html

17. ”Unit tests provide quality assurance to the engineer.
Functional tests provide quality assurance to the user”
– From an unknown Tweet

18. Challenges

19. “It works on my machine, why is it failing on CI?”

20. ”How can I run my tests in an isolated and
a clean environment?”

21. ”Tests take too long to finish!!”

22. ”We deployed our apps on Kubernetes. How do
we run our post-deploy tests?”

23. Solution
Concord E2E
Test plugin

24. Pre-requisites
• Docker images to run tests
• Authentication mechanism for K8
cluster
• Namespace created on the cluster

25. Overview
Pod
Container
BYOC Shard 1
Pod
Container
BYOC Shard 2
Pod
Container
BYOC Shard n
Pod
Container
orchestra
Concord Hook
Trigger
Docker Registry
Pull App Image
CI
Manual
Commit
Test log
Git Hook

26. Plugin Usage

27. Parameters

28. Parameters

29. Authentication

30. Sharding

31. Example

32. Logs & Reporting

33. Key Takeaways
Hooks for Github
and Concord
Clean environment
on Kubernetes
Shard jobs on
Kubernetes

34. Thanks!