1. Resilient Data Pipelines with
Docker and Docker-
Compose
Heeren Sharma
Software Engineer @Cliqz
heeren@cliqz.com
@heerensharma

2. 80+ - Team size!
!
500,000 - DAU!
!
3 Million+ - Downloads (Germany only)!
!
1 billion+ - Indexed pages (We do not believe
in indexing the web.)!
!
5 TB - In-Memory indexed (Based on open
source and in-house build NoSQL stores.)

3. Storyfile
FROM Introduction
RUN Data-pipelines
RUN whats-docker
ADD use-case ./wrap-up
CMD ./demo

4. “Data and data everywhere”
“Design and development of Data Pipelines”
“Cloud deployment has its own needs”

5. One’s processed Data is another
system’s input data

6. What’s that whale named
Docker ?
"Docker allows you to package an application with all of its dependencies into a
standardised unit for software development."

7. A little bit more Docker
• Dockerfile
!
FROM ubuntu:14.04!
RUN pip install requests!
ADD . /awesome-code!
WORKDIR /awesome-code!
CMD python revolutionary_app.py
$ docker build -t=“your-awesome-image:v1” .
$ docker run -d -p 80:5000 —name container-name your-awesome-image:v1
$ docker push your-awesome-image:v1
$ docker pull myregistry.com:8080/your-awesome-image:v1
$ docker ps!
!
$ docker inspect <container-name/ID>!
!
$ docker log <container-name/ID>!
!
$ docker stop <container-name/ID>!
!
$ docker exec -it <container-name/ID> your_command!

8. Simplified Use Case
• Streaming data from different sources - Twitter,
FB, feeds and customised scraping engine.
• Different processing engines
• Fast iterations over new requirements
• Resilient system with focus over easy
deployment

9. Use case - News Articles
• Trending news articles from
different domains
• News Categorisation
• Relevant news over search
query
• Traffic of news content
fluctuates - pretty dynamic.
• There is no universal right
answer

10. System Design (1)
Data
Stream
Processing
Engine
Processed
Data
(Redis)
$ docker run -d —-name processed_data redismaster:v1 redis-server
$ docker build -t=“redismaster:v1” .
$ docker build -t=“datastream:twitter” .
$ docker run -d —-name twitter_queue datastream:twitter python /code/format_data.py
$ docker build -t —-name processing_engine .!
$ docker run -d —-name magic_powerhouse —-link processed_data:db processing_engine python /code/
magic_script.py

11. New State of design

12. System Design (2)
Data
Stream
Processing
Engine
Processed
Data
(Redis)
News
Fetcher
Requests

13. System Design (3)
Data
Stream
Processing
Engine
Processed
Data
(Redis)
FB
Engine
News
Fetcher
Requests

14. System Design (4)
Data
Stream
Processing
Engine
Processed
Data
(Redis)
FB
Engine
News
Fetcher
Requests
NewsLetters
Engine
3rd
Party

15. docker-compose to rescue
queue:!
build: .!
command: python /news-swimlane/twitter-queue/read_queue.py!
volumes:!
- /ebs/data:/data!
environment:!
DATA_DIR: /data!
PYTHONPATH: /news-swimlane!
restart: always!
update:!
build: .!
command: python /news-swimlane/server-redis/build_redis_exc.py!
volumes:!
- /ebs/data:/data!
links:!
- redismaster!
- fetcher!
environment:!
REDIS_HOST: redismaster_1!
DATA_DIR: /data!
PYTHONPATH: /news-swimlane!
restart: always!
redismaster:!
build: redis3-container!
ports:!
- "6379:6379"!
volumes:!
- /ebs/backup:/data!
restart: always!
command: /usr/local/bin/redis-server /master.conf!
fetcher:!
build: .!
command: python /news-swimlane/fetcher/server_fetcher.py!
ports:!
- "80:5000"!

16. Key points: Design &
Development
• Micro - Services oriented design and henceforth
development
• Old/new components can be realised in form of
docker containers
• Different containers can readily interact among
each other
• Ease to test (Local environment) and no worries if
it bursts out in production.

17. Deployment
• Just install docker on remote (to be done with
care)
• Docker images can be pushed to remote
repository (better say registry).
• Make your instances autoscale in any cloud IaaS.
• If instance go down, then new instance just pull
docker images, and start docker containers.

18. Resources
• Docker - https://www.docker.com/whatisdocker
• Boot2docker - https://docs.docker.com/installation/mac/
• Docker-compose - https://docs.docker.com/compose/
• Very short video about Docker (American Style) - https://
www.youtube.com/watch?v=aLipr7tTuA4 - American style
• https://www.youtube.com/watch?v=FdkNAjjO5yQ - Good
resource to have a little deep insight about Containers

19. http://www.cliqz.com/en
THANKQZ