Slides for the talk given on 20-07-2019 at Nairobi JVM. It was a talk about building data pipelines with Apache Kafka as a message broker or enterprise bus and Apache spark as a distributed computing engine that enables processing of large volume of data efficiently.

1. BUILDING SCALABLE DATA PIPELINES
WITH KAFKA AND SPARK

2. WHAT IS A DATA PIPELINE?

3. Who am i?
Babatunde Ekemode
Product & Data Engineer at Africastalking Ltd
@babatush_ babatundeekmode

4. WHAT IS A DATA PIPELINE?
Simply put, A data pipeline facilitates the movement and
optional transformation of data from one point to another.

5. DATA PIPELINE ORG

6. DATA PIPELINE ORG
CAPTURE DATAFEEDBACK
ANALYZE DATAPLAN AND DECIDE
DERIVE ACTIONABLE
INSIGHTS
DATA DRIVEN
ORG.

7. Role of data pipeline in data driven org
E- COMMERCE
ACCOUNTING
CRM
PM
DRUID.IO
HADOOP
BIGQUERY
QUERIES
VISUALIZE
OLAP
OLTP

8. USE CASES
Cloud Migration
Data Backup
Data Processing

9. 7 Vs of Big data
● Volume
● Velocity
● Veracity
● Variety
● Variability
● Visualization
● Value

10. What could go wrong
● Data loss
● Data duplication
● Data corruption
● Data Formatting
● Latency
● Process Failure
● Flow control
● Data velocity
● Data volume

11. DATA PIPELINE GUARANTEES
● Scalability
● Reliability
● Durability
● Traceability
● Timeliness
● Security
● ....

12. DATA PIPELINE ARCHITECTURE
SOURCE INGESTION PROCESSING STORAGE VISUALIZATION

13. Apache kafka in ARCHITECTURE
SOURCE INGESTION PROCESSING STORAGE VISUALIZATION
Apache Kafka is a distributed streaming platform capable of handling trillions of
events a day. Initially conceived as messaging system.

14. Apache kafka
Kafka guarantees the following 3 capabilities of streaming systems:
● Publish and subscribe to streams of records, similar to a message queue
or EMS.
● Store streams of records in a fault tolerant durable way.
● Process streams of records as they occur.
Kafka is generally used for 2 broad classes of applications:
● Building real-time streaming data pipelines that reliably get data
between systems or applications
● Building real-time streaming applications that transform or react to
the streams of data

15. APACHE KAFKA ARCHITECTURE

16. APACHE KAFKA ARCHITECTURE

17. APACHE KAFKA ARCHITECTURE

18. APACHE KAFKA ARCHITECTURE

19. KAFKA APIS
● Producer API: allows applications to publish a stream of records to one or more
kafka topics.
● Consumer API: allows applications to subscribe to one or more topics and
process the stream of records provided to them.
● Streams API: allows an application to act as a stream processor, consuming an
input stream from one or more topics and producing an output stream to one or
more output topics, effectively transforming the input streams to output
streams
● Connector API: allows building and running reusable producers or consumers that
connect Kafka topics to existing applications or data systems.
● AdminClient API: supports managing and inspecting topics, brokers, acls, and
other kafka objects.

20. KAFKA CLI TOOLS
● Topic creation: bin/kafka-topics.sh --create --zookeeper
localhost:2181 --replication-factor 1 --partitions 5 --topic test
● Publish to topic: bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic test
● Consume from topic: bin/kafka-console-consumer.sh --bootstrap-server
localhost:9092 --topic test --from-beginning

21. KAFKA API EXAMPLES

22. Apache SPARK in ARCHITECTURE
SOURCE INGESTION PROCESSING STORAGE VISUALIZATION
Apache Spark is a unified analytics engine for large-scale data processing. Its
design is focused on speed, ease of use, generality, and multiple environments
deployment. It provides support for the following languages: Scala, Java, Python, R
and SQL.

23. APACHE SPARK ARCHITECTURE

24. APACHE SPARK STACK

25. Apache spark programming abstractions
● Resilient Distributed Dataset
● DataFrame
● Dataset

26. APACHE SPARK EXAMPLES

27. Apache cassandra in ARCHITECTURE
SOURCE INGESTION PROCESSING STORAGE VISUALIZATION
Apache Cassandra is a distributed nosql database for managing large amounts of
structured data across many commodity servers, while providing highly available
service and no single point of failure.

28. APACHE CASSANDRA ARCHITECTURE

29. BEST PRACTICES
Data serialization
Data Evolution
Schema Registry
....