1. The document summarizes the evolution of data architectures from siloed databases to streaming data pipelines using Apache Kafka. It notes how traditional architectures led to latency issues and tight coupling. 2. It introduces Apache Kafka as a distributed streaming platform that can decouple applications and enable low-latency data streams. Kafka allows building streaming data pipelines without Java coding. 3. KSQL is presented as an open source streaming SQL engine for Apache Kafka that allows processing streams of data in real-time with zero coding required, simply using configuration files and SQL.

1. 1° Meetup Kafka a Roma
01/02/18

2. https://www.meetup.com/it-IT/Roma-Kafka-meetup-group/

3. Big Data Streaming Analysis without code
STEFANO PAMPALONI
stefano.pampaloni@seacom.it
&
DARIO BALINZO
dariob@seacom.it

4. 2
Let’s take a trip back in time. Each application has its
own database for storing information. But we want
that information elsewhere for analytics and
reporting.

5. 3
We don't want to query the transactional system, so
we create a process to extract from the source to a
data warehouse / lake

6. 4
Let’s take a trip back in time
We want to unify data from multiple systems, so
create conformed dimensions and batch processes
to federate our data. This is all batch driven, so
latency is built in by design.

7. 5
Let’s take a trip back in time
As well as our data warehouse, we want to use our
transactional data to populate search replicas,
Graph databases, noSQL stores…all introducing
more point-to-point dependencies in our system

8. 6
Let’s take a trip back in time
Ultimately we end up with a spaghetti architecture. It
can't scale easily, it's tightly coupled, it's generally
batch-driven and we can't get data when we want it
where we want it.

9. 7
But…there's hope!

10. 8
Apache Kafka, a distributed streaming platform,
enables us to decouple all our applications creating
data from those utilising it. We can create low-
latency streams of data, transformed as necessary.

11. Before

12. After

13. Kafka concepts

14. 9
But…to use stream processing, we need to be Java
coders…don't we?

15. 7
But…there's hope!

16. 10
Happy days! We can actually build streaming data
pipelines using just our bare hands, configuration
files, and SQL.

23. KSQL
An Open Source Streaming SQL
Engine for Apache Kafka

24. • Enables stream processing with zero coding required
• The simplest way to process streams of data in real-time
• Powered by Kafka: scalable, distributed, battle-tested
• All you need is Kafka–No complex deployments