From ksqlDB with Love: Detecting 007’s presence with a dash of machine learning, Hans-Peter Grahsl, Technical Trainer at NETCONOMY https://www.meetup.com/Berlin-Apache-Kafka-Meetup-by-Confluent/events/271635420/

1. From ksqlDB with ♥ LOVE ♥
Detecting 007 with a
dash of machine learning

2. Hans-Peter Grahsl
‣ technical trainer
‣ independent engineer & consultant
‣ associate lecturer
‣ occasional conference speaker
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 2

3. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 3

4. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 4

5. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 5

6. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 6

7. Kafka's
streaming
SQL
engine
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 7

8. declarative
stream
processing
language
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 8

9. KSQL in a Nutshell
‣ ANSI SQL inspired
‣familiar syntax & semantics
‣concise & expressive
‣ built on top of Kafka Streams
‣ NO(!) coding skills required
‣ entry barrier? "none"
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 9

10. KSQL in a Nutshell
‣ usual suspects OOTB:
‣projections, filters
‣joins, aggregations
‣windowing
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 10

11. Query Examples
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 11

12. Criteria 1 - Learning Curve
‣ SQL == widespread & successful 4GL
‣ algorithms & data structures handled for us
‣ KSQL
‣ very easy to learn
‣ quick implementation cycles
‣ productivity-wise hard to beat
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 12

13. "Lingua Franca" effect
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 13

14. Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 14

15. Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 15

16. Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 16

17. Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 17

18. Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 18

19. NOT the ETL
of our ancestors
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 19

20. Complex Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 20

21. KSQL ➔ originally the [T] only
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 21

22. What if I told you it's as
easy to build a
STREAMING app
as it is to build a
CRUD app?
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 22

23. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 23

24. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 24

25. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 25

26. Connecting with Sources
HINT: connector examples shown in demo later
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 26

27. Connecting with Sinks
HINT: connector examples shown in demo later
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 27

28. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 28

29. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 29

30. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 30

31. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 31

32. Unified Streaming Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 32

33. Instead, only try to realize
the truth... ksqlDB is
NO DATABASE
as we know it.
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 33

34. Data Concepts in ksqlDB
STREAM
‣ immutable append-only sequence
‣ captures events representing a series of facts
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 34

35. Data Concepts in ksqlDB
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 35

36. Data Concepts in ksqlDB
TABLE
‣ mutable collection of events
‣ holds the last known value for each key
‣ also result from stateful operations
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 36

37. Data Concepts in ksqlDB
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 37

38. Data Concepts in ksqlDB
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 38

39. ksqlDB: PUSH Queries
‣ act as subscription to query results
‣ fit asynchronous & reactive data flows
‣ run indefinitely
‣ new data causes continuous updates
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 39

40. ksqlDB: PUSH Query
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 40

41. ksqlDB: PULL Queries
‣ fetch point-in-time results
‣ fit request / response data flows
‣ terminate immediately
‣ lookup current state of materialized views
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 41

42. ksqlDB: PULL Query
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 42

43. ksqlDB Functions
‣ choose from three categories
‣ scalar ➔ UDF
‣ aggregation ➔ UDAF
‣ table ➔ UDTF
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 43

44. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 44

45. Criteria 2: Extensibility Options
‣ custom functions (UDFs, UDFAs & UDTFs)
‣ enable flexbile & powerful capabilities
‣ but Java code needed
HINT: custom UDF example shown in demo later
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 45

46. Fictional
Use Case
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 46

47. ksqlDB Use Case
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 47

48. ksqlDB Use Case
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 48

49. ksqlDB Use Case
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 49

50. ksqlDB Use Case
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 50

51. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 51

52. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 52

53. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 53

54. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 54

55. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 55

56. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 56

57. Data Architecture
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 57

58. Criteria 3: ML Integration Paths
‣ call fully-managed ML services (external)
‣ run your own model server (co-located)
‣ package home-brewed model into UDF (embedded)
‣ completely separated: integrate ML results via Connectors
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 58

59. Everywhere...
ksqlDB runs everywhere
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 59

60. Criteria 4: Deployment Options
‣ run however you want
‣ bare metal, VMs, containers
‣ deploy wherever you need
‣ on-premises, private / public cloud, hybrid
‣ something fully-managed?
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 60

61. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 61

62. Unfortunately, no one can
be told what ksqlDB
is...You have to TRY IT
for yourselves!
https://ksqldb.io
@hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 62

63. @hpgrahsl | #ConfluentVUG #ksqlDB #ApacheKafka | 2020-07-16 63

64. reach out to me
@hpgrahsl