Talk I gave at the Voxxed Days Thessaloniki 2018 conference

1. 1
Stream Processing with
Hazelcast Jet
Rafał Leszko
@RafalLeszko

2. © 2018 Hazelcast Inc. Confidential & Proprietary
Agenda
● Introduction to Jet
○ What is Hazelcast?
○ What is Stream Processing and Hazelcast Jet?
○ Example 1: Word Count
● Jet Under the Hood
○ How does it work?
○ Infinite Streams
○ Example 2: Twitter Cryptocurrency Analysis
● Jet Features & Use Cases
○ Jet Features
○ Why would I need it?
○ Example 3: Web Crawler

3. © 2018 Hazelcast Inc. Confidential & Proprietary
Introduction to Jet

4. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?

5. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?
Products:

6. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?
Products:

7. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast?
Products:
My Role:
● Cloud Software Engineer

8. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?

9. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?
DAG - Direct Acyclic Graph

10. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?

11. © 2018 Hazelcast Inc. Confidential & Proprietary
What is Hazelcast Jet?

12. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Problem:
Count the number of occurrences of each word in the given text.
Sample Input:
Lorem ipsum dolor, dolor.
Sample Output:
lorem=1
ipsum=1
dolor=2

13. © 2018 Hazelcast Inc. Confidential & Proprietary
Pure Java
Pattern delimiter = Pattern.compile("W+");
return lines.entrySet().stream()
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> Arrays.stream(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.collect(
groupingBy(
identity(),
counting()));
Example 1: Word Count

14. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

15. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

16. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

17. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

18. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

19. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Hazelcast Jet
Pattern delimiter = Pattern.compile("W+");
Pipeline pipeline = Pipeline.create();
pipeline.drawFrom(Sources.<Long, String>map(LINES))
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> traverseArray(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.groupingKey(wholeItem())
.aggregate(counting())
.drainTo(Sinks.map(COUNTS));
return pipeline;

20. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Pure Java
Pattern delimiter = Pattern.compile("W+");
return lines.entrySet().stream()
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> Arrays.stream(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.collect(
groupingBy(
identity(),
counting()));

21. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Hazelcast Jet
Pattern delimiter = Pattern.compile("W+");
Pipeline pipeline = Pipeline.create();
pipeline.drawFrom(Sources.<Long, String>map(LINES))
.map(e -> e.getValue().toLowerCase())
.flatMap(t -> traverseArray(delimiter.split(t)))
.filter(word -> !word.isEmpty())
.groupingKey(wholeItem())
.aggregate(counting())
.drainTo(Sinks.map(COUNTS));
return pipeline;

22. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

23. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count

24. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 1: Word Count
Demo:
https://github.com/hazelcast/hazelcast-jet-code-samples

25. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Under the Hood

26. © 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?

27. © 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?

28. © 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?

29. © 2018 Hazelcast Inc. Confidential & Proprietary
How does it work?
Under the Hood:
● Generate DAG representation from Pipeline
● Serialize DAG
● Send DAG to every Node
● Deserialize DAG
● Executes DAG on each Node

30. © 2018 Hazelcast Inc. Confidential & Proprietary
Infinite Streams

31. © 2018 Hazelcast Inc. Confidential & Proprietary
Infinite Streams
Examples:
● Currency Exchange Rates
● Tweets from Twitter
● Events in some Event-Based system
● ...

32. © 2018 Hazelcast Inc. Confidential & Proprietary
Infinite Streams
Windowing
pipeline.drawFrom(...)
.addTimestamps()
.window(sliding(30_000, 10_000))

33. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 2: Twitter Cryptocurrency Analysis
Problem:
Present in real-time the sentiments about cryptocurrencies
Input:
Tweets are streamed from Twitter and categorized by coin type
(BTC, ETC, XRP, etc)
Output:
Tweets sentiments (last 30 sec, last minute, last 5 minutes)

34. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 2: Twitter Cryptocurrency Analysis
Demo:
https://jet.hazelcast.org/demos/

35. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features & Use Cases

36. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features
Categories of Features
● Easy to Use
● Performance

37. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: Performance

38. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: Performance

39. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: other features

40. © 2018 Hazelcast Inc. Confidential & Proprietary
Jet Features: discovery

41. © 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?

42. © 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?
● Big Data Projects

43. © 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?
● Big Data Projects
● Speed up Everything

44. © 2018 Hazelcast Inc. Confidential & Proprietary
Why would I need it?
● Big Data Projects
● Speed up Everything

45. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 3: Web Crawler
Problem:
Parse all blog posts from the webpage
Input:
URL of Blog Trips
Output:
All the content from the Blog

46. © 2018 Hazelcast Inc. Confidential & Proprietary
Example 3: Web Crawler
Demo:
https://github.com/leszko/geodump

47. © 2018 Hazelcast Inc. Confidential & Proprietary
Thank You!