The presentation covers problems of high-performance reliable stream processing and associated time and synchronization related issues.

1. PATTERNS OF RELIABLE IN-STREAM
PROCESSING @ SCALE
Alexey Kharlamov

3. 50+ Billion events/day

4. 20+ Terabytes/day

5. Up to 1.5M event/second

6. 10+ Data Centers

7. Why did we decide to go real-Gme?

8. 300 servers
50 servers
λ
LAMBDA Architecture
Realtime
View
Historical
View
KAFKA CLUSTER
Query
Layer
HADOOP

10. Could we lower costs?
λ

11. κ
Aggregated
Data
KAFKA CLUSTER
Query
Layer
KAPPA Architecture

12. κ
Log
Checkpoint N - 1
Transaction N
Processing
failure
Rollback

13. κ
Log
Session 1
Session 2
Session 3
TX N-1 TX N TX N+1
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Session 1

14. κ
50 servers
50 servers
Aggregated
Data
KAFKA CLUSTER
Query
Layer
STATE STORAGE

16. All data transformaGons are
idempotent and state is volaGle.
κ

17. κ
Input Topic
State S1
State S2
Output
S1=F1(Input)
S2=F2(S1)
Output=F3(S2)

18. κ
Log
Checkpoint N - 1
State loss
position
Rollback

19. Are we all set?
κ

20. Oops! Something wrong with Gme!
κ

21. Event Time
Log
κ

22. Actually… More like that
κ
Log

23. In reality…
κ
Log

24. In reality…
κ
Log

25. Logical Gme is cumulaGve maximum of
observed event Gmestamps
κ

26. In reality…
κ
Log
Session Window

27. In reality…
κ
Log
Session Window

28. Log
Log
Log
Log
Log
C1 C2 C3 C4 C5
Rogue Broker
κ

29. Rogue Broker
κ
Log
LOST DATA

30. Limit logical Gme progress across
consumers
κ

31. Logical Gme progress control
κ
TXN
TXN
TXN
TXNTXN+1
TXN+1
TXN+1
TXN
Progress limit

32. TXN
TXN
TXN
TXNTXN+1
TXN+1
TXN+1
TXN
Progress limit
Logical Gme progress control
κ

33. Global synchronizaGon

36. Assume data distributed evenly

37. Zipf distribuGon

38. Hardware maYers!

39. QuesGons?

40. THANK YOU!
@aih1013
alexey@kharlamov.biz

41. Logical Time (watermark)
κ
Log