This document discusses a new technology called Streaming Ledger that provides ACID transaction guarantees for streaming data. Streaming Ledger allows applications to read and modify multiple data rows/keys within a transaction, sharing state between streams. It uses logical clocks to define a conflict-free processing schedule and handles key contention well. Streaming Ledger is built on Apache Flink and provides a new capability beyond Flink's existing exactly-once processing of single keys.

1. Stephan Ewen – data Artisans CTO, Apache Flink PMC
UNLOCKING THE NEXT WAVE
OF STREAMING APPLICATIONS

2. © 2018 data Artisans2
Streaming Applications over Time
Real time
Approximate
Analytics Fraud
Detection
Online
Machine Learning
Data pipelines
Realtime ETL
Intrusion
Prevention
Financial Risk &
Reporting
Real time
dashboards
Anomaly
Detection
Logistics
Tracking
Recommender
Systems
Web
Applications
Masterdata
Management
Continuous
Processing
Continuous Processing
and Analytics
Unification of
Analytics and Applications
Data-driven
Applications

3. © 2018 data Artisans3
Enablers of new Applications
Abstractions, APIs Consistency
Event-time, streaming SQL,
state & time, CEP
Exactly-once, savepoints
Scalability
high parallelism, large state
Interoperability
Deployment, Connectors, Operations

4. © 2018 data Artisans4
Enablers of new Applications
Abstractions, APIs Consistency
Event-time, streaming SQL,
state & time, CEP
Exactly-once, savepoints
Interoperability
Deployment, Connectors, Operations
Steam SQL,
Time-versioned Table Joins,
SQL+CEP, …
Framework and Library,
REST-ified Flink,
SQL Client, …Scalability
high parallelism, large state
Scalable timers,
dynamic scaling,
local recovery, …

5. © 2018 data Artisans5
Exactly-once Changed Applications
Stateless
Application
K/V Store
CRUD / request/response
Applications
Streaming
Application
State
Stateful Stream
Processing Applications
… become …

6. © 2018 data Artisans6
Some Applications don't move to Stream Processing
Application
Relational Database

7. © 2018 data Artisans7
Limitation of Current Stream Processors
Transferring money from one account (key) to another
with transactional guarantees is not feasible
The Limitation Example
All stream processors so far can
update a single key-at-a-time
with correctness guarantees
(exactly once)

8. © 2018 data Artisans8
Wouldn't it be great if Stream Processors could…
• … access and update state with multiple keys at the same time
• … maintain full isolation/correctness for the multi-key operations
• … operate on multiple states at the same time
• … share the states between multiple streams

9. © 2018 data Artisans9
TODAY WE ANNOUNCE
The first system for
serializable multi-row ACID transactions
on streaming data

10. © 2018 data Artisans10

11. © 2018 data Artisans11
‒ Atomicity: the transfer affects
either both accounts or none
‒ Consistency: the transfer must
only happen if the account have
sufficient funds
‒ Isolation: no other operation can
interfere and cause an incorrect
result
‒ Durability: the result of the
transfer is durable
ACID Transactions for Multi-key Stream Processing
Streaming Ledger provides ACID guarantees
across multiple states, rows, and streams

12. © 2018 data Artisans12
The Evolution of Stream Processing
accurate single-key applications
Flink pioneered exactly once guarantees on true streaming:
accurate analytics and applications, consistent single key at a time
At-least-once
guarantees
Exactly-once
guarantees
approximate real time analytics
No data loss but possible duplications to support real-time
approximate analytics in a "speed layer" (lamda)
ACID
guarantees
consistent general applications
Streaming Ledger provides ACID guarantees supporting
applications that read and modify several keys

13. © 2018 data Artisans13
Stateful Streaming vs. Streaming Ledger
Storage
Computing
Single datum /
key at a time
Consistent
total view
data Artisans
Streaming
Ledger
exactly-once
stateful
streaming
Key/Value
Stores
Relational
Databases

14. © 2018 data Artisans14
Tables, Streams, Transactions
Account Balance Asset Qty.
Deposit Function
Transfer Function
Xfer: A  B: $
Swap: X Y
Deposit: A: $
SUCCESS
A=$, B=$
Transaction Event Streams Transaction Functions Result Streams

15. © 2018 data Artisans15
Example: Transferring Cash/Assets between Accounts

16. © 2018 data Artisans16
Example: Position-keeping, Reporting, Risk Management
in Investment Banking

17. © 2018 data Artisans17
A Library on top of Apache Flink
• No additional dependencies needed
• Seamlessly integrates and composes with
DataStream API and SQL
• Read from- and write to all Flink connectors
• Supports savepoints for upgrades

18. © 2018 data Artisans18
How does it work?

19. © 2018 data Artisans19
Relational Database embedded? No!

20. © 2018 data Artisans20
First-of-a-kind - Unique Approach
Logical Clocks
to define schedule
event re-ordering &
out-of-order processing
Iterative streaming
dataflows
(0xf876ab78|0x0b7cc7a3)

21. © 2018 data Artisans21
Conflict-free Schedule
Txn: (A,B,C) -> A
Txn: (A,B,C) -> C
Txn: (D,E) -> (D,F)
transaction events
define ordering
for schedule
reorder events to obey schedule
B CA A
D E D F
A CB CA

22. © 2018 data Artisans22
Performance (early results) - Scalability
(parallelism)
200 million rows
100% update queries
4 rows written/query

23. © 2018 data Artisans23
Performance (early results) – Key Contention
Artificial case of
extreme contention:
4 x 200,000
= 800,000 updates/sec
on the same 1,000 keys
Slowdown, but does not
break like optimistic
concurrency approaches
100% update queries
4 written/query

24. © 2018 data Artisans24
Running a distributed setup
… and because we can…
…we run a globally
distributed setup

25. © 2018 data Artisans25

26. © 2018 data Artisans26
Apache Flink: The heavy lifter
This technology is possible, because Apache Flink offers such
powerful building blocks
• Continuous processing
• Iterative flows
• Flexible state abstraction
• Asynchronous checkpoints
• Sophisticated event-time/watermarks

27. © 2018 data Artisans27
Part of data Artisans Platform River Edition
Streaming Ledger is part of data Artisans Platform
• Stream Edition: Apache Flink, Application Manager
• River Edition: Apache Flink, Application Manager, Streaming Ledger
API and single-node implementation under Apache 2.0 license

28. © 2018 data Artisans28
Learn more at
At the booth

29. Thank you!
Enjoy the conference!