Over the past year, we've seen users build entire event-driven applications such as social networks on top of Apache Flink (Drivetribe.com), elevating the importance of state management in Flink to a whole new level. Users are placing more and more data as state in Flink, using it as a replacement to conventional databases. With such mission-critical data entrusted to Flink, we need to provide similar capabilities of a database to users. One of such capabilities is being flexible in how data is persisted and represented. Specifically, how can I change how my state is serialized and stored, or even the schema of my state, as business logic changes over time? In this talk, we'll provide details on the latest state management features in Flink that allows users to do exactly that. We'll talk about how Flink manages the state for you, how it provides flexibility to the user to adapt to evolving state serialization formats and schemas, and the best practices when working with it.

1. Tzu-Li (Gordon) Tai
Flink Committer / PMC Member
tzulitai@apache.org
@tzulitai
Managing State in Apache Flink

2. Original creators of Apache
Flink®
Providers of
dA Platform 2, including
open source Apache Flink +
dA Application Manager

3. Users are placing more and more
of their most valuable assets within
Flink: their application data

4. @
Complete social network implemented
using event sourcing and
CQRS (Command Query Responsibility Segregation)

5. Classic tiered architecture Streaming architecture
database
layer
compute
layer
compute
+
stream storage
and
snapshot storage
(backup)
application
state

6. Streams and Snapshots
6
Version control
Exactly-once guarantees Job reconfiguration
Asynchronous, pipelined
State
Snapshots

7. … want to ensure that users can
fully entrust Flink with their state,
as well as good quality of life with
state management

8. State management FAQs ...
● State declaration best practices?
● How is my state serialized and persisted?
● Can I adapt how my state is serialized?
● Can I adapt the schema / data model of
my state?
8

9. Recap: Flink’s Managed State
9

10. Flink Managed State
10
...
...
Your
Code
Your
Code
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(
“my-value-state”,
MyPojo.class
);
ValueState<MyPojo> state =
getRuntimeContext().getState(desc);
MyPojo p = state.value();
...
state.update(...)
...

11. Flink Managed State
11
...
...
Your
Code
Your
Code
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(
“my-value-state”,
MyPojo.class
);
ValueState<MyPojo> state =
getRuntimeContext().getState(desc);
MyPojo p = state.value();
...
state.update(...)
...
unique identifier
for this managed
state

12. Flink Managed State
12
...
...
Your
Code
Your
Code
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(
“my-value-state”,
MyPojo.class
);
ValueState<MyPojo> state =
getRuntimeContext().getState(desc);
MyPojo p = state.value();
...
state.update(...)
...
type information
for the state

13. Flink Managed State
13
...
...
Your
Code
Your
Code
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(
“my-value-state”,
MyPojo.class
);
ValueState<MyPojo> state =
getRuntimeContext().getState(desc);
MyPojo p = state.value();
...
state.update(...)
...
Local
State Backend
registers
state

14. Flink Managed State
14
...
...
Your
Code
Your
Code
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(
“my-value-state”,
MyPojo.class
);
ValueState<MyPojo> state =
getRuntimeContext().getState(desc);
MyPojo p = state.value();
...
state.update(...)
...
Local
State Backend
read
write

15. Flink Managed State (II)
15
State
State
State
State
State
State
State
DFS
Checkpoints
- Pipelined
checkpoint barriers
flow through the
topology
- Operators
asynchronously
backup their state in
checkpoints on a
DFS
checkpointed state

16. Flink Managed State (II)
16
State
State
State
State
State
State
State
DFS
Restore
- Each operator is
assigned their
corresponding
file handles, from
which they
restore their state
checkpointed state

17. State Serialization Behaviours
17
State
State
State
State
State
State
State
DFS
JVM Heap backed
state backends
(MemoryStateBackend,
FsStateBackend)
⇒ lazy serialization +
eager deserialization
checkpointed state

18. State Serialization Behaviours
18
State
State
State
State
State
State
State
DFS
JVM Heap backed
state backends
(MemoryStateBackend,
FsStateBackend)
⇒ lazy serialization +
eager deserialization
Java object read / writes
checkpointed state

19. State Serialization Behaviours
19
State
State
State
State
State
State
State
DFS
JVM Heap backed
state backends
(MemoryStateBackend,
FsStateBackend)
⇒ lazy serialization +
eager deserialization
Java object read / writes
serialize
on chckpts /
savepoints
Checkpoint
checkpointed state

20. State Serialization Behaviours
20
State
State
State
State
State
State
State
DFS
JVM Heap backed
state backends
(MemoryStateBackend,
FsStateBackend)
⇒ lazy serialization +
eager deserialization
deserialize
to objects
on restore
Restore
checkpointed state

21. State Serialization Behaviours
21
State
State
State
State
State
State
State
DFS
Out-of-core state backends
(RocksDBStateBackend)
⇒ eager serialization +
lazy deserialization
checkpointed state

22. State Serialization Behaviours
22
State
State
State
State
State
State
State
DFS
Out-of-core state backends
(RocksDBStateBackend)
⇒ eager serialization +
lazy deserialization
De-/serialize on every
local read / write
checkpointed state

23. State Serialization Behaviours
23
State
State
State
State
State
State
State
DFS
Out-of-core state backends
(RocksDBStateBackend)
⇒ eager serialization +
lazy deserialization
De-/serialize on every
local read / write
exchange
of state bytes
Checkpoint
checkpointed state

24. State Serialization Behaviours
24
State
State
State
State
State
State
State
DFS
Out-of-core state backends
(RocksDBStateBackend)
⇒ eager serialization +
lazy deserialization
De-/serialize on every
local read / write
exchange
of state bytes
Restore
checkpointed state

25. State Declaration
25
Matters of State (I)

26. State Access Optimizations
26
ValueStateDescriptor<Map<String, MyPojo>> desc =
new ValueStateDescriptor<>(“my-value-state”, MyPojo.class);
ValueState<Map<String, MyPojo>> state =
getRuntimeContext().getState(desc);
● Flink supports different state “structures” for
efficient state access for various patterns
Map<String, MyPojo> map = state.value();
map.put(“someKey”, new MyPojo(...));
state.update(map);
X don’t do

27. State Access Optimizations
27
MapStateDescriptor<String, MyPojo> desc =
new MapStateDescriptor<>(“my-value-state”, String.class, MyPojo.class);
MapState<String, MyPojo> state =
getRuntimeContext().getMapState(desc);
● Flink supports different state “structures” for
efficient state access for various patterns
MyPojo pojoVal = state.get(“someKey”);
state.put(“someKey”, new MyPojo(...));
optimized for
random key access

28. Declaration Timeliness
28
● Try registering state as soon as possible
• Typically, all state declaration can be done in the “open()”
lifecycle of operators

29. Declaration Timeliness
29
public class MyStatefulMapFunction extends RichMapFunction<String, String> {
private static ValueStateDescriptor<MyPojo> DESC =
new ValueStateDescriptor<>(“my-pojo-state”, MyPojo.class);
@Override
public String map(String input) {
MyPojo pojo = getRuntimeContext().getState(DESC).value();
…
}
}

30. Declaration Timeliness
30
public class MyStatefulMapFunction extends RichMapFunction<String, String> {
private static ValueStateDescriptor<MyPojo> DESC =
new ValueStateDescriptor<>(“my-pojo-state”, MyPojo.class);
private ValueState<MyPojo> state;
@Override
public void open(Configuration config) {
state = getRuntimeContext().getState(DESC);
}
@Override
public String map(String input) {
MyPojo pojo = state.value();
…
}
}

31. Eager State Declaration
31
● Under discussion with FLIP-22
● Allows the JobManager to have knowledge on
declared states of a job
(under discussion,
release TBD)

32. Eager State Declaration
32
(under discussion,
release TBD)
public class MyStatefulMapFunction extends RichMapFunction<String, String> {
@KeyedState(
stateId = “my-pojo-state”,
queryableStateName = “state-query-handle”
)
private MapState<String, MyPojo> state;
@Override
public String map(String input) {
MyPojo pojo = state.get(“someKey”);
state.put(“someKey”, new MyPojo(...));
}
}

33. State Serialization
33
Matters of State (II)

34. How is my state serialized?
34
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(“my-value-state”, MyPojo.class);
● Provided state type class info is analyzed by Flink’s own serialization
stack, producing a tailored, efficient serializer
● Supported types:
• Primitive types
• Tuples, Scala Case Classes
• POJOs (plain old Java objects)
● All non-supported types fallback to using Kryo for serialization

35. Avoid Kryo for State Serde
35
● Kryo is generally not recommended for use on persisted data
• Unstable binary formats
• Unfriendly for data model changes to the state
● Serialization frameworks with schema evolution support is
recommended: Avro, Thrift, etc.
● Register custom Kryo serializers for your Flink job that uses these
frameworks

36. Avoid Kryo for State Serde
36
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// register the serializer included with
// Apache Thrift as the standard serializer for your type
env.registerTypeWithKryoSerializer(MyCustomType, TBaseSerializer.class);
● Also see: https://goo.gl/oU9NxJ

37. Custom State Serialization
37
● Instead of supplying type information to be analyzed, directly
provide a TypeSerializer
public class MyCustomTypeSerializer extends TypeSerializer<MyCustomType> {
...
}
ValueStateDescriptor<MyCustomType> desc =
new ValueStateDescriptor<>(“my-value-state”, new MyCustomTypeSerializer());

38. State Migration
38
Matters of State (III)

39. State migration / evolution
39
● Upgrading to more efficient serialization schemes for performance
improvements
● Changing the schema / data model of state types, due to evolving
business logic

40. Upgrading State Serializers
40
● The new upgraded serializer would either be compatible, or not
compatible. If incompatible, state migration is required.
● Disclaimer: as of Flink 1.3, upgraded serializers must be compatible, as
state migration is not yet an available feature.
ValueStateDescriptor<MyCustomType> desc =
new ValueStateDescriptor<>(“my-value-state”, new UpgradedSerializer<MyCustomType>());
ValueStateDescriptor<MyPojo> desc =
new ValueStateDescriptor<>(“my-value-state”, MyPojo.class); // modified MyPojo type
Case #2: New custom serializer
Case #1: Modified state types, resulting in different Flink-generated serializers

41. Upgrading State Serializers (II)
41
● On restore, new serializers are checked against the metadata of the
previous serializer (stored together with state data in savepoints) for
compatibility.
● All Flink-generated serializers define the metadata to be persisted. Newly
generated serializers at restore time are reconfigured to be compatible.Savepoint
state
data
serializer
config
metadataField “A”
Field “B”
Old serializer at time
of savepoint
Field “B”
Field “A”
New serializer at
restore time

42. Upgrading State Serializers (II)
42
● On restore, new serializers are checked against the metadata of the
previous serializer (stored together with state data in savepoints) for
compatibility.
● All Flink-generated serializers define the metadata to be persisted. Newly
generated serializers at restore time are reconfigured to be compatible.Savepoint
state
data
serializer
config
metadataField “A”
Field “B”
Old serializer at time
of savepoint
Field “B”
Field “A”
New serializer at
restore time
reconfigure

43. Upgrading State Serializers (II)
43
● On restore, new serializers are checked against the metadata of the
previous serializer (stored together with state data in savepoints) for
compatibility.
● All Flink-generated serializers define the metadata to be persisted. Newly
generated serializers at restore time are reconfigured to be compatible.Savepoint
state
data
serializer
config
metadataField “A”
Field “B”
Old serializer at time
of savepoint
Field “A”
Field “B”
New serializer at
restore time

44. Upgrading State Serializers (III)
44
● Custom serializers must define the metadata to write:
TypeSerializerConfigSnapshot
● Also define logic for compatibility checks
public class MyCustomTypeSerializer extends TypeSerializer<MyCustomType> {
...
TypeSerializerConfigSnapshot snapshotConfiguration();
CompatibilityResult<MyCustomType> ensureCompatibility(
TypeSerializerConfigSnapshot configSnapshot);
}

45. Closing
45

46. TL;DR
46
● The Flink community takes state management in Flink very seriously
● We try to make sure that users will feel comfortable in placing their
application state within Flink and avoid any kind of lock-in.
● Upcoming changes / features to expect related to state management:
Eager State Declaration & State Migration

47. 4
Thank you!
@tzulitai
@ApacheFlink
@dataArtisans

48. We are hiring!
data-artisans.com/careers