Introduction to FlatBuffers

1. Beyond JSON
Maxim Zaks
@iceX33

2. Agenda
• War Stories
• And very technical details

3. 2007

5. 2007

7. JSON vs. XML
• much more human readable
• no schema
• no streaming parser
• too early for financial sector

8. 2015

10. Simulation Games

11. Deterministic
simulaiton running
on mobile device
And validated on the BackEnd

13. Share code between
BackEnd & FrontEnd

14. Stateless Worker

17. Parsing a JSON
config (400kb) and
game state (30kb)
takes 60 ms

20. Did we gave up?

21. No!
We Google

23. FlatBuffers
is an efficient cross
platform serialization
library for C++, Java, C#,
Go, Python and JavaScript (C,
PHP & Ruby in progress).

24. How does it work?

25. Define a schema
table Person {
name: string;
age : int;
}
root_type Person;

26. Call code generator
./flatc -n person01.fbs

27. Use generated code for
reading
public sealed class Person : Table {
public static Person GetRootAsPerson(ByteBuffer _bb) {...}
public static Person GetRootAsPerson(ByteBuffer _bb, Person obj) {...}
public string Name { get { ... } }
public int Age { get { ... } }
};
byte[] data = ...;
Person person = Person.GetRootAsPerson(new ByteBuffer(data));

28. Suffer through generated code
for writing
public sealed class Person : Table {
public static Offset<Person> CreatePerson(FlatBufferBuilder builder,
StringOffset name = default(StringOffset),
int age = 0) {
...
}
public static void FinishPersonBuffer(FlatBufferBuilder builder, Offset<Person> offset) { ... }
};
FlatBufferBuilder fbb = new FlatBufferBuilder(1);
var nameOffset = fbb.CreateString("maxim");
var personOffset = Person.CreatePerson(fbb, nameOffset, 34);
Person.FinishPersonBuffer(fbb, personOffset);
byte[] data = fbb.SizedByteArray();

29. What is
underneath?
Why is it so fast to read
And cumbersome to write

32. Call FlatBuffers compiler
./flatc -b person01.fbs person01.json

33. And if you want binary to
JSON
./flatc -t person01.fbs -- person01.bin --raw-binary --strict-json
=
{
"name": "maxim",
"age": 34
}

35. FlatBuffers has
evolution strategy

36. table Person {
name: string;
age : int;
}
root_type Person;
->
table Person {
name: string;
age : int (deprecated);
birthday : double;
}
root_type Person;

37. table Person {
name: string;
age : int;
}
root_type Person;
->
table Person {
name: string (id: 0);
birthday : double (id: 2);
age : int (deprecated, id: 1);
}
root_type Person;

38. It's backwards and
forwards
compatible
new code can read old data
&
old code can read new data

40. I wanted to make
writing more
usable
specially for unit testing

42. Eager serialization
public sealed class Person : Table {
public string name;
public int age;
public static Person FromByteArray(byte[] data) {...}
public byte[] ToByteArray(){...}
}
// var person = new Person();
byte[] data = ...;
var person = Person.FromByteArray(data);
person.name = "Alex";
byte[] data2 = person.ToByteArray();

44. let person = Person(name: "Maxim", age: 34)
let fbData = person.toByteArray
NSData( bytes: UnsafePointer<UInt8>(fbData),
length: fbData.count)
.writeToFile("person.bin", atomically: true)

46. Facebook is using
FlatBuffers !

47. FlatBuffersSwift benchmark
• 65.015 mseconds for parsing JSON (3.6MB)
• 36.478 mseconds for eager FlatBuffers (2.1MB)
• 00.002 mseconds for lazy FlatBuffers (2.1MB)

48. Some rain on FlatBuffers
parade
• Zip JSON: 3.6MB -> 739KB
• Zip FB: 2.1MB -> 1.1MB

49. Outlook for FlatBuffers
• real Graph not a Tree
• could support streaming
• is easy to encrypt
• could be very memory efficient with custom
compression

50. Questions?

51. Thank you!
@iceX33
https://google.github.io/flatbuffers
https://github.com/mzaks/FlatBuffersSchemaEditor
https://github.com/mzaks/FlatBuffersSwift