Erlang is successfully used for game servers. Lua is a friendly, small language that is big in game scripting. And VoltDB is a new high speed ACID database from PostgreSQL inventor Mike Stonebreaker. Add some JSON, Websockets and JavaScript and you have an exciting HTML5 game stack. This talk is about research and results from the work on a very high throughput game server with embedded Lua for game logic and VoltDB as SQL database backend. The stack aims to maximize robustness, throughput and ease of scripting. Its special feature is linearly scaling true ACID with a goal of a million transactions per second, while being easy on the game logic scripter. This stack is part of the development effort of games start-up Eonblast for the cross-media Science Fiction franchise Solar Tribes, which will feature film elements merged with cross-platform online games. The stack is created to allow for a unified, non-sharded game world experience, thus for a unified story line and also otherwise transcends current MMO conventions. Players will share the same stories at the same time and be empowered to leave a stronger imprint on their environment. Interaction with other players is taken beyond the limitations currently seen in social games. All of which are technical challenges, asking for fastest possible data access and better data integrity guarantees. A triple asynchronous server side architecture is described that communicates non-blocking with a fat JS/HTML5 browser client. The related Open Source projects Erlvolt and Fleece are introduced, which were created for this stack. The embedded Lua driver Erlualib and Robert Virding's new project, the Lua VM emulator Luerl are compared and evaluated. A brief overview of capabilities of new databases is given to explain why VoltDB was chosen for the task. (c) 2012 Eonblast Corporation

1. Lua + Erlang + VoltDB
An Erlang Game Stack
An Erlang Game Stack
Erlang User Conference 2012
H. Diedrich
http://www.eonblast.com – twitter @hdiedrich
1 Eonblast

2. An Erlang Game Stack
Henning Diedrich
• Maintainer Emysql, Erlvolt, Erlualib
• Eonblast: game + film
• NewTracks: game + music
• Bricks and mortar
• Insurance tariff language
Octocat says: try markedoc!
2 Eonblast

3. An Erlang Game Stack
Eonblast
3 Eonblast

4. An Erlang Game Stack
This Talk
• A game server architecture
• Its components
• The glue
4 Eonblast

5. An Erlang Game Stack
This Talk
• Work in progress
• Unproven concepts
• Lies and conjecture
andalso
• 3 genius tools
• 10,000 lines of glue code
5 Eonblast

6. An Erlang Game Stack
Questions
Please throw them in like Torben
Mail me at hdiedrich at eonblast.com
Erlang Mailing List
6 Eonblast

7. An Erlang Game Stack
Erlang Game Servers
Zynga: FarmVille via membase, Activision Blizzard: Call of Duty, Bigpoint: Battle Star Galactica, Wooga: Magic Land
7 Eonblast

8. An Erlang Game Stack
Erlang + MySQL
http://eonblast.github.com/Emysql
Electronic Arts contributed emysql.
8 Eonblast

9. An Erlang Game Stack
The Hurt
• Leaks, Deadlocks
• Bad Productivity
• Game Design Language Gap
• Less Features
• Less Players
• Lesser Product
9 Eonblast

10. An Erlang Game Stack
Target
„The perfect game server“
(makes no sense)
10 Eonblast

11. An Erlang Game Stack
Generals’ Problem
State
Two generals must agree on a time.
11 Eonblast

12. An Erlang Game Stack
Generals’ Problem
State
Send a messenger.
12 Eonblast

13. An Erlang Game Stack
Generals’ Problem
State
Acknowledge.
13 Eonblast

14. An Erlang Game Stack
Generals’ Problem
State
ACK the ACK. Etc.
14 Eonblast

15. An Erlang Game Stack
Generals’ Problem
State
The messenger may get lost.
15 Eonblast

16. An Erlang Game Stack
Byzantine Generals
State
The generals, too.
16 Eonblast

17. An Erlang Game Stack
The Hunt
Server Stack
• Language
• Database
• Protocol
• Game Logic
17 Eonblast

18. An Erlang Game Stack
Spec
“Table Game”
• 1 million active players
• 1 million transactions per second
• 1 second latency
• Linear scale
• 100% data integrity
18 Eonblast

19. An Erlang Game Stack
Client Architecture
• Fat JavaScript Browser Client
• COMET
Not discussed in this talk.
19 Eonblast

20. An Erlang Game Stack
Language
Wishlist
• Robustness
• Deadlock-free
• Multi-core
• Linear Scale
er?
20 Eonblast

21. An Erlang Game Stack
Database
CAP
• Distributed
• Consistent
• Highly-available
• Partition-tolerant
All of it!
Brewer on CAP 2012: http://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed
21 Eonblast

22. An Erlang Game Stack
Database
ACID
• Atomicity
• Consistency
• Isolation
• Durability
For Granted?
22 Eonblast

23. An Erlang Game Stack
Database
Double Bookkeeping
• Not every game needs it
• Requires ACID Transactions
• Neigh impossible to emulate
• Impossible with BASE /
Eventual Consistency
23 Eonblast

24. An Erlang Game Stack
Database
Upcoming Blog Post
• Looking at 14 databases
• Riak, Cassandra, Membase, H-Base, Voldemort, MySQL, MySQL Cluster, Redis, Redis
Cluster, Tokyo Cabinet, Memcached, CouchDB, Couchbase, VoltDB, MongoDB, Postgres
• In the light of what games need
• Unbiased comparison
• Twitter @hdiedrich
24 Eonblast

25. An Erlang Game Stack
Protocol
Wishlist
• Small footprint
• Fast to encode
• Human-readable
25 Eonblast

26. An Erlang Game Stack
Logic
Wishlist
• Human Readable
• Stable
• Fast
• Concise
• Small Footprint
26 Eonblast

27. An Erlang Game Stack
The Answers
• Client
• Protocol
• Server
• Logic
• Data
Client: HTML5/JS, Protocol: JSON, Server: Erlang, Logic: Lua, Data: VoltDB.
27 Eonblast

28. An Erlang Game Stack
“Embeddable scripting language”
28 Eonblast

29. An Erlang Game Stack
Lua
”Lua is strange.“
R. Virding
“It has mutable variables, what more need I say.”
29 Eonblast

30. An Erlang Game Stack
Lua
a = “Roberto”
if a ~= nil then
print “Hello ” .. a .. “!”
end
30 Eonblast

31. An Erlang Game Stack
Lua
• Created 1993
• Simple, imperative syntax
• Stable syntax, stable VMs
• Two main VMs: ISO C, JIT
• Game scripting champion
• Open Source, 100% dictatorial*
*benevolent, of course
Site: http://www.lua.org
List: lua-l@lists.lua.org
31 Eonblast

32. An Erlang Game Stack
Lua
• Made for non-programmers - imperative syntax
• Powerful for programmers - Scheme semantics
• Made for DSLs - extensible semantics
• Made to be extended - simple C API
• Made to be embedded - small VM footprint
• Now with a native Erlang VM!
32 Eonblast

33. An Erlang Game Stack
Lua Caveats
• Deceiving Looks
• Encourages magic
• Simple but not for beginners
• Syntax forked
• Only one collection type
33 Eonblast

34. An Erlang Game Stack
Lua VMs
• PUC Reference VM pure ISO C
Lua 5.2 super compatible
• JIT VM X86, ARM, PPC, MIPS
remains at Lua 5.1
super fast
adds extensions
• Luerl native Erlang VM
Lua 5.2 super stable
incomplete
34 Eonblast

35. An Erlang Game Stack
Architecture
35 Eonblast

36. An Erlang Game Stack
Perceived reaction to asking about benchmarks on the Erlang mailing list.
36 Eonblast

37. An Erlang Game Stack
World State
State
Where does the state go? Into the Logic core, for speed.
37 Eonblast

38. An Erlang Game Stack
World State
State State State State State
State
38 Eonblast

39. An Erlang Game Stack
World State
State State State State State
State
39 Eonblast

40. An Erlang Game Stack
World State
State State State State
State
State
40 Eonblast

41. An Erlang Game Stack
World State
State State State State State
State
41 Eonblast

42. An Erlang Game Stack
“High Velocity Database”
42 Eonblast

43. An Erlang Game Stack
VoltDB
• Created 2009
• Simple SQL syntax
• Stable 2.5
• Commercial developer, support
• High Velocity Secret Tip
• Open Source, 100% dictatorial*
*benevolent, of course
Site: http://www.voltdb.com
Help: http://community.voltdb.com/forum
43 Eonblast

44. An Erlang Game Stack
VoltDB
• ACID transactions – double bookkeeping
• SQL – subset of '92
• Linear scale - „unlimited“ data (but ...)
• Made for OLTP - fast, cheap writes, high throughput
• “More SQL than SQL” – invites clean sep. of data
• In-memory - 100x faster than MySQL
• Replication, Snapshots – 'hot backup built in'
44 Eonblast

45. An Erlang Game Stack
VoltDB Caveats
• Less agile than MySQL/NoSQL
• Need to understand partitions
• Must program SPs in Java
• Not elastic (yet)
• No online schema change (yet)
• There is always a bottleneck
• Simple but not for beginners
45 Eonblast

46. An Erlang Game Stack
Horizonzal Partitions
State
#
46 Eonblast

47. An Erlang Game Stack
Replication
State
#
47 Eonblast

48. An Erlang Game Stack
Actual Connections
State State State State State
State
48 Eonblast

49. An Erlang Game Stack
The Mix
Erlang + Lua + VoltDB
49 Eonblast

50. An Erlang Game Stack
How do we connect these components?
50 Eonblast

51. An Erlang Game Stack
Fleece
Erlualib
Erlvolt
51 Eonblast

52. An Erlang Game Stack
Fleece
Luerl
Erlvolt
52 Eonblast

53. An Erlang Game Stack
The Packages
• Protocol Fleece Lua to JSON
• Logic Erlualib Lua port
• Logic Luerl Lua VM
• Data Erlvolt VoltDB driver
53 Eonblast

54. An Erlang Game Stack
Erlualib
• Lua embedding library for Erlang
• Lua state in original PUC Lua VM
• A fast way
• A not secure way: can crash Erlang
54 Eonblast

55. An Erlang Game Stack
Erlualib
hello() ->
% get handle of the Lua engine
{ok, L} = lua:new_s tate(),
% put " print" global on top of s tack
lua:getfield(L, global, " print" ),
% put hello on top
lua:pus hs tring(L, " Hello from Lua!" ),
% exec ute on top 2 values on s tac k
lua:c all(L, 1, 0).
55 Eonblast

56. An Erlang Game Stack
Erlualib
hello() ->
{ok, L} = lua:new_s tate(),
lua:getfield(L, global, " print" ),
lua:pus hs tring(L, " Hello from Lua!" ),
lua:c all(L, 1, 0).
56 Eonblast

57. An Erlang Game Stack
Luerl
• New Lua VM programmed in Erlang
• Perfectly secure way
• Precompiled Chunks
• Re-usable State
• Work in progress
57 Eonblast

58. An Erlang Game Stack
Luerl
hello() ->
L = luerl:init(),
luerl:do(" print('Hello from Lua!')" , L).
58 Eonblast

59. An Erlang Game Stack
Luerl
hello() ->
L = luerl:init(),
C hunk = luerl:eval(" print('Hello from Lua!')" , L),
luerl:call(C hunk, L)
% C hunk and L c an be re-us ed.
59 Eonblast

60. An Erlang Game Stack
Benchmark Luerl vs Erlualib
Erlualib / PUC Luerl
270 µs state Init 250 µs
10 µs parse 1 + 1 15 µs
2.5 µs execute 1 + 1 13 µs
23 µs parse formula 63 µs
2.5 µs execute formula 35 µs
2.5 µs call a function 28 µs
Formula: a = 7.33; b = 9000; c = (33 * a / b) ^ 15 * a + b
60 Eonblast

61. An Erlang Game Stack
Fast Lua to JSON encoder
61 Eonblast

62. An Erlang Game Stack
Fleece
• Fastest Lua JSON encoder
• C+ASM or ISO C
• Faster than Lua JIT
• Faster than concat
• 10x faster than other fastest
62 Eonblast

63. An Erlang Game Stack
63 Eonblast

64. An Erlang Game Stack
Erlvolt
• Native Erlang VoltDB driver
• Async parallel data shuffling
• Coming next quarter
• Your help is wanted
• Connect over github
64 Eonblast

65. An Erlang Game Stack
Server Architecture
• Erlang • Luerl
• Lua • Erlualib
• JSON • Fleece
• VoltDB • Erlvolt
65 Eonblast

66. An Erlang Game Stack
Invitation
• All at github
• All w/samples & docs
• All maintained by Eonblast
• Luerl maintained by R. Virding
• Updates @hdiedrich, @rvirding
66 Eonblast

67. An Erlang Game Stack
Questions
Mail: hdiedrich ∂ eonblast.com
Skype: eonblast
Twitter: @hdiedrich
67 Eonblast