At Scale With Style (Erlang User Conference 2012)
•
20 likes•3,399 views
In the world of social gaming, the classic 2-tier of web application does not cut it anymore. We need new and better solutions. Follow along the evolution of game servers at Wooga and get an in-depth look into the next-generation backend putting the combined forces of Erlang and Ruby to work. Learn how scalability, reliability, concurrency control and beautiful code do not need to be mutually exclusive.
Recommended
More Related Content
What's hot
What's hot (18)
Similar to At Scale With Style (Erlang User Conference 2012)
Similar to At Scale With Style (Erlang User Conference 2012) (20)
More from Wooga
More from Wooga (20)
Recently uploaded
Recently uploaded (20)
At Scale With Style (Erlang User Conference 2012)
- 1. AT SCALE WITH STYLE Erlang <3 <3 <3 Ruby Mar$n Rehfeld, @klickmich
- 2. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 4. Typical game architecture Client HTTP Backend API
- 5. Typical game architecture Backend State Changes ValidaHon Persistence
- 6. The scale is interesHng 14 billion requests / month >100,000 DB operaHons / second >50,000 DB updates / second
- 7. Wooga’s approach to development Small independent teams for each game Team gets to choose tools and technologies Same team also does ops a,er going live Culture of sharing Look around what is there, pick/adapt exis;ng solu;ons, but take ownership for what you include
- 8. ExisHng backends – Technology landscape
- 9. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 10. Most games use stateless applicaHon servers Server Database
- 11. And then there’s sharding lb app app app app app app app app app My My SQL SQL slave slave
- 12. More app servers, more sharding lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
- 13. Wait, seriously?! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slave
- 14. Find the flaw “Stateless applicaHon servers guarantee one thing: The data is never where you need it!” Paolo Negri, Developer @ Wooga
- 15. Strong session pa`ern User starts playing many transformaHons of the same set of data User stops playing
- 16. Stateful servers and DBs Server Database One Game Session
- 17. Stateful game server One process per acHve user gaming session ... holds the current state and is the only one that can modify it (strong encapsulaHon) ... handles all API calls for the given user one a,er the other (concurrency control through actor model) ... loads the game state from storage and writes it back periodically and on process termina;on (;meout = user stopped playing)
- 18. The DB is no longer the bo`leneck Stateless Stateful 30.000 22.500 15.000 700 7.500 0 DB operations / second
- 19. Magic Land uses Erlang Details: Awesome presentaHon on the Magic Land game server by @knuHn & @hungryblank h`p://www.slideshare.net/wooga/from-‐0-‐to-‐1000000-‐daily-‐users-‐with-‐erlang
- 20. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 22. Erlang & Ruby Erlang is great Concurrent, robust Great for opera;on Ruby is great Concise, expressive, flexible Great for development
- 24. Bringing two worlds together Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 25. Do you know Mongrel2? Web Server that hooks up applicaHons via ➡ We chose the same queue setup & message format Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 27. ConnecHng the dots Mongrel2 h`p://mongrel2.org/ protocol & ZeroMQ setup Erlang: h`ps://github.com/hungryblank/emongrel2 Ruby rack-‐mongrel2 fork hKps://github.com/khiltd/khi-‐rack-‐mongrel2 rack protocol hKp://rack.rubyforge.org Sinatra hKp://www.sinatrarb.com/ ➡ essenHally we are speaking HTTP over ZeroMQ and can hook up any Rack-‐based Ruby web framework
- 28. Example controller in Ruby app.game_action '/:actor/fruit_tree/self/shake', :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do x, y = params[:x], params[:y] fruit_trees[x, y].shake end DSL-‐like definiHon of game acHon Skinny as controllers should be
- 29. Example model in Ruby class FruitTree < Tree property :last_shake_time, :type => Integer, :default => 0 property :collectable_fruit_count, :type => Integer, :default => 0 def shake raise G8::Error::Validation, "no fruit!" unless carries_fruit? session.user.xp += 1 session.user.energy -= 1 self.last_shake_time = game_time self.collectable_fruit_count = config.fruit_count end end Easily unit testable Minimal amount of code
- 30. Game state Game state is split in mulHple parts user, map, fruit_trees etc. Erlang does not care about content Serialized Ruby objects Erlang does know mapping of state parts to URLs
- 31. Looking back at the game acHon app.game_action '/:actor/fruit_tree/self/shake', :observable => true, :affects => [:fruit_trees, :user], :params => [:x, :y] do x, y = params[:x], params[:y] fruit_trees[x, y].shake end Mapping of state parts to game acHons Worker knows mapping Worker pushes mapping to Erlang on startup Erlang can query mapping if needed
- 32. NICE! http://www.flickr.com/photos/aigle_dore/
- 33. At Scale With Style How we roll Server architectures How to innovate Mind the limits
- 34. Performance impact for large payloads 200 kB payload 6000 5000 Requests / s 4000 Pure 3000 Erlang 2000 Ruby 1000 ZMQ 400 rps 0 1 # of workers 8
- 35. NOT CPU bound 25% Fixed bandwith limit @ 300 MB/s
- 36. This has happened before 2,000,000" 1,500,000" 1,000,000" 500,000" 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
- 38. Example controller in Erlang shake(Args, State) -> Coords = g8_map:coords(Args), Map = g8_game:get_map(State), Tree = g8_map:find(Coords, Map), NewTree = g8_fruit_tree:shake(Tree), UserEffects = [incr_xp, decr_energy], NewMap = g8_map:place(Coords, NewTree, Map), [observable, {state, g8_game:set_map(NewMap, g8_user:apply(UserEffects, State))}].
- 39. Example model in Erlang shake(Tree) -> validate_carries_fruit(Tree), FruitCount = g8_fruit_tree_config:fruit_count(Tree), Tree#fruit_tree{last_shake_time = g8_game:gametime(), collectable_fruit_count = FruitCount}.
- 40. ✓ Y ES
- 41. QuesHons? MarHn Rehfeld @klickmich slideshare.net/wooga wooga.com/jobs