TERA Server Architecture

2. Table of Contents Introduction – TERA Warming Up Architecture – TERA Server Test Summary TERA’s Server Architecture 2

3. • Introduction • Warming UpT.E.R.A. • Architecture • Test • Summary The Exiled Realm of Arborea – Open-beta test within this year TERA via Mass Media – Blockbuster MMORPG using Unreal Engine 3 – Development cost about 200,000,000 RMB during 3.5 years – Various contents on the Seamless world – Non-targeting, Non-targeting, Non-targeting, … TERA via Developers’ View – Is it possible? • (X) Converting Action-heavy console game into MMORPG • (O) Making MMORPG with strong action elements TERA’s Server Architecture 3

4. • Introduction • Warming UpAction-Heavy MMORPG • Architecture • Test • Summary Realistic Combat – Dodge or block by observing the attacks of enemies – Control the direction and distance to attack enemies – Natural in Console or MO games – High expectations in regard to high sync rate and low latency Non-targeting Combat System – A way to support action-heavy combat Server-side Support for Action-Heavy Combat – Space and collision calculation very rapidly – Send and receive packets much more  Work load proportionate to the level of action-heaviness TERA’s Server Architecture 4

5. • Introduction • Warming UpTargeting vs. Non-Targeting • Architecture • Test • Summary Targeting Non-targeting T6 T5 T4 Weapon T3 Direction T2 T1 Player Ally In Fact, 3D Volume for Collision Check Enemy  Optimized space search and collision check are needed TERA’s Server Architecture 5

6. • Introduction • Warming UpTerminology • Architecture • Test • Summary Channel – The space where players actually explore Continent – Literally, a large area of land having one or more channels World – Server contains a group of continents Planet – Group of worlds – (aka) Realm server – The unit of user clients’ view TERA’s Server Architecture 6

7. • Introduction • Warming UpServer Architecture Requirement • Architecture • Test • Summary Action-Heavy Combat – Non-targeting battle in the overall combat system • Increasing space-search and hit-judgment Seamless World – No loading and teleport within a world – No limitation to deploy contents anywhere • Creature, hunting-ground, and any type of content • Implementation should not restrict contents-deployment Inter-Server Contents – Competitive contents among planets – Community contents TERA’s Server Architecture 7

8. • Introduction • Warming UpSupporting Action-Heavy Combat • Architecture • Test • Summary Frequent Space Calculation – Object-search & Hit-judgment • CPU-intensive work – No physics engine in the server • Simplify collision volume – Using lock-free manner [4] • Lock radically increase the CPU usage Frequent Transmission – Aggregate broadcast • In inverse proportion to action-heaviness – The more frequent send, the better action-heaviness • Important to find an adequate value for aggregation TERA’s Server Architecture 8

9. • Introduction • Warming UpSupporting Seamless World • Architecture • Test • Summary Popular Thread Model for Zone-based World – A dedicated thread to a particular content or area • The decreasing complexity of source-codes – Work load could converge on a specific thread • Inappropriate for implementing a seamless world To Make Seamless World – TERA takes on (homogeneous) worker thread model [1] – No dedicated thread that handles specific contents or areas • One thread pool including symmetric worker-threads • Somewhat difficult and complex to implement – No blockings among threads via lock-free implementation [2, 3] TERA’s Server Architecture 9

10. • Introduction • Warming UpSupporting Inter-Server Contents • Architecture • Test • Summary Types – Community contents – Shared world contents • Common instance-dungeon pool • Battleground server Simple Structure – To reduce inter-server complexity – To regard the shared world as clients’ own planet – No handover of network connection Scalability – Easy to add a new world server with new contents TERA’s Server Architecture 10

11. • Introduction • Warming UpServer Architecture History (1/2) • Architecture • Test • Summary First Architecture – Project S1 prototyping version – Front Server: world container • Support action-heavy combat on a seamless world – Back Server: DB cache server – Inappropriate to inter-server contents Front Server Client Back Server Planet TERA’s Server Architecture 11

12. • Introduction • Warming UpServer Architecture History (2/2) • Architecture • Test • Summary Second Architecture – Broadcaster • To support community contents (chat-channels, party, …) • To reduce world servers’ network I/O – DB proxy • To support data-sharing contents (auction, warehouse, … ) • Serialize and validate the DB tasks – Complex: difficult and annoying to implement Broadcaster DB Proxy Client World Server Planet TERA’s Server Architecture 12

13. • Introduction • Warming UpFinal TERA Server Architecture • Architecture • Test • Summary Split by Role – World server for CPU intensive tasks – Arbiter server for I/O intensive tasks • The role of “DB proxy + Broadcaster” • In fact, DB proxy and Broadcaster have a lot in common – Shared code, similar task-style – Reduced the complexity of server architecture Client Arbiter World Server Server Planet TERA’s Server Architecture 13

14. • Introduction • Warming UpArbiter Server • Architecture • Test • Summary Single System Image [6] – The only server that game-clients access • Originated in Starcraft unit – No handover of network connection – Delegated for packet-broadcast by World servers • Dispersing network I/O load – Handling common contents among World servers • Chatting, guild, party, … Implementation Features – Using locks for common contents – Two thread pools • Blocking I/O (e.g. DB operation) • Non-blocking I/O (e.g. packet broadcast) TERA’s Server Architecture 14

15. • Introduction • Warming UpWorld Server • Architecture • Test • Summary Seamless World Container – Space where players explore • A world includes multiple continents • Specialized for space search and assessment of the combat – Sharable among multiple Arbiter servers • Battle-group, common instance dungeon pool Implementation Features – Single thread pool consists of symmetric worker-threads • No thread that exclusively handles particular contents – Robust code base for multi-threading [2, 3] • Making all tasks non-block-able through lock-free manner [4] • (c.f.) Proactor [1] & Active-object pattern [5] TERA’s Server Architecture 15

16. • Introduction • Warming UpScalability • Architecture • Test • Summary Server Topology – Can attach World servers to a single Arbiter server – Convenient to arrange continents • Different combinations of continents in multiple World servers World Server Side – Identical worker-threads as the number of CPU cores • Direct benefits from CPU core count and speed Arbiter Server Side – Possibility of bottleneck • Less critical of an issue than allocated bandwidth for service – Utilizing the surplus CPU power • Packet optimization to reduce bandwidth TERA’s Server Architecture 16

17. • Introduction • Warming UpAdditional Issues • Architecture • Test • Summary Considering Contents Deployment – Inter-server contents influence how to deploy World servers – Considering the stability • Can keep a Planet live even if a specific World server down • Quarantine unstable contents (e.g. latest patched) Split Servers Depending on Purpose – Dedicated server for users’ convenience • Chatting, searching, and so on Order of Priority – Not to put restriction on elements of fun TERA’s Server Architecture 17

18. • Introduction • Warming UpTest with Dummy-Clients • Architecture • Test • Summary Sisyphus – Client program operating dummy-bots for server-load test • Operating 1000~1500 bots per one Sisyphus • Recording real-players’ behavior patterns WAN Simulator – Give virtual latency to mimic real-network environment • Randomized latency around 200ms • Required high performance machine • Required dedicated line for load-test Sisyphus Sisyphus WAN Simulator Planet Sisyphus Sisyphus Dedicated-line for test TERA’s Server Architecture 18

19. • Introduction • Warming UpStress Test • Architecture • Test • Summary 3rd Closed Beta Test – Last Spring in Korea involving about 24000 users – Events designed to complement daily stress test • A sizable number of concurrent users were provided • Server-status goes on smoothly throughout the stress test Server Status – Very low CPU usage on World servers – Not any congestion handling network I/O on the Arbiter server – Monitoring via VTune [7] • Latest version to support Nehalem CPU [8] TERA’s Server Architecture 19

20. • Introduction • Warming UpSummary • Architecture • Test • Summary Impossible is Nothing – Not easy to make non-targeting battle in a seamless world – But, it is certainly possible Server Architecture Case by Case – Optimized for each different type of MMOG – Arbiter-World architecture for TERA Complexity – Support the “simplicity first” policy – For stability, ease of management, and … TERA’s Server Architecture 20

21. References[1] Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Object Douglas Schmidt et al. John Wiley & Sons, 2000.[2] Thread-Specific Storage for C/C++ Douglas Schmidt et al. C++ Report, 1997.[3] The Art of Multiprocessor Programming Maurice Herlihy et al. Morgan Kaufmann, 2008.[4] Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms Maged M. Michal et al. ACM symposium on Principles of distributed computing, 1996.[5] Active Object: An object behavioral pattern for concurrent programming R. Greg Lavender et al. Pattern languages of program design, 1996.[6] Single System Image R. Buyya et al. The international journal of high performance computing applications, 2001.[7] VTune: Performance Analyzer Intel Software Network. http://software.intel.com/en-us/intel-vtune/[8] Nehalem micro-architecture http://en.wikipedia.org/wiki/Nehalem_(microarchitecture) TERA’s Server Architecture 21

22. 감사합니다 谢谢! Thanks! Q&A TERA’s Server Architecture 22