2018 아이펀팩토리 데브데이 서버위더스 (iFunFactory DevDay Server,Withus) 제목 : Scalable 한 게임 서버 만들기 발표자 : 문대경 CEO 일정 : 2018년 03월 28일 개요 : 게임 서버의 고확장성을 확보하기 위한 기법들을 설명하고, 각 기법들의 서버 장애에 대한 대응 전략을 공개합니다. 더 많은 정보를 원하신다면 아래를 참고해주세요 *아이펀팩토리 홈페이지 : https://ifunfactory.com/ *아이펀팩토리 기술 블로그 : https://blog.ifunfactory.com/

1. Scalable
(dkmoon@ifunfactory.com)

2. 2018 iFunFactory Dev Day
Scalability?
Scalability is the capability of a system, network, or process
to handle a growing amount of work, or its potential to be
enlarged to accommodate that growth.
- Wikipedia

3. 2018 iFunFactory Dev Day
Scalability?
Scalability is the capability of a system, network, or process
to handle a growing amount of work, or its potential to be
enlarged to accommodate that growth.
- Wikipedia
Scalability is the capability of a system, network, or process
to handle a growing amount of work, or its potential to be
enlarged to accommodate that growth.
Scalability , ,
,

4. 2018 iFunFactory Dev Day
Scalability Question in Gaming
Q) ?

5. 2018 iFunFactory Dev Day
...

6. 2018 iFunFactory Dev Day
Scalability Question in Gaming
Q) ?
Q) ?

7. 2018 iFunFactory Dev Day
Scalability Question in Gaming
Q) ?
Q) ?
A1)

8. 2018 iFunFactory Dev Day
Scalability Question in Gaming
Q) ?
Q) ?
A1)
A2)

9. 2018 iFunFactory Dev Day
Contents.
01
02
03
04 /
05 DB

10. 2018 iFunFactory Dev Day
Offhand Approach
“Scale-up”
SW HW

11. 2018 iFunFactory Dev Day
Scale-up
for High Per-server Capacity
Pros:
(SW)

12. 2018 iFunFactory Dev Day
Scale-up
for High Per-server Capacity
Pros:
(SW)
(via container/VM)

13. 2018 iFunFactory Dev Day
Intel Xeon E5-46XX v4 Family Retail Prices
(Launched in Q2’16)
USD 0
USD 2000
USD 4000
USD 6000
USD 8000
E5-4610
v4
E5-4620
v4
E5-4627
v4
E5-4640
v4
E5-4650
v4
E5-4660
v4
E5-4667
v4
E5-4669
v4

14. 2018 iFunFactory Dev Day
Scale-up
for High Per-server Capacity
Cons:

15. 2018 iFunFactory Dev Day
Scale-up
for High Per-server Capacity
Cons:
HW

16. 2018 iFunFactory Dev Day
Scale-up
for High Per-server Capacity
Cons:
HW

17. 2018 iFunFactory Dev Day
Harmonious Approach
“ SW ”

18. 2018 iFunFactory Dev Day
Golden Rule
Minimize Inefficiency
Maximize Concurrency

19. 2018 iFunFactory Dev Day
Inefficiency = Wasting CPU time

20. 2018 iFunFactory Dev Day
Inefficiency = Wasting CPU time

21. 2018 iFunFactory Dev Day
Inefficiency = Wasting CPU time

22. 2018 iFunFactory Dev Day
Inefficiency = Wasting CPU time
) Indexing vs. Looping
Guide #1.
big O
coefficient
Guide #2. Busy-Waiting
Asynchronous ( )

23. 2018 iFunFactory Dev Day
Code Efficiency vs. Readability
Q) 10%
50% ?

24. 2018 iFunFactory Dev Day
Readability = Reliability
=
=
= @_@/
( , !)

25. 2018 iFunFactory Dev Day
Efficiency vs. Readability
trade-off
➡ efficiency

26. 2018 iFunFactory Dev Day
Concurrency
( ..)

27. 2018 iFunFactory Dev Day
Concurrency Units
Threads (preemptive or cooperative)
Processes (local or remote)

28. 2018 iFunFactory Dev Day
More Units != More Concurrency
By Benjamin D. Esham / Wikimedia Commons, CC BY-SA 3.0
https://commons.wikimedia.org/w/index.php?curid=56559

29. 2018 iFunFactory Dev Day
Concurrency
( ..)

30. 2018 iFunFactory Dev Day
For Better Concurrency
,
Blocking

31. 2018 iFunFactory Dev Day
For Better Concurrency
Guide #1.
Guide #2.

32. 2018 iFunFactory Dev Day
Concurrency Control
Concurrent
sequential

33. 2018 iFunFactory Dev Day
Concurrency Control
Concurrent
sequential
➡ Shared data

34. 2018 iFunFactory Dev Day
Pessimistic Concurrency Control

35. 2018 iFunFactory Dev Day
Pessimistic Concurrency Control
•
• Shared data
• ) lock
RDBMS xaction

36. 2018 iFunFactory Dev Day
Optimistic Concurrency Control
•
• Shared data ,
Roll-back & retry

37. 2018 iFunFactory Dev Day
Q) ? DBMS ?
Optimistic Concurrency Control

38. 2018 iFunFactory Dev Day
Q) ? DBMS ?
A) ?
Optimistic Concurrency Control

39. 2018 iFunFactory Dev Day
synchronization primitive
primitive
Optimistic CC ( )

40. 2018 iFunFactory Dev Day
Pessimistic CC ( - )
Shared ObjectThread A Thread B
Write Lock
Read
Write
Unlock
Process Locked

41. 2018 iFunFactory Dev Day
Pessimistic CC ( - )
Shared ObjectThread A Thread B
Write Lock
Read
Write
Unlock
Process Locked
Read Lock Req
Read Lock Return
blocked

42. 2018 iFunFactory Dev Day
Optimistic CC ( - )
Shared ObjectThread A Thread B
Read Lock
Read (Rev=1)
Commit (Rev=1)
Unlock
Process
Locked
Write Lock
Unlock
Locked

43. 2018 iFunFactory Dev Day
Optimistic CC ( - )
Shared ObjectThread A Thread B
Read Lock
Read (Rev=1)
Commit (Rev=1)
Unlock
Process
Locked
Write Lock
Unlock
Read Lock
Read (Rev=1)
Unlock

44. 2018 iFunFactory Dev Day
Pessimistic CC ( - )
Shared ObjectThread A Thread B
Write Lock
Read
Write
Unlock
Process Locked
Write Lock Req
Write Lock Return
blocked

45. 2018 iFunFactory Dev Day
Optimistic CC ( - )
Read Lock
Read (Rev=1)
Commit (Rev=1)
Unlock
Process
Locked
Write Lock
Unlock
Read Lock
Read (Rev=1)
Unlock

46. 2018 iFunFactory Dev Day
Optimistic CC ( - )
Read Lock
Read (Rev=1)
Commit (Rev=1)
Unlock
Process
Locked
Write Lock
Unlock
Read Lock
Read (Rev=1)
Unlock
Write Lock
Commit (Rev=1, REJECT)
Locked
Process

47. 2018 iFunFactory Dev Day
Optimistic CC ( - )
Read Lock
Read (Rev=1)
Commit (Rev=1)
Unlock
Process
Locked
Write Lock
Unlock
Read Lock
Read (Rev=1)
Unlock
Write Lock
Commit (Rev=1, REJECT)
Locked
Read (Rev=2)
Unlock Roll back & Retry
Process
Process

48. 2018 iFunFactory Dev Day
Pessimistic CC vs. Optimistic CC

49. 2018 iFunFactory Dev Day
Pessimistic CC vs. Optimistic CC
Pessimistic Optimistic
Low contention
High contention progress
rollback&retry
(progress )

50. 2018 iFunFactory Dev Day
•High contention
•Low contention
Pessimistic CC vs. Optimistic CC
➡ Pessimistic
➡ Optimistic

51. 2018 iFunFactory Dev Day
Optimistic CC ( )
•
• in-game
( : )

52. 2018 iFunFactory Dev Day
Optimistic CC ( )
•
• in-game
( : )
•

53. 2018 iFunFactory Dev Day
Key Questions…
• workload ?

54. 2018 iFunFactory Dev Day
Key Questions…
• workload ?
• ?

55. 2018 iFunFactory Dev Day
Key Questions…
• workload ?
• ?
• ?

56. 2018 iFunFactory Dev Day
Key Questions…
• workload ?
• ?
• ?
• ?

57. 2018 iFunFactory Dev Day
Workload Distribution
Domain-
agnostic
Domain-
specific

58. 2018 iFunFactory Dev Day
Domain-agnostic Distribution
instance
stateless processing unit
) web server scaling
PUBG

59. 2018 iFunFactory Dev Day
Domain-specific Distribution
in-memory state
) RPG
,

60. 2018 iFunFactory Dev Day
Workload Distribution
Domain-Agnostic Domain-Specific
state
Scale-out
reconfiguration
State
DB/Cache (centralized)
RPC (distributed)
in-memory
master-slave
replication

61. 2018 iFunFactory Dev Day
Workload Distribution Guideline

62. 2018 iFunFactory Dev Day
Workload Distribution Guideline

63. 2018 iFunFactory Dev Day
Workload Distribution Guideline
✓ Concurrency
• Unit: thread → (remote) process

64. 2018 iFunFactory Dev Day
Workload Distribution Guideline
✓ Concurrency
• Unit: thread → (remote) process
➡ , blocking

65. 2018 iFunFactory Dev Day
Dos & Don’ts
✓
✓ : Lobby vs. Room → specific
Room instances → agnostic

66. 2018 iFunFactory Dev Day
Dos & Don’ts
✓
✓ : Lobby vs. Room → specific
Room instances → agnostic
✓ Stateless stateful
✓ Stateful stateless
✓ : RPG workhorse

67. 2018 iFunFactory Dev Day
Static Peer Configuration
Pros
•
•

68. 2018 iFunFactory Dev Day
Static Peer Configuration
Cons
• Dev Live
• Live

69. 2018 iFunFactory Dev Day
Dynamic Peer Discovery
• registry
: Zookeeper, Redis, DB, …
•
• RPC endpoint
• Server tags

70. 2018 iFunFactory Dev Day
Dynamic Peer Discovery
RPC=<IP, port>
TAGS=[“lobby”]

71. 2018 iFunFactory Dev Day
Dynamic Peer Discovery
RPC=<IP, port>
TAGS=[“lobby”]
RPC=<IP, port>
TAGS=[“game”, “chobo"]

72. 2018 iFunFactory Dev Day
Dynamic Peer Discovery
RPC=<IP, port>
TAGS=[“lobby”]
RPC=<IP, port>
TAGS=[“game”, “chobo"]

73. 2018 iFunFactory Dev Day
Dynamic Peer Discovery
RPC=<IP, port>
TAGS=[“lobby”]
RPC=<IP, port>
TAGS=[“game”, “chobo"]

74. 2018 iFunFactory Dev Day
Pitfall: DB as Sync Point
DB
•
• atomicity

75. 2018 iFunFactory Dev Day
Pitfall: DB as Sync Point
...
DB ( )

76. 2018 iFunFactory Dev Day
Pitfall: DB as Sync Point

77. 2018 iFunFactory Dev Day
Pitfall: DB as Sync Point
Solution #1. DB Cache
Solution #2.

78. 2018 iFunFactory Dev Day
Optimistic CC ( )
• RPC
• PCC
•OCC

79. 2018 iFunFactory Dev Day
RECAP
✓ 2
• Minimize Inefficiency
• Maximize Concurrency

80. 2018 iFunFactory Dev Day
RECAP
✓ 2
• Minimize Inefficiency
• Maximize Concurrency
✓ , Concurrency

81. 2018 iFunFactory Dev Day
RECAP
✓ 2
• Minimize Inefficiency
• Maximize Concurrency
✓ , Concurrency
✓ Concurrency 2
•
•

82. Q&A
2018 iFunFactory Dev Day
2018 iFunFactory Dev Day
Thank You!
660, 1 B 606
+82-70-4923-6566 www.ifunfactory.cominfo@ifunfactory.com