【TECH×GAME COLLEGE#32】ゼロからリアルタイムサーバーを作るまで

1. 1
TECH×GAME COLLEGE#32

2. 2

3. 3
2007 6 13
XR (VR, AR, MR )

4. ▸
▸ 2011
▸ 2015 R&D
4

5. 5
, , ,

6. ARK
ArkPubSub Server
6

7. 7
2 VS 2

8. 8

9. 9

10. 10
1 VS 1
2 ?

11. 11
2
COPY
COPY
4

12. ▸
▸
12
.
OS

13. 13

14. 14

15. 15

16. 16
HTTP API

17. 17

18. 18

19.
19

20. 20
COPY
COPY

21. 21
COPY
COPY

22. ▸
▸
▸
▸
22

23. ▸
▸
▸
▸
▸
▸
23

24. 24

25. 25

26. 26

27. 27
, 1

28. 28

29. 29
COPY
COPY

30. 30
COPY
COPY

31. 31
COPY
COPY

32. 32

33. 33

34. 34

35. 35

36. 36

37. ▸
▸
▸
37

38. 38

39. ▸
▸
▸
▸
▸
39

40. 40

41. 41

42. ▸
▸
▸
▸
42

43. ▸
▸
▸
▸
▸
43

44. 44
/CHATROOM/AAA
A
B
C
D
PubSub
Subscribe

45. 45
/CHATROOM/AAA
A
B
C
D
PubSub
Publish

46. 46
/CHATROOM/AAA
/TOPIC/BBB
A
B
C
D
PubSub
Subscribe

47. 47
A
B
C
D
PubSub
Publish
/CHATROOM/AAA
/TOPIC/BBB

48. ▸
▸
▸
48

49. Room/1234
A
B
C
PubSub
Topic

50. Room/1234
Room/1234/User/A
A
B
C
PubSub
Room/1234/User/B
Room/1234/User/C
Topic

51. Room/1234
Room/1234/User/A
A
B
C
PubSub
Room/1234/User/B
Room/1234/User/C
Topic
Topic Subscribe
Room/1234/User/D
D

52. ▸
▸
▸
52

53. ▸
▸
▸
53

54. ▸
▸
▸
▸
54

55. 55
ranch
ErlangVM

56. 56
ranch
ErlangVM
session
<0.99.0>

57. 57
ranch
ErlangVM
session
<0.99.0>
Topic/AA <0.80.0>ETS

58. 58
ranch session
<0.99.0>
ErlangVM
Write <0.99.0>
Topic/AA <0.80.0>ETS

59. 59
ranch
ErlangVM
Topic/AA <0.80.0>, <0.99.0>
session
<0.99.0>
ETS

60. ranch
ErlangVM
Topic/AA <0.80.0>, <0.99.0>
session
<0.99.0>
ETS
60

61. 61
ranch session
<0.99.0>
ErlangVM
link
Topic/AA <0.80.0>, <0.99.0>
Read session
<0.80.0>
Send
ETS

62. 62
ranch session
<0.99.0>
ErlangVM
link
Topic/AA <0.80.0>, <0.99.0>
Read session
<0.80.0>
Send
ETS

63. 63

64. 64

65. ▸
▸
▸
▸
65

66. 66
1bit 16
MsgType DUP QoS Retain length
MSB LSB
Payload
Protocol Version
User
Name
Pass
word
Will
retain
Will QoS Will
Clean
session
Keepalive MSB Keepalive LSB

67. ▸
▸
▸
67

68. ▸
▸
▸
▸
68

69. 69
1 bit 16 32
Type ( ) Length ( )
SenderTimestamp ( ) ※
ReceiverTimesamp ( ) ※
Payload ( )

70. 70
1 16 32
SectionType SectionLength
Value

71. 71
1 16 32
Type Length
SenderTimestamp
ReceiverTimesamp
Payload

72. 72
1 16 32
Type Length
SenderTimestamp
ReceiverTimesamp
SectionType SectionLength
Value

73. 73
1 16 32
Type Length
SenderTimestamp
ReceiverTimesamp
SectionType SectionLength
Value
SectionType SectionLength
Value

74. ▸
▸
▸
74

75. 75
CONNECT / CONNECT TOKEN RECONNECT TOKEN
CONNACK SUCCESS CONNACK FAILED / CONNACK FAILED UNAVAILABLE etc
DISCONNECT SUBSCRIBE
UNSUBSCRIBE PUBLISH / PUBLISH ASYNC
PUBLISHED / PUBLISHED ASYNC INFO SUBSCRIBER REQUEST
RTT NOTIFY START / RTT NOTIFY STOP MESSAGE ACK
RESEND REQUEST PING REQUEST / PING RESPONSE
TIME REQUEST / TIME RESPONSE
DGRAM SETUP REQUEST / DGRAM SETUP REQUEST RECEIVED /
DGRAM SETUP COMPLETE
DGRAM REFRESH / DGRAM RE-SETUP REQUEST DGRAM DISCONNECT
ADD SUBSCRIBER / REMOVE SUBSCRIBER
SUBSCRIBE PERMISSION ERROR
/ PUBLISH PERMISSION ERROR
INFO SUBSCRIBER INFO CLIENT RTT

76. 76
1 16 32
Type: Subscribe (0x0400) Length
SenderTimestamp
ReceiverTimesamp
SectionType: MessageID (0x0400) SectionLength: 0x0004
Value: 0x00000001
SectionType: Topic (0x0002) SectionLength
Value: topic/AAA (UTF-8)

77. ▸
▸
▸
77

78. 78
1 bit 16 32
Type ( ) Length ( )
SenderTimestamp ( ) ※
ReceiverTimesamp ( ) ※
Payload ( )

79. 79
Client
Timer
Timer1ms
1000ms
Sender
1
Receiver
301 1300ms
321ms
PubSub
300ms

80. 80
Client
Timer Timer
1
1000
Sender Receiver
1 0 ※
Sender Receiver
1300 301 1300
321
421
Sender Receiver
421 1400
1420
521
1520Sender Receiver
521 1500
PubSub

81. ▸
▸
81

82. 82

83. 83

84. ▸
▸
▸
▸
▸
84

85. 85
ELB
App
Job
MQ
Batch
log

86. ▸
▸
▸
▸
86

87. 87
PubSub
App
Battle
HTTP API
Ark

88. 88
App
1. game start API
2. IP:Port, Token, Topic
3. Connect Token
4. Token 5. OK
6. OK

89. HW
89

90. 90
App
PubSub
PubSub
PubSub
HTTP API
A 2019-01-01 00:00:00 alive
B 2019-01-01 00:00:00 alive
C 2019-01-01 00:00:00 alive
-4 sec
-4 sec
-4 sec
(5 1 ) API API
RDBMS ,

91. 91
App
PubSub
PubSub
PubSub
HTTP API
A 2019-01-01 00:00:00 dead
B 2019-01-01 01:00:00 alive
C 2019-01-01 01:00:00 aliveA
B
C
-3604 sec
-4 sec
-4 sec

92. 92
App
PubSub
PubSub
PubSub
HTTP API
A 2019-01-01 00:00:00 dead
B 2019-01-01 01:00:00 alive
C 2019-01-01 01:00:00 aliveA
B
C
-3604 sec
-4 sec
-4 sec

93. 93
App
PubSub
PubSub
PubSub
HTTP API
A 2019-01-01 00:00:00 dead
B 2019-01-01 01:00:00 alive
C 2019-01-01 01:00:00 aliveA
B
C
-3604 sec
-4 sec
-4 sec

94. 94
App
PubSub
PubSub
PubSub
HTTP API
A 2019-01-01 00:00:00 dead
B 2019-01-01 01:00:00 alive
C 2019-01-01 01:00:00 aliveA
B
C
-3604 sec
-4 sec
-4 sec

95. ▸
95

96. 96
App
Connection
1. heartbeat 2. OK
Server A
Server B
RoomA Server A

97. 97
App
Connection
1. heartbeat 2. OK
Server A
Server B
RoomA Server A

98. 98
App
1. RoomA Server?
2. Server B IP, Port …
Server A
Server B
RoomA Server B
Connection

99. A
99

100. 100
App
Connection
1. heartbeat 2. OK
Server A
Server B
RoomA Server A

101. 101
App
Connection
1. heartbeat 2. OK
Server A
Server B
RoomA Server A
1. RoomA Server?
2. Server B IP, Port …

102. 102
App
Connection
1. heartbeat 2. OK
Server A
Server B
RoomA Server B
1. RoomA Server?
2. Server B IP, Port …
Connection

103. ▸
▸
▸
▸
103

104. 104

105. ▸
▸
▸
▸
▸
105

106. ▸
▸
▸
▸
▸
106

107. 107
Locust Slaves
x8
Locust Master
m4.2xlarge
CPU: 8
RAM: 32GB
m4.xlarge
CPU: 4
RAM: 16GB
PubSub

108. 108
PubSub
1 1
8 x 8 = 64

109. 109
user/room 8
message/user/sec 0.2 waittime[ms]5000
rooms 100 1000 2000 3000 3500 3750
( ) users 800 8000 16000 24000 28000 30000
( ) message/sec 1280 12800 25600 38400 44800 48000
( ) message/sec 1270 12783 25560 38328 44787 21054
(Server)CPU [%] 66.5 360.4 404.7 560.9 635.8 444.4
(Server)RAM [%] 0.6 2.4 3.7 6.2 7.2 16.5
RTT Med [ms] 1 2 5 12 19 34
RTT Avg [ms] 14 11 12 14 27 1260
RTT Max [ms] 44 49 75 356 536 62798
End to End Med [ms] 1 3 8 18 39 460
End to End Avg [ms] 2 12 13 21 50 3017
End to End Max [ms] 747 738 754 1003 1315 100979
66.5 360.4 404.7 560.9 635.8 444.4
1270 12783
25560
38328
44787
21054
0
100
200
300
400
500
600
700
800
0
10000
20000
30000
40000
50000
60000
70000
80000
800 8000 16000 24000 28000 30000
8user/room 1msg/5s
(Server)CPU [%] ( ) message/sec

110. 110
user/room 8
message/user/sec 1 waittime[ms]1000
rooms 100 400 800 900 1000
( ) users 800 3200 6400 7200 8000
( ) message/sec 6400 25600 51200 57600 64000
( ) message/sec 6360 25527 48195 53240 25154
(Server)CPU [%] 275.8 618.3 709.3 774.7 766.7
(Server)RAM [%] 0.5 1.1 2.7 3 11.2
RTT Med [ms] 1 1 5 12 5
RTT Avg [ms] 10 7 24 40 494
RTT Max [ms] 45 48 491 1100 34295
End to End Med [ms] 2 11 36 80 310
End to End Avg [ms] 10 13 54 120 970
End to End Max [ms] 554 550 981 2923 44120
275.8
618.3
709.3
774.7 766.7
6360
25527
48195
53240
25154
0
100
200
300
400
500
600
700
800
0
10000
20000
30000
40000
50000
60000
70000
80000
800 3200 6400 7200 8000
8user/room 1msg/s
(Server)CPU [%] ( ) message/sec

111. 111
user/room 8
message/user/sec 15 waittime[ms]66.6666666666667
rooms 10 50 70 80 90 100
( ) users 80 400 560 640 720 800
( ) message/sec 9600 48000 67200 76800 86400 96000
( ) message/sec 9096 46047 58600 60764 63539 26390
(Server)CPU [%] 229.8 694.7 776.6 775.6 787.9 786.2
(Server)RAM [%] 0.4 0.5 0.6 0.7 0.7 8
RTT Med [ms] 38 2 7 10 15 24
RTT Avg [ms] 30 14 15 17 23 194
RTT Max [ms] 45 67 93 122 522 20686
End to End Med [ms] 19 19 24 33 46 73
End to End Avg [ms] 18 21 27 37 50 293
End to End Max [ms] 618 631 657 718 1231 39470
229.8
694.7
776.6 775.6 787.9 786.2
9096
46047
58600
60764
63539
26390
0
100
200
300
400
500
600
700
800
0
10000
20000
30000
40000
50000
60000
70000
80000
80 400 560 640 720 800
8user/room 15msg/s
(Server)CPU [%] ( ) message/sec

112. TCP
5 MESSAGE / SEC
112

113. ▸
▸
113

114. ▸
▸
▸
114

115. 115
user/room 8
message/user/sec 0.2 waittime[ms]5000
rooms 100 2000 3750 6250 7500 8750
( ) users 800 16000 30000 50000 60000 70000
( ) message/sec 1280 25600 48000 80000 96000 112000
( ) message/sec 1281 25639 48124 78742 94555 97731
(Server)CPU [%] 46.9 312.2 426.8 625 723.2 718
(Server)RAM [%] 2 6.8 9.8 14 17.3 18.7
1281
25639
48124
78742
94555 97731
0
100
200
300
400
500
600
700
800
0
20000
40000
60000
80000
100000
120000
140000
160000
800 16000 30000 50000 60000 70000
8user/room 1msg/5s
(Server)CPU [%] ( ) message/sec

116. 116
user/room 8
message/user/sec 1 waittime[ms]1000
rooms 100 400 800 1000 1200 1400 1600
( ) users 800 3200 6400 8000 9600 11200 12800
( ) message/sec 6400 25600 51200 64000 76800 89600 102400
( ) message/sec 6356 25472 51202 63890 75912 88143 99033
(Server)CPU [%] 139.4 269.1 425 500.3 625.8 699.2 765.3
(Server)RAM [%] 2 3.6 5 5 5.2 5.2 6.6
6356
25472
51202
63890
75912
88143
99033
0
100
200
300
400
500
600
700
800
0
20000
40000
60000
80000
100000
120000
140000
160000
800 3200 6400 8000 9600 11200 12800
8user/room 1msg/s
(Server)CPU [%] ( ) message/sec

117. 117
user/room 8
message/user/sec 15 waittime[ms]66.6666666666667
rooms 10 50 70 80 90 100 110 120 130 140
( ) users 80 400 560 640 720 800 880 960 1040 1120
( ) message/sec 9600 48000 67200 76800 86400 96000 105600 115200 124800 134400
( ) message/sec 8912 45967 64648 74035 83211 91581 99596 106775 112889 116739
(Server)CPU [%] 161.5 374.2 489.7 535.8 582.1 632.1 666.4 703.7 726.9 784.7
(Server)RAM [%] 2.1 2.3 2.3 2.3 2.3 2.2 1.8 2.6 3 3.4
8912
45967
64648
74035
83211
91581
99596
106775
112889 116739
0%
100%
200%
300%
400%
500%
600%
700%
800%
0
20000
40000
60000
80000
100000
120000
140000
10 50 70 80 90 100 110 120 130 140
8user/room 15msg/s
(Server)CPU [%] ( ) message/sec

118. 10 MESSAGE / SEC
118

119. ▸
▸
▸
▸
119

120. ▸
▸
▸
120

121. 121