The document discusses logging and monitoring techniques for services. It provides examples of using common Unix tools like grep, awk, sed etc to extract and analyze log data. It also covers issues that can occur like slow uploads, storage degradation, network problems and ways to identify them through logging. Video processing and uploads are given as an example throughout.

1. Как писать сервис,
п ржка которого
не превращается в ад
Антон Резников
Игорь Мунькин

2. Syllabus
✘ Code
✘ Config
✘ Theory
✔ Experience
2 / 105

3. 3 / 105

4. Why do we need logs?
✱ Investigation
✱ Debugging
✱ Statistics
✱ Monitoring
4 / 105

5. 5 / 105

6. 6 / 105

7. 7 / 105

8. 8 / 105

9. 9 / 105

10. 10 / 105

11. 11 / 105

12. 12 / 105

13. 13 / 105

14. 14 / 105

15. 15 / 105

16. Our tools
16 / 105

17. Our tools
grep
17 / 105

18. Our tools
grep
awk
18 / 105

19. Our tools
grep
awk
sed
19 / 105

20. Our tools
grep
awk
sed
sort
20 / 105

21. Our tools
grep
awk
sed
sort
cut
21 / 105

22. Our tools
grep
awk
sed
sort
cut
xargs
22 / 105

23. Our tools
grep
awk
sed
sort
cut
xargs
tail
23 / 105

24. Our tools
grep
awk
sed
sort
cut
xargs
tail
head
24 / 105

25. Our tools
grep
awk
sed
sort
cut
xargs
tail
head
uniq
25 / 105

26. Our tools
grep
awk
sed
sort
cut
xargs
tail
head
uniqPERL
26 / 105

27. Who framed Roger Rabbit
2017-10-31 17:34:04 Jessica Rabbit <SAVE>
2017-10-31 17:34:04 Judge Doom <SAVE>
2017-10-31 17:34:04 Roger Rabbit <SAVE>
27 / 105

28. Who framed Roger Rabbit
2017-10-31 17:34:04 Jessica Rabbit <SAVE>
2017-10-31 17:34:04 Judge Doom <SAVE>
2017-10-31 17:34:04 Roger Rabbit <SAVE>
28 / 105

29. Who framed Roger Rabbit
2017-10-31 17:34:04.010 Jessica Rabbit <SAVE> #1
2017-10-31 17:34:04.990 Judge Doom <SAVE> #3
2017-10-31 17:34:04.550 Roger Rabbit <SAVE> #2
29 / 105

30. 30 / 105

31. Race
***
***
***
2017-10-31 17:34:04.545 [END=500] user/create ...
***
2017-10-31 17:34:04.595 [END=200] user/create ...
***
***
31 / 105

32. Race
***
2017-10-31 17:34:04.540 Duplicate key exists in
unique index 0
2017-10-31 17:34:04.545 [END=500] user/create ...
***
2017-10-31 17:34:04.595 [END=200] user/create ...
***
***
32 / 105

33. Race
***
2017-10-31 17:34:04.540 Duplicate key exists in
unique index 0
2017-10-31 17:34:04.545 [END=500] user/create time:
0.300s ...
2017-10-31 17:34:04.595 [END=200] user/create time:
0.400s ...
***
33 / 105

34. Race
34 / 105

35. Race
35 / 105

36. Race
36 / 105

37. Race
2017-10-31 17:34:04.195 START user/create
***
2017-10-31 17:34:04.245 START user/create
***
***
***
***
***
***
***
***
***
***
***
***
2017-10-31 17:34:04.545 [END=500] user/create time: 0.300s ...
***
2017-10-31 17:34:04.595 [END=200] user/create time: 0.400s ...
37 / 105

38. Race
2017-10-31 17:34:04.195 START user/create
2017-10-31 17:34:04.217 Auth
2017-10-31 17:34:04.245 START user/create
2017-10-31 17:34:04.247 Cache lookup
2017-10-31 17:34:04.256 Auth
2017-10-31 17:34:04.256 Select
2017-10-31 17:34:04.256 Cache lookup
2017-10-31 17:34:04.256 Get profile
2017-10-31 17:34:04.256 Select
2017-10-31 17:34:04.256 Lock
2017-10-31 17:34:04.256 Insert
2017-10-31 17:34:04.256 Get profile
2017-10-31 17:34:04.256 Unlock
2017-10-31 17:34:04.256 Lock
2017-10-31 17:34:04.256 Cache update
2017-10-31 17:34:04.545 [END=500] user/create time: 0.300s ...
2017-10-31 17:34:04.569 Init tree
2017-10-31 17:34:04.595 [END=200] user/create time: 0.400s ...
38 / 105

39. Race
2017-10-31 17:34:04.195 [Jgz36] START user/create
2017-10-31 17:34:04.217 [Jgz36] Auth
2017-10-31 17:34:04.245 [W4IL6] START user/create
2017-10-31 17:34:04.247 [Jgz36] Cache lookup
2017-10-31 17:34:04.256 [W4IL6] Auth
2017-10-31 17:34:04.256 [Jgz36] Select
2017-10-31 17:34:04.256 [W4IL6] Cache lookup
2017-10-31 17:34:04.256 [Jgz36] Get profile
2017-10-31 17:34:04.256 [W4IL6] Select
2017-10-31 17:34:04.256 [Jgz36] Lock
2017-10-31 17:34:04.256 [Jgz36] Insert
2017-10-31 17:34:04.256 [W4IL6] Get profile
2017-10-31 17:34:04.256 [Jgz36] Unlock
2017-10-31 17:34:04.256 [W4IL6] Lock
2017-10-31 17:34:04.256 [Jgz36] Cache update
2017-10-31 17:34:04.545 [W4IL6] [END=500] user/create time: 0.300s ...
2017-10-31 17:34:04.569 [Jgz36] Init tree
2017-10-31 17:34:04.595 [Jgz36] [END=200] user/create time: 0.400s ...
39 / 105

40. Race
2017-10-31 17:34:04.195 [Jgz36] START user/create
2017-10-31 17:34:04.217 [Jgz36] Auth
2017-10-31 17:34:04.245 [W4IL6] START user/create
2017-10-31 17:34:04.247 [Jgz36] Cache lookup
2017-10-31 17:34:04.256 [W4IL6] Auth
2017-10-31 17:34:04.256 [Jgz36] Select
2017-10-31 17:34:04.256 [W4IL6] Cache lookup
2017-10-31 17:34:04.256 [Jgz36] Get profile
2017-10-31 17:34:04.256 [W4IL6] Select
2017-10-31 17:34:04.256 [Jgz36] Lock
2017-10-31 17:34:04.256 [Jgz36] Insert
2017-10-31 17:34:04.256 [W4IL6] Get profile
2017-10-31 17:34:04.256 [Jgz36] Unlock
2017-10-31 17:34:04.256 [W4IL6] Lock
2017-10-31 17:34:04.256 [Jgz36] Cache update
2017-10-31 17:34:04.545 [W4IL6] [END=500] user/create time: 0.300s ...
2017-10-31 17:34:04.569 [Jgz36] Init tree
2017-10-31 17:34:04.595 [Jgz36] [END=200] user/create time: 0.400s ...
40 / 105

41. Race
41 / 105

42. Race
42 / 105

43. Race
43 / 105

44. Race
44 / 105

45. Race
45 / 105

46. Race
46 / 105

47. Race
47 / 105

48. Race
48 / 105

49. Race
49 / 105

50. Race
50 / 105

51. Race
51 / 105

52. Race
52 / 105

53. Race
53 / 105

54. Race
54 / 105

55. Race
55 / 105

56. Race
56 / 105

57. Race
57 / 105

58. 58 / 105

59. Theory of relativity
... [END=200] t=0.850s ...
normal too long
59 / 105

60. Theory of relativity
... [END=200] t=0.850s ...
normal too long
60 / 105

61. 61 / 105

62. Long connection story
17:30:05.010 GET /HLJ2017.hief
17:30:05.054 Auth
***
***
***
***
***
18:37:39.990 END=200
62 / 105

63. Long connection story
17:30:05.010 +0.000 GET /HLJ2017.hief
17:30:05.054 +0.044 Auth
***
18:00:00.010 +1795.000 done:45%, wb:100%, r:860kB/s
***
18:20:00.010 +2995.000 done:85%, wb:50%, r:460kB/s
***
18:37:39.990 +4027.980 END=200
63 / 105

64. Long connection story
wbuf=0% wbuf=100%
64 / 105

65. Client side problems
Long connection story
wbuf=0% wbuf=100%
65 / 105

66. Client side problems
Long connection story
wbuf=0% wbuf=100%
wbuf=0%
66 / 105

67. Client side problems
Storage degradation
Long connection story
wbuf=0% wbuf=100%
wbuf=0%
67 / 105

68. Client side problems
Storage degradation
Long connection story
wbuf=0% wbuf=100%
wbuf=0%
wbuf=100% wbuf=0%
68 / 105

69. Client side problems
Storage degradation
Network problems
Long connection story
wbuf=0% wbuf=100%
wbuf=0%
wbuf=100% wbuf=0%
69 / 105

70. 70 / 105

71. Video
71 / 105

72. Video
72 / 105

73. Video
73 / 105

74. Video
74 / 105

75. Video
75 / 105

76. Video
Options
Storage degradation
Video conversion problems
Network problems
Cache node degradation
76 / 105

77. Video
Options
Storage degradation
Video conversion problems
Network problems
Cache node degradation
... video processing: ffmpeg: 3.3s, upload: 3.5s, size: 5Mb ...
77 / 105

78. grep | grep | sort | sed | xargs | awk |
sort | uniq | sort | head | xargs |
grep | cut | awk
Video
Options
Storage degradation
Video conversion problems
Network problems
Cache node degradation
78 / 105

79. Video
Let's log it!
... ffmpeg t: 3.3s, q: 1080p, p: 27 ...
... upload t: 3.5s, sz: 5Mb, r: 1.4kB/s n: 3 ...
79 / 105

80. Video
Let's log it!
... ffmpeg t: 3.3s, q: 1080p, p: 27 ...
... upload t: 3.5s, sz: 5Mb, r: 1.4kB/s n: 3 ...
Let's graph it!
80 / 105

81. 81 / 105

82. And video again
***
18:37:39.980 [END=200] GET /video.hief
18:37:39.990 [END=200] GET /video.hief
***
82 / 105

83. And video again
Error
No Access-Control-Allow-Origin
83 / 105

84. And video again
18:37:39.980 [END=200] GET ... Origin='mail.ru'
18:37:39.990 [END=200] GET ... Origin=''
84 / 105

85. And video again
18:37:39.980 [END=200] GET ... Origin='mail.ru'
18:37:39.990 [END=200] GET ... Origin=''
Everybody Lies
85 / 105

86. And video again
Host: cloud.mail.ru
Referer: https://cloud.mail.ru/
X-Real-Ip: 8.9.8.9
Content-Length: 575
Cookie: session_id
Connection: close
Accept: application/octet-stream
86 / 105

87. And video again
Host: cloud.mail.ru
Referer: https://cloud.mail.ru/
X-Real-Ip: 8.9.8.9
Content-Length: 575
Cookie: session_id
Connection: close
Accept: application/octet-stream
87 / 105

88. And video again
88 / 105

89. And video again
89 / 105

90. And video again
90 / 105

91. And video again
91 / 105

92. 92 / 105

93. Query of death
SMS: 2017-11-04T19:19:10 App worker ended with SIGABRT
Email: http://store.local/lightning.core
93 / 105

94. Query of death
SMS: 2017-11-04T19:19:10 App worker ended with SIGABRT
Email: http://store.local/lightning.core
What's going on?
94 / 105

95. Query of death
SMS: 2017-11-04T19:19:10 App worker ended with SIGABRT
Email: http://store.local/lightning.core
What's going on?
12 servers * 10 workers * 10 rps = 1200 requests
95 / 105

96. Query of death
12002017-10-31 17:34:04.195 [Query] START ..
1200 / 12 = 100.. websrv9 ..
1200 / 12 / 10 = 10C2[10542]
96 / 105

97. Query of death
SMS: 2017-11-04T19:19:10 <websrv9> Lightning worker[29453] ended with
SIGABRT
Email: http://store.local/websrv9.lightning.29453.core
97 / 105

98. 98 / 105

99. Again request id
99 / 105

100. Tracking request id
100 / 105

101. Client id
101 / 105

102. 102 / 105

103. 103 / 105

104. 104 / 105

105. 105 / 105