Backup automation in KAKAO

1. Backup Automation
SUNG DONGCHAN From KAKAO

2. Backup is for..

3. 2. Replication
1. Recovery
3. etc...

4. Backup Method?

5. 2. xtrabackup
1. mysqldump
3. snapshot

6. Backup Storage?

7. Local Disk
Easy, No scheduling
But.. Recovery?

8. Remote Storage
Network Traffic?
Rare affect on Recovery
NIC

9. Backup Start Time?

10. • Schedule with CRONTAB
• Start from manager server on DEFINED TIME
(BUT, Everyday it’s same time..)
Same time

11. No Problem
if it's on LOCAL BACKUP

12. Think about
REMOTE BACKUP

13. A lot of Servces

14. Multi-IDC
IDC1 IDC2
IDC3

15. Frequently
Master/Slave Switch

16. X
Schedule Manually?

17. We need..
Scheduling Automatically

18. TODAY!!
I’ll talk about remote backup scheduling

19. Backup Files?
• Big.. but Rarely Used
• Transaction Log exists 2~3 Days

20. THEN!!!!

21. MUST "Same Backup Server"
As Yesterday?
Question 1.
NO

22. Question 2.
MUST "Today's Backup File"?
NO

23. Question 3.
MUST “Backup on Same Time"?
NO

24. Think about..
Backup File is in Cloud
A B
C D E F

25. Today Yesterday
The day before yesterday
Everyday, Different

26. Let's start backup on
Unhurried Server

27. Consists of..
• Service Configuration
• Backup Servers
• Backup History

28. Service Configuration
Queue Time?
Backup job could be started
ANY TIME after QUEUE TIME

29. Backup Server
Quota?
0/3
3/3
1/3
Quota Full
I'm So Busy!!

30. Backup Server
Located on “Specific RACK”

31. But..
Database could be on..
“Different” IDC!!

32. need logical Concept..

33. Service Zone!!
Virtual Service Group
ex) Common Zone, Security Zone, ThirdParty Zone..

34. IDC1 IDC2 IDC3
Common Service Zone
Security Zone
ThirdParty Zone
Service Zone

35. Backup History
Today? Not Yesterday Server!!
BACKUP001.kr
Yesterday
NOT!!
BACKUP001.kr
so.. BACKUP001.kr
BACKUP002.kr
BACKUP003.kr
Service
Candidate Today's Backup Server

36. Backup Step
• Choose Service base on Queue Time
• Choose Target Backup Server
(Service Zone + IDC Location + Quota)
• Start Backup

37. Choose Service (1/3)
SVC01 SVC02 SVC03 SVC0400:00
Smaller Queue Time, Backup First
01:59 02:00 02:00 03:00

38. Choose Service (2/3)
SVC01 SVC02 SVC03 SVC0400:00
Bigger Data File, Backup First
02:00 02:00 02:00 02:00
500G 300G 200G 100G

39. Choose Service (3/3)
SVC01 SVC02 SVC03 SVC0400:00
Different Switch, Backup First
02:00 02:00 02:00 02:00
100G 100G 100G 100G
SW01 SW02 SW03 SW01

40. Choose Backup Server (1/3)
Target Backup Server List?
• Same Service Zone
• Same IDC

41. Choose Backup Server (2/3)
Less than Quota
0/3
3/3
1/3
Quota Full
I'm So Busy!!
First
Candidate

42. Choose Backup Server (3/3)
Not Used Yesterday
Backuped
Yesterday
Candidate Today's Backup Server

43. So..
Backup Process is..

44. Start Backup
At Service after queue time
on Unhurried and Valid Backup Server
Backup ServerService Zone

45. Backup Service Architect
Databases
Zone1
Databases
Databases
Zone2
Databases
Manager
Schedule Automatically
Backup Servers Backup Servers
Backup Servers Backup Servers
IDC1 IDC2

46. Backup Timeline

47. Restore Sample
ssh root@BACKUP_SERVER
'cat /data/backup/SERVICE_HOST/DATE.xbs' | xbstream -x
ssh root@backup001
'cat /data/backup/svc01/20150427.xbs' | xbstream -x
Generated By Manager

48. With LZOP?
Recommand! If InnoDB :)
Compressed

49. After Queue Time
Same IDC
Same Service Zone
Unhurried Backup Server
Backup Scheduled..

50. Q/A

51. Thank you~!
< http://gywn.net >