In this talk we'll talk in details on how modern databases are capable of performing backups without downtimes and how these backups later can be used to restore database to any point in time. While the talk describes generally applicable approach, a Litestream (SQLite backup service) is used for reference implementation.

1. Databases: zero-downtime
backup & point-in-time
recovery

2. INTRODUCTION
Bartosz Sypytkowski
▪ @Horusiath
▪ b.sypytkowski@gmail.com
▪ bartoszsypytkowski.com

4.  How databases persist data on disk
 How to backup changes
 How to restore database
 Optimizations and tricks
AGENDA

5. B+TREE

6. B+TREE
P1
k6 k8 k10 k11
P2
k2 k4 k6
P3
k7 v7 k8 v8
P4
k9 v9 k10 v10
P5
k11 v11
P6
k1 v1 k2 v2
P7
k3 v3 k4 v4
P8
k5 v5 k6 v6

7. B+TREE
ON DISK
P1
P2 P3 P8 P9
P5 P6 P10
P1 P2 P3 P5 P6 P8 P9
Database file
P10
Free pages
Root page
4KiB 4KiB 4KiB
4KiB 4KiB
4KiB

8. COMPAR
ING
VECTOR
CLOCKS
SQLITE
ARCHITECTURE
Core
Backend
Compiler
Interface
SQL Command
Processor
Virtual Machine
B+Tree
Pager
Virtual File System
Tokenizer
Parser
Code Generator
Rollback
Journal
Write-Ahead
Log

9. ROLLBACK JOURNAL

10. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal

11. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal
Locate page
with the
record

12. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal
P2
Copy pages to
Rollback
Journal
P1

13. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal
P2
P1
Modify
affected
pages

14. SQLITE
ROLLBACK
JOURNAL P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Rollback
Journal

15. WRITE-AHEAD LOG

16. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log

17. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
Locate page
with the
record

18. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
P1
P2
Write new
page versions

19. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
UPDATE users SET name = ‘Joe’ WHERE id = 1;
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
P1
P2
Pager
redirects
pages to new
versions

20. WRITE-AHEAD LOG
CHECKPOINT

21. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
PRAGMA checkpoint(TRUNCATE);
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
P1
P2

22. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
PRAGMA checkpoint(TRUNCATE);
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log
P1
P2
Override DB
file with log
data

23. SQLITE
WRITE-AHEAD
LOG P1
P2 P3
PRAGMA checkpoint(TRUNCATE);
B+Tree
Pager
Database
File
P1
P2
P3
Write-Ahead
Log

24. EVERY CHANGED PAGE MUST BE
FIRST APPENDED AT THE END OF
THE WRITE-AHEAD LOG FILE

25. EVERY CHANGE IN THE DATABASE
FILE COMES FROM READING
FRONT-TO-BACK WRITE-AHEAD
LOG FILE
EVERY CHANGED PAGE MUST BE
FIRST APPENDED AT THE END OF
THE WRITE-AHEAD LOG FILE

26. LOG CAN BE PRETTY USEFUL

27. SQLITE BACKUP

28. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive

29. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1

30. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1 P4
F2

31. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1 P4
F2 P3
F3

32. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1 P4
F2 P3
F3 P1
F4

33. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1 P4
F2 P3
F3
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4

34. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1 P4
F2 P3
F3
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4

35. BACKING UP CHANGED PAGES
101
Write-Ahead Log
Database process
Backup service Backup drive
P3
F1 P4
F2 P3
F3
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4

36. RESTORE

37. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00

38. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
restore(2024/02/11/10:00:00)

39. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00

40. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1

41. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1
P1
F4
P2
F5
F4-F5-2024/02/10/07:54:10

42. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5

43. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
PRAGMA CHECKPOINT(TRUNCATE);

44. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5

45. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5

46. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5

47. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5

48. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5

49. POINT-IN-TIME RECOVRERY
101
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00

50. WHAT IF LOG FILE IS TOO BIG?

51. COMPAR
ING
VECTOR
CLOCKS
WRITE-AHEAD
LOG FRAMES
https://www.sqlite.org/fileformat.html#the_write_ahead_log
Magic number File version
Database page size Checkpoint Seq. No.
0 4B 8B
Salt
Checksum
WAL HEADER
Page number DB size after commit
Salt
Checksum
Page data (4KiB)
Page number DB size after commit
WAL FRAME 1
WAL FRAME 2

52. COMPAR
ING
VECTOR
CLOCKS
WRITE-AHEAD
LOG FRAMES
https://www.sqlite.org/fileformat.html#the_write_ahead_log
Magic number File version
Database page size Checkpoint Seq. No.
0 4B 8B
Salt
Checksum
Page number DB size after commit
Salt
Checksum
Page data (4KiB)
Page number DB size after commit
if NOT 0, current
frame marks
transaction commit

53. ROLLBACK ISSUE

54. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
BEGIN DEFERRED;
T1
EOF

55. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1
INSERT INTO t(name) VALUES(‘John Doe’);
size_after=0
T1
T1
EOF

56. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1
COMMIT;
P1
F2
size_after=0 size_after=3
T1
T1
EOF

57. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1 P1
F2
size_after=0 size_after=3
T1
BEGIN DEFERRED;
T2
EOF

58. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1 P1
F2
size_after=0 size_after=3
T1
T2
UPDATE t SET banned = 1;
P3
F3 P1
F4
size_after=0 size_after=0
EOF

59. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1 P1
F2
size_after=0 size_after=3
T1
P3
F3 P1
F4
size_after=0 size_after=0
ROLLBACK;
EOF

60. TRANSACTION
COMMIT & ROLLBACK
Write-Ahead Log
Database process
P3
F1 P1
F2
size_after=0 size_after=3
T1
P3
F3 P1
F4
size_after=0 size_after=0
ROLLBACK;
EOF
what if we already backed up these
frames? :/

61. DON’T BACKUP FRAMES THAT WERE NOT CONFIRMED AS
COMMITTED

62. POINT-IN-TIME RECOVRERY PROBLEM
Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
restore(2024/02/11/10:00:00)

63. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
P3
F1 P3
F1 P3
F1 P1
F4 P2
F5
POINT-IN-TIME RECOVRERY PROBLEM
F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10

64. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
INSERT INTO t(name)
VALUES(‘John Doe’);

65. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2

66. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
P5
F1
P1
F2
F1-F2-2024/02/13/09:30:00

67. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
P5
F1
P1
F2
F1-F2-2024/02/13/09:30:00

68. Write-Ahead Log
Database process
Backup service Backup drive F1-F3-2024/02/10/07:54:00
F4-F5-2024/02/10/07:54:10
F6-F7-2024/02/11/11:02:00
F8-F9-2024/02/12/20:30:00
POINT-IN-TIME RECOVRERY PROBLEM
P5
F1 P1
F2
P5
F1
P1
F2
F1-F2-2024/02/13/09:30:00
Database history is no longer linear!

69. GENERATIONS
SPLIT LOG HISTORY INTO SESSIONS

70. COMPAR
ING
VECTOR
CLOCKS
GENERATIONS
G1/F1-F3-2024/02/10/07:54:00
Backup drive
G1/.snapshot
G1/F4-F6-2024/02/10/09:30:00
G1/F7-F8-2024/02/11/03:11:00
G1/F9-F10-2024/02/12/14:44:10
G2/.parent
G2/F1-F2-2024/02/11/04:39:13
G2/F3-F8-2024/02/12/05:12:55
G1
G2

71. DELTAS
COMPACTING HISTORY

72. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process

73. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process
Visited pages:
P3
P3
H1
Delta file
Artificial Delta
Header

74. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process
Visited pages:
P2 P3
P3
H1
Delta file
P2
H1

75. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process
Visited pages:
P1 P2 P3
P3
H1
Delta file
P2
H1
P1
H3

76. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process
Visited pages:
P1 P2 P3
Skip P3 since it’s
already present in
visited pages
P3
H1
Delta file
P2
H1
P1
H3

77. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process
Visited pages:
P1 P2 P3 P4
P3
H1
Delta file
P2
H1
P1
H3
P4
H4

78. DELTAS
Backup drive
P3
F1
P4
F2
P3
F3
F1-F3-2024/02/10/07:54:00
P1
F4
P2
F5
P3
F6
F4-F6-2024/02/10/09:33:00
Compacting process
P3
H1
Delta file
Visited pages:
P1 P2 P3 P4
P2
H1
P1
H3
P4
H4
Skip P3 since it’s
already present in
visited pages

79. DELTAS CAN BE EASILY MERGED

80. COMPAR
ING
VECTOR
CLOCKS
DELTA MERGE
P3
H1
Delta
file
P2
H1
P1
H3
P4
H4
F1-F6-2024/02/10/09:33:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F7-F13-2024/02/10/12:02:00

81. COMPAR
ING
VECTOR
CLOCKS
DELTA MERGE
P3
H1
Delta
file
P2
H1
P1
H3
P4
H4
F1-F6-2024/02/10/09:33:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F7-F13-2024/02/10/12:02:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F1-F13-2024/02/10/12:02:00
Latest delta can be
copied as is
Visited pages:
P1 P3 P4 P5

82. COMPAR
ING
VECTOR
CLOCKS
DELTA MERGE
P3
H1
Delta
file
P2
H1
P1
H3
P4
H4
F1-F6-2024/02/10/09:33:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F7-F13-2024/02/10/12:02:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F1-F13-2024/02/10/12:02:00
Visited pages:
P1 P3 P4 P5
Skip P3 since it’s
already present in
visited pages

83. COMPAR
ING
VECTOR
CLOCKS
DELTA MERGE
P3
H1
Delta
file
P2
H1
P1
H3
P4
H4
F1-F6-2024/02/10/09:33:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F7-F13-2024/02/10/12:02:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F1-F13-2024/02/10/12:02:00
Visited pages:
P1 P2 P3 P4 P5
P2
H1

84. COMPAR
ING
VECTOR
CLOCKS
DELTA MERGE
P3
H1
Delta
file
P2
H1
P1
H3
P4
H4
F1-F6-2024/02/10/09:33:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F7-F13-2024/02/10/12:02:00
F1-F13-2024/02/10/12:02:00
Visited pages:
P1 P2 P3 P4 P5
Skip P1 since it’s
already present in
visited pages
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
P2
H1

85. COMPAR
ING
VECTOR
CLOCKS
DELTA MERGE
P3
H1
Delta
file
P2
H1
P1
H3
P4
H4
F1-F6-2024/02/10/09:33:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F7-F13-2024/02/10/12:02:00
P5
H1
Delta
file
P1
H1
P3
H3
P4
H4
F1-F13-2024/02/10/12:02:00
Visited pages:
P1 P3 P4 P5
Skip P4 since it’s
already present in
visited pages

86. SUMMARY

87.  How does continuous backup and point-in-time recovery work in databases:
https://www.bartoszsypytkowski.com/db-backup-point-in-time-recovery
 Litestream: https://litestream.io/
 SQLite write-ahead log docs: https://www.sqlite.org/wal.html
REFERENCES