Are you tired of managing backup and recovery processes of your Postgres server using custom scripts? Are you worried about being called in the night or worse whilst on holiday to perform a recovery operation of your company's business critical PostgreSQL server? Barman, Backup and Recovery Manager, standardises backup and recovery operations, allowing database administrators and system administrators to easily integrate their PostgreSQL solutions in their disaster recovery plan. Barman is distributed as open source under GNU GPL 3 terms. Further information available at www.pgbarman.org.

1. COLD SWEAT

4. GABRIELE BARTOLINI
PGConf.EU 2013 - Dublin, 30 October 2013

5. GABRIELE BARTOLINI
• Co-Founder
and Manager of 2ndQuadrant Italia
• Data Architect, Business
• Data
critical environments
warehousing
• Co-Founder
Italian PostgreSQL Users Group
• Co-Founder
PostgreSQL Europe
• PostgreSQL
Contributor and Advocate

6. DISCLAIMER
This talk assumes you are familiar with disaster
recovery concepts and PostgreSQL
implementation of Point In Time Recovery

7. BE AWARE
In 2ndQuadrant, all these concepts usually fit in a
2 day workshop on Disaster Recovery and a 1 day
workshop on Barman alone

8. OUTLINE
• Business
continuity / Disaster recovery for databases
• Disaster
recovery with Barman for PostgreSQL

9. PART I
Business continuity / Disaster recovery for databases

10. BUSINESS CONTINUITY
activity performed by an organisation to ensure that critical
business functions will be available to customers, suppliers,
regulators, and other entities that must have access to those
functions - Wikipedia

11. INFORMATION TECHNOLOGY
•
Business continuity
•
High availability
•
Disaster recovery

12. LAW REQUIREMENTS
In Italy, the “Codice dell’Amministrazione Digitale”
defines business continuity requirements for public
administrations

13. WARNING FOR THE
SUPERSTITIOUS
You are now allowed to touch wood

14. DISASTER
(too late for touching wood now)
system/hardware failures
unintentional errors
natural disaster

15. REACT TO A DISASTER
Recover systems, data and infrastructures

16. TOO LATE!
Do not wait for a disaster to happen

17. PLAN FOR DISASTERS
“Disasters” will happen. Be prepared.

18. “Plans are worthless, but planning is
everything. There is a very great distinction
because when you are planning for an emergency
you must start with this one thing: the very definition
of "emergency" is that it is unexpected,
therefore it is not going to happen the way you are
planning.”
- Dwight D. Eisenhower

19. REGULAR CRASH TESTS

20. DATABASE
DISASTER RECOVERY
Let’s just focus on databases!

21. REQUIREMENTS
•
Automated backups
•
Retention policies
•
Notifications (anomalies)
•
Data protection
•
Frequency of backups
•
Availability for recovery

22. TYPES OF BACKUP
•
Full backup
•
Hot backup
•
Incremental backup
•
Logical backup
•
Differential backup
•
Physical backup

23. POSTGRES BACKUP
• Hot
backup
• MVCC
• Logical
(core)
Backup
• pg_dump
• Physical
• Full
Backup
backup (base backup)
• Differential
• Incremental
• N/A
backup (WAL)
backup

24. TRADITIONAL DR
WITH POSTGRESQL
• PostgreSQL
primitives for
DR are robust and reliable
• High
level skills
• DBA
• Sysadmins
• Custom
• Hard
scripts
to integrate in:
• Backup
solutions
• Disaster
• Hard
Recovery plans
to test!

25. EXISTING TOOLS
• Omni-PITR
• WAL
centric
• WAL-E
• EC2
centric, but ...
• WALmgr
• good
• WAL
• came
centric
• pg-rman
• Server
centric
later

26. NONE FOR DR
None of them was a pure disaster recovery solution.
We wanted something similar to Oracle’s RMAN.

27. FILLING A HOLE
The lack of a DR solution is a barrier towards the adoption
of PostgreSQL from Oracle users’ point of view.

28. OUR GOALS
• Hot, Full, Differential
Incremental backups
and
• Multiple
servers
• Remote
backup & recovery
• Backup
catalogues
• Retention
policies
• Archival
• WAL
and compression
segments
• Periodical
backups
• Automation
• Integration
• Usability

29. WWW.PGBARMAN.ORG

30. PART II
Disaster recovery with Barman for PostgreSQL

31. BARMAN
• GNU
GPL 3
• Hosted
on Sourceforge.net
• Linux
• Python
package
• Debian/Ubuntu
package
• Designed, developed,
2.6/2.7 (3.0 exp.)
• PostgreSQL
• PyPI
• RPM
8.4 to 9.3
package
maintained by 2ndQuadrant

32. Postgres
Postgres
Barman
tape
LAN, hybrid architecture
centralised architecture
Postgres
Barman

33. data centre 1
data centre 2
Postgres
Barman
rsync
Geographic redundancy
Barman

34. Postgres
Continuous archiving
(WAL shipping via SSH)
SSH commands
SQL commands
Barman

35. Postgres
WAL
Barman
secure channel
cron
Barman’s
WAL archive

36. Postgres
Periodical backup (weekly)
Differential backup
Backup catalogue
Barman
Full backup - Sat 1, 4AM
Full backup - Sat 8, 4AM
Full backup - Sat 15, 4AM
Full backup - Sat 22, 4AM

37. WAL archive
Backup 1
(100MB)
WAL
Backup 2
(105MB)
WAL
WAL
WAL
WAL
WAL
WAL
WAL
WAL
WAL
Size: 100MB + 80MB == 260MB Size: 105MB + 80MB = 185MB
160MB 180MB

38. CONFIGURATION FILE
[barman]
barman_home = /srv/barman
barman_user = barman
log_file = /var/log/barman/barman.log
log_level = NOTICE
compression = gzip
[production]
description = Production PostgreSQL
ssh_command = ssh postgres@pg.2ndQuadrant.it
conninfo = host=pg.2ndQuadrant.it user=postgres
compression = bzip2

39. MULTI-SERVER
CONFIGURATION
[barman]
; General configuration
; …
[server_one]
; Configuration for Server 1
; …
[server_two]
; Configuration for Server 2
; …
[server_X]
; …

40. MULTIPLE FILES INCLUSION
[barman]
; General configuration
; …
configuration_files_directory = /etc/barman.d

41. CONVENTION OVER
CONFIGURATION
global/per server options
default directory layout

42. CONVENTIONAL
DIRECTORIES FOR BARMAN
• barman_home
• server
(/srv/barman)
directory (/srv/barman/production)
• base
directory (/srv/barman/production/base)
• WAL
directory (/srv/barman/production/wals)
• incoming
directory (/srv/barman/production/incoming)

43. GLOBAL COMMANDS
• List
of managed servers
• barman
list-server
• Maintenance
• barman
cron
operations

44. SERVER COMMANDS
• Information
diagnostics
and
• barman
status
• barman
check
• barman
show-server
• barman
list-backup
• Backup
control
• Recovery
control

46. BACKUP CONTROL
• barman
backup
• barman
show-backup
• barman
list-files
• standalone, data, wal, full
• barman
delete

48. SHOW BACKUP
• General
• Server
name, Postgres
version, status, ...
• Base
backup
• Start/End
time, first/last
WAL, disk usage, ...
• WAL
• Number
• disk
of associated files
usage
• Context
• Previous/Next
backup

50. RECOVERY CONTROL
• Recovery
• Local
target (full / point in time)
recovery
• barman
• Remote
• barman
recover
recovery
recover --remote-ssh-command

51. ADVANCED RECOVERY
• Point
In Time
Recovery
• --target-time
• --target-xid
= TIME
= XID
• --target-name
(for 9.1+)
= NAME
• Relocation
of
tablespaces
• --tablespace
NAME:LOCATION [...]

52. COMMON USE CASES
• Unintentional
errors recovery
• Disaster
recovery
• Sandbox
server (BI, staging, ...)

54. RETENTION POLICIES
• User-defined
policy
• How
long backups are retained for recovery
• Point
of recoverability
• REDUNDANCY
• RECOVERY WINDOW

55. RETENTION POLICY
CONFIGURATION
; Base backup retention policy
retention_policy = 'redundancy 3'
retention_policy = 'recovery window of 3 months'

56. BANDWIDTH CONTROL
• You
can limit I/O bandwidth usage
• bandwidth_limit
global/server option
• tablespace_bandwidth_limit
• Unit
on a per tablespace basis
of measure: kilobytes (default 0, no limits)

57. BANDWIDTH LIMIT
CONFIGURATION
[barman]
bandwidth_limit = 4096
[server_one]
;...
bandwidth_limit = 1024
[server_two]
;...
tablespace_bandwidth_limit = tbs01:4096,tbs02:2048

58. BACKLOG
•
Incremental backup
•
SSH only connections
•
Better recovery support
•
Replication protocol support
•
Sandbox recovery
•
libpq only connections
•
Logical backup integration
•
pg_basebackup
•
WAL streaming (0 Data Loss)
•
Backup server
•
pg_dump on sandbox instances
•
Backup from standby
•
More hook scripts
•
Windows support
•
TAR format for backups
•
JSON output for full automation
•
Export/Import of backups
•
Backup validation
•
External backups
•
...

59. OUR COMMITMENT
• Keep
it open source
• Reinvest
money from sale of DR turnkey solutions in R&D
• Support
and maintain RPM/Debian packages
• Accept
sponsorships for new features development

60. CSI PIEMONTE
(One of the top 10 ICT companies in Italy for revenue)
“We found in Barman the optimal solution for physical backup
and disaster recovery of PostgreSQL databases. Barman is
robust and easy to use. Its command interface allows an easy
integration with the existing management tools in our
enviroment.”
Sponsors of RPM package and WAL compression

61. CONCLUSIONS
• Hides
• Not
complexity of PITR / Keeps unaltered PITR strenghts
invasive
• Fosters
migrations from Oracle
• “Standard
de facto” for PostgreSQL Disaster Recovery
• Advice: plan
for DR (if you have not done it yet)

62. TIME TO ...
yum install barman
apt-get install barman

63. QUESTIONS?
Gabriele.Bartolini@2ndQuadrant.it
Twitter: _GBartolini_
www.pgbarman.org

64. THANK YOU!

65. LICENSE
This work is licensed under a Creative Commons AttributionNonCommercial-ShareAlike 3.0 Unported License
Copyright (c) 2012, 2013 - 2ndQuadrant.it