1. High Availability and PostgreSQL
Fujitsu Coffee Morning — July 28th, 2006
Gavin Sherry
gavin@alcove.com.au
High Availability and PostgreSQL – p.

2. What is high availability?
A system to reduce application down time
Usually consists of
Master server(s)
Slave server(s)
Software to detect the failure of a master
Software to promote a slave to master status
Software or hardware to ensure data consistency
between the master(s) and slave(s)
Sometimes technologies provide HA incidentally (more
later)
High Availability and PostgreSQL – p.

3. What high availability is not
... a mechanism to increase performance
Usually HA comes at the price of performance
... a way to simplify your network
... cheap
... easy to implement
High Availability and PostgreSQL – p.

4. Choosing a HA solution
How much down time can you afford?
How much data (how many seconds, minutes) can you
afford to lose?
Do you need a geographically dispersed solution?
Are slaves online or offline?
Are online upgrades possible?
High Availability and PostgreSQL – p.

5. Failure detection
‘heartbeat’ / ‘ping’
Can be performed via: serial, fibre, Ethernet. . .
Red Hat Cluster Suite, Linux-HA
More complex OS level interconnect
HACMP (AIX), Lifekeeper (Linux), Sun Cluster
(Solaris)
High Availability and PostgreSQL – p.

6. Switch over
IP take over
Managed or automatic?
Application reconfiguration
Middleware reconfiguration
Shoot the master in the head
High Availability and PostgreSQL – p.

7. Data availability mechanisms
Historically seen to be the most complex part of
database HA
Replicating data is easy, doing it with transactional
semantics is hard
Minimising performance impact is also hard
The slave server(s) need access to the data
How far out of date can the slave(s) be?
High Availability and PostgreSQL – p.

8. Shared storage
Master and slave(s) connect to shared disk/SAN
Pros
Little to no performance impact
Simple design
May utilise existing infrastructure
Theoretically no data loss in failure scenario
Cons
SANs can fail — especially cheap ones
No off site capability
Upgrades can be painful
Slave(s) are offline
Homogenous cluster
High Availability and PostgreSQL – p.

9. Data replication: Slony I
Trigger-based row level replication
Data is pulled by slave server(s)
Pros
Implemented by PostgreSQL developers
Online upgrades are a feature
Data is replicated with transaction semantics
Online slaves
Off site capable
Heterogeneous cluster
Cons
Performance impact: 1% to 5%
Complex administration
High Availability and PostgreSQL – p.

10. Data replication: file system
Block based OS file system copy on write
DRBD on Linux
Pros
Theoretically, no data loss
Simplified implementation
Cons
Significant performance impact — more than 10%
Off site will increase performance impact
No online upgrade
Offline slaves
Homogenous cluster
High Availability and PostgreSQL – p. 1

11. Data replication: log shipping
Point in time recovery (PITR)
Write ahead logs (WAL) are shipped to slave and
replayed
Pros
<1% performance impact
Simplified design
Off site capable
Cons
Offline slaves
Minimising data loss is tricky and error prone
Lacks tools
No online upgrade
High Availability and PostgreSQL – p. 1

12. SQL proxy: PG Cluster
Write statements are sent to all nodes via a proxy
Pros
Online slaves
Simplified design
Cons
Data inconsistencies across servers
Not two-phase commit — query may fail a server
Does not handle non-deterministic queries
INSERT INTO SALES VALUES(1001,
current_timestamp);
High Availability and PostgreSQL – p. 1

13. SQL proxy: PgPool
Much like PG Cluster
Other cons
One slave
High Availability and PostgreSQL – p. 1

14. SQL proxy: Sequoia, p/cluster
J2EE/JDBC two phase commit statement based
replication
Derived from c-JDBC by Continuent
Pros
Data consistency with 2PC
p/cluster does complete HA, pretty tools
Simplified design
Online slaves
Cons
Only supports JDBC (other interfaces planned)
Off site may have a huge performance impact
Performance impact would be non-trivial, especially
with more slaves
High Availability and PostgreSQL – p. 1

15. Testing
Test your HA solution(s)
... but thoroughness is hard
High Availability and PostgreSQL – p. 1

16. The future
Lots of next generation HA for PostgreSQL on the
horizon
PgPool 2.0
PgCluster 2.0 — shared disk, shared memory
Postgres-R/Slony-2
High Availability and PostgreSQL – p. 1

17. Cautionary notes
HA solutions can still fail
SANs can blowup
Split brain
Replication software can fail — corrupt data or stop
replicating
Most common issue
1. Network/software/hardware blip
2. Heart beat to fails
3. Switch over to takes place
4. Something goes wrong
Network reconfiguration doesn’t work
Slave server is misconfigured
Slave server doesn’t kill master properly
High Availability and PostgreSQL – p. 1

18. References
Lifekeeper, by SteelEye - http://www.steeleye.com
Linux-HA - http://www.linux-ha.org/
Red Hat Cluster Suite -
http://www.redhat.com/solutions/clustersuite/
Slony - http://www.slony.info
DRBD - http://www.drbd.org
PG Cluster - http://pgcluster.projects.postgresql.org
PgPool - http://pgpool.projects.postgresql.org/
Continuent - http://www.continuent.com
Postgres-R(8) - http://www.postgres-r.org
High Availability and PostgreSQL – p. 1