This document provides an overview of setting up high availability for PostgreSQL using Linux-HA, DRBD, and PostgreSQL. It describes the goals of inspiring more replication discussions and informing admins. It then covers installing and configuring Linux-HA and DRBD for data mirroring across two servers, setting up PostgreSQL to use the mirrored storage, some performance tests showing a 30% write degradation, and tips for improving performance like using writeback mode on the RAID cards.

1. Linux‐HA + DRBD + PostgreSQL
Jesse Young

2. Target Audience
• System Administrators
• DBAs looking into replicaBon

3. Goals
• Inspire more replicaBon discussions
• Inform System Admins/DBAs
• Describe components
• Simple Setup
• DemonstraBon
• Performance Tips

4. Intro
• Zonar Systems, Inc.
– 45,000 vehicles sending data in 24/7 and growing
– 15 database servers
– 5 TB data, growing ~25GB daily
– PostgreSQL + PostGIS

5. Why use Linux‐HA + DRBD?
• Simple replicaBon
• GPS data coming into the system 24/7
• MiBgate problems from server hardware
• Users and IntegraBon soYware access

6. Linux‐HA
• “Provide a high availability (clustering)
solu6on for Linux which promotes reliability,
availability, and serviceability (RAS) through a
community development effort.”
• System Admins: You can conBnue to (half)
sleep, even if a server dies.

7. DRBD
• Distributed Replicated Block Device
• Mirrored storage across servers
Image from h_p://www.drbd.org

8. PostgreSQL
• Database engine we all know and love

9. Setup
• Dual‐proc Quad‐core Intel Xeon @ 2.66 GHz
• 16GB RAM
• 2x 74 GB SCSI @ 15kRPM (RAID 1 ‐ OS)
• 6x 300 GB SCSI @ 10kRPM (RAID 10 – DRBD)
• LSI Megaraid – 2 Channel, 512 MB Cache w/
BBU

10. Quick Install
• Versions
– heartbeat‐2.0.8
– drbd‐8.0.5
– PostgreSQL – Any version

11. Install DRBD
• Compile from source
or
• yum install drbd kmod‐drbd

12. Configure DRBD
resource postgres {
on node‐001 {
device /dev/drbd0;
disk /dev/sdb1;
address 10.40.0.1:7788;
meta‐disk internal;
}
on node‐002 {
device /dev/drbd0;
disk /dev/sdb1;
address 10.40.0.2:7788;
meta‐disk internal;
}
}

13. Begin DRBD
• Each node:
– modprobe drbd
– drbdadm create‐md postgres
– drbadm up all

14. Create ParBBon on DRBD Device
• Primary node:
– drbdadm ‐‐ ‐‐overwrite‐data‐of‐peer primary all
– mke2fs ‐j /dev/drbd0
– mount device
– Edit /etc/init.d/postgresql
• PGDATA to DRBD device
– initdb or sync/copy exisBng database

15. DRBD status
• cat /proc/drbd
version: 8.0.5 (api:86/proto:86)
SVN Revision: 3011 build by jesse@node‐001.zonarsystems.net, 2007‐09‐02 23:16:51
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r‐‐‐
ns:191681144 nr:108784 dw:191798660 dr:1228217841 al:962834152 bm:9054 lo:0
pe:0 ua:0 ap:0
resync: used:0/31 hits:34633 misses:4367 starving:0 dirty:0 changed:4367
act_log: used:0/127 hits:11433119293 misses:1051777270 starving:11558178
dirty:81149096 changed:962834152

16. Install Linux‐HA
• Compile from source
or
• yum install heartbeat

17. Configure Linux‐HA
• ha.cf
• haresources
• authkeys

18. ha.cf
deadBme 120
auto_failback off
udpport 694
ucast eth1 10.30.0.10
node node‐001.zonarsystems.net
node node‐002.zonarsystems.net

19. haresources
cluster‐001.zonarsystems.net IPaddr::10.30.0.101/24 drbddisk::postgres
Filesystem::/dev/drbd0::/mnt/rod1::ext3
postgresql
MailTo::jesse@zonarsystems.com::'Zonar_Cluster‐001_Switch'

20. authkeys
auth 1
1 sha1 f93e5e562d430dae92b7fd7b272c13bb0c5de0e2

21. Start Linux‐HA
• /etc/init.d/heartbeat start

22. DEMO

23. Problems?
• Secondary server always stand‐by
• Failed node can cause slow downs
• Kernel module

24. Performance?
• dd if=/dev/zero of=/mnt/drbd/pgwest/test bs=8192
count=2000000
• DRBD device
– ~85 MB/s
• Disk device
– 121MB/s
30% write performance degradaBon

25. Problems?

27. Performance Improvements

28. Ethernet
• Direct cross over cable
• GigE, 10 GigE
• 1500 MTU vs 9000 MTU

29. RAID Cards
• Capable of WRITEBACK
• Write caching
• Ba_ery backup unit (BBU)

30. Writeback VS. Writethrough
100
• Writeback 90
80
– 80MB/s – 94MB/s 70
60
50
40
• Writethrough 30
– 61MB/s – 71MB/s 20
10
0
Run Run Run Run Run Run Run Run
1 3 5 7 9 11 13 15

31. DRBD Protocols
• Protocol A
– DRBD waits for local disk and local TCP send buffer
• Protocol B
– DRBD waits for local disk and remote buffer cache
• Protocol C
– DRBD waits for both local and remote disk
– SAFEST

32. YMMV
• Remember to test yourself
• Fedora/e1000 network stack issues
• DRBD is soYware too, bugs come up

33. Other Uses
HA‐Linux + DRBD + ???
• Mail
• NFS
• MySQL

34. QuesBons?

35. Links
• Linux‐HA: h_p://www.linux‐ha.org
• DRBD: h_p://www.drbd.org

37. Demo (backup plan)

38. Host: c6a

39. Host: c6a

40. Host: c6a

41. Host: c6a

42. Host: c6a

43. Host: c6b

44. Host: c6b

45. Host: c6a