1. Denish Patel
Database Architect, OmniTI
Deploying PostgreSQL on
Amazon EC2
A case study
1
Tuesday, May 7, 13

2. Story teller?
✤ Denish Patel
✤ denish@omniti.com
✤ Database Architect at OmniTI
✤ Heterogeneous Environment
✤ PostgreSQL, Oracle , MySQL, NoSQL
✤ Scalability, Performance & Simplicity
✤ OmniTI is hiring Database Engineer!
2
Tuesday, May 7, 13

3. Situation
✤ Startup
✤ Ruby on Rails Shop
✤ Entire IT architecture on Amazon Cloud
✤ 2 Identical Database Environments on EC2
✤ 4 Production Instances
✤ 50GB Each PostgreSQL 9.0 Master - Slave ( 2 Pairs)
✤ 2 QA and 2 Dev environments
3
Tuesday, May 7, 13

4. Situation
✤ Weekly flake out issues on EC2 instances
✤ Weekly backups using pg_dump
✤ No file system level backup
✤ Master-slave wasn’t working for months!
✤ Schema mismatch between 2 environments
✤ Almost NO monitoring in-place for DBs
4
Tuesday, May 7, 13

5. Situation
✤ Prod , QA and Dev ...
✤ M1 Extra large instances (m1.xlarge)
✤ 15 GB RAM, 4 Virtual Cores with 2 EC2 compute units each
✤ Configuration Management tool - Puppet
5
Tuesday, May 7, 13

6. The Amazon Cloud
✤ Business Benefits
✤ Offloading heavy lifting
✤ Lowers cost
✤ Reduces time to market
6
Tuesday, May 7, 13

7. Amazon Data Centers
7
Tuesday, May 7, 13

8. Amazon Instances
✤ High CPU
✤ High IO
✤ High Memory
8
Tuesday, May 7, 13

9. Amazon StorageTypes
9
Tuesday, May 7, 13

10. Network and Security
✤ Amazon Security Groups
✤ Keep your Amazon EC2 firewall permissions as restrictive as
possible.
✤ Elastic IP Address (EIP)
10
Tuesday, May 7, 13

11. Deployment Strategies
✤ Sizing Instances and Storage Types
✤ Configuration Management
✤ High Availability
✤ Replication
✤ Failover
✤ Operations
✤ Upgrade, Backups, Maintenance tools
✤ Monitoring
11
Tuesday, May 7, 13

12. Sizing Instances/Storage
✤ Understand your database , schema and data
✤ Env 1: 50GB , Env 2: 25GB
✤ 2 large archival tables
✤ Working Set about 20-30GB in total
✤ Both environments are inter-related.
✤ Decided to consolidate both environment into single cluster
12
Tuesday, May 7, 13

13. Sizing Instances/Storage
✤ m2.2xlarge: 34.2GB RAM, 4 Virtual Core
✤ Storage
✤ EBS
✤ RAID 10
✤ Not EBS optimized ( Cost Savings!)
✤ 850 GB
13
Tuesday, May 7, 13

14. Deployment Phase #1
us-east-1
AV1
AV2
us-west-1
App
Servers
db-master db-slave-
backup
db-slave
Streaming Replication
WAL shipping (OmniPITR)
14
Tuesday, May 7, 13

15. Configuration Management
✤ Automation
✤ Repeatability
✤ Maintainability
✤ Security
15
Tuesday, May 7, 13

16. Configuration Management
16
Tuesday, May 7, 13

17. Puppetizing Postgres
✤ DB server role
✤ PostgreSQL installation
✤ Installing common tools
✤ 3 Roles : prod-master role , prod-slave role, prod-slave-backup role
✤ PostgreSQL Configuration Files: postgresql.conf, pg_hba.conf
✤ Scripts, crontabs
17
Tuesday, May 7, 13

18. High Availability
✤ Replication
✤ Failover
18
Tuesday, May 7, 13

19. Replication
✤ Streaming replication
✤ WAL shipping , OmniPITR
✤ Different AV zones
✤ At least 1 slave in different Region
19
Tuesday, May 7, 13

20. Failover
✤ EIP - about 3 minutes to propagate changes after failover
✤ Failover tools
✤ Semi-automated script
✤ pgbouncer
✤ puppet
20
Tuesday, May 7, 13

21. Deployment Phase #2
us-east-1
AV1
AV2
us-west-1
App
Servers
db-master
db-slave-
backup
db-slave
Streaming Replication
WAL shipping (OmniPITR)
pgbouncer
21
Tuesday, May 7, 13

22. pgbouncer
✤ Session Pooling
✤ Supports all Postgres features
✤ Transactions Pooling
✤ Does NOT support PREPARE transactions
22
Tuesday, May 7, 13

23. Operations
23
Tuesday, May 7, 13

24. Database Upgrade
✤ Upgraded from PostgreSQL 9.0 to PostgreSQL 9.2
✤ Minimal outage upgrade using Mimeo
✤ http://www.pateldenish.com/2012/10/an-easy-way-to-reduce-
outage-window-for-postgresql-upgrade.html
24
Tuesday, May 7, 13

25. Beware!!
✤ Sept 24th, 2012
✤ PostgreSQL 9.1.0 to 9.1.5 & 9.2.0
✤ VACUUM all tables in your database & Rebuild Indices
✤ http://wiki.postgresql.org/wiki/20120924updaterelease
✤ April 4th, 2013
✤ PostgreSQL 9.0.* to 9.1.* & 9.2.*
✤ Security release
✤ http://www.postgresql.org/support/security/faq/2013-04-04/
25
Tuesday, May 7, 13

26. Backups
✤ Online Backups
✤ Daily, db-slave , OmniPITR
✤ Test the backups!!!!
✤ Schema Backups
✤ pg_extractor
✤ Offsite Backup
✤ sending backups to Amazon S3
26
Tuesday, May 7, 13

27. Tools
✤ tracking schema in git : pg_extractor
✤ pg_bloat_report
✤ Pg log errors reporting : tail_n_mail
✤ Log analysis: pgbadger
✤ log_min_duration_statement = 50 ms
✤ log_line_prefix= '%t [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
27
Tuesday, May 7, 13

28. Monitoring
28
Tuesday, May 7, 13

29. Database Checks
29
Tuesday, May 7, 13

30. Database Checks
30
Tuesday, May 7, 13

31. System Checks
31
Tuesday, May 7, 13

32. Business Metrics
32
Tuesday, May 7, 13

33. Histograms!
33
Tuesday, May 7, 13

34. References
✤ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/
Storage.html
✤ OmniTI labs tools
✤ https://github.com/omniti-labs/pgtreats
✤ OmniPITR
✤ https://github.com/omniti-labs/omnipitr
34
Tuesday, May 7, 13