Pro Postgres 9

Pro Postgres 9

Robert Treat, JDCon East 2011

Thursday, March 31, 2011

Who Am I? (Why Listen To Me)

PostgreSQL User Since 6.5.x

DBA of High Trafﬁc / Large PostgreSQL Instances

Long Time Contributor to PostgreSQL Project

Contribute / Maintain Several Open Source Projects

Co-Author Beginning PHP & PostgreSQL 8 (Apress)

I like to call it “Postgres”

Robert Treat | Pro Postgres 9

Outline

What you need to know
about the project Backups

Getting Started Replication

Upgrading Query Tuning

Conﬁguring Your Server Other stuff you should know
about
Hardware


Know Your Way Around The Project



www.postgresql.org

downloads security alerts

documentation wiki

bug reports support companies

rss -> news - events - version



planet.postgresql.org

project news
helpful tips / examples
community news



www.pgfoundry.org

URI Type

SkyTools
modules
Npgsql
programs
pg_bulkload
resources
plpgsql-debugger

pagila, sample database



git.postgresql.org

check_postgres

modules edb-installers

programs website code

resources json type

works in progress pg_top

user repo’s
see also github.com


archives.postgresql.org

mailing list archives back to 1997

full text search via postgresql FTS

keyword search suggestions

lists for users, developers, regional, user groups



#postgresql

irc.freenode.net

real time help

pg_docbot - ??help



project management

core team

committers

-hackers

road map

web team


Get Off To A Good Start



Use Package Management

Consistent

Standardized

Simple




Different across systems

Upgrades can be a problem

Trust your packager?




Different across systems

Upgrades can be a problem

Trust your packager?

Don’t Be Afraid To Roll Your Own



A Note About Versions

Just playing? Default packages
probably OK, (min 8.2)

Any production level work
should start on 9.0

Any project not due to launch
for 6 months, 9.1



So What Do You Get?
pg_control version number: 903
Catalog version number: 201008051
Database system identifier: 5546136951888795324
Database cluster state: in archive recovery
pg_control last modified: Tue Mar 22 11:50:24 2011
Latest checkpoint location: 7C/1D000058
Prior checkpoint location: 7C/1C000058
Latest checkpoint's REDO location: 7C/1D000020
Latest checkpoint's TimeLineID: 1
Latest checkpoint's NextXID: 0/4924
Latest checkpoint's NextOID: 16608
Latest checkpoint's NextMultiXactId: 1
Latest checkpoint's NextMultiOffset: 0
Latest checkpoint's oldestXID: 654
Latest checkpoint's oldestXID's DB: 1

pg_controldata Latest checkpoint's oldestActiveXID:
Time of latest checkpoint:
4924
Tue Mar 22 11:46:19 2011
Minimum recovery ending location: 7C/16000000
Backup start location: 0/0
Current wal_level setting: hot_standby
Current max_connections setting: 500
Current max_prepared_xacts setting: 0
Current max_locks_per_xact setting: 64
Maximum data alignment: 8
Database block size: 8192
Blocks per segment of large relation: 131072
WAL block size: 8192
Bytes per WAL segment: 16777216
Maximum length of identifiers: 64
Maximum columns in an index: 32
Maximum size of a TOAST chunk: 1996
Date/time type storage: 64-bit integers
Float4 argument passing: by value
Float8 argument passing: by value



So What Do You Get?
-rw------- 1 postgres postgres 4 Dec 2 12:23 PG_VERSION
drwxr-xr-x 6 postgres postgres 6 Dec 2 13:41 base
drwxr-xr-x 2 postgres postgres 42 Jan 28 17:49 global
drwxr-xr-x 2 postgres postgres 3 Dec 2 12:23 pg_clog
-rw------- 1 postgres postgres 3938 Dec 19 11:25 pg_hba.conf
-rw------- 1 postgres postgres 1636 Dec 19 11:25 pg_ident.conf
drwxr-xr-x 3 postgres postgres 113 Mar 22 00:00 pg_log
drwxr-xr-x 4 postgres postgres 4 Dec 2 12:23 pg_multixact
drwxr-xr-x 2 postgres postgres 3 Jan 12 14:23 pg_notify
drwxr-xr-x 2 postgres postgres 3 Jan 12 14:23 pg_stat_tmp
drwxr-xr-x 2 postgres postgres 3 Dec 2 12:23 pg_subtrans
drwxr-xr-x 2 postgres postgres 2 Dec 2 12:23 pg_tblspc
drwxr-xr-x 2 postgres postgres 2 Dec 2 12:23 pg_twophase
lrwxrwxrwx 1 postgres postgres 30 Dec 19 11:13 pg_xlog
-rw------- 1 postgres postgres 17915 Mar 3 21:52 postgresql.conf
-rw------- 1 postgres postgres 63 Jan 12 14:23 postmaster.opts
-rw------- 1 postgres postgres 54 Jan 12 14:23 postmaster.pid
-rw-r--r-- 1 postgres postgres 336 Dec 20 16:14 recovery.conf



Conﬁgure Your Logging

$PGDATA/pg_log

/var/log/postgresql

when in doubt...
(postgresql.conf)

separate disk

Logging is often overlooked,
but it is the ﬁrst step toward troubleshooting!



Conﬁgure Authentication

most systems have different defaults

ﬁrewalls / selinux (FATAL?)

rtfm (pg_hba.conf, grant, revoke)



Authentication Methods

TRUST

md5

IDENT



/contrib

Trust these more than your own code

package dependent

use different schemas (when able)

pgcrypto



Additional Supplied Modules

“contrib”

Trust these more than your own code

package dependent

use different schemas (when able)

pgcrypto, pg_upgrade

http://www.postgresql.org/docs/current/interactive/contrib.html


Let’s Talk About Upgrades



Versioning

First Digit (7.4.16 -> 8.2.0)

Second Digit (8.3.6 -> 8.4.0)

Third Digit (9.0.0 -> 9.0.2)



Achtung!!

Make Backups!

Read the Release Notes!

Read the Commit Logs!

Test!



pg_dump/pg_restore

simple

-Fc is your friend

dump with new version of pg_dump

pitfalls: (time, hdd)



the “slony” method

not simple

create slave on new version

switchover (switch back?)

pitfalls: (initial synch, compatibility)



pg_upgrade

in place upgrades

rewrites system catalog info

no way to go back (fs snapshots)

first official release was 9.0



upgrading legacy databases

< 8.2 is no longer supported (upgrade now!)

pg_upgrade only works on 8.3 or greater

older slony can work back to 7.3ish

pre-7.3 ?



is it worth it?

any database with significant transaction
workload (read or write) >= 8.3 (vxid)

most systems will want at least 8.4 (fsm)

replication is nice too though...



minor release versioning policy

drop in replacement

no feature changes

bug fixes only (includes security fixes)

sometimes involves extra steps

also referred to as service packs, hot fixes, fix packs, etc...

It is considered more dangerous to not upgrade!

Figure Your Conﬁgure



the basics: performance

shared_buffers

effective_cache_size

work_mem

checkpoint_segments



the basics: logging

stderr/pg_log vs. syslog/eventlog

log_min_duration_statement

log_line_prefix (%t, %p, %d, %u, %r)



other stuff worth looking at

maintenance_work_mem

max_prepared_transactions

wal_buffers

checkpoint_completion_target

autovacuum_freeze_max_age



the basics: replication

primary: wal_level = ‘hot_standby’

primary: max_wal_senders = n

slave: hot_standby = on


Hardware For Software


Hardware for Software People

cpu and you

connections, autovacuum, bgwriter,
stats collector, replication

Multiple CPU’s work wonders, up
to 32 processors

http://tweakers.net



disk does and don’ts

Put WAL on it’s own disk (RAID 1)

Put DATA directory on it’s own disk (RAID 10)

More Spindles is Good

More Controllers Even Gooder (tablespaces)

Battery Backed Controllers, Write Cache Enabled



disk don’ts

NFS == Not Fully Safe!

RAID 5

Beware disks / controllers that lie


Can You Back That Up?



types of backups

logical (sql export)

physical (files on disk)



logical backups

slow to create

slow to restore

“pure”, no system level corruption

susceptible to database level corruption

pg_dump is your friend
pg_dumpall for globals



physical backups

replication / failover machine

“tarball” (pitr)

filesystem snapshots (pitr)



replication / failover node

good idea, but... offsite? fragile?

we’ll talk more in replication
section :-)



“tarball” (pitr)

basic idea is to copy all database
files and relevant xlogs

use multiple machines if able

use rsync if able

copy the slave if able!!



ﬁlesystem snapshots (pitr)

basic idea is to copy all database
files and relevant xlogs

use multiple machines if able

use rsync if able

copy the slave if able!!


Replication, Replication



9.0 “Introduced” Replication In Postgres

Built In Solution

Based on WAL replay

True Binary Replication

1st Cut, Still A Little Rough



However, There Are Lots Of Alternatives!

Most Are Trigger Based:

Slony,

Bucardo,

Londiste,

RubyRep,

etc...

Statement Based (PGPoolII)



Quick Rundown: Slony

Trigger Based

Asynchronous

Master -> (Multiple, Cascading) Slave

Switchover

“A Pain To Work With”



Quick Rundown: Londiste

Trigger Based

Asynchronous

Master -> (Multiple) Slave

Switchover

“I Don’t Like Slony”



Quick Rundown: Bucardo

Trigger Based

Asynchronous

Master -> (Multiple) Slave

Master -> Master

“Small Community”



Quick Rundown: Ruby Rep

Trigger Based

Asynchronous

Master -> Slave

Master -> Master

“Huh?”



Determine Your Needs: Failover

Simple 2 Server Setup?

9.0 Streaming Replication

Multiple Slaves?

Slony



Determine Your Needs: Switchover
Master -> Slave

Slony

Londiste

Master -> Master

Bucardo

Ruby Rep



Determine Your Needs: Single Read Slave

Postgres Streaming Replication

no changes to the slave

Any of the Trigger Based Solutions



Determine Your Needs: Multiple Read Slaves

Slony (Cascading)

Bucardo (Split)

Londiste (meh)



A Note On Scaling Writes

Scaling Writes Is A Hard Problem

Bucardo/RubyRep Help IF CPU Bound

If you are bound on write i/o, you probably need to
federate your database


Query Your Queries


Query Your Queries

ﬁnding slow queries:
log_min_duration_statement

-1, 0, n

superuser only

alter user

LOG: duration: 5005.273ms statement: select pg_sleep(5);


Query Your Queries

log analyzers

command line
tools
i/o load
generate
reports cpu load

archive reports


Query Your Queries

log analyzers

pqa

pgfouine

maatkit (percona)

pgsi (endpoint)


Query Your Queries

ﬁxing slow queries:
explain analyze

universal tool

good for specific queries

“explain” for large queries

could be it’s own talk


Query Your Queries

explain analyze

universal tool

good for specific queries

“explain” for large queries

could be it’s own talk

http://wiki.postgresql.org/Using_EXPLAIN


Query Your Queries

indexing (basic)

use explain to find large sequential reads

use pg_stat_* tables to find numerous reads

use btree (gist/gin)

dual column vs. single column


Query Your Queries

indexing (partial)

restrain index to rows that matter

can give significant speed improvements

where clause of index should match where
clause of query

create index address_ba_part_idx
on address (district)
where district = ‘Buenos Aires’;


Query Your Queries

indexing (partial)

restrain index to rows that matter

can give significant speed improvements

where clause of index should match where
clause of query

create index customer_active_part_idx
on customer (customer_id)
where active is true;


Query Your Queries

indexing (functional)

some people prefer to call these expressional indexes


Query Your Queries

indexing (expressional)

push expensive functions into your index

system sees just WHERE indexedcolumn = ‘constant’

expression of index should match expression of queries

narrow scope, but nice gains

create unique index one_true_email_xidx on customer (lower(email));


Query Your Queries

indexing (expressional)

push expensive functions into your index

system sees just WHERE indexedcolumn = ‘constant’

expression of index should match expression of queries

narrow scope, but nice gains

create index fullname_xidx on customer ((firstname||’ ‘||lastname));


Other Stuff I Should Mention



pgcrypto

cryptography

/contrib (export issues)

md5, sha1, blowfish, many more



dblink

pgsql -> pgsql connections

/contrib (still under development)

can have performance issues on large queries

make it live in it’s own schema

may be replaced by foreign data wrappers



*-link

heterogenous connections for postgresql

db specific and db independent options

any pl/u language can implement this

similar performance issue as dblink

dblink-tds, dbi-link, oralink, odbclink



autonomous logging tool

persistent logging for postgresql functions

built on top of dblink

make it live in it’s own schema

https://labs.omniti.com/trac/pgtreats


check_postgres

nagios oriented monitoring script

comprehensive list of items for alterting

adapted for other systems, some trending

http://bucardo.org/check_postgres


phpPgAdmin

web based gui for postgresql

remote administration of multiple servers

implements much of postgresql functionality

http://phppgadmin.sourceforge.net/


omnipitr

advanced WAL file management scripts

pitr archiving, pitr restore

complete filesystem based backup

can create backups from a slave server

https://labs.omniti.com/trac/pgtreats


books

PostgreSQL 9.0 High Performance, by Greg Smith

PostgreSQL 9.0 Admin Cookbook, by Simon Riggs



we’re hiring :-)

Database Operations

Big Systems

High Volume Workloads

Keggerator!


El Fin

Thanks For Coming!


Pro Postgres 9

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Pro Postgres 9

Similar a Pro Postgres 9 (11)

Más de Robert Treat

Más de Robert Treat (20)

Último

Último (20)

Pro Postgres 9