In this session, we provide an overview of the PostgreSQL options available on AWS, and do a deep dive on Amazon Relational Database Service (Amazon RDS) for PostgreSQL, a fully managed PostgreSQL service, and Amazon Aurora, a PostgreSQL-compatible database with up to 3x the performance of standard PostgreSQL. Learn about the features, functionality, and many innovations in Amazon RDS and Aurora, which give you the background to choose the right service to solve different technical challenges, and the knowledge to easily move between services as your requirements change over time. 

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep Dive on PostgreSQL
Databases on Amazon RDS
Jim Mlodgenski
Principal Database Engineer
Amazon Web Services
D A T 3 2 4
Jeremy Schneider
Database Engineer
Amazon Web Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Wednesday, November 28
Ask Me Anything about Amazon Aurora
2:30 p.m. - 3:30 p.m. | MGM, Level 1, South Concourse 104
Thursday, November 29
Deep Dive on Amazon Aurora PostgreSQL Performance Tuning
12:15 p.m. - 1:15 p.m. | Aria West, Level 3, Starvine 2
Thursday, November 29
Amazon Aurora Storage Demystified: How It All Works
2:30 p.m. - 3:30 p.m. | Venetian, Level 4, Lando 4305

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Relational Database Services (Amazon RDS)
Cloud native engine Open source engines Commercial engines
RDS platform
• Automatic fail-over
• Backup & recovery
• X-region replication
• Isolation & security
• Industry compliance
• Automated patching
• Advanced monitoring
• Routine maintenance
• Push-button scaling
Image credit: By Mackphillips - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=55946550

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Running PostgreSQL on AWS
Self-managed on Amazon
Elastic Compute Cloud
(Amazon EC2)
AWS managed DB services
Amazon RDS for PostgreSQL
Amazon Aurora with PostgreSQL-
compatibility

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Choosing your Amazon PostgreSQL deployment
Amazon RDS for
PostgreSQL
Self-
managed on
Amazon EC2

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Support for the latest minor releases
• 10.5
• 9.6.10
• 9.5.14
• 9.4.19
Version 11.0 available in preview
https://aws.amazon.com/rds/databasepreview/
Amazon RDS for PostgreSQL

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Built from the ground up to leverage AWS
• PostgreSQL 10.4 compatible with up to two-
three times better performance on the same
hardware
• Scalable with up to 64 TB in single database
• Highly available, durable, and fault-tolerant
custom SSD storage layer: six-way replicated
across three Availability Zones
Amazon Aurora

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Scheduled daily volume
backup of entire instance
• Archive database change
logs
• 35-day maximum retention
• Negligible impact on
database performance
• Taken from standby when
running Multi-AZ
Automated backups
Every day during your backup
window, Amazon RDS creates a
storage volume snapshot of your
instance
Every five minutes, Amazon RDS
backs up the transaction logs of your
database

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Choose the point in time
of the database to restore
• The restored instance size
can be different than the
backed up instance
• Storage can be adjusted
to different IOPS
Point in time restore

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fault tolerance across
multiple data centers
• Automatic failover
• Synchronous replication
• Enabled with a few clicks
Redirection to the new
primary instance is
provided through DNS
Multi-AZ configuration

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network encryption (server side)
An SSL certificate is available
on Amazon RDS instances
Used to encrypt network traffic
Also used to verify the end point to
guard against spoofing attacks
By default, SSL is optional
Set rds.force_ssl to 1 to force SSL
The client requests the type of
SSL connection

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Network encryption (client side)
psql -h $RDSHOST -p 5432
"dbname=postgres user=jim
sslrootcert=rds-bundle.pem
sslmode=verify-full"
Setting SSL Mode on the
connection string determines
the SSL connection
SSL Mode Options
disable require
allow verify-ca
prefer verify-full
psql -h $RDSHOST -p 5432
"dbname=postgres user=jim"
DefaultGuards against disabling SSL on
the server
psql -h $RDSHOST -p 5432
"dbname=postgres user=jim
sslmode=require"
Checks the certificate is from a
known certificate authority
ensures the client is connecting to an
RDS server
Verifies the host name in
addition to the certificate
psql -h $RDSHOST -p 5432
"dbname=postgres user=jim
sslrootcert=rds-bundle.pem
sslmode=verify-ca"

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authentication
Ensures only authorized users
connects to the database
Default method uses MD5 with
a salt for authentication
CREATE USER jim
PASSWORD 'strongpassword'
VALID UNTIL '2018-11-28 12:15';

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authentication
MD5 authentication works well
for many cases
Some organizations need
Guaranteed SSL network communication
Centralized user management
Complex passwords
AWS Identity and Access
Management (IAM) for
authentication

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Identity and Access Management (IAM) authentication
PostgreSQL authentication is managed
externally using IAM
Available for Amazon RDS PostgreSQL and Aurora
PostgreSQL
Authentication tokens are used to
validate the user
Tokens have a lifetime of 15 minutes
Generated using AWS Signature Version 4

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configuring AWS Identity and Access Management
(IAM) authentication
postgres=> GRANT rds_iam TO jim;
GRANT ROLE
Add the rds_iam role to the user
postgres=> CREATE USER jeremy WITH LOGIN;
CREATE ROLE
postgres=> GRANT rds_iam TO jeremy;
GRANT ROLE

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configuring IAM authentication
Add an IAM user with the same
user name as the database user
Using the same name simplifies
management
The user needs programmatic
access to generate the token

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configuring AWS Identity and Access Management
(IAM) authentication
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:us-east-1:1234567890:dbuser:*/jim"
]
}
]
}
Create an IAM policy to authenticate the database user

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Connecting with AWS Identity and Access
Management (IAM) authentication
export RDSHOST="reinevent.cniebfuzszeq.us-east-1.rds.amazonaws.com“
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname
$RDSHOST --port 5432 --region us-east-1 --username jim )"
echo $PGPASSWORD
reinevent.cniebfuzszeq.us-east-1.rds.amazonaws.com:5432/?Action=connect&DBUser=jim&
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-Credential=AKIAI6CMH6GLNT2
WNJVQ%2F20181114%2Fus-east-1%2Frds-db%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz
-Date=20181114T014814Z&X-Amz-Signature=7daa6565759ec540e2effce731bd4ecb3ae3de6b41ec
1cd22b9cdeaa21b1e5f1
psql "host=$RDSHOST sslmode=require user=jim dbname=postgres"
Getting and using a token

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Authorization
Object level
Controls privileges on objects such as
tables, views and functions
GRANT UPDATE (last_viewed) ON
TABLE sales_history TO biuser;
REVOKE SELECT (ssn) ON
TABLE employees FROM public;
GRANT SELECT ON login TO webuser;
REVOKE UPDATE ON sales FROM biuser;
GRANT EXECUTE ON FUNCTION f1() TO user1;
ALTER TABLE sales
ENABLE ROW LEVEL SECURITY;
CREATE POLICY region_policy ON sales
FOR SELECT
USING (region_owner = CURRENT_USER);
Column level
Privileges can be controlled down to
the column
Row level
Table level restriction on data based on
a user’s identity

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database server instance types
General purpose (T2)
• 1 vCPU / 1 GB RAM to
8 vCPU 32 GB RAM
• Moderate networking
performance
• Good for smaller or variable
workloads
• T2.micro is eligible for free tier
General purpose (M4/M5)
• 2 vCPU / 8 GiB RAM to
96 vCPU 384 GiB RAM
• High performance
networking
• Good for running CPU
intensive workloads (e.g.
WordPress)
Memory optimized (R4)
• 2 vCPU / 16 GiB RAM to
64 vCPU 488 GiB RAM
• High performance
networking
• Good for query intensive
workloads or high
connection counts

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High performance database storage
General purpose (GP2)
• SSD storage
• Maximum of 32 TB
• Latency in milliseconds
• IOPS determined by volume
size
• Bursts to 3,000 IOPS
(applicable below 1.3 TB)
• Affordable performance
Provisioned IOPS (IO1)
• SSD storage
• Maximum of 32 TB
• Single digit millisecond
latencies
• Maximum of 40 K IOPS
• Delivers within 10% of the
IOPS performance 99.9%
of the time
• High performance and
consistency

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning
Split a large table to smaller
more manageable chunks
Increased performance by
pruning away large amounts of
data
Eases management by allowing
operations to be performed on
a subset of the table

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning (before PostgreSQL 10)
CREATE TABLE orders (
o_orderkey INTEGER,
o_custkey INTEGER,
o_orderstatus CHAR(1),
o_totalprice NUMERIC,
o_orderdate DATE,
o_orderpriority CHAR(15),
o_clerk CHAR(15),
o_shippriority INTEGER,
o_comment VARCHAR(79));
CREATE TABLE orders_199201 (
CHECK ( o_orderdate >= DATE '1992-01-01'
AND o_orderdate < DATE '1992-02-01')
) INHERITS (orders);
CREATE TABLE orders_199201 (
CHECK ( o_orderdate >= DATE '1992-02-01'
AND o_orderdate < DATE '1992-03-01')
) INHERITS (orders);
CREATE TABLE orders_199201 (
CHECK ( o_orderdate >= DATE '1992-02-01'
AND o_orderdate < DATE '1992-03-01')
) INHERITS (orders);
...

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning (before PostgreSQL 10)
CREATE OR REPLACE FUNCTION order_part_trg()
RETURNS TRIGGER AS $$
BEGIN
EXECUTE 'INSERT INTO orders_' ||
to_char(NEW.o_orderdate, 'YYYYMM') ||
' VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)'
USING NEW.o_orderkey, NEW.o_custkey, NEW.o_orderstatus, NEW.o_totalprice,
NEW.o_orderdate, NEW.o_orderpriority, NEW.o_clerk, NEW.o_shippriority,
NEW.o_comment;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER order_part_trg(
BEFORE INSERT ON orders
FOR EACH ROW EXECUTE PROCEDURE order_part_trg();

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning (PostgreSQL 10)
CREATE TABLE orders (
o_orderkey INTEGER,
o_custkey INTEGER,
o_orderstatus CHAR(1),
o_totalprice NUMERIC,
o_orderdate DATE,
o_orderpriority CHAR(15),
o_clerk CHAR(15),
o_shippriority INTEGER,
o_comment VARCHAR(79))
PARTITION BY RANGE (o_orderdate);
CREATE TABLE orders_199201
PARTITION OF orders
FOR VALUES FROM ('1992-01-01')
TO ('1992-02-01');
CREATE TABLE orders_199202
PARTITION OF orders
FOR VALUES FROM ('1992-02-01')
TO ('1992-03-01');
CREATE TABLE orders_199203
PARTITION OF orders
FOR VALUES FROM ('1992-03-01')
TO ('1992-04-01');
...

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning (PostgreSQL 10)
=> CREATE INDEX orders_o_custkey_idx
-> ON orders (o_custkey);
ERROR: cannot create index on partitioned
table “orders“
=> CREATE INDEX o_199201_ _o_custkey_idx
-> ON orders_199201 (o_custkey);
CREATE INDEX
Partitioning syntax added
Triggers are no longer needed
Support range and list
partitioning

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning
0.0
100.0
200.0
300.0
400.0
500.0
600.0
1 2 3
Data Load (seconds)
COPY command
to load 7.5M rows
About 1GB of data

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning (PostgreSQL 11)
=> CREATE INDEX lineitem_l_orderkey_idx
-> ON lineitem (l_orderkey);
CREATE INDEX
Support for PRIMARY KEY,
FOREIGN KEY, indexes, and
triggers on partitioned tables
UPDATE statements move
partitions when necessary
Allows for a “default” partition
Partition elimination during
query execution (joins)
Hash partitioning

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Table partitioning
SELECT o_orderdate,
sum(o_totalprice)
FROM orders
INNER JOIN dates d
ON (o_orderdate = d.date)
WHERE d.year = 1992
AND d.quarter = 2
AND d.dow = 1
GROUP BY 1
ORDER BY 1;
0
200
400
600
800
1000
1200
1400
1600
1800
1 2 3
Query Time (ms)
74%
35%

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement based
• Trigger BasedLogical - SQL
• Standard PostgreSQL
• Extension “pglogical”
• AWS DMS
• Third-party
Logical - Engine
• Read replicas
• Multi-AZPhysical - Engine

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement based
• Trigger basedLogical - SQL
• Standard PostgreSQL
• Extension “pglogical”
• DMS
• Third-Party
Logical - Engine
• Read Replicas
• Multi-AZPhysical - Engine
CREATE TRIGGER delta
AFTER INSERT OR UPDATE OR
DELETE ON "$schema"."$table”
• Significant
development effort

38. “The SQL based replication pattern is
the longest used and best known, but
not the most efficient.”

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Physical replication
CREATE EXTENSION pageinspect;
UPDATE mytable SET id=2 WHERE id=1;
SELECT lp,t_ctid,t_xmax,t_data from heap_page_items(get_raw_page('mytable',0));
lp | t_ctid | t_xmax | t_data
----+--------+--------+--------
1 | (0,2) | 1113 | x0100
2 | (0,2) | 0 | x0200

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement Based
• Trigger BasedLogical - SQL
• Standard PostgreSQL
• Extension “pglogical”
• DMS
• Third-Party
Logical - Engine
• Read replicas
• Multi-AZPhysical - Engine
• All-or-nothing
• Same version only
• Read only

41. “The physical replication pattern is
efficient and powerful, but not flexible
enough for every use case.”

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Engine based logical
Write-Ahead Log
select name,size from pg_ls_waldir();
name | size
--------------------------+----------
0000000100000001000000ED | 16777216
0000000100000001000000EC | 16777216
0000000100000001000000EB | 16777216
0000000100000001000000EA | 16777216

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Engine based logical
Streaming Replication
Write-Ahead Log
select pid, application_name, state, sent_lsn
from pg_stat_replication;
pid | application_name | state | sent_lsn
-------+------------------+-----------+------------
31812 | subtl | streaming | 0/B20004D8
13681 | walreceiver | streaming | 0/B20004D8
select plugin, slot_type, active_pid, restart_lsn
from pg_replication_slots;
plugin | slot_type | active_pid | restart_lsn
----------+-----------+------------+-------------
pgoutput | logical | 31812 | 0/B7000028
[NULL] | physical | 13681 | 0/B7000108

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Engine based logical
Logical Decoding
Streaming Replication
Write-Ahead Log
select plugin, slot_type, active_pid, restart_lsn
from pg_replication_slots;
plugin | slot_type | active_pid | restart_lsn
----------+-----------+------------+-------------
pgoutput | logical | 31812 | 0/B7000028
[NULL] | physical | 13681 | 0/B7000108

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Engine based logical
Publications and
Subscriptions
Logical Decoding
Streaming Replication
Write-Ahead Log
CREATE PUBLICATION ...
FOR TABLE ...;
CREATE SUBSCRIPTION ...
CONNECTION ... PUBLICATION ...;

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement Based
• Trigger BasedLogical - SQL
• Standard PostgreSQL
• Extension “pglogical”
• DMS
• Third-Party
Logical - Engine
• Read replicas
• Multi-AZPhysical - Engine
Streaming Replication
Write Ahead Log

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement Based
• Trigger BasedLogical - SQL
• AWS DMS
• Third-party
Logical - Engine
• Read Replicas
• Multi-AZPhysical - Engine
Logical Decoding
Streaming Replication
Write Ahead Log

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement Based
• Trigger BasedLogical - SQL
• Standard PostgreSQL
• Extension “pglogical”
Logical - Engine
• Read Replicas
• Multi-AZPhysical - Engine
Publications and
subscriptions
Logical Decoding
Streaming Replication
Write Ahead Log

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement Based
• Trigger BasedLogical - SQL
• Standard PostgreSQL
Logical - Engine
• Read Replicas
• Multi-AZPhysical - Engine
Standard PostgreSQL:
• Efficient
• Replicate between
different versions
• Replicate only part of
the database
• Queues transactions
and sends on COMMIT
• Primary or unique key
is recommended
CREATE PUBLICATION ...
FOR TABLE ...;
CREATE SUBSCRIPTION ...
CONNECTION ... SUBSCRIPTION ...;

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Replication in Amazon RDS PostgreSQL
• Statement Based
• Trigger BasedLogical - SQL
• Extension “pglogical”
Logical - Engine
• Read Replicas
• Multi-AZPhysical - Engine
Features of pglogical:
• Conflict Handling
• PostgreSQL 9.x
• DDL and Sequences
• Advanced Filtering
• Delay
• Third-party JSON
consumers
-- modify parameter group:
-- shared_preload_libraries
CREATE EXTENSION pglogical;

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring Amazon RDS
Performance
Insights
Amazon
CloudWatch
Third-party
Tools
Enhanced
Monitoring

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring Amazon RDS
Performance
Insights
• Measures database load to help
you identify bottlenecks
• Top SQL/most intensive queries
• Adjustable timeframe: hour, day, week, longer
• Compliments other key tools
• query execution plans
• pg_stat_statements

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring Amazon RDS
Enhanced
Monitoring
Enhanced Monitoring for Amazon RDS
Access to over 50 CPU, memory, file system,
and disk I/O metrics
Access to top processes
As low as 1 second intervals

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring Amazon RDS
Amazon
CloudWatch Amazon CloudWatch metrics
Displayed in the Amazon RDS Console or
personalized CloudWatch dashboards
As low as one minute intervals
Amazon CloudWatch alarms
Trigger actions based on a metric value
relative to a threshold you set

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Production happiness hints from DB Engineering
Using Multi-AZ for high availability
Enable automated backups
Keep 3 weeks of backup
Regularly test restores
Take periodic logical exports
Force SSL connections
Apply Postgres quarterly updates
Stable minor releases for security and bug fixes
Upgrade extensions too
Enable Performance Insights
Keep the history
Setup Enhanced Monitoring
10 second collection
Enable “pg_stat_statements”
Configure alarms on
Maximum used transaction IDs
DBLoad
Free disk space
Memory / swap

57. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jim Mlodgenski
mlodj@amazon.com
Jeremy Schneider
schnjere@amazon.com

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.