What's New in Amazon Aurora - ADB203 - Anaheim AWS Summit

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’s new in Amazon Aurora
A D B 2 0 3
Anoop Gupta
Software development manager
AWS
Vlad Vlasceanu
Principal database specialist SA
AWS

Amazon Relational Database Service
(Amazon RDS)
Choice Value Innovation

Amazon RDS
Choice ofopen sourceand commercialdatabases
Amazon RDS platform
Open-source engines Commercial engines
Advanced monitoring
Routine maintenance
Push-button scaling
Automatic failover
Backup & recovery
X-region replication
Isolation & security
Industry compliance
Automated patching
Cloud-native engine

Amazon Aurora
Enterprise database at opensourceprice
Delivered as a managed service
Amazon Aurora
Speed and availability of high-end commercial databases
Simplicity and cost-effectiveness of open source databases
Drop-in compatibility with MySQL and PostgreSQL
Simple pay-as-you-go pricing

Amazon Aurora innovations
Re-imagining databases forthe cloud
Automate administrative tasks—fully managed service
Scale-out, distributed, multi-tenant design
Service-oriented architecture leveraging AWS services

Scale-out, distributed architecture
Purpose-built log-structured distributed
storage system designed for databases
Storage volume is striped across
hundreds of storage nodes distributed over
three different Availability Zones
Six copies of data, two copies
in each Availability Zone to protect
against AZ+1 failures
Shared storage volume
Storage nodes with SSDs
Availability
Zone 1
SQL
Transactions
Caching
Availability
Zone 2
SQL
Transactions
Caching
Availability
Zone 3
SQL
Transactions
Caching

Leveraging AWS services
Invoke AWS Lambda events from
stored procedures/triggers
Load data from Amazon Simple
Storage Service (Amazon S3),
store snapshots and backups in
Amazon S3
Lambda
function
Amazon
S3
IAM
Amazon
CloudWatch
Use AWS Identity and Access
Management (IAM) roles to manage
database access control
Upload systems metrics
and audit logs to Amazon
CloudWatch

Automate administrative tasks
Automatic failover
Backup & recovery
Isolation & security
Industry compliance
Push-button scaling
Automated patching
Advanced monitoring
Routine maintenance
Takes care of your time-consuming database management tasks,
freeing you to focus on your applications and business
You AWS
Schema design
Query construction
Query optimization

Security management
Encryption to secure data at rest using customer-
managed keys
• AES-256; hardware accelerated
• All blocks on disk and in Amazon S3 are encrypted
• Key management via AWS KMS
Encrypted cross-region replication, snapshot
copy—SSL to secure data in transit
Advanced auditing and logging without
any performance impact
Database activity monitoring
Database
engine
*NEW*
Customer master key(s)
Data key 1
Storage
node
Storage
node
Storage
node
Storage
node
Data key 1 Data key 1 Data key 1

Industry certifications
Amazon Aurora gives each database
instance IP firewall protection
Aurora offers transparent encryption at
rest and SSL protection for data in
transit
Amazon VPC lets you isolate and control
network configuration and connect
securely to your IT infrastructure
IAM provides resource-level permission
controls
*New* *New* *New*

Aurora customer adoption
Fastest growing service in AWS history
Aurora is used by three-quarters of the top 100 AWS customers

Who is moving to Aurora and why?
Customers using
open source engines
• Higher performance—up to 5x
• Better availability and durability
• Reduces cost—up to 60%
• Easy migration; no
application change
Customers using
commercial engines
• One-tenth of the cost; no licenses
• Integration with cloud ecosystem
• Comparable performance and availability
• Migration tooling and services

Aurora migration options
Source database From where Recommended option
Amazon RDS
Amazon EC2, on premises
Amazon EC2, on premises,
Amazon RDS
Console-based automated
snapshot ingestion and catch up
via binlog replication
Binary snapshot ingestion
through Amazon S3 and catch
up via binlog replication
Schema conversion using AWS
SCT and data migration via AWS
DMS

What’s new in AWS DMS and AWS SCT?
Workload qualification
Allows customers to estimate and manage the efforts required to migrate Oracle
and SQL Server workloads to Aurora
Migration playbook
Step-by-step instructions on how to migrate from Oracle and SQL Server to Aurora
Schema conversion
Automation from Oracle and SQL Server to Aurora is over 90%
Native start point
Customers can use engine-native utilities to copy data, such as PG dump and restore, and
replicate the changes using AWS DMS

S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Aurora manageability
Serverless, data API, advanced monitoring

Aurora Serverless use cases
Infrequently used applications
(e.g., low-volume blog site)
Applications with variable load—peaks
of activity that are hard to predict (e.g.,
news site)
Development or test databases not
needed on nights or weekends
Consolidated fleets of multi-tenant SaaS
applications

Aurora Serverless
Starts up on demand, shuts down when
not in use
Scales up/down automatically
No application impact when scaling
Pay per second, one minute minimum
Warm pool
of instances
Application
Database storage
Scalable DB capacity
Request routers

Scale up and down with load
1
2
4
8
16
32
64
128
0
500
1000
1500
2000
2500
3000
1
12
23
34
45
56
67
78
89
100
111
122
133
144
155
166
177
188
199
210
221
232
243
254
265
276
287
298
309
320
331
342
353
364
375
386
397
408
419
430
441
452
463
474
485
496
507
518
529
540
551
562
573
584
595
606
617
628
639
650
661
672
683
694
705
716
727
TPS ACU

What’s new in Aurora Serverless?
New regions
Seoul, Singapore, Sydney, Mumbai, London, N. California, Paris, Frankfurt, Canada Central
Compliance and audit
FedRAMP, HIPPA, PCI, SOC, ISO & HITRUST, publish logs to Amazon CloudWatch
Preview
Support for Aurora PostgreSQL
Preview
Support for REST Data API

Amazon RDS Data API for serverless applications
Millions of IoT/mobile
devices Data API fleet
API
endpoint
Aurora
Serverless
Access through simple web interface
• Public endpoint addressable from anywhere
• No client configuration required
• No persistent connections required
Ideal for serverless applications (AWS Lambda)
Ideal for light-weight applications (IoT)

Performance insights
Dashboard showing database load
▪ Easy—e.g., drag and drop
▪ Powerful—drill down using zoom-in
Identifies source of bottlenecks
▪ Sort by top SQL
▪ Slice by host, user, wait events
Adjustable timeframe
▪ Hour, day, week, month
▪ Up to 2 years of data; 7 days free
Max vCPU
CPU bottleneck
SQL w/ high CPU

CloudWatch
consumer
Database activity monitoring
Search: Look for specific events across log files
Metrics: Measure activity in your Aurora DB cluster
• Continuously monitor activity in your DB clusters by sending these audit logs to Amazon CloudWatch Logs
• Export to Amazon S3 for long-term archival; analyze logs using Amazon Athena; visualize logs with Amazon QuickSight
Visualizations: Create activity dashboards
Alarms: Get notified or take action
Amazon
Aurora
Amazon
CloudWatch
Third-party
consumer
Amazon
Kinesis

Aurora performance
Five times faster than MySQL
Three times faster than PostgreSQL

Write and read throughput
Aurora MySQLis 5x faster than MySQL
0
50,000
100,000
150,000
200,000
250,000
MySQL 5.6 MySQL 5.7 MySQL 8.0
Aurora 5.6 Aurora 5.7
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
MySQL 5.6 MySQL 5.7 MySQL 8.0
Aurora 5.6 Aurora 5.7
Write throughput Read throughput
Using SysBench with 250 tables and 200,000 rows per table on R4.16XL

Copy in
Copy in
Vacuum
Vacuum
Index build
Index build
0 500 1000 1500 2000 2500 3000 3500 4000
PostgreSQL
Amazon Aurora
Runtime (seconds)
pgbench initialization, scale 10,000 (150 GiB)
86% reduction in vacuum time
Bulk data load performance
AuroraPostgreSQLloads data 2x fasterthan PostgreSQL

Bulk data load performance
AuroraMySQLloads data 2.5x fasterthan MySQL
Data loading
Data loading
Index build
Index build
0 100 200 300 400 500 600 700 800
MySQL
Amazon
Aurora
Runtime (sec.)
Ten SysBench tables, 10MM rows each

0.00
100.00
200.00
300.00
400.00
500.00
600.00
700.00
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Responsetime,ms
SysBench response time (p95), 30 GiB, 1,024 clients
PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)
Performance variability under load
AuroraPostgreSQLis ~10x moreconsistent than PostgreSQL

SysBench write-only workload with 250 tables and 200,000 initial rows per table
0
500
1,000
1,500
2,000
2,500
0 100 200 300 400 500 600
Time from start of run (sec.)
Write response time (ms.) Amazon Aurora MySQL
Performance variability under load
Aurora MySQLis ~25x moreconsistent than MySQL

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
50
100
150
200
250
2015 2016 2017 2018
Max write throughput—up 100%
0
100
200
300
400
500
600
700
800
2015 2016 2017 2018
Max read throughput—up 42%
Launched with R3.8xl
32 cores, 256-GB memory
Now supports R4.16xl
R5.24xl coming soon
Besides many performance optimizations, we are also upgrading HW platform
Performance improvement over time
AuroraMySQL,2015–2018

How did we achieve this?
Do less work
• Do fewer I/Os
• Minimize network packets
• Cache prior results
• Offload the database engine
Be more efficient
• Process asynchronously
• Reduce latency path
• Use lock-free data structures
• Batch operations together
• Databases are all about I/O
• Network-attached storage is all about packets/second
• High-throughput processing is all about context switches

Aurora I/O profile
MySQL with replica Aurora
Amazon EBS mirrorAmazon EBS mirror
AZ 1 AZ 2
Amazon EBSAmazon EBS
Primary
instance
Replica
instance
1
2
3
4
5
Amazon
S3
MySQL I/O profile for 30-min SysBench run
780K transactions
7,388K I/Os per million transactions (excludes mirroring, standby)
Average 7.4 I/Os per transaction
AZ 1 AZ 3
Primary
instance
AZ 2
Replica
instance
Async 4/6 quorum
Distributed writes
Replica
instance
Amazon
S3
Aurora I/O profile for 30-min SysBench run
27,378K transactions—35x more
0.95 I/Os per transaction (6x amplification)—7.7x less
Binlog Data Double-writeLog From files

Aurora lock management
MySQL lock manager Aurora lock manager
Scan
Delete
Insert
Scan
Scan
Delete
Scan
Insert
Insert
Delete
Insert
Scan
Same locking semantics as MySQL
Concurrent access to lock chains
Multiple scanners in individual lock chains
Lock-free deadlock detection
Needed to support many concurrent sessions, high update throughput

Asynchronous
key prefetch
▪ Works for queries using Batched
Key Access (BKA) join algorithm
+ Multi-Range Read (MRR)
optimization
▪ Performs a secondary to primary
index lookup during JOIN evaluation
▪ Uses background thread
to asynchronously load pages
into memory
Latency improvement factor vs. BKA
Join algorithm; decision support benchmark, R3.8xlarge
14.57
-
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
Cold buffer
AKP used in queries 2, 5, 8, 9, 11, 13, 17, 18, 19, 20, 21

Batched scans
Standard MySQL processes rows one-at-a-time
resulting in high overhead due to
• Repeated function calls
• Locking and latching
• Cursor store and restore
• InnoDB to MySQL format conversion
Aurora scan tuples from
the InnoDB buffer pool in batches for
• Table full scans
• Index full scans
• Index range scans
1.78x
-
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
Latency improvement factor vs. BKA
Join algorithm; decision support benchmark, R3.8xlarge

Parallel query processing
Aurora storage has thousands of CPUs
• Opportunity to push down and parallelize query
processing
• Moving processing close to data reduces network
traffic and latency
However, there are significant challenges
• Data is not range-partitioned—require full scans
• Data may be in-flight
• Read views may not allow viewing most recent data
• Not all functions can be pushed down
Database node
Storage nodes
Push down
predicates
Aggregate
results

Well-known decision support benchmark
0x
20x
40x
60x
80x
100x
120x
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22
Query response time reduction
▪ Peak speed up ~120x
▪ >10x speedup: 8 of 22 queries
We were able to test Aurora’s parallel query feature and the performance gains were very
good. To be specific, we were able to reduce the instance type from r3.8xlarge to
r3.2xlarge. For this use case, parallel query was a great win for us.
Jyoti Shandil, Cloud Data Architect
Performance results

What about availability?
“Performance matters only if your database is up.”

Six-way replicated storage
Survives catastrophic failures
• Six copies across three Availability
Zones
• Four out of six write quorum;
three out of six read quorum
• Peer-to-peer replication
for repairs
• Volume striped across hundreds
of storage nodes
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching
SQL
Transaction
AZ 1 AZ 2 AZ 3
Caching

Read replica and custom endpoint
Master
Read
replica
Read
replica
Read
replica
Shared distributed storage volume
Reader endpoint #1 Reader endpoint #2
Up to 15 promotable read replicas across multiple Availability Zones
Re-do log based replication leads to low replica lag—typically <10 ms
Custom reader endpoint with configurable failover order

Instant crash recovery
Traditional database
Must replay logs since
the last checkpoint
Typically five minutes between checkpoints
Single-threaded in MySQL; requires a large
number of disk accesses
Amazon Aurora
Underlying storage replays redo records on
demand as part of a disk read
Parallel, distributed, and asynchronous
No replay for startup
Checkpointed Data Redo Log
Crash at T0 requires
a re-application of the
SQL in the redo log since
last checkpoint
T0 T0
Crash at T0 results in redo logs applied to each
segment on demand, in parallel, asynchronously

Continuous availability with multi-master
Master Master Master Master
Shared distributed storage volume
Application #1 Application #2

Scaling write workload
Optimistic conflict management
There are many “oases” of consistency in
Aurora
Database nodes know transaction orders
from that node
Storage nodes know transactions orders
applied at that node
Only have conflicts when data changed at
both multiple database nodes and
multiple storage nodes
Much less coordination required
Near linear throughput scaling for
workloads with no or low conflict
Lower commit latency for workloads with
low conflict

Transactions with no conflict
▪ Transactions T1 and T2 from blue and
orange masters update different tables
(pages) Table 1 and Table 2
▪ No logical or physical conflicts—no
coordination required
▪ Near linear scaling with number of masters
for this type of sharded or partitioned
workloads
1 1 1 1 11 2 2 2 2 22
Page1 Page2
Master
Master
Begin transaction (T1)
Update (Table1)
Commit (T1)
Update (Table2)
Commit (T2)
Table 1 Table 2

Conflict resolution
1 1 11
Page1 Page2
Master
Master
Update (Table1)
Commit (T1)
Update (Table1)
Rollback (T2)
Table 1 Table 2
▪ Transactions T1 and T2 from blue and orange
masters update the same table (page) Table 1
▪ Transaction T1 from blue master achieves
quorum – transaction T1 is committed
▪ Transaction T2 from orange master fails to
achieve quorum—transaction T2 is rolled back
▪ Storage nodes act as arbitrator for conflict
resolution
▪ Logical conflicts are handled through MVCC
X
2 2

Aurora multi-master: Scaling and availability
0
10000
20000
30000
40000
50000
60000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101103105107109111113115117119121123125
Aggregatedthroughput
Time in minutes
SysBench workload on 4 R4.XL nodes
Adding a node Adding a node Node going down Node recovering

Global replication: Logical
Faster disaster recovery and enhanceddata locality
▪ Promote read replica to a master
for faster recovery
in case of a disaster
▪ Bring data close to your customer’s
applications
in different regions
▪ Promote to a master
for easy migration

High throughput: Up to 150K writes/sec.—negligible performance impact
Low replica lag: <1 sec. cross-country replica lag under heavy load
Fast recovery: <1 min. to accept full read/write workloads after region failure
Global replication: Physical
MR R
Region 1
AZ 1 AZ 2 AZ 3
Shared storage
R
Region 2
AZ 1 AZ 2 AZ 3
Shared storage
Replication
fleet
Replication
fleet

Global replication performance
Logical vs. physical replication
Logical replication Physical replication
0
100
200
300
400
500
600
0
50,000
100,000
150,000
200,000
250,000
seconds
QPS
QPS
Lag
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
0
50,000
100,000
150,000
200,000
250,000
seconds
QPS
QPS
Lag

Database backtrack
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible
Backtrack brings the database to a point in time without requiring restore from backups
• Backtracking from an unintentional DML or DDL operation
• Backtrack is not destructive. You can backtrack multiple times to find the right point in time
• Also useful for QA (rewind your DB between test runs)

Thank you!
Vlad Vlasceanu
vladv@amazon.com
Anoop Gupta
gupanoop@amazon.com

What's New in Amazon Aurora - ADB203 - Anaheim AWS Summit

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a What's New in Amazon Aurora - ADB203 - Anaheim AWS Summit

Similar a What's New in Amazon Aurora - ADB203 - Anaheim AWS Summit (20)

Más de Amazon Web Services

Más de Amazon Web Services (20)

What's New in Amazon Aurora - ADB203 - Anaheim AWS Summit