Learn about the new AWS Database Migration Service, which helps you migrate databases with minimal downtime from on-premises and Amazon EC2 environments to Amazon RDS, Amazon Redshift, Amazon Aurora and EC2 databases.
Exploring the Future Potential of AI-Enabled Smartphone Processors
Database Migration – Simple, Cross-Engine and Cross-Platform Migration
1. Database Migration: Simple, Cross-Engine and Cross-Platform Migrations with Minimal Downtime
Prahlad Rao,
Solutions Architect
2. Agenda
• How does the cloud help?
• How do I get there?
• How does it work?
• What have others done?
• Are there any tricks? Where is the magic?
• Show me!
4. How does the cloud help?
• Provision a database in 6 minutes
• Provision an MAZ database with a few mouse clicks
• Scale a database up/down with 60–90 seconds of downtime
• Apply patches with 60–90 seconds of downtime
• Add Read Replicas with a few mouse clicks
• Protect your backups and logs with 11 9s of durability
• Recover to any point in time from nightly backups + logs
• Detailed metrics, down to 1-second intervals
• Secure your data with single-click encryption at rest
Amazon
RDS
5. How can I get to the cloud?
• How will my on-premises data migrate to the cloud?
• How can I make it transparent to my users?
• Afterwards, how will on-premises and cloud data interact?
• How can I integrate my data assets within AWS?
• Can I get help moving off of commercial databases?
6. Migration used to be cost + complexity + time
• Commercial data migration and replication software
• Complex to set up and manage
• Application downtime
• Database-engine-specific application code
7. Start your first migration in 10 minutes or less
Keep your apps running during the migration
Replicate within, to, or from Amazon EC2 or RDS
Move data to the same or a different database engine
AWS
Database Migration
Service
(AWS DMS)
8. Customer
premises
Application users
AWS
Internet
VPN
• Start a replication instance
• Connect to source and target
databases
• Select tables, schemas, or
databases
Let AWS DMS create tables,
load data, and keep them in
sync
Switch applications over to
the target at your convenience
Keep your apps running during the migration
AWS
DMS
9. Load is table by table
Replication instance
Source Target
10. Change data capture (CDC) and apply
Replication instance
Source Target
Update
t1 t2
t1
t2
Transactions Change
apply after
bulk load
11. Multi-AZ option for high availability
Customer
premises
or AWS
AWS
Internet
VPN
AWS DMS
AWS DMS
15. AWS Schema Conversion Tool
Features
• Oracle and Microsoft SQL Server schema conversion to MySQL, Amazon Aurora, MariaDB, and
PostgreSQL
• Or convert your schema between PostgreSQL and any MySQL engine
• Database Migration Assessment report for choosing the best target engine
• Code browser that highlights places where manual edits are required
• Secure connections to your databases with SSL
• Cloud native code optimization
The AWS Schema Conversion Tool helps automate
many database schema and code conversion tasks
when migrating between database engines or data
warehouse engines
21. Other database migration use cases
Migration of business-critical applications
Migration from Classic to VPC
Cheap Read Replicas for Oracle
Read Replicas for other engines
Cross-region Read Replicas for Oracle and SQL Server
Analytics in the cloud
Dev/test and production environment sync
Ongoing replication for BI
22. Sources for AWS Database Migration Service
Customers can use the following databases as a source for data migration using AWS DMS:
On-premises and Amazon EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.4–9.5
• SAP ASE 15.7+
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008R2–2014. CDC operations are not supported yet.
• MySQL versions 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.4–9.5. CDC operations are not supported yet.
• Amazon Aurora (MySQL-compatible data source)
23. Targets for AWS Database Migration Service
Customers can use the following databases as a target for data replication using AWS
DMS:
On-premises and EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2016
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.5
• SAP ASE 15.7+
RDS instance databases:
• Oracle Database 11g–12c
• Microsoft SQL Server 2008 R2 - 2014
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3–9.5
• Amazon Aurora (MySQL-compatible data source)
Amazon Redshift
24. AWS Database Migration service pricing
• T2 for developing and periodic data migration
tasks
• C4 for large databases and minimizing time
• T2 pricing starts at $0.018 per hour for T2.micro
• C4 pricing starts at $0.154 per hour for C4.large
• 50 GB GP2 storage included with T2 instances
• 100 GB GP2 storage included with C4 instances
•
• Data transfer inbound and within AZ is free
• Data transfer across AZs starts at $0.01 per GB
26. Expedia: Online travel marketplace
Migrating some databases to Amazon Aurora
Kuldeep Chowhan, Principal Engineer, Expedia, Inc.:
“The ease by which we can do this using the AWS Database
Migration Service has simplified this process for us and
enabled us to accelerate our migration efforts. The ability to
closely monitor the process, the detailed logging feature, and
the support we received from AWS have given us a great deal
of confidence in a successful migration.”
World’s leading online travel company,
with a portfolio that includes 150+ travel
sites in 70 countries.
27. Thomas Publishing: Digital-friendly business
Needed to grow database footprint but using Oracle
would require significant up front investment in both
infrastructure and license expense
Wanted to migrate to Amazon Aurora
AWS DMS automated most of the work and dramatically
reduced the manual effort involved in the code migration
Hans Wald, Chief Technology Officer, Thomas Publishing:
"The AWS Database Migration Service will be a key
enabler for our plans to migrate more databases to
Amazon Aurora in 2016.”
Connecting buyers and suppliers across
all industrial sectors, evolving from an
industrial trade print publisher into
industry’s most respected group of
digital-friendly businesses.
28. Resources available to customers—DMS
• Getting Started Guide: Review
technical documentation.
• Features and benefits: Highlights
DMS features.
• Pricing: Prices for replication
instances, storage, and data transfer.
• Support: Post your questions to our
Support forum.
• Java SDK: Java-based API for
creating and managing data migration
tasks.
• AWS Command Line Interface: Start
and stop replication tasks with simple
commands.
29. Resources available to customers—AWS Schema Conversion Tool
• User Guide: Review technical docs at
• aws.amazon.com/documentation/SchemaConversion
Tool/
• or choose the Download button.
• Download area: Get installation files for
the Schema Conversion Tool.
•
• Support forums: Ask questions and
review how-to guides.
• https://forums.aws.amazon.com/forum.jspa?forumID=
208.
Start with some audience interaction… How many of you have been to the AWS Database Migration website? Has anyone tried using DMS yet?
When migrating your database to the cloud you do more than just change location. You leverage the power of the cloud. Customers appreciate the flexibility that platforms like EC2 offer, and many do run database engines there, but the RDS engines take things to the next level.
Other great features our customers rave about include the ability to get up an running quickly, deploy a redundant system easily, scale as required, have guaranteed backups, and secure information with a click.
- Responsive to customers
- Migrations easier and less intrusive
- Migrate will low downtime to any RDS engine type
- Post migration, customers asked for ongoing sync
- Sync between on prem and also between engines
- This mobility addresses one of the big requests: The ability to migrate off of commercial, license-intensive database engines, onto cloud-native, non-commercial engines.
- Pre DMS, costly, license-intensive replication software to move data
- Were not simple to setup or use required expert resources
- Long downtimes the other option
- Customers who wanted to switch engines had to deal with differences in datatypes, table syntax, sp’s, functions, triggers
So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine type
Let’s explore how it is that DMS can enable near zero downtime migration. The customer starts by launching a DMS instance in their AWS account. (click) Then the customer provides database connection information to connect out to their on-premises database, and in to their AWS database. (click) Next the customer will select which tables, schemas or databases they want to migrate, (click) and DMS will load the data, and keep it in sync on an ongoing basis. (click) Finally, at the time of the customer’s choosing, they can simply change the application to point to the new AWS database, instead of their old on-premises database. (click)
Creates tables at target database
Sets up metadata required at target
Populates data from source
Each process loads one entire table
Can use multiple processes
Can be paused
When restarted will continue from where it was stopped
Will reload any tables that were currently in progress
Oracle supplemental logging
MySQL row-level bin logging
SQL Server bulk logged/full recovery
Postgres WAL
No agent
Uses recovery log
Native change data capture API
When you either migrate or replicate, you can choose the Multi-AZ option. This option provides DMS in a high availability mode–in case the primary DMS server fails, the secondary one kicks in.
So by now it should be pretty obvious that data can move from A to B, and many have used DMS to do just that. Some customers have been pretty ingenious with DMS and have fanned in, or consolidated systems. DMS will put data where you ask it to, so if you want to merge data from a number of legacy systems into a single repository, DMS can help with this. And remember, DMS can just as easily move data between like databases and unlike databases, so you can take information from a number of different platforms and consolidate it in a single, cloud database.
In addition to “fanning-in” or consolidating, we’ve had a customer who has taken their database and “fanned-out” different portions to different targets. For example, from their ERP system they replicate they customer list out to the CRM system and also take some of the general ledger transactions and funnel them over to a BI system.
We’ve also had a customer who, for legacy reasons, used a single database for three different applications. With DMS you can split this information out to different databases. One could be PostgreSQL, one SQL Server, and one Aurora
So what happens if you are converting from one database engine to another? Where does the magic happen? This is where the AWS Schema Conversion Tool steps in.
Some organisations have many years of historical data, and for those that are willing to make the leap, filtering can be done during a migration so that only newer information is kept. Filtering can be done on any field in the source database–not just dates.
Just in case it wasn’t abundantly clear–DMS can move your data from pretty much anywhere to pretty much anywhere. Sort of like a courier company for the cloud
Mention why no older engines as sources
Mention submitted slides show only SQL 2014
MySQL to Aurora. They are moving to Aurora because they have to store more data for compliance.
Since they operate in every country there are lots of compliacnes they hae to adhere to.
Currently 120 TB of data for 1.5 years. They need to start holding for 2 years. 120 TB.
Reduce cost and improve performance.
Lodging events and changes are stored in the database.
Thomas Publishing is a family-owned company headquartered in New York City. They began in 1898 by publishing the Thomas Register of American Manufacturers, a 34-volume buying guide offering sourcing information and product specifications for industrial products and services. Thomas Publishing’s homegrown applications ran on a single, monolithic, Oracle database that could not support their growing user base with dynamic usage. Their application performance declined. Thomas Publishing came to Apps Associates looking for an alternative to the high cost associated with adding Oracle licenses, and the inflexible IT infrastructure necessary to maintain acceptable application performance.
Solution
Amazon Aurora
AWS Database Migration Service
S3
EC2
VPC
Using the AWS Database Migration Service and Schema Conversion Tool, the migration was completed in less than 4 weeks and all the data was successfully imported into Amazon Aurora.
Where can you learn more?
(CLICK) The User Guide and download option are available from the DMS section of the AWS Console
(CLICK) Yeah, there’s the Download button. Click it and you’ll see something like this.
(CLICK) Information on installation and use
(CLICK) The Schema Conversion Tool has its own forum to ask questions and get help