Simplify Your Database Migration to AWS | AWS Public Sector Summit 2016
•Descargar como PPTX, PDF•
8 recomendaciones•2,153 vistas
Migrating a database from one platform to another has been a pain point for many organizations for a long time. Often times, it involves weeks of careful planning and a migration strategy to minimize impact to the business. Many organizations are locked into a database platform even when there are better options available because they don’t want to take up the migration challenge. AWS Data Migration Service helps with live migration of databases across homogenous or heterogeneous database platforms. The service supports homogenous migrations such as Oracle to Oracle, and also heterogeneous migrations between different database platforms, such as Oracle to Amazon Aurora or Microsoft SQL Server to MySQL. The AWS Schema Conversion Tool is a desktop application that makes heterogeneous database migrations easy by automatically converting the source database schema to a format compatible with the target database. The tool helps with conversion of a database schema from an Oracle or Microsoft SQL Server database to an Amazon RDS MySQL DB instance or an Amazon Aurora DB cluster. Join us in this session to explore how these capabilities can simplify your database migration challenge.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Simplify Your Database Migration to AWS | AWS Public Sector Summit 2016
Similar a Simplify Your Database Migration to AWS | AWS Public Sector Summit 2016 (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
Simplify Your Database Migration to AWS | AWS Public Sector Summit 2016
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Atul Ganatra, AWS Solutions Architecture June 20, 2016 Simplify Your Database Migration to AWS
- 2. Session outline • The need for a database migration tool • AWS Database Migration Service (AWS DMS) overview • AWS Schema Conversion Tool overview • Demo • Q&A
- 3. Database migration challenges • How will my on-premises data be migrated to the cloud? • How can I make it transparent to my users? • How do I migrate schema objects and database code? • Can I get help moving off of commercial databases? • Can I avoid complex setup and migration procedures? • Can I do it cost effectively?
- 4. AWS Database Migration Service
- 5. Database Migration Service (DMS) goals Minimize disruption to the applications Make it easier to migrate databases to AWS Provide choice and flexibility
- 6. Start your first migration in minutes Keep your apps running during the migration Migrate to/from Amazon EC2 or Amazon RDS Move data to the same or different database engine Learn more at aws.amazon.com/dms AWS Database Migration Service
- 7. How does DMS work?
- 8. Create a replication instance Replication instance
- 9. Source Target Create source and target endpoints Replication instance
- 10. Source Target Create a task to migrate the data Task Replication instance
- 11. Source Target Load starts—captures changes from source Replication instance
- 12. Source Target Load complete—apply captured changes Replication instance Updates
- 13. Source Target Changes reach steady state Replication instance Updates
- 14. Source Target Stop writes on source Replication instance Updates
- 15. Source Target Allow in-flight transactions to finish Replication instance
- 16. Source Target Point application to the target Replication instance Updates
- 17. Full load is table by table Creates tables on target database Sets up metadata required on target Populates data from source Each process loads one entire table Can use multiple processes Task can be paused Replication instance Source Target
- 18. Changes are transactional Source Target Updates t1 t2 t1 t2 Target Replication instance
- 21. Homogenous or heterogeneous Replication instance Microsoft SQL Server MySQL Replication instance Oracle Oracle Replication instance Oracle Amazon Aurora
- 22. Key features • Homogenous or heterogeneous migrations • Full load migration, ongoing replication, or both • Supports variety of source and target databases • Allows migrating portion of database • Encryption of the data on replication instance • No agent on source or target database server
- 23. Security • Uses AWS Identity and Access Management (IAM) to control access • Secure resources within Amazon Virtual Private Cloud (Amazon VPC) • Uses AWS Key Management Service (AWS KMS) to encrypt data at rest on replication instance
- 24. Migration phases • Apply/create the empty database schema minus foreign key constraints and secondary indexes • Use DMS to move the data • Apply the remaining schema objects • Stop writes on source database • Let the changes synchronize to target • Change the app to point to target
- 25. Best practices • Load multiple tables in parallel • Remove bottlenecks on the target • Use multiple tasks for parallel load • Manage load on the source database • Enable logging for the migration tasks • Plan schema conversion
- 26. Supported databases Source • Oracle, Microsoft SQL Server, MySQL, MariaDB, PostgreSQL Target • Oracle, Microsoft SQL Server, MySQL, MariaDB, PostgreSQL, Amazon Aurora, Amazon Redshift
- 27. AWS Database Migration Service pricing T2 pricing starts at $0.018 per hour C4 pricing starts at $0.154 per hour (Pricing as of June 2016. See website for latest) 50 GB GP2 storage included with T2 instances 100 GB GP2 storage included with C4 instances Data transfer inbound and within AZ is free Data transfer across AZs starts at $0.01 per GB Data transfer between AWS Regions and leaving AWS costs the same as when using EC2
- 28. • Downloadable client install • Converts your database schema including tables, stored procedures, views, and DML AWS Schema Conversion Tool
- 29. Key features • Point-and-click migration of schema • Migration Assessment Report • Script to generate schema • Apply converted schema to target database • Converts tables, stored procedures, views, and other objects • Provides action items for manual changes
- 30. Get help with converting tables, views, and code Schemas Tables Indexes Views Packages Stored procedures Functions Triggers Sequences User defined types Synonyms
- 31. Supported conversions Source Database Target Database on Amazon RDS Microsoft SQL Server Amazon Aurora, MySQL, PostgreSQL MySQL PostgreSQL Oracle Amazon Aurora, MySQL, PostgreSQL PostgreSQL Amazon Aurora, MySQL
- 32. Demo
- 33. Expedia: On-line travel marketplace Migrating some databases to Amazon Aurora Kuldeep Chowhan, Principal Engineer, Expedia, Inc.: “The ease by which we can do this using the AWS Database Migration Service has simplified this process for us and enabled us to accelerate our migration efforts. The ability to closely monitor the process, the detailed logging feature, and the support we received from AWS have given us a great deal of confidence in a successful migration.” World’s leading online travel company, with a portfolio that includes 150+ travel sites in 70 countries.
- 34. Thomas Publishing: Digital-friendly business Needed to grow database footprint but using Oracle would require significant upfront investment in both infrastructure and license expense. Wanted to migrate to Amazon Aurora Database Migration Service automated most of the work and dramatically reduced the manual effort involved in the code migration. Hans Wald, Chief Technology Officer, Thomas Publishing: "The AWS Database Migration Service will be a key enabler for our plans to migrate more databases to Amazon Aurora in 2016.” Connecting buyers and suppliers across all industrial sectors, evolving from an industrial trade print publisher into industry’s most respected group of digital-friendly businesses.
- 36. Thank You!
Notas del editor
- We will be looking at some of the scenarios in which customers realize need to migrate the databases. We will have high level overview of what the database migration service has to offer and how it works. We will look at Schema Conversion Tool and see when you need to use it and how. This is also part of Database Migration Service, but works slightly differently. And at the end we will see a demo of actual migration that uses both DMS and SCT. We will leave few minutes at the end for questions.
- As we all know the databases and the applications that use relational databases have been around for decades. As such, some decisions to use the databases have been dated and may not hold to be the best choice for the current situations. The landscape of the database world keeps changing and with that your needs and desire to use a particular platform are likely to change. E.g. now you have an option to use a database in AWS with RDS that takes away lot of burden of doing that you had no choice but to do it on your own. So may customers see value in offering such as RDS or for that matter continue to run it like they have always run but want to run it in AWS using EC2 instances instead of managing those instances on their own. AWS offers very attractive value proposition to run the IT infrastructure which certainly includes your database. Often times customers are comfortable with the challenge it involves to move that database to AWS. Database migration regardless of from where you are moving to where you are moving often requires careful planning and even with that involves experiencing decent amount of outage and impact to your customers. You may not be happy about that impact to your customers. When you move the how your source database and target databases are going to communicate. What type of networking and security requirements you are going to have. Depending on the type of the data you may be concerned about that. Add to that the complexity of setting up that connection and tools to do the migration. And last but not the least many customers feel that they need to move off of the commercial databases in order to stay competitive or continue to grow the usage of databases without incurring cost of expensive licenses.
- Welcome to the AWS Database Migration Service. AKA DMS. The service was introduced earlier this year and has received great response from our customers. We have thousands of customers already using DMS.
- As you seen with many AWS services that our goal is to relieve you from that undifferentiated heavy lifting so you can focus on what you are good at doing. You can focus on what your core competency is. We want to do it so that has min impact to your business. And another primary reason is that we want our customers to have choice of using various database options on top of AWS. You can migrate to different databases on RDS or run them on EC2. It’s your choice. You can use DMS to move your data in or move it out. The requirement is that one end of the migration has to be in the AWS. So for whatever reason if your migration doesn’t work out, you can use the same mechanism to get it out of AWS and take back to your on-prem environment or move it from RDS to database that is running on EC2. So we want to keep the migration options like some of the options that you may be familiar with that it can be done both ways.
- * Like all AWS services, it is easy and straightforward to get started. You can get started with your first migration task few minutes. You simply connect it to your source and target databases, and it copies the data over, and begins replicating changes from source to target. *That means that you can keep your apps running during the migration, then switch over at a time that is convenient for your business. * DMS can be used to replicate within, to or from AWS EC2 or RDS databases. Basically, from anywhere to anywhere as long as one of them is an AWS resource. *Again- it is heterogeneous ~. With DMS, you can move data between engines. Supports Oracle, Microsoft SQL Server, MySQL, PostgreSQL, MariaDB, Amazon Aurora, Amazon Redshift * If you would like to learn more, go to…
- You start out your DMS migration effort by replication instance. A replication instance is your main compute unit that does the job of doing your migration. We have several different types of replication instance you can choose depending on how large your migration project is. Mainly replication instance comes in two families. T2 and C4. We recommend using one of the T2 instance while you are checking your database migration out to see if things are migrating. Or when you are running small jobs on regular basis you can use T2 family. For large migration in production we recommend using one of the C4 instances. As you know the T2 instances comes with the baseline cpu performance which can burst depending the CPU credits that you accumulate. Certainly okay to use them for your dev/test scenarios. The repl instance is created inside your VPC. You can put it in default VPC or on the VPC you create. It’s going to need access to Internet or your on-prem resource depending where your source target are located. We will look at that little bit later. But you should keep it mind as you create your replication instance as to it’s going to need connectivity with your source and target. Instance can be public or private depending on whether your source and target databases are available through the private connectivity from the VPC. I.e. VPN connection or Direct Connect or have the db in the same VPC, or peered VPC. Similar concepts that apply to network configuration of an instance inside of a VPC applies also applies to replication instance. Just like an EC2 instance, the replication instance is also protected by Security Group which needs to allow traffic between source and target endpoints.
- Next you want to create your source and target endpoints. The source endpoints can be any of the supported databases. You don’t have to worry about managing odbc or jdbc driver or anything like that. When you create a source and target endpoint mainly you are required to provide is the host where your database is running, the port, the username and the password. There are additional things you can specify by custom parameters to influence the behavior the source or the target. For example, Oracle requires the addition of supplemental logging and MySQL requires row-level bin logging. When using Amazon RDS as a source, we recommend ensuring that backups are enabled and that the source database is configured to retain change logs for a sufficient time (24 hours is usually enough). The documentation describes all endpoint requirements for each source and target database engine. The connection information you provide is encrypted by the service using a key in the KMS. By default the service uses an account specific default. You have an option to use a different key.
- now you are ready to create a task. Task is unit of your migration activity. Task has mainly three key parameters: - source and target - type of task - tables to migrate There are three types of tasks you can define: Full load: transfers the data from source to target and stops Full load and on-going replication: starts with the full load and once finishes continues with on-going replication using Change Data Capture technique. Replication only: Captures only new transaction from the source and applied them to target While creating the task you can also specify which schema and tables you want to load as part of this task. We will look at it little bit later as to how you can create multiple tasks for migration of the same database. You can also specify certain transformation at the metadata level. Changing column names, or changing case of the table, adding prefix, etc. You can also drop a column during migration. There are several other parameters you can specify such as how many tables you want loaded in parallel. The default number is 8, which you can change depending on how you want to perform the task. You can put multiple tasks on the same replication instance. You should test your configuration out.
- Once you have the tasks defined, you basically start your task. Database Migration Service starts with what we call full load. While it’s doing the full load the new transaction that occur on the source database are captured using Change Data Capture on the source and staged on the migration instance. Task caches the data mostly in the memory of the replication instance but depending how fast the target is able to take in the data and also the size of the data being transferred it may need to write to the disk of the migration instance. You get 50 GB or 100 GB depending which family you used for the replication instance
- Let’s say after some time depending on the size of the database the full load is complete. The source and target reaches steady state or equilibrium. At this point the service is going start applying captured changes since it started the full load. Depending on the source, you are going to have to enable transaction logs. E.g. in case of Oracle you have to enable archive logs and supplemental logging at the database level and also for each table. If you don't have that enabled, you are going to see some type of error in the logs and the task will fail.
- Once it catches up the source and the target will reach a steady state. What this means is that only the delta that is applied to the source is being moved to the target. There could be some replication lag but not much. You also have a choice of how frequently you want DMS to check for new changes. It can be as small as 1 sec. The default is 5 sec. It’s one of the parameter you can specify while creating the task. So you can see you still have the updates or the write going on with your source database.
- You can stop the update on the source and make sure all the changes are flushed through the replication tasks and applied to the target
- You can stop the update on the source and make sure all the changes are flushed through the replication tasks and applied to the target
- At this point you are ready to move
- 8 processes at a time by default. Parameter that can be controlled by customer - Mention the phases in the project e.g.) Setup an empty shell, migrate, apply foreign keys, secondary indexes, etc. Task loads data table by table during the full-load. You need to make sure foreign keys are disabled on the target database. E.g. there is a parameter for MySQL target to ignores the foreign key checks. You can also drop the foreign keys and recreate them when your full load is done. If the target doesn’t have the necessary tables to transfer the data it will create them including the primary keys. Once the necessary table is created the task will transfer the data. Each process will load the entire table. You can configure multiple processes to load tables in parallel. The default is 8, but you can adjust this number to either speed up the task or adjust the load on your source and target database. You should be mindful of how many tasks you run. The table copy requires a full table scan on the source and you want to make sure it doesn’t adversely impact your source application. You can also pause the task. If for any reason if you want to pause the task to check on something or if you have issue with connectivity that disrupts the tasks. It can be restarted. When the task restarts it will start from where it left. By default it will reload the table which it was in the middle of loading. You can configure it so that it doesn’t reload the table.
- As we saw earlier, mainly there are two phases of migration tasks. The full-load and then synchronize the transaction for ongoing replication. The replication starts once the full-load is complete and it’s going to take little bit of time for it to be in sync with source because it would have built up some backlog of transactions that will apply in the order they were received. The changes applied during this phase are transactional. So you want to make sure that if you are going to have on-going replication in your task, you include tables that can provide consistent transactions. In other words, you don’t want to not include some tables that are in transactions and leave behind the others. This could create an error while applying this transactions on target if the tables don’t exist and you may lose those transactions.
- The migration performed with DMS can be homogenous or heterogeneous. Meaning it can be same engine or different.
- Here is the summary of key features that we reviewed. You can use DMS for both homogenous and heterogeneous migrations There three type of migrations it supports. Variety of options for both source and target. Allows you to migrate portion of your db by just including certain tables and columns. Encryption of data on replication instance and endpoints using KMS It’s easy to configure because you don’t need to install any software or agent on either the source or the target database.
- Just like any other service in AWS you need proper IAM permission assigned to your user or role to be able to access DMS. You can carry out a completely private migration project by keeping your resources private. The replication instance is launched inside of a VPC and you have an option to keep it private. You can create private connectivity from the replication instance to the your source and target and keep everything private. DMS uses encryption using AWS Key Management Service to encrypt the data that is stored on the instance and also encrypt the endpoint information like user and password.
- JW - The AWS Schema Conversion Tool helps you convert your database schema from an Oracle or Microsoft SQL Server database, to a MySQL DB instance, an Amazon Aurora DB cluster, or a PG instance. The target can be either RDS or EC2 based Desktop application available for Windows, Mac, and Linux