SlideShare a Scribd company logo
1 of 48
Download to read offline
Replicate Oracle to Oracle,
Oracle to MySQL ,
and Oracle to Analytics
Linas Virbalas, Senior Software Engineer
Robert Hodges, CEO
©Continuent 2014
Introducing Continuent

•

The leading provider of clustering and
replication for open source DBMS

•

Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL

• Replication - Flexible, high-performance data
movement

©Continuent 2014

2
What is Tungsten Replicator?
Tungsten Replicator:
a high-performance
open source database
replication engine
!

GPL V2 license - 100% open source	

Download from https://code.google.com/p/tungsten-replicator/	

Annual support subscription available from Continuent
® without the Price Tag”
“GoldenGate
©Continuent 2014

3
Tungsten Replicator Overview
Master
Download
transactions 	

via network
DBMS	

Logs

Slave

Replicator

(Transactions + Metadata)

Replicator

Apply using JDBC

©Continuent 2014

THL

THL
(Transactions + Metadata)

4
Tungsten Replication Service
Pipeline
Stage
Extract Filter

Master	

DBMS

©Continuent 2014

Stage
Apply

Extract Filter

Transaction	

History Log

Stage
Apply

Extract Filter

Queue

5

Apply

Slave	

DBMS
Filters and Parallel Apply
Pipeline
Stage
Extract Filter

Master	

DBMS

©Continuent 2014

Stage

Stage
Apply

Extract Filter

Transaction	

History Log

Apply

Extract Filter
Extract Filter
Extract Filter

In-Memory	

Queue

6

Apply
Apply
Apply

Slave	

DBMS
Multiple Services per Replicator
Replicator
Service
frommysql

Replicator
frommysql
fromoracle

Aggregated
Service
fromoracle

Replicator

©Continuent 2014
master-slave
MySQL

Oracle
fan-in slave

©Continuent 2014

Oracle

MySQL
all-masters

Heterogeneous

Oracle

MySQL

Oracle
star

MySQL
Heterogeneous
Replication

©Continuent 2014
Steps to Heterogeneous Replication
1. Prepare (translate) schema for
the slave DBMS
2. Set up replication

ddlscan
tpm

3. Provision initial data



MySQL Blackhole Parallel Apply

Parallel Extract

4. Continue real-time replication
Tungsten Replicator
©Continuent 2014
Replicating from
MySQL to Oracle

©Continuent 2014
Use Case: Real-Time Sales/Order
Fulfillment
Web-based Sales

~1M	

Xacts/	

Day

©Continuent 2014

Purchase Ordering

Real-Time Replication

3M
Order
Items
1. Translating schema
for the slave

©Continuent 2014
Translating Schema

•

Goal - set up heterogeneous replication

Replication

©Continuent 2014

14
Translating Schema

•

Beginning - how to convert tables?

empty

Tables

©Continuent 2014

15
Translating Schema

•

Beginning - how to convert tables?

empty

•Data types?	

•Column lengths?	

•Naming conventions?	

•Reserved words?

Tables

©Continuent 2014

16
Translating Schema

•

ddlscan automates these conversions

ddlscan

Tables

©Continuent 2014

17

empty
ddlscan

•
•
•
•

©Continuent 2014

Part of Tungsten Replicator, GPL v2
Translates schema with replication in mind
Provides errors and warnings
Can rename schema/tables/columns

18
Usage (MySQL to Oracle Example)
$ cd tungsten-replicator/bin
!

$ ./ddlscan 
-db test 
-template ddl-mysql-oracle.vm 
-user tungsten 
-pass secret

©Continuent 2014

19
Translating Schema

•

ddlscan looks into source schema

empty

Tables

©Continuent 2014

20
Translating Schema

•

ddlscan translates and renders DDL commands

empty

Tables
oracle-ddl.sql
©Continuent 2014

21
Result of ddlscan
CREATE TABLE 3colors
/* ERROR: table starts with a number, use rename option and a
filter */
(
id NUMBER(10, 0),
color VARCHAR2(1) /* ENUM('R','G','B') */,
enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */,
acolumnthatdoesntfittooracleat NUMBER(10, 0)
/* WARN: truncated column name exceeding 30 characters
(acolumnthatdoesntfittooracleatall) */
!
/* ERROR: table must have a primary key! */
);
!
CREATE TABLE talks
...

©Continuent 2014

22
Translating Schema

•

You run resulting SQL file on Oracle

empty

Tables
oracle-ddl.sql
©Continuent 2014

23
Translating Schema

•

Tables are ready!

Tables

©Continuent 2014

Translated	

tables (empty)
24
2. Set Up Replication

©Continuent 2014
Replicator Installation
./tools/tpm configure my_ora 
--enable-heterogenous-service=true 
--install-directory=/opt/mysql_to_ora/continuent 
--members=alpha,bravo 
--master=alpha
!
./tools/tpm configure my_ora --hosts=alpha 
--replication-user=tungsten 
--replication-password=secret
!
./tools/tpm configure my_ora --hosts=bravo 
--replication-user=tungsten_frommysql 
--replication-password=secret 
--datasource-type=oracle 
--datasource-oracle-service=ORCL 
--svc-applier-filters=dropstatementdata
!
./tools/tpm install my_ora
!
./tools/tpm start my_ora

!
©Continuent 2014

26
Replication Under the Hood
MySQLExtractor

OracleApplier

Tungsten Master
Replicator
Service
my_ora

MySQL
Binlog

binlog_format=row

©Continuent 2014

Tungsten Slave
Replicator
Service
my_ora

MySQLExtractor
Special Filters
• Transform
ENUM to string
• Transform SET
to string
• Column names
and signed flag

Special Filters
• Drop DDL
• Map names to upper case
• Rename too long or reserved
objects
• Optimize updates to remove
unchanged columns
• Ignore extra tables

27
3. Provisioning an
Oracle slave

©Continuent 2014
Provisioning (2)

ddlscan

table structure
table structure
table data
table structure
table data
table data

©Continuent 2014

29

table structure
table structure
table structure
Provisioning (3)

MySQL Sandbox
table structure
table structure
table data
table structure
table data
table data

table structure
table structure
table structure

CONVERT to
BLACKHOLE

©Continuent 2014

30
Provisioning (4)
not replicating

MySQL Sandbox
table structure
table structure
table data
table structure
table data
table data
backup

©Continuent 2014

31
Provisioning (5)

(ROW)

backup

©Continuent 2014

Master	

Tungsten
Replicator

THL

32

4

bin
logs

1
2
3

MySQL
Sandbox

Slave	

Tungsten
Replicator
4. Continue Real-Time
Replication

©Continuent 2014
Provisioning (6)

MySQL Sandbox
table structure
table structure
table data
table structure
table data
table data

Master	

Tungsten
Replicator

bin
logs

THL

©Continuent 2014

34

Slave	

Tungsten
Replicator
Replicating from
Oracle to MySQL

©Continuent 2014
Use Case: Web Content Publishing
Backend Office

Web-Based Catalog

Real-Time Publication

©Continuent 2014
How Does Tungsten Do That?
OracleCDCExtractor

MySQLApplier

Tungsten Master
Replicator

Tungsten Slave
Replicator

Service
ora_my

Service
ora_my

OracleCDCExtractor
No Special Filters

demo	

schema

demo_pub	

schema

Oracle CDC
(Synchronous or
Async Hotlog)
©Continuent 2014

37

Special Filters
• Map names to lower case
• Ignore extra tables
• Heartbeat table renaming
Replicating from
Oracle to Oracle

©Continuent 2014
How Does Tungsten Do That?
OracleCDCExtractor

OracleApplier

Tungsten Master
Replicator

Tungsten Slave
Replicator

Service
ora_ora

Service
ora_ora

OracleCDCExtractor
No Special Filters

demo	

schema

demo_pub	

schema

Oracle CDC
(Synchronous or
Async Hotlog)
©Continuent 2014

39

Special Filters
No Special Filters
DEMO

DEMO.*
DEMO2.*

©Continuent 2014

demo.*
demo2.*
Replicating from
Oracle to Vertica

©Continuent 2014
Use Case: Web Content Publishing
Backend Office

Analytic Reports

Near Real-Time	

Publication
Feeds from other data sources
©Continuent 2014
How Does Tungsten Do That?
OracleCDCExtractor

SimpleBatchApplier

Tungsten Master
Replicator

Tungsten Slave
Replicator

Service
ora_ver

Service
ora_ver

OracleCDCExtractor
No Special Filters

demo	

schema

demo_pub	

schema

Oracle CDC
(Synchronous or
Async Hotlog)
©Continuent 2014

43

Special Filters
• Map names to lower case
• Ignore extra tables
• Rename data tables (if required)
Batch Loading--The Gory Details

Replicator
Transactions
from master

Service ora_ver
COPY to
stage tables

CSV	

CSV	

CSV	

Files
Files
Files
Merge
Script

©Continuent 2014

44

(or)	

COPY 	

directly to
base tables

Staging	

Staging	

Staging	

Tables
Tables
Tables
SELECT to
base tables

Base
Base
Base
Tables
Tables
Tables
DEMO

©Continuent 2014
Coming
Attractions

©Continuent 2014
New Replication Features in 2014

•

•

©Continuent 2014

Parallel extraction for fast provisioning

•
•
•

Select data as of specific SCN
Pull data across in multiple streams
Matches parallel apply on slave side

!

Hadoop data loading

•
•
•

Direct loading from MySQL/Oracle to HDFS
View data using Hive
Generation of materialized views

47
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com

Our Blogs:
http://flyingclusters.blogspot.com
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://continuent-tungsten.blogspot.com

Continuent Web Page:	

http://www.continuent.com	

!

Documentation: 	

https://docs.continuent.com	

!

Tungsten Replicator:	

http://code.google.com/p/tungsten-replicator	

©Continuent 2014

48

More Related Content

What's hot

What's hot (20)

Scylla Summit 2017: Scylla for Mass Simultaneous Sensor Data Processing of ME...
Scylla Summit 2017: Scylla for Mass Simultaneous Sensor Data Processing of ME...Scylla Summit 2017: Scylla for Mass Simultaneous Sensor Data Processing of ME...
Scylla Summit 2017: Scylla for Mass Simultaneous Sensor Data Processing of ME...
 
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and Beyond
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and BeyondScylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and Beyond
Scylla Summit 2022: The Future of Consensus in ScyllaDB 5.0 and Beyond
 
Wayfair Use Case: The four R's of Metrics Delivery
Wayfair Use Case: The four R's of Metrics DeliveryWayfair Use Case: The four R's of Metrics Delivery
Wayfair Use Case: The four R's of Metrics Delivery
 
Latest performance changes by Scylla - Project optimus / Nolimits
Latest performance changes by Scylla - Project optimus / Nolimits Latest performance changes by Scylla - Project optimus / Nolimits
Latest performance changes by Scylla - Project optimus / Nolimits
 
Postgres clusters
Postgres clustersPostgres clusters
Postgres clusters
 
Apache Sqoop: A Data Transfer Tool for Hadoop
Apache Sqoop: A Data Transfer Tool for HadoopApache Sqoop: A Data Transfer Tool for Hadoop
Apache Sqoop: A Data Transfer Tool for Hadoop
 
Nov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars georgeNov. 4, 2011 o reilly webcast-hbase- lars george
Nov. 4, 2011 o reilly webcast-hbase- lars george
 
Scylla Summit 2017: The Upcoming HPC Evolution
Scylla Summit 2017: The Upcoming HPC EvolutionScylla Summit 2017: The Upcoming HPC Evolution
Scylla Summit 2017: The Upcoming HPC Evolution
 
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, AdjustShipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
Shipping Data from Postgres to Clickhouse, by Murat Kabilov, Adjust
 
Transparent sharding with Spider: what's new and getting started
Transparent sharding with Spider: what's new and getting startedTransparent sharding with Spider: what's new and getting started
Transparent sharding with Spider: what's new and getting started
 
Performance improvements in PostgreSQL 9.5 and beyond
Performance improvements in PostgreSQL 9.5 and beyondPerformance improvements in PostgreSQL 9.5 and beyond
Performance improvements in PostgreSQL 9.5 and beyond
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
 
Go Programming Patterns
Go Programming PatternsGo Programming Patterns
Go Programming Patterns
 
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
Flink Forward Berlin 2017: Stefan Richter - A look at Flink's internal data s...
 
Case Studies on PostgreSQL
Case Studies on PostgreSQLCase Studies on PostgreSQL
Case Studies on PostgreSQL
 
Performance tuning ColumnStore
Performance tuning ColumnStorePerformance tuning ColumnStore
Performance tuning ColumnStore
 
Building Spark as Service in Cloud
Building Spark as Service in CloudBuilding Spark as Service in Cloud
Building Spark as Service in Cloud
 
Replicating in Real-time from MySQL to Amazon Redshift
Replicating in Real-time from MySQL to Amazon RedshiftReplicating in Real-time from MySQL to Amazon Redshift
Replicating in Real-time from MySQL to Amazon Redshift
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBase
 
PostgreSQL 9.5 - Major Features
PostgreSQL 9.5 - Major FeaturesPostgreSQL 9.5 - Major Features
PostgreSQL 9.5 - Major Features
 

Similar to Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

MySQL 5.7: Core Server Changes
MySQL 5.7: Core Server ChangesMySQL 5.7: Core Server Changes
MySQL 5.7: Core Server Changes
Morgan Tocker
 
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
LarryZaman
 

Similar to Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics (20)

Real-Time Data Loading from MySQL to Hadoop with New Tungsten Replicator 3.0
Real-Time Data Loading from MySQL to Hadoop with New Tungsten Replicator 3.0Real-Time Data Loading from MySQL to Hadoop with New Tungsten Replicator 3.0
Real-Time Data Loading from MySQL to Hadoop with New Tungsten Replicator 3.0
 
Tungsten University: Load A Vertica Data Warehouse With MySQL Data
Tungsten University: Load A Vertica Data Warehouse With MySQL DataTungsten University: Load A Vertica Data Warehouse With MySQL Data
Tungsten University: Load A Vertica Data Warehouse With MySQL Data
 
MySQL 5.7: Core Server Changes
MySQL 5.7: Core Server ChangesMySQL 5.7: Core Server Changes
MySQL 5.7: Core Server Changes
 
Setup & Operate Tungsten Replicator
Setup & Operate Tungsten ReplicatorSetup & Operate Tungsten Replicator
Setup & Operate Tungsten Replicator
 
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdfDataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
DataEng Mad - 03.03.2020 - Tibero 30-min Presentation.pdf
 
Set Up & Operate Open Source Oracle Replication
Set Up & Operate Open Source Oracle ReplicationSet Up & Operate Open Source Oracle Replication
Set Up & Operate Open Source Oracle Replication
 
Set Up & Operate Real-Time Data Loading into Hadoop
Set Up & Operate Real-Time Data Loading into HadoopSet Up & Operate Real-Time Data Loading into Hadoop
Set Up & Operate Real-Time Data Loading into Hadoop
 
Set Up & Operate Tungsten Replicator
Set Up & Operate Tungsten ReplicatorSet Up & Operate Tungsten Replicator
Set Up & Operate Tungsten Replicator
 
Breaking the-database-type-barrier-replicating-across-different-dbms
Breaking the-database-type-barrier-replicating-across-different-dbmsBreaking the-database-type-barrier-replicating-across-different-dbms
Breaking the-database-type-barrier-replicating-across-different-dbms
 
介绍 Percona 服务器 XtraDB 和 Xtrabackup
介绍 Percona 服务器 XtraDB 和 Xtrabackup介绍 Percona 服务器 XtraDB 和 Xtrabackup
介绍 Percona 服务器 XtraDB 和 Xtrabackup
 
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
Business_Continuity_Planning_with_SQL_Server_HADR_options_TechEd_Bangalore_20...
 
DBCC - Dubi Lebel
DBCC - Dubi LebelDBCC - Dubi Lebel
DBCC - Dubi Lebel
 
Accessing Databases from R
Accessing Databases from RAccessing Databases from R
Accessing Databases from R
 
Accessing Databases from R
Accessing Databases from RAccessing Databases from R
Accessing Databases from R
 
Denver SQL Saturday The Next Frontier
Denver SQL Saturday The Next FrontierDenver SQL Saturday The Next Frontier
Denver SQL Saturday The Next Frontier
 
What’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStoreWhat’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStore
 
GoldenGate CDR from UKOUG 2017
GoldenGate CDR from UKOUG 2017GoldenGate CDR from UKOUG 2017
GoldenGate CDR from UKOUG 2017
 
Db2 V12 incompatibilities_&_improvements_over_V11
Db2 V12 incompatibilities_&_improvements_over_V11Db2 V12 incompatibilities_&_improvements_over_V11
Db2 V12 incompatibilities_&_improvements_over_V11
 
Whats new in Oracle Database 12c release 12.1.0.2
Whats new in Oracle Database 12c release 12.1.0.2Whats new in Oracle Database 12c release 12.1.0.2
Whats new in Oracle Database 12c release 12.1.0.2
 
Copy Data Management for the DBA
Copy Data Management for the DBACopy Data Management for the DBA
Copy Data Management for the DBA
 

Recently uploaded

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 

Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

  • 1. Replicate Oracle to Oracle, Oracle to MySQL , and Oracle to Analytics Linas Virbalas, Senior Software Engineer Robert Hodges, CEO ©Continuent 2014
  • 2. Introducing Continuent • The leading provider of clustering and replication for open source DBMS • Our Product: Continuent Tungsten • Clustering - Commercial-grade HA, performance scaling and data management for MySQL • Replication - Flexible, high-performance data movement ©Continuent 2014 2
  • 3. What is Tungsten Replicator? Tungsten Replicator: a high-performance open source database replication engine ! GPL V2 license - 100% open source Download from https://code.google.com/p/tungsten-replicator/ Annual support subscription available from Continuent ® without the Price Tag” “GoldenGate ©Continuent 2014 3
  • 4. Tungsten Replicator Overview Master Download transactions via network DBMS Logs Slave Replicator (Transactions + Metadata) Replicator Apply using JDBC ©Continuent 2014 THL THL (Transactions + Metadata) 4
  • 5. Tungsten Replication Service Pipeline Stage Extract Filter Master DBMS ©Continuent 2014 Stage Apply Extract Filter Transaction History Log Stage Apply Extract Filter Queue 5 Apply Slave DBMS
  • 6. Filters and Parallel Apply Pipeline Stage Extract Filter Master DBMS ©Continuent 2014 Stage Stage Apply Extract Filter Transaction History Log Apply Extract Filter Extract Filter Extract Filter In-Memory Queue 6 Apply Apply Apply Slave DBMS
  • 7. Multiple Services per Replicator Replicator Service frommysql Replicator frommysql fromoracle Aggregated Service fromoracle Replicator ©Continuent 2014
  • 10. Steps to Heterogeneous Replication 1. Prepare (translate) schema for the slave DBMS 2. Set up replication ddlscan tpm 3. Provision initial data
 
 MySQL Blackhole Parallel Apply Parallel Extract 4. Continue real-time replication Tungsten Replicator ©Continuent 2014
  • 11. Replicating from MySQL to Oracle ©Continuent 2014
  • 12. Use Case: Real-Time Sales/Order Fulfillment Web-based Sales ~1M Xacts/ Day ©Continuent 2014 Purchase Ordering Real-Time Replication 3M Order Items
  • 13. 1. Translating schema for the slave ©Continuent 2014
  • 14. Translating Schema • Goal - set up heterogeneous replication Replication ©Continuent 2014 14
  • 15. Translating Schema • Beginning - how to convert tables? empty Tables ©Continuent 2014 15
  • 16. Translating Schema • Beginning - how to convert tables? empty •Data types? •Column lengths? •Naming conventions? •Reserved words? Tables ©Continuent 2014 16
  • 17. Translating Schema • ddlscan automates these conversions ddlscan Tables ©Continuent 2014 17 empty
  • 18. ddlscan • • • • ©Continuent 2014 Part of Tungsten Replicator, GPL v2 Translates schema with replication in mind Provides errors and warnings Can rename schema/tables/columns 18
  • 19. Usage (MySQL to Oracle Example) $ cd tungsten-replicator/bin ! $ ./ddlscan -db test -template ddl-mysql-oracle.vm -user tungsten -pass secret ©Continuent 2014 19
  • 20. Translating Schema • ddlscan looks into source schema empty Tables ©Continuent 2014 20
  • 21. Translating Schema • ddlscan translates and renders DDL commands empty Tables oracle-ddl.sql ©Continuent 2014 21
  • 22. Result of ddlscan CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */ ( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0) /* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */ ! /* ERROR: table must have a primary key! */ ); ! CREATE TABLE talks ... ©Continuent 2014 22
  • 23. Translating Schema • You run resulting SQL file on Oracle empty Tables oracle-ddl.sql ©Continuent 2014 23
  • 24. Translating Schema • Tables are ready! Tables ©Continuent 2014 Translated tables (empty) 24
  • 25. 2. Set Up Replication ©Continuent 2014
  • 26. Replicator Installation ./tools/tpm configure my_ora --enable-heterogenous-service=true --install-directory=/opt/mysql_to_ora/continuent --members=alpha,bravo --master=alpha ! ./tools/tpm configure my_ora --hosts=alpha --replication-user=tungsten --replication-password=secret ! ./tools/tpm configure my_ora --hosts=bravo --replication-user=tungsten_frommysql --replication-password=secret --datasource-type=oracle --datasource-oracle-service=ORCL --svc-applier-filters=dropstatementdata ! ./tools/tpm install my_ora ! ./tools/tpm start my_ora ! ©Continuent 2014 26
  • 27. Replication Under the Hood MySQLExtractor OracleApplier Tungsten Master Replicator Service my_ora MySQL Binlog binlog_format=row ©Continuent 2014 Tungsten Slave Replicator Service my_ora MySQLExtractor Special Filters • Transform ENUM to string • Transform SET to string • Column names and signed flag Special Filters • Drop DDL • Map names to upper case • Rename too long or reserved objects • Optimize updates to remove unchanged columns • Ignore extra tables 27
  • 28. 3. Provisioning an Oracle slave ©Continuent 2014
  • 29. Provisioning (2) ddlscan table structure table structure table data table structure table data table data ©Continuent 2014 29 table structure table structure table structure
  • 30. Provisioning (3) MySQL Sandbox table structure table structure table data table structure table data table data table structure table structure table structure CONVERT to BLACKHOLE ©Continuent 2014 30
  • 31. Provisioning (4) not replicating MySQL Sandbox table structure table structure table data table structure table data table data backup ©Continuent 2014 31
  • 34. Provisioning (6) MySQL Sandbox table structure table structure table data table structure table data table data Master Tungsten Replicator bin logs THL ©Continuent 2014 34 Slave Tungsten Replicator
  • 35. Replicating from Oracle to MySQL ©Continuent 2014
  • 36. Use Case: Web Content Publishing Backend Office Web-Based Catalog Real-Time Publication ©Continuent 2014
  • 37. How Does Tungsten Do That? OracleCDCExtractor MySQLApplier Tungsten Master Replicator Tungsten Slave Replicator Service ora_my Service ora_my OracleCDCExtractor No Special Filters demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) ©Continuent 2014 37 Special Filters • Map names to lower case • Ignore extra tables • Heartbeat table renaming
  • 38. Replicating from Oracle to Oracle ©Continuent 2014
  • 39. How Does Tungsten Do That? OracleCDCExtractor OracleApplier Tungsten Master Replicator Tungsten Slave Replicator Service ora_ora Service ora_ora OracleCDCExtractor No Special Filters demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) ©Continuent 2014 39 Special Filters No Special Filters
  • 41. Replicating from Oracle to Vertica ©Continuent 2014
  • 42. Use Case: Web Content Publishing Backend Office Analytic Reports Near Real-Time Publication Feeds from other data sources ©Continuent 2014
  • 43. How Does Tungsten Do That? OracleCDCExtractor SimpleBatchApplier Tungsten Master Replicator Tungsten Slave Replicator Service ora_ver Service ora_ver OracleCDCExtractor No Special Filters demo schema demo_pub schema Oracle CDC (Synchronous or Async Hotlog) ©Continuent 2014 43 Special Filters • Map names to lower case • Ignore extra tables • Rename data tables (if required)
  • 44. Batch Loading--The Gory Details Replicator Transactions from master Service ora_ver COPY to stage tables CSV CSV CSV Files Files Files Merge Script ©Continuent 2014 44 (or) COPY directly to base tables Staging Staging Staging Tables Tables Tables SELECT to base tables Base Base Base Tables Tables Tables
  • 47. New Replication Features in 2014 • • ©Continuent 2014 Parallel extraction for fast provisioning • • • Select data as of specific SCN Pull data across in multiple streams Matches parallel apply on slave side ! Hadoop data loading • • • Direct loading from MySQL/Oracle to HDFS View data using Hive Generation of materialized views 47
  • 48. 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com Continuent Web Page: http://www.continuent.com ! Documentation: https://docs.continuent.com ! Tungsten Replicator: http://code.google.com/p/tungsten-replicator ©Continuent 2014 48