Countless petabytes of data are sitting in MySQL database where they are perfectly useless for PostgreSQL users. Fortunately there is a solution: Tungsten Replicator can move data from MySQL to PostgreSQL, and in real time, too. In this talk we'll describe how to design cross-database replication, then set it up using Tungsten Replicator. We will cover some of the pitfalls and corner cases like SQL dialect differences, data types, character sets, and MySQL bugs that make implementation both exciting and fun. We'll conclude with a demo of database updates moving in real time between databases.

1. Robert Hodges and Linas Virbalas
Continuent, Inc.
© Continuent 2010

2. / Introductions
/ What is Tungsten?
/ A Word About MySQL Replication
/ Tungsten’s MySQL to PostgreSQL Replication
/ Demo!
/ Questions and Answers
© Continuent 2010

3. © Continuent 2010

4. / Our Value: Make open source as robust as
commercial databases
• Enterprise capability
• Cloud flexibility
• Open source cost
/ Our Solution: Tungsten Database Clustering
/ Our Company:
• Founded in Finland, HQ in USA
• US/European presence
• Deep experience in replication, distributed systems, and RDBMS
© Continuent 2010

5. What is Tungsten and How
Does It Work?
© Continuent 2010

6. What Is Tungsten?
/ Tungsten creates data services using off-
the-shelf open source databases to:
1. Ensure reliable, fast access to data
2. Simplify common management tasks
3. Enable disaster recovery
…Without changing applications
© Continuent 2010

7. Apache/Mod_PHP Apache/Mod_PHP
Connector Connector
Replicator Replicator Replicator
Manager Manager Manager
© Continuent 2010

8. Technology: Replication Pipelines
© Continuent 2010

9. So Why Are We Here?
© Continuent 2010

10. Why Would You Want to Do That?
/ Feed MySQL web transactions to PostgreSQL
applications
/ Feed PG-based data warehouse and reporting
/ Migrate MySQL data to PG with minimum downtime
/ Scott McNealy says we should share (a PG-West joke)
© Continuent 2010

11. © Continuent 2010

12. Slave readable Slave
writeable
MySQL Statement Based + +
MySQL Row Based + +
MySQL Mixed + +
PostgreSQL WAL Shipping - -
PostgreSQL Streaming + -
Replication (Hot Standby)
/ Tungsten for MySQL
• You can both read and write to the DB operating in slave mode
• This, potentially, allows you to break consistency
• In some scenarios that is a plus
/ Tungsten for PostgreSQL
• In Streaming Replication you can read from slaves
• Still, you cannot write to them
• In effect, you cannot have different data/structure on a slave
© Continuent 2010

13. / Statement Based Replication
/ Row Based Replication
© Continuent 2010

14. Logical Physical
MySQL Statement Based x
MySQL Row Based x
MySQL Mixed x
PostgreSQL WAL Shipping x
PostgreSQL Streaming Replication x
Filters (data transformation) possible + -
Different data/structure on slave + -
possible
© Continuent 2010

15. © Continuent 2010

16. © Continuent 2010

17. Tungsten Tungsten
Replicator Replicator
© Continuent 2010

18. Master Slave
Replicator Replicator
Transaction Transaction
History Log History Log
Filters Filters
MySQL PostgreSQL
Extractor Applier
© Continuent 2010

19. / Provisioning
/ Data Type Differences
/ Database vs. Schema
/ Default (Implicitly Defined) Schema Selection
/ SQL Dialect Differences
• Statement Replication vs. Row Replication
/ Character Sets and Binary Data
/ Old Versions of MySQL
© Continuent 2010

20. Provisioning
/ Harder way: Dump data explicitly
/ Easier way: Replicate a mysqldump backup
Replicator
© Continuent 2010

21. / Note the type differences between MySQL and PG
MySQL PostgreSQL
! TINYINT SMALLINT
SMALLINT SMALLINT
INTEGER INTEGER
BIGINT BIGINT
! CHAR(1) CHAR(5) = {‘true’, ‘false’}
CHAR(x) CHAR(x)
VARCHAR(x) VARCHAR(x)
DATE DATE
TIMESTAMP TIMESTAMP
! TEXT (diff. sizes) TEXT
! BLOB BYTEA
…
© Continuent 2010

22. Database vs. Schema
/ In MySQL these are the same:
! !CREATE DATABASE foo!
! !CREATE SCHEMA foo!
/ In PostgreSQL these are very different:
CREATE DATABASE foo!
! !CREATE SCHEMA foo!
/ Tungsten uses filters to rectify MySQL databases to
PostgreSQL schemas
© Continuent 2010

23. / MySQL: Trivial to use `USE`
/ MySQL: Going without `USE` generates different
events
MySQL Implicit MySQL Explicit
CREATE SCHEMA s; CREATE SCHEMA s;
USE s;
! CREATE TABLE t (i int); CREATE TABLE s.t (i int);
! INSERT INTO t (1); INSERT INTO s.t (1);
/ PG: Extract the default schema from the event
/ PG: Set it before applying
MySQL PostgreSQL
USE s; > SET search_path TO s, "$user”;
© Continuent 2010

24. / Differences between DDL and DML statement SQL
dialects
/ Row Replication resolves issues rising from
differences in DML, but still leaves DDL to handle
/ Tungsten Replicator Filters come to the rescue!
• Simple to develop Java or JavaScript extensions
• Event structure IN -> Filter -> Event structure OUT
MySQL PostgreSQL
CREATE TABLE complex (id CREATE TABLE complex (id
INTEGER AUTO_INCREMENT SERIAL PRIMARY KEY, i INT);
PRIMARY KEY, i INT);
CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT);
…
© Continuent 2010

25. / Statement replication: MySQL syntax is “permissive”
/ Embedded binary / alternate charsets
/ Different charsets for different clients
/ Row replication: database/table/column charsets
may differ
/ Answer: Stick with one character set throughout; use
row replication to move binary data
MySQL PostgreSQL
INSERT INTO embedded_blob ARGH!!! (SQL statement fails)
(key, data) VALUES (1, ‘?0^Es
0^0’’)
create table xlate(id int, d1 ARGH!!! (no way to translate to
varchar(25) character set common charset)
latin1, d2 varchar(25)
character set utf8);
© Continuent 2010

26. MySQL Versions
/ Problem: Data stored on hard-to-replicate MySQL
versions or configurations
• Row replication not enabled (5.1)
• No row replication support (5.0, 4.1)
• Tungsten cannot read binlog (4.1)
/ Answer: MySQL blackhole replication
• (Blackhole = no store, just a binlog)
• Caveat: Check MySQL docs carefully
Replicator
© Continuent 2010

27. What’s Left?
/ We have covered the basics but there is more…
/ Data transformation (more Javascript filtering)
/ Making it go *really* fast (think parallel replication)
© Continuent 2010

28. © Continuent 2010

29. © Continuent 2010

30. HQ and Americas EMEA and APAC
560 S. Winchester Blvd., Suite 500 Lars Sonckin kaari 16
San Jose, CA 95128 02600 Espoo, Finland
Tel (866) 998-3642 Tel +358 50 517 9059
Fax (408) 668-1009 Fax +358 9 863 0060
e-mail: sales@continuent.com
Continuent Web Site:
http://www.continuent.com
© Continuent 2010