6. 技术发展路线图
Rapidly increasing heterogeneous platform and IMS
EMC
application support and developing additional high McKesson
availability solutions Convergys
Fidelity
Comverse
Netezza
IBM Info Srv. Infosphere
Ingres Ingres
Sybase Sybase
Cerner Cerner
Microsoft Microsoft Microsoft
Initial Support
t a Suppo t SQLServer SQLServer SQLServer
Application Eclipsys Eclipsys Eclipsys
IBM DB2 IBM DB2 IBM DB2
Platform
Oracle-Siebel Oracle-Siebel Oracle-Siebel
In progress
Amdocs Amdocs Amdocs
DST
DST DST DST
Teradata Teradata Teradata Teradata
MySQL MySQL MySQL MySQL MySQL
Oracle Oracle Oracle Oracle Oracle
GE Medical GE Medical GE Medical GE Medical GE Medical GE Medical
IDX IDX IDX IDX IDX IDX
ACI ACI ACI ACI ACI ACI ACI ACI
HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop HP Non-Stop
1995 1996 - 1998 1999 2000 - 2002 2003 2004 2005 2006 2007 2008- 2009
7. 议程/Agenda
历史 定位和发展
• GoldenGate历史,定位和发展
• GoldenGate基本架构,应用场景
Go de Gate基本架构,应用场景
• GoldenGate成功案例
• GoldenGate技术特点
• GoldenGate外争和内比
• GoldenGate实施经验分享
8. 整合后的产品和解决方案
GoldenGate Products Real-time Data Integration
• Oracle GoldenGate • Real-Time Data Warehousing
• Oracle GoldenGate for
Mainframe p g
• Live Reporting
• Oracle GoldenGate Veridata
• Management Pack for Oracle • Transactional Data Integration
GoldenGate
• Oracle GoldenGate Adapters High Availability & Disaster Tolerance
• Live Standby
• Zero-Downtime Operations
e o o t e Ope at o s
• Active-Active
10. 架构中的组件(进程)
Capture Deliver
• Log based • Native database API
• VAM based • ODBC
• Local/Remote Queuing • Batched Operations
• Filtering • Filtering
• Parallel Coordination • Transaction/Operations Grouping
• Local/Remote* • Parallel Coordination
• Local/Remote*
PUMP MANAGER
• Local/Remote
oca / e ote Queue (Trail) Management
( a ) a age e t
• Filtering Process Management (Dynamic)
• TCP/IP Monitoring and Administration (Lag reports)
• Compression Reporting (Events, Errors, Thresholds)
• Encryption
E ti Receive/Route requests f
R i /R t t from UI
On-disk Components
Trail Fil
T il Files Configuration Files
C fi ti Fil
Checkpoint Files Output Files
- Capture, Pump, Delivery Report/Log Files
Data Definition Files (SOURCEDEFS) Binaries
- Utility to generate definitions
11. 交易数据管理
Transactional Data Management
T ti lD t M t
TDM提供异构环境下交易数据的实时捕捉 变
GoldenGate TDM提供异构环境下交易数据的实时捕捉、变
换、投递,主要是支持异构数据库环境下的变化数据同步。
TDM 具有 : 特性:
实时性 高性能
秒一级延迟 能够以低资源消耗完成每秒数千交易的复制
异构环境支持 可扩展
在不同平台和数据库环境下复制数据 开放的结构使客户适应各种异构数据平台
以事务为单位复制 可靠性
维护交易一致性 保证数据的连续可用
12. GoldenGate TDM软件支持的环境
Databases O/S and Platforms
Capture: Windows 2000, 2003, XP
Oracle Linux
DB2 UDB
Sun Solaris
Microsoft SQL Server
Sybase ASE HP NonStop
Teradata HP-UX
Ingres
HP TRU64
Enscribe
SQL/MP IBM AIX
SQL/MX IBM z/OS
Delivery:
All listed above
MySQL, HP Neoview, Netezza
And ODBC compatible databases
ETL products
JMS message queues
MySQL
TimesTen
Ti T
14. GoldenGate: 关键的架构优势
• Distributed Loosely Coupled Process Based Architecture
• Heterogeneous Initial Load
• Asynchronous Change Propagation
– Committed Transactions Only
– DML, Metadata, DDL
• Universal Trail Format
– Interoperable across Heterogeneous Systems
• Data Transformation Rules and Filtering (table row column)
(table, row,
• Archival and Audit Capabilities
– Trail backups
– Audit – History of all changes (Insert all records)
15. GoldenGate: 关键的架构优势
• Conflict Detection and Resolution: for dual-active environments
• Parallel Coordination (Source, Target)
16. Oracle GoldenGate 典型应用场景
企业级的实时数据集成解决方案
• 一套产品满足多种业务需求
套产品满足多种业务需求
Disaster Recovery,
Data Protection Standby • 支持高可用性和实时数据集成
(Open & Active)
Zero Downtime
Migration and
Upgrades
Log Based, Real-
Time Change Data Operational
Capture Reporting
Reporting Database
OGG
ETL
ODS EDW
ETL • 高度灵活
Heterogeneous
Real-time BI EDW • 部署迅速
Source Systems
• 运维简单
Query Offloading
Data Distribution
25. 议程/Agenda
历史 定位和发展
• GoldenGate历史,定位和发展
• GoldenGate基本架构,应用场景
Go de Gate基本架构,应用场景
• GoldenGate成功案例
• GoldenGate技术特点
• GoldenGate外争和内比
• GoldenGate实施经验分享
26. 全球范围内成功案例
• Over 400 customers with 4,000+ implementations
across Fortune 500 companies: Financial Services,
Communications, Healthcare
Communications Healthcare, Public Sector Retail &
Sector,
Utilities industries
• Top 3 of 5 largest commercial banks
• Top 3 of 3 busiest ATM networks
• Top 7 of 10 financial data services companies
• Top 4 of 5 telecommunications providers
• Top 3 of 5 largest food & drug stores
36. Oracle GoldenGate Capture
GoldenGate
GoldenGate
Event Log
Parameter
Messaging Subsystem
File
Checkpoint Manager
Parser
Memory Manager
GoldenGate
Transaction Checkpoint
Logs File
VAM
API/Redo
Transaction API Transformation Serialization
Formatting
g Filtering
g
Log Reader and Mapping and Routing
(log reader
interface)
GoldenGate
Trail Files
M t d t
Metadata Metadata
M t d t SQL T f ti
Transformation
Manager Manager Interface Engine
Command
Database Access
and
Control
Capture GoldenGate
Source Command
Database Interpreter
37. Oracle GoldenGate Delivery
GoldenGate
GoldenGate
Event Log
Parameter
Messaging Subsystem
File
Checkpoint Manager
Parser
Memory Manager
GoldenGate
Checkpoint
File
GoldenGate
Transformation SQL
Trail Read API Formatting
g Filtering
g
and Mapping Generation
Reader
GoldenGate
Trail Files
Metadata
M t d t SQL T f ti
Transformation
Manager Interface Engine
Command
and
Control Database Access
GoldenGate Delivery
Command Target
Interpreter Database
38. GoldenGate Capture捕捉
• Log-based change data capture
– High volume
– Low overhead
• Decoupled architecture
– Multiple capture processes may be used to scale, but generally
not required
– Possible to split “hot” tables into a separate capture process
• Filtering and Compression
– Unnecessary change data if discarded immediately
– U d t and d l t are “
Updates d deletes “compressed” b d f lt
d” by default
– Before images are discarded by default
39. GoldenGate Delivery传送
• Decoupled architecture
– Multiple delivery processes may be used to scale
– Possible to split “hot” tables into a separate delivery process
– Possible for multiple delivery p
p y processes to split the work for a single
p g
table
• T
Transaction grouping
ti i
– Small transactions are grouped by default to reduce commit overhead
40. GoldenGate初始化
• Start change data capture on Source Database
– Current source Commit Sequence Number (CSN) is 222
Current CSN is
222 CSNs 222
through …
Commit Ordered Commit Ordered
Capture Source Trail Pump Target Trail
Source Target
Database Database
41. GoldenGate初始化
• Wait for any open transactions to close
• Use a backup (or export/import) to copy source to target as of a
consistency point
– Current source Commit Sequence Number (CSN) is 245
Consistent
Current CSN is as of CSN
245 CSNs 222 245
through …
Commit Ordered Commit Ordered
Capture Source Trail Pump Target Trail
Source Target
Database Database
Take Backup Apply
as of CSN Backup as of
245 CSN 245
42. GoldenGate初始化
• Start delivery at or after given backup CSN (245 in this case)
– Current source Commit Sequence Number (CSN) is 356
• Once Delivery is current the source and target are in sync
Consistent
Current CSN is Deliver as of CSN
356 CSNs 222 transactions 356
through 356 after CSN In Sync!
245
Commit Ordered Commit Ordered
Capture Source Trail Pump Target Trail Delivery
Source Target
Database Database
Take Backup Apply
as of CSN Backup as of
245 CSN 245
43. GoldenGate 检查点
• Capture, Pump, and Delivery save positions to a checkpoint file
so they can recover in case of failure
Start of Oldest Open (Uncommitted)
Begin,
Begin TX 1 Transaction
T ti
Insert, TX 1
Begin, TX 2 Begin, TX 2
Update, TX 1 Insert, TX 2
Insert, TX 2 Commit, TX 2
Commit, TX 2 Capture
Checkpoint Begin, TX 3
Begin, TX 3 Insert, TX 3
Insert, TX 3 Commit, TX 3
Current
Begin, TX 4 Write
Commit, TX 3 Position
Delete,
Delete TX 4
Current Read
Position
Commit Ordered
Capture Source Trail
Source
Database
44. GoldenGate 检查点
• Capture, Pump, and Delivery save positions to a checkpoint file so
they can recover in case of failure
Start of Oldest Open (Uncommitted)
Begin,
Begin TX 1 Transaction
T ti
Insert, TX 1
Begin, TX 2 Begin, TX 2
Pump
Begin, TX 2
Checkpoint
Update, TX 1 Insert, TX 2 Insert, TX 2
Insert, TX 2 Commit, TX 2 Commit, TX 2
Commit, TX 2 Capture Current Current
Checkpoint Begin, TX 3 Read Write
Begin, TX 3 Insert, TX 3 Position Position
Insert, TX 3 Commit, TX 3
Current
Begin, TX 4 Write
Commit, TX 3 Position
Delete,
Delete TX 4
Current Read
Position
Commit Ordered Commit Ordered
Capture Pump
Source Trail Target Trail
Source
Database
45. GoldenGate 检查点
• Capture, Pump, and Delivery save positions to a checkpoint file
so they can recover in case of failure
Start of Oldest Open (Uncommitted)
Begin,
Begin TX 1 Transaction
T ti
Insert, TX 1
Begin, TX 2 Pump Delivery
Begin, TX 2 Begin, TX 2
Checkpoint Checkpoint
Update, TX 1 Insert, TX 2 Insert, TX 2
Insert, TX 2 Commit, TX 2 Commit, TX 2
Commit, TX 2 Capture Current Current Current
Checkpoint Begin, TX 3 Read Write Read
Begin, TX 3 Insert, TX 3 Position Position Position
Insert, TX 3 Commit, TX 3
Current
Begin, TX 4 Write
Commit, TX 3 Position
Delete,
Delete TX 4
Current Read
Position
Commit Ordered Commit Ordered Delivery
Capture Pump
Source Trail Target Trail
Source Target
Database Database
47. 数据的选择和过滤
• GoldenGate provides the ability to select or filter out data based on a
variety of levels and conditions
Parameter Clause Selects
TABLE or MAP –— Table
WHERE Row
FILTER Row, Operation
Row Operation, Range
TABLE COLS | COLSEXCEPT Columns
48. 列映射
• GoldenGate provides the capability to map columns from one table to another
• Data can be transformed between dissimilar database tables
• Mapping can be applied either when extracting or replicating data
This example uses special values to build history of operations data
INSERTALLRECORDS
MAP SALES.ACCOUNT, TARGET REPORT.ACCTHISTORY,
COLMAP (USEDEFAULTS,
TRAN_TIME = @GETENV(“GGHEADER”,”COMMITTIMESTAMP”),
OP_TYPE = @GETENV(“GGHEADER”, “OPTYPE”),
_
BEFORE_AFTER_IND = @GETENV(“GGHEADER”,
“BEFOREAFTERINDICATOR”),
);
49. Functions函数功能
• GoldenGate provides the capability to transform columns by using a set of built-
in functions (currently 35)
• Transformation functions can be applied either for Extract or Replicat
• If you require more, you also have the ability to call your own logic through user
exits
MAP SALES.ACCOUNT, TARGET REPORT.ACCOUNT,
COLMAP ( USEDEFAULTS,
TRANSACTION_DATE = @DATE (“YYYY-MM-DD”,
“YY”, YEAR,
YY ,
“MM”, MONTH,
“DD”, DAY),
AREA_CODE = @STREXT (PHONE-NO, 1, 3),
PHONE_PREFIX = @STREXT (PHONE-NO, 4, 6),
PHONE_NUMBER = @STREXT (PHONE-NO, 7, 10) );
_
50. 议程/Agenda
历史 定位和发展
• GoldenGate历史,定位和发展
• GoldenGate基本架构,应用场景
Go de Gate基本架构,应用场景
• GoldenGate成功案例
• GoldenGate技术特点
• GoldenGate外争和内比
• GoldenGate实施经验分享
55. 根据解决方案来选择
数据移动和转换的Oracle产品
Golden Active Data
Gate Data Integrator
Guard EE
Disaster Recovery / Data Protection Non-
Cont
Oracle Oracle
tinuous Ava
Data Distribution / Synchronization
Zero Downtime Migrations / Upgrades
ailability
Query Offloading Hetero- Oracle
geneous to Oracle
Grid & Scalability (Oracle RAC)
Operational Reporting
Data Inte
Real Time BI
Typical Data Warehouse
yp
egration
Data Integration for SOA
Migrating Between Different Apps
56. 针对数据集成的Oracle产品
Comparing How They Work
Retrieve Data Load Data
Disaster Recovery & Data Protection
Redo in
Active Data
Memory Guard
Buffer Direct Direct
Memory Write to Logs
Access
增加转换能力
减少延迟
Real Time Data Integration & High Availability
g g y
GoldenGate
Read Fast SQL
异构
On-Disk Logs
g
Data Integration for Data Warehouse & SOA
Data
D t
Integrator Set-based,
SQL Query Complex SQL
56
57. GoldenGate and Streams
Oracle GoldenGate Oracle Streams
• GoldenGate is Oracle’s strategic replication solution
Oracle s
• Streams continues to be a supported Oracle Database feature
• Best Streams technology will be integrated into GoldenGate
gy g
58. GoldenGate & Streams
GoldenGate is Oracle’s strategic Streams continues to be a
replication solution
li ti l ti supported Oracle Database feature
t dO l D t b f t
Oracle GoldenGate Oracle Streams
• Out-of-the-box solutions • Highly customizable and flexible
• Broad heterogeneous support • Optimized for Oracle Database
• Excellent reference base • Few large references
Question: I already have Streams for free? Why buy GoldenGate?
• Better ROI overall: GoldenGate is lower cost to implement and maintain
• Enterprise-wide solution: GoldenGate easily expands to new use cases
• Support for Streams continues: Continue to use Streams where it’s deployed
Lead with GoldenGate
59. Comparing GoldenGate & Streams
• GoldenGate
– Solution for all heterogeneous real-time data access and delivery
– N i
Non-invasive, t
i transactionally consistent delivery of data across
ti ll i t t d li fd t
WAN
– Very low latency - extreme data volumes
– Solve wide range of HA challenges (active-active, multi-master, etc.)
– Strategic Data Replication solution for Oracle DB and
heterogeneous DBs
g
• Oracle Streams
– Built in feature of Oracle DB allowing info sharing between
Built-in
databases
– Can create data replication in homogeneous Oracle DB
environment
en ironment
– Existing customers will continue to be supported
– Continue and Converge: over time, best of Streams and GG
products will converge
60. GoldenGate and Data Guard
Heterogeneous
Distributed Oracle
Oracle Active Data
Fully-active
y
GoldenGate
G ld G t Guard
Subset
Replicas
Oracle DB Primary Oracle DB Standby
• Information Distribution & Consolidation, Application Upgrades
& Changes
– Use GoldenGate - heterogeneous, active-active, transformations, subsetting
g , , , g
• Disaster Recovery / Data Protection / HA
– Simple Full Oracle Database Protection
• Use Active Data Guard
• High-performance, simple, drop-in solution for HA and DR, readable at standby
• Works for all apps and data types, including packaged apps that can’t be
changed
– Application desiring flexible HA, active-active, schema changes, platform changes
• Use GoldenGate
• Avoid or manage active-active conflicts at application level
active active
61. Product Comparison
Active Data Guard and GoldenGate
Oracle Active Data Guard Oracle GoldenGate
Target Any distributed, heterogeneous
Oracle Data Guard users
Environment computing environment
Physical replica maintained by Logical replica maintained by
Technology shipping redo blocks replaying transactions
Bi-directional or uni-directional
Readable Data Guard standby
Capture once, apply many (N-way)
Benefits Highest transaction rates
Supports different schemas
Automatic repair of corrupt blocks
Supports heterogeneous platforms*
Oracle-only
Some data types unsupported
Restrictions Identical schemas
Manual switch-over / fail-over
“Similar” platforms
Similar
* Supported DBMS today: Oracle, MySQL, DB2, SQL Server, Sybase, Nonstop SQL, Enscribe, Teradata, Ingres, Greenplum, Netezza
* Supported DBMS today: Oracle MySQL DB2 SQL Server Sybase Nonstop SQL Enscribe Teradata Ingres Greenplum Netezza
Supported Platforms today: IBM AIX, IBM Z/OS, HP‐UX, HP NonStop , Solaris, Linux Windows
61
62. Complementary and Used Together
A ti D t Guard and G ld G t
Active Data G d d GoldenGate
Heterogeneous
Distributed Oracle Active
Subset Oracle Data Guard
Replicas GoldenGate
Oracle DB Primary O l DB St db
Oracle Standby
Disaster Recovery & Data Distribution /
Data Protection
D t P t ti Synchronization
S h i ti
• Fast failover to physical standby database • Enterprise-wide heterogeneous replication
• Fastest solution for Oracle • Zero downtime operations
• Oracle Database only • Non-invasive by reading database logs
• Allows standby to be open for reads to • Supports active-active scenarios
leverage customer investment • Ensures transactional consistency
• Transparent data protection to repair • High volume; low latency
corrupted blocks on primary or standby • Guaranteed delivery across wide area
networks
Oracle Active Data Guard Oracle GoldenGate
62
63. Oracle GoldenGate产品包
New Products Based on GoldenGate Technology
Oracle Products Description 收购之前的产品
Oracle GoldenGate v10.4 Includes GoldenGate Capture, Transactional Data
Deliver and Active Data Guard and Management (TDM)
XStream on Oracle DB and support Capture & Delivery,
for other open systems databases.
databases Rollback
Oracle GoldenGate for Includes GoldenGate Capture and TDM Capture&Delivery,
Mainframe v10 Deliver for HP Non-Stop and IBM Syncfile,
DB2 on Mainframe, plus Syncfile. Rollback
Oracle GoldenGate Veridata v3 Add-on capability to validate data in GoldenGate Veridata
replicated systems.
Management Pack f
M t P k for Add-on management pack (
Add t k (was GoldenGate Di
G ld G t Director
t
GoldenGate v2 “Director”). Restricted use
WebLogic SE.
Oracle GoldenGate Adapters (for Pre-packaged application content. Base 24 Add-on Modules,
Siebel, Cerner, Amdocs, Base24, Integration Edition for
JMS/Flat File, Logger-Enscribe) Java & Flat Files,
Logger
64. 议程/Agenda
历史 定位和发展
• GoldenGate历史,定位和发展
• GoldenGate基本架构,应用场景
Go de Gate基本架构,应用场景
• GoldenGate成功案例
• GoldenGate技术特点
• GoldenGate外争和内比
• GoldenGate实施经验分享