OracleGoldenGate PDF
OracleGoldenGate PDF
OracleGoldenGate PDF
Real-Time Replication
Fred Louis
Oracle Enterprise Architect
Ohio Valley Region
Agenda
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Key Differentiators:
Performance
Extensible &
Flexible
Reliable
Standardize on Single
Disaster
Recovery, Data
Protection
Standby
(Open & Active)
Zero Downtime
Migration and
Upgrades
Log Based, RealTime Change Data
Capture
Operational
Reporting
OGG
Reporting
Database
ETL
ODS
ETL
Heterogeneous
Source Systems
Real-time BI
Query
Offloading
EDW
EDW
Highly Flexible
Fast Deployments
Lower TCO & Improved ROI
Data
Distribution
Agenda
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Oracle GoldenGate
Simple, Flexible Instantiation
Transaction Logs
Capture
Trail
File
LAN / WAN /
Internet
(TCP/IP)
Pump
Collector
Online Backups
GG Initial Load
3rd party Tools
Trail
File
Deliver
Capture
Source
Database
Source Trail
LAN / WAN /
Internet
Target Trail
Deliver
Target
Database
Capture
Source
Database
Source Trail
LAN / WAN /
Internet
Target Trail
Deliver
Target
Database
10
Capture
Source
Database
Source Trail
LAN / WAN /
Internet
Target Trail
Target Trail
Deliver
Source Trail
Deliver
Capture
Bi-directional
Target
Database
11
Capture:
Oracle
DB2
Microsoft SQL Server
Sybase ASE
Teradata
Enscribe
SQL/MP
SQL/MX
Delivery:
All listed above, plus:
HP NonStop
HP-UX
HP TRU64
HP OpenVMS
IBM AIX
IBM z/OS
12
Broadcast
Data Distribution
Bi-Directional
Live Standby or
Active-Active for HA
Integration/Consolidation
Data Warehouse
Peer-to-Peer
Load Balancing,
Multi-Master
Cascading
Data Marts
13
Agenda
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
14
CAPTURE
Log-based
VAM-based
Local/Remote Queuing
Filtering
Parallel Coordination
Local/Remote*
PUMP
Filtering
Local/Remote
TCP/IP
Compression
Encryption
MANAGER
Queue (Trail) Management
Process Management (Dynamic)
Monitoring and Administration (Lag reports)
Reporting (Events, Errors, Thresholds)
Receive/Route requests from UI
On-disk Components
Trail Files
Checkpoint Files (Capture, Pump, Delivery)
Data Definition Files (SOURCEDEFS/TARGETDEFS)
Configuration Files
Discard Files
Report/Log Files
Binaries
15
GGSCI
GGSCI
Manager
Interactive
Interface
Manager
Interactive
Interface
Network
Source
Database
Capture
Source Trail
Pump
Source
Collector
Target Trail
Delivery
Target
Database
Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
16
GGSCI
GGSCI
Manager
Interactive
Interface
Manager
Interactive
Interface
Network
Source
Database
Capture
Source Trail
Pump
Source
Collector
Target Trail
Delivery
Target
Database
Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
17
GGSCI
GGSCI
Manager
Interactive
Interface
Manager
Interactive
Interface
Network
Source
Database
Capture
Source Trail
Pump
Source
Collector
Target Trail
Delivery
Target
Database
Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
18
GGSCI
GGSCI
Manager
Interactive
Interface
Manager
Interactive
Interface
Network
Source
Database
Capture
Source Trail
Pump
Source
Collector
Target Trail
Delivery
Target
Database
Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
19
GGSCI
GGSCI
Manager
Interactive
Interface
Manager
Interactive
Interface
Network
Source
Database
Capture
Source Trail
Pump
Source
Collector
Target Trail
Delivery
Target
Database
Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
20
GoldenGate
Parameter
File
Messaging Subsystem
Checkpoint Manager
Parser
Memory Manager
GoldenGate
Checkpoint
File
Transaction
Logs
Transaction
Log Reader
VAM
API/Redo
API
Formatting
Filtering
(log reader
interface)
Metadata
Manager
Metadata
Manager
Database Access
Source
Database
Capture
Transformation
and Mapping
SQL
Interface
Serialization
and Routing
Transformation
Engine
GoldenGate
Trail Files
Command
and
Control
GoldenGate
Command
Interpreter
21
GoldenGate
Parameter
File
Messaging Subsystem
Checkpoint Manager
Parser
Memory Manager
GoldenGate
Trail
Reader
Read API
Formatting
GoldenGate
Checkpoint
File
Filtering
Transformation
and Mapping
SQL
Generation
GoldenGate
Trail Files
Metadata
Manager
Command
and
Control
GoldenGate
Command
Interpreter
SQL
Interface
Transformation
Engine
Database Access
Delivery
Target
Database
22
Agenda
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
23
Decoupled architecture
Multiple capture processes may be used
to scale, but generally not required
Possible to split hot tables into a
separate capture process
Decoupled architecture
Multiple delivery processes may be used to scale
Possible to split hot tables into a separate delivery process
Possible for multiple delivery processes to split the work for a
single table
Transaction grouping
Small transactions are grouped by default to reduce commit
overhead
Record batching
Records are batched by table and operation within a single
SQL execution
Automatic reordering of batches
25
Agenda
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
26
Transaction boundaries
Recorded in GoldenGate trail file
Transactions ordered in commit sequence
Boundaries adjusted automatically due to
record filtering or trail splits
Transaction integrity
Checkpointing and recovery are based on
transaction boundaries
Original commit sequence is maintained
by the delivery process
27
Differentiator: Reliability
Decoupled architecture
Individual processes can be restarted automatically
Tolerance to network outages (configurable)
Recovery
Recovery ensures that no operations are skipped or
duplicated after failure of any kind
Recovery of the Capture process is more involved than
recovery the Delivery process
Although GoldenGate processes are completely decoupled,
the Delivery process must be aware of a recovery performed
by the Capture process or pump
28
GoldenGate Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Begin, TX 1
Insert, TX 1
Begin, TX 2
Begin, TX 2
Update, TX 1
Insert, TX 2
Insert, TX 2
Commit, TX 2
Commit, TX 2
Capture
Checkpoint
Begin, TX 3
Insert, TX 3
Insert, TX 3
Begin, TX 4
Commit, TX 3
Begin, TX 3
Current
Write
Position
Commit, TX 3
Delete, TX 4
Current Read
Position
Source
Database
Capture
Commit Ordered
Source Trail
29
GoldenGate Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Begin, TX 1
Insert, TX 1
Begin, TX 2
Begin, TX 2
Update, TX 1
Insert, TX 2
Insert, TX 2
Commit, TX 2
Insert, TX 3
Insert, TX 3
Commit, TX 3
Begin, TX 3
Current
Write
Position
Begin, TX 2
Insert, TX 2
Commit, TX 2
Capture
Checkpoint
Begin, TX 3
Begin, TX 4
Pump
Checkpoint
Commit, TX 2
Current
Write
Position
Current
Read
Position
Commit, TX 3
Delete, TX 4
Current Read
Position
Source
Database
Capture
Commit Ordered
Source Trail
Pump
Commit Ordered
Target Trail
30
GoldenGate Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Begin, TX 1
Insert, TX 1
Begin, TX 2
Begin, TX 2
Update, TX 1
Insert, TX 2
Insert, TX 2
Commit, TX 2
Insert, TX 3
Insert, TX 3
Commit, TX 3
Begin, TX 3
Current
Write
Position
Delivery
Checkpoint
Begin, TX 2
Insert, TX 2
Commit, TX 2
Capture
Checkpoint
Begin, TX 3
Begin, TX 4
Pump
Checkpoint
Commit, TX 2
Current
Write
Position
Current
Read
Position
Current
Read
Position
Commit, TX 3
Delete, TX 4
Current Read
Position
Source
Database
Capture
Commit Ordered
Source Trail
Pump
Commit Ordered
Target Trail
Delivery
Target
Database
31
Agenda
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
32
Differentiator: Heterogeneity
Core Infrastructure
Core product components are utilized for all databases
Canonical trail file format
Canonical metadata definitions
Automatic mapping of common datatypes
Capture Methods
Log scraping (direct file access)
Database built-in log interface (transaction log API)
Intercepts/exits
Vendor-provided access modules
33
Differentiator: Heterogeneity
Oracle Capture
Capture
Archived Log
Online Log
Capture
OCI/Oracle Net
Commit Ordered
Trail
34
Differentiator: Heterogeneity
Microsoft SQL Server Capture
Capture
Backups
Transaction Log
Capture
Commit Ordered
Trail
ODBC
35
Differentiator: Heterogeneity
DB2 LUW Capture
Capture
Archived Log
Online Log
DB2READLOG
Capture
Commit Ordered
Trail
DB2 CLI
36
Differentiator: Heterogeneity
DB2 z/OS Capture
Capture
Archived Log
Online Log
Capture
Commit Ordered
Trail
DB2 CLI
Unix System Services (USS)
37
Differentiator: Heterogeneity
HP NonStop Capture (Enscribe and SQL/MP)
Capture
TMF Audit Dump
ARLIB2
AUDSERV
Intercept
Capture
Embedded SQL
Logger
Commit Ordered
Trail
38
Differentiator: Heterogeneity
HP NonStop Capture (SQL/MX)
Capture
TMF Audit Dump
ARLIB2
VAMSERV
Capture
Commit Ordered
Trail
Embedded SQL
Open Systems Services (OSS)
39
Differentiator: Heterogeneity
Teradata Capture
Replication Group
RSG
RSG
vproc
vproc
CDC
CDC
Capture
T
A
M
Capture
Unsorted Trail
Sort/Pump
Commit Ordered
Trail
ODBC
40
Differentiator: Heterogeneity
Delivery
Native APIs
Transaction grouping
Transaction splitting (if necessary)
Array operations
Integration with load utilities
Delivery
Native Interface
Commit Ordered
Trail
Delivery
Target Database
41
Agenda
42
GoldenGate Instantiation
Start change data capture on Source Database
Current source Commit Sequence Number (CSN) is 222
Current CSN is
222
Source
Database
CSNs 222
through
Capture
Commit Ordered
Source Trail
Pump
Commit Ordered
Target Trail
Target
Database
43
GoldenGate Instantiation
Wait for any open transactions to close
Use a backup (or export/import) to copy source to target as of a consistency point
Current source Commit Sequence Number (CSN) is 245
Consistent as
of CSN 245
Current CSN is
245
Source
Database
CSNs 222
through
Capture
Take Backup
as of CSN
245
Commit Ordered
Source Trail
Pump
Commit Ordered
Target Trail
Target
Database
Apply Backup
as of CSN
245
44
GoldenGate Instantiation
Start delivery at or after given backup CSN (245 in this case)
Current source Commit Sequence Number (CSN) is 356
Current CSN is
356
Source
Database
CSNs 222
through 356
Capture
Take Backup
as of CSN
245
Commit Ordered
Source Trail
Pump
Commit Ordered
Target Trail
Consistent as
Deliver
of CSN 356
transactions
In Sync!
after CSN 245
Delivery
Target
Database
Apply Backup
as of CSN
245
45
Agenda
46
Standalone execution
Add parameter to the root level of a parameter file
Executes independently of a table or map statement
Execution order is based on the order it appears in the parameter file
Execute once
SQLEXEC call prc_job_count ()
SQLEXEC truncate table scott.target_table
SQLEXEC SET TRIGGERS OFF
47
Inline execution
Add a clause to the table/map statement in the parameter file
Executes for records that meet the map condition
The procedure or query can accept input parameters and return
output parameters
Executed within the target database transaction context
Execute once (always execute by default)
MAP sales.cust, TARGET sales.cust_extended, &
SQLEXEC (SPNAME lookup, PARAMS &
(long_name = birth_state),EXEC ONCE), &
COLMAP (custid = custid, birth_state_long = lookup.long_name);
48
Agenda
49
Reports
Event
Log
Discard
Checkpoint
File
Reports
Source
Database
Source Trail
Discard
Checkpoint
File
EVENT
PROCESSING
EVENT
PROCESSING
Capture
Event
Log
Pump
Target Trail
Delivery
Target
Database
50
51
Agenda
52
Integration via:
Staging tables
Flat files
Messaging
ETL
Integration via Staging Tables
Delivery
Capture
Trail File
Delivery
data
data
data
files
files
files
ETL
OLTP
Integration via Messaging
Delivery
ETL
JMS / MoM
Queue or
Topic
53
Writes as:
Transactions (with guaranteed integrity)
Individual DB Operations (insert, delete)
Trail
XML
Capture
XML
XML
tx
tx
logs
tx
logs
logs
source
defs
(meta
data)
Data
Pump
JMS
UE
JMS
XML
Queue or Topic
Third Party
Systems
Partners
Customers
Applications
etc
54
Time/size-based rollover
One file, or one table
per file
55
Sun/Oracle Information
http://tinyurl.com/sunorcl
or
oracle.com
Oracle GoldenGate
Other Oracle Products..
Oracle GoldenGate
High
Availabili
ty
Oracle Streams
58
Oracle GoldenGate
Other Oracle Products..
Oracle Data
Integrator EE
RealTime
Data
Integratio
n
Oracle GoldenGate
59
GoldenGate Information
http://oracle.com/goldengate
Customer Case
Studies
COMPANY OVERVIEW
DIRECTV is a $17 billion provider of
satellite-based television services.
DIRECTVs 7,500 employees operate the
companys broadcast centers, monitor
satellites, and deliver service to about 17
million U.S. and over 5 million Latin
American customers.
CHALLENGES / OPPORTUNITIES
Maintain high quality customer
service in competitive market
reduce churn!
Centralize customer information for a
single view to support sales,
marketing, support & field service
Significantly reduce data latency in
central data warehouse for all
queries & reports edict for < 15
minutes!
RESULTS
Significantly reduced churn by 25%
All business units have access to realtime business data.
COMPANY OVERVIEW
Overstock.com is an online closeout
retailer offering high-quality, brand-name
merchandise, including bed-and-bath
goods, home dcor, kitchenware,
watches, jewelry, electronics and
computers, sporting goods, apparel and
designer accessories among other
products at discount prices
CHALLENGES / OPPORTUNITIES
Better understand customer
purchasing behavior
Provide timely information to support
marketing, merchandising and
operational decisions
Improve shopping database
reliability and uptime
RESULTS
Customer analysis now done in minutes,
rather than days
Dashboard reports using real-time data
for P&L & customer profitability
Google Adwords adjusted based on realtime data reports
Ranked #4 in customer service by
National Retail Federation
COMPANY OVERVIEW
A world leader in the travel marketplace,
Sabre Holdings merchandises and retails
travel products and provides distribution
and technology solutions for the travel
industry
CHALLENGES / OPPORTUNITIES
Optimize OLTP system performance
offload all query activity
Reduce TCO via platform changes and
segment lookers from bookers
Handle growing data volumes and
support heterogeneous systems over
life-cycle of Air Travel Shopping
Engine (ATSE)
Maintain data integrity across all
systems
RESULTS
80% TCO Reduction Millions $$ saved
Bookers vs Lookers
COMPANY OVERVIEW
Research In Motion (RIM) is a leading
designer, manufacturer and marketer of
innovative wireless solutions for the
worldwide mobile communications
market.
CHALLENGES / OPPORTUNITIES
Experiencing exponential growth and
stringent carrier SLAs to meet growing
demand for Blackberry devices
Maintain continuous availably of critical
data across all global data centers (4)
for manufacturing plants
All plants see same data and can move
devices based on regional demand
COMPANY OVERVIEW
Retail Decisions (ReD) is a payment card
issuer and a world leader in card fraud
prevention and payment processing. A
specialist supplier to the payments
industry worldwide. Its blue-chip
international clients come from the
global telecommunications, retail, travel,
petroleum, banking and the broader ecommerce sectors
CHALLENGES / OPPORTUNITIES
Typical Service Level Agreements
dictate 99.95% availability & aggressive
sub-second average response times
Must ensure quick, massive scalability
High cost of downtime -- ReDs clients
lose millions of dollars per hour
Global clients; data centers on 4
continents
Oracle PROVIDES:
Fraud Detection (ReDShield) using Oracle 9i
and 10g databases
U.S. Payment Processing system, using SQL
Server databases also supports data
access for Web apps and Reporting
RESULTS
Lightning Fast implementation
Time to recover: ZERO minutes
Reduced database license &
infrastructure costs
66
COMPANY OVERVIEW
HSN is an interactive lifestyle network
and retail destination, offering a curated
assortment of exclusive products
combined with top brand names. HSN
incorporates experts, entertainment,
inspiration, solutions, tips and ideas to
provide an entirely unique shopping
experience for its customers.
Oracle PROVIDES:
Continuous availability during major
migration project.
Active-Active configuration provided data
synchronized across old & new systems
Provided upgrade & downgrade logic
between old and new Siebel versions
CHALLENGES / OPPORTUNITIES
Eliminate downtime during major Siebel
upgrade from version 6.2 to 8.0
Additionally, upgrade Oracle database 8i
to 10g with no downtime
Conduct a phased migration approach
to allow time for training & testing
Minimal business disruption
RESULTS
Enabled migration of 1500+ users to new
system
Reduced risk with live parallel running of
both old and new
Zero downtime
No disruption to the business &
continuous order processing on mission
critical system
67