Best Practices for protecting Oracle RAC with NetBackup Issue Date: 29th July 2009 Version number:

1.0 Applies to: All

supported versions of NetBackup up to version 6.5.4 Note: This is a living document and will be subject to periodic updates.
Please check the data and version number to ensure you are referencing the latest version. If you have any feedback or
questions about this whitepaper please email them to [email protected] This document is provided for
informational purposes only and is intended for distribution only by Symantec employees to selected partners and customers.
All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent
allowed by law. The information in this document is subject to change without notice. Copyright 2009 Symantec Corporation.
All rights reserved. Symantec, the Symantec logo and NetBackup are trademarks or registered trademarks of Symantec
Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.
Contents 1.0 INTRODUCTION .................................................................................................................. 1 1.1 INTENDED
AUDIENCE ........................................................................................................... 1 1.2 SOLUTION
OVERVIEW ........................................................................................................... 1 1.2.1 Oracle Recovery Manager
(RMAN) ........................................................................... 1 1.2.2 NetBackup for
Oracle................................................................................................. 1 1.3
GLOSSARY ............................................................................................................................... 1 1.4 ADDITIONAL
READING................................................................................................................ 2 2.0 3.0 3.1 3.2 3.3 3.4 3.5 4.0 5.0
OPERATIONAL CHARACTERISTICS ................................................................................ 2 RAC BACKUP
OPTIONS .................................................................................................... 3 FAILOVER VIP EXISTS, BACKUP IS NOT LOAD
BALANCED......................................................... 3 FAILOVER VIP EXISTS, BACKUP IS LOAD
BALANCED ............................................................... 4 VIP DOES NOT FAILOVER, BACKUP IS NOT LOAD
BALANCED .................................................... 6 VIP DOES NOT FAILOVER, BACKUP IS LOAD BALANCED, 1 POLICY WITH
CUSTOM SCRIPT ........... 7 VIP DOES NOT FAILOVER, BACKUP IS LOAD BALANCED, SEPARATE AUTOMATIC
POLICIES .......... 8 CATALOG CONSIDERATIONS .......................................................................................... 9 STEPS TO
CONFIGURE LOAD BALANCED BACKUPS ................................................ 10 Best Practices for protecting Oracle RAC with
NetBackup Page i Issue date: 29th July 2009
1.0 Introduction Oracle database backup and recovery becomes more difficult as databases grow in size and as increasing
demands on database availability limit the time available to perform backups. Often the backup window is too short to
accommodate a complete backup process. Database administrators are looking for methods that will enable them to complete
these large backups in the time available. For the Oracle Real Application Clusters (RAC) database, NetBackup provides the
ability to split backups in pieces and send them in parallel across multiple nodes, therefore shortening the time needed to
complete a backup of the whole database. NetBackup software (NBU) is an enterprise wide backup and recovery solution.
Symantec Software has worked closely with Oracle Corporation to develop a highly scalable and reliable online backup and
recovery solution for Oracle databases. NetBackup for Oracle software protects both data and the availability of Oracle
applications. This paper will discuss methods that can be used to balance the backup load across multiple nodes of an Oracle
RAC database. This is an advanced technical paper. Familiarity with Oracle RAC, Oracle RMAN and NetBackup is expected
from the reader. 1.1 Intended Audience Backup administrators and database administrators can use this paper to determine the
best solution for protecting their Oracle RAC environments. 1.2 Solution overview Solutions described in this paper his paper
utilize Oracle Recovery Manager and NetBackup for Oracle. Oracle System Backup to Tape (SBT) API is used to enable direct
access to the functionality of NetBackup (NBU). 1.2.1 Oracle Recovery Manager (RMAN) Oracle databases can be protected
with the out-of-the-box Oracle Recovery Manager (RMAN) utility. RMAN provides the interface to the databases, and performs
backup and recovery functions. It controls data streams going in or out of a database for the purpose of backup and recovery.
RMAN by itself can save to or restore from disk only. RMAN can only connect to one target instance in a RAC database at a
time. However, this does not preclude load balancing a backup across all instances. This is because RMAN makes a distinction
between target connections and subsequent channel allocation connections. RMAN connections initiated via the allocate
channel command can be distributed among all instances in a RAC database. This is the key to the load balancing of data files
and archive logs between instance and channels. 1.2.2 NetBackup for Oracle NetBackup (NBU) provides a comprehensive data
protection solution, including centralized administration and reporting, media management, automated policy based backups,
and simplified restores. NetBackup for Oracle Agent (NBU Oracle) extends the capabilities of NetBackup to include online
backups and restores of Oracle databases. NetBackup for Oracle tightly integrates with RMAN and tailors NetBackup
capabilities to the uniqueness of an Oracle environment. Through the GUI or command line interface it provides a complete,
flexible database protection solution for a variety of platforms. NBU Oracle provides a media management to RMAN enabling
backup to media other than disk. 1.3 Glossary The following descriptive terms are used throughout this document: Backup piece
- The physical file format used to store RMAN backup sets. Best Practices for protecting Oracle RAC with NetBackup Page 1
Issue date: 29th July 2009
Backup set - A backup of one or more datafiles, control files, SPFILEs and archived redo log files. Each backup set consists of
one or more binary files called backup pieces. Backup pieces are written in a proprietary format that can only be created or
restored by RMAN. Instance - A system global area (SGA) and the Oracle background processes constitute an Oracle database
instance. Every time a database is started, a system global area is allocated and Oracle background processes are started. The
SGA is de-allocated when the instance shuts down. Real Application Clusters (RAC) - Option that allows multiple concurrent
instances to share a single physical database. Recovery Manager (RMAN) - A utility that backs up, restores, and recovers
Oracle databases. It can be used with or without the central information repository called a recovery catalog. If you do not use a
recovery catalog, RMAN uses the database's control file to store information necessary for backup and recovery operations.
You can use RMAN in conjunction with a media manager to back up files to tertiary storage. SBT - System Backup to Tape.
This term specifies a nondisk backup device type, typically a tape library or tape drive. RMAN supports channels of type disk
and SBT. TNSPING Oracle utility that determines whether or not a service (for example, an Oracle database, an Oracle
Names server, or any other Oracle service) on a Oracle Net network can be successfully reached. VIP Virtual IP address is an
IP address that is not connected to a specific computer or network interface card (NIC) on a computer. VIP address will still be
available if a computer or NIC fails because an alternative computer or NIC replies to connections. 1.4 Additional reading
NetBackup 5.1 for Oracle System Administrator's Guide (http://support.veritas.com/docs/268099). NetBackup 6.0 for Oracle
System Administrator's Guide (http://support.veritas.com/docs/279281). NetBackup 6.5 for Oracle System Administrator's Guide
for Windows (http://support.veritas.com/docs/290215). NetBackup 6.5 for Oracle System Administrator's Guide for UNIX and
Linux (http://support.veritas.com/docs/290216). 2.0 Operational Characteristics This section lists background information for
NetBackup for ORACLE general operation. Initiating RMAN: The NBU Oracle policy can contain one or more client
names and one or more backup scripts to execute. The NBU master is going to use the Automatic schedules in the Oracle
policy to determine when the scripts should be executed on the client(s). The NBU scheduler will start one Automatic job for
each client in the policy, the jobs can run concurrently. The NBU scheduler will execute each script, in sequence specified, on
each client. The all the scripts for one client will run in the same Automatic job. The script(s) start RMAN which gathers the data,
sends a user-directed backup request to NBU, and then transfers the data to NBU for storage. If an Automatic schedule and
script does not exist in the policy, a process on the client can still initiate RMAN when necessary. RMAN Requests the Backup:
RMAN will connect to the appropriate Oracle instance(s) for the backup. Issue date: 29th July 2009 Best Practices for
protecting Oracle RAC with NetBackup Page 2
 Hence the script may execute on one host, but the backup may take place on a different host. RMAN will
allocate one or more channels per the backup script. RMAN will send one or more backupset pieces on each channel, in
sequence. Each channel will send a user-directed backup request to the NBU master for each backupset piece. Each request
will become a separate NBU Application Backup job. Hence there can be one NBU Application Backup job queued or active,
concurrently, per allocated channel. This will be in addition to an active job if an Automatic schedule initiated the script
execution. For the Application Backup jobs, by default the NBU master will select the first policy it finds of type Oracle for this
client and use the first schedule of type Application Backup within the policy. The NBU master must be able to match the
requesting client name to an Oracle policy and Application Backup schedule or the job will fail. RMAN can SEND the variables
NB_ORA_CLIENT and/or NB_ORA_POLICY and/or NB_ORA_SCHED to the NBU master to override the defaults otherwise
selected for the job. Receiving the Data from RMAN: Once the Application Backup job(s) go active, they will connect back
to the provided client host to receive the data. Hence the client name that is sent in the user-directed request must bring the
data connection back to the requesting host. RMAN will then send the appropriate data on the appropriate channel and it will be
transferred to storage. 3.0 RAC backup options The following sections describe five potential RAC configurations and presents
various options for the backup of each one. 3.1 Failover VIP exists, backup is not load balanced In this configuration, a failover
VIP name exists such that the NBU media server can always reach a host which is up to execute the backup script. Further,
since load balancing is disabled, RMAN will allocate the channels on a single host, typically the same host where the script
executes. The configuration is as follows: The policy should specify the failover VIP name as the client name so that the
Automatic schedule executes the backup script on a host that is currently operational. The backup script, or an identical copy,
must be accessible to all hosts in the cluster. The clustered file system makes a good location. The backup script should be
configured so that RMAN sends to the NBU master (regardless of active host) the same client name to be used to both store the
images and for the connect-back from the NBU media server to the client for the data transfer. The VIP name from the policy is
ideal since that name is already in the policy, will float to the active instance/host and ensure successful data transfer, and all
the backups will be stored under that single client name. ALLOCATE CHANNEL ... ; SEND 'NB_ORA_CLIENT='; BACKUP ... ;
The NBU master should be configured to know about the relationship between the physical and virtual names. This is required
for backups, such as this one, where the user-directed request originates from a hostname that does not match the virtual name
Issue date: 29th July 2009 Best Practices for protecting Oracle RAC with NetBackup Page 3
used for NB_ORA_CLIENT. This configuration also makes it possible to perform alternate client/instance restores from either
node in the cluster. cd /usr/openv/netbackup/db/altnames echo "hostname1" >> hostname1 echo "vip_name1" >> hostname1
echo "hostname2" >> hostname1 echo "vip_name2" >> hostname1 echo "vip_name" >> hostname1 cp hostname1 hostname2
If REQUIRED_INTERFACE or another means is being utilized on the client hosts to force NBU to use the IP addresses
associated with the VIP names for the outbound userdirected backup requests, then the NBU master configuration must be
extended in the reverse direction. cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2
The net result is that the backup script will run on the active host currently hosting the VIP name, RMAN will allocate the
channels on that host to perform the backup, the Application Backup jobs will queue to the VIP name, the NBU media server will
connect back to the VIP name for the data transfer. The backup images will be stored under the VIP name regardless of which
host performed the backup. Restores can take place from either host as long as the restore request is configured to SEND
'NB_ORA_CLIENT=vip_name'; 3.2 Failover VIP exists, backup is load balanced In this configuration, a failover VIP name exists
such that the NBU master can always reach an active host to execute the backup script. However, RMAN will allocate the
channels on both hosts and the NBU media server must connect-back to the correct host to obtain the data for each request.
Because of this, the backup images will be stored under two different client names which also differ from the VIP name used to
execute the backup script. The policy should specify the failover VIP name as the client name so that the Automatic
schedule executes the backup script on a host that is currently operational. The backup script, or an identical copy, must be
accessible to both hosts in the cluster. The clustered file system makes a good location. The backup script must NOT be
configured to send a single value for NB_ORA_CLIENT because the NBU media server needs to connect back to the correct
host depending on which channel/host originated the user-directed backup request. By default, the backup will use the
CLIENT_NAME from the bp.conf file which should be distinct for each host and is typically the physical hostname. If desired, the
persistent parameters in the Oracle configuration can be utilized to bind specific channels to specific hosts. CONFIGURE
CONFIGURE CONFIGURE CONFIGURE CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 4; CHANNEL 1 DEVICE
TYPE 'SBT_TAPE' CONNECT CHANNEL 2 DEVICE TYPE 'SBT_TAPE' CONNECT CHANNEL 3 DEVICE TYPE 'SBT_TAPE'
CONNECT CHANNEL 4 DEVICE TYPE 'SBT_TAPE' CONNECT 'sys/passwd@tns1'; 'sys/passwd@tns2'; 'sys/passwd@tns1';
'sys/passwd@tns2'; Alternatively, RMAN can be configured to bind specific channels to specific instances and send specific
client names on each channel for backup image storage and for connect-back to the requesting host for the data transfer. The
failover VIP name should not be used since it will be active on only one of the hosts. ALLOCATE CHANNEL 1 ...
PARMS='ENV=(NB_ORA_CLIENT=hostname1)' CONNECT='sys/passwd@tns1'; ALLOCATE CHANNEL 2 ...
PARMS='ENV=(NB_ORA_CLIENT=hostname2)' CONNECT='sys/passwd@tns2'; Best Practices for protecting Oracle RAC with
NetBackup Page 4 Issue date: 29th July 2009
ALLOCATE CHANNEL 3 ... PARMS='ENV=(NB_ORA_CLIENT=hostname1)' CONNECT='sys/passwd@tns1'; ALLOCATE
CHANNEL 4 ... PARMS='ENV=(NB_ORA_CLIENT=hostname2)' CONNECT='sys/passwd@tns2'; Because the CLIENT_NAME
or NB_ORA_CLIENT differs from the VIP name in the policy, the NBU master will not accept the user-directed backup request
unless one of the following two configurations is implemented: Configuration A: 1. Coordinate the policy and backup script to
handle multiple client names. 2. Add both host names to the policy along with the failover VIP name. 3. Modify the script so that
it exits with status 0 if the client name is not the VIP name. Configuration B: 1. Create a second policy to receive the backup
requests from RMAN. 2. The policy needs to be of type Oracle. 3. The policy must contain the client/host names used by each
host in the backup requests from RMAN. 4. The Application Backup schedule must have an open window to accept the
backups. 5. The policy does not need either a backup script or an Automatic schedule. 6. The backup script executed by the
first policy should be configured to send the name of the second policy with each user-directed backup request; ALLOCATE
CHANNEL ... PARMS='ENV=(NB_ORA_POLICY=)'; or SEND 'NB_ORA_POLICY='; The NBU master should be configured to
know about the relationship between the physical and virtual names to facilitate browsing and restoring images from the other
host. cd /usr/openv/netbackup/db/altnames echo "vip_name" >> hostname1 echo "hostname1" >> hostname1 echo
"vip_name1" >> hostname1 echo "hostname2" >> hostname1 echo "vip_name2" >> hostname1 cp hostname1 hostname2If
REQUIRED_INTERFACE or another means is being utilized on the client hosts to force NBU to use the IP addresses
associated with the VIP names for the outbound user-directed backup requests, then the NBU master configuration must be
extended in the reverse direction. cd /usr/openv/netbackup/db/altnames cp hostname1 vip_name1 cp hostname1 vip_name2
For Configuration A, the net result is that the NBU scheduler will start three automatic jobs, and each will execute the backup
script (two of them on the same host). The two jobs started using the host names will exit immediately with status 0 and not
retry. The third job will execute the backup script and RMAN will send the data for backup using the appropriate client name for
the host originating each channel. The backup images will be stored under the initiating policy using both host names. For
Configuration B, the net result is that the first policy executes the backup script using the failover VIP name. RMAN sends the
local hostname with the user-directed request from each host. The second policy stores the backup images using both host
names. Either client can initiate a restore. RMAN can be configured with 'SET AUTOLOCATE ON;' to request the backupset
pieces from the appropriate instance/host that performed the backup. Alternatively, restores can take place from either
host/instance as long as the restore request is configured to include the same client name used at the time the backupset piece
was transferred to storage. SEND 'NB_ORA_CLIENT=backup_hostnameX_or_vip_nameX' Best Practices for protecting Oracle
RAC with NetBackup Page 5 Issue date: 29th July 2009
3.3 VIP does not failover, backup is not load balanced In this configuration, hostname1 and VIP name1 both allow connections
to one host in the cluster and hostname2 and VIP name2 both allow connections to the other host in the cluster. Special
configuration will need to ensure that the backup script executes on at least one of the hosts but not on both hosts. Otherwise
either a backup will not occur (1 of 1 specified instances is down) or a redundant backup will occur (2 of 2 specified instances
are active). For ease of discussion, the term primary refers the instance on which the backup would normally occur and
secondary refers to the other instance which could be used for the backup of the primary is down. In addition, because the
backup may occur on either host, the backup images have the potential to be stored under both client names depending on the
host which is active at the time of the backup. The configuration is as follows: The policy should specify both hosts, either
hostname1 & hostname2 or VIP name1 & VIP name2 to ensure that the backup is attempted on a host which is currently
operational. The backup script must be accessible to both hosts in the cluster, the clustered file system makes a good location.
The backup script should be customized so that it will perform the backup on only one of the two instance if they are both active.
If executed on the primary, then start RMAN and perform the backup. If executed on the secondary and the primary is up, then
exit with status 0 so that the NBU scheduler does not retry this client. If executed on the secondary and the primary is down,
then start RMAN and perform the backup. The script customization could be built around a ping to the primary or even a query
of the database to see if the other instance is open and able to perform the backup, e.g. $ select
INST_ID,STATUS,STARTUP_TIME, HOST_NAME from gv$instance; INST_ID STATUS STARTUP_T HOST_NAM ---------- ----
-------- --------- --------1 OPEN 13-JAN-06 CAMS1 2 OPEN 13-JAN-06 CAMS2 The user-directed backup request must
utilize a client name which allows the NBU media server to connect back to the correct host for the data transfer. By default, the
backup will use the CLIENT_NAME from the bp.conf file which will be distinct for each host. Alternatively, RMAN can be
configured to send the appropriate hostname or VIP name from the policy. If executing on hostname1: SEND
'NB_ORA_CLIENT=> peername1 echo "vip2" >> peername1 echo "peername1" >> peername1 echo "peername2" >>
peername1 cp peername1 peername2 Please note that static routes or the use of REQUIRED_INTERFACE on the client hosts
may cause the traffic from the client hosts to the NBU servers to originate from an unexpected IP address. The hostnames to
which those IPs resolve are the peernames to use. 9) Image storage, catalog maintenance, and restore operations The above
configuration will have RAC backup images stored under multiple client names. This is not of concern if Oracle is configured to
autolocate when restores or catalog reconciliation occurs between NetBackup and Oracle. SET AUTOLOCATE ON; However,
this can present challenges when performing alternate client restores or restores utilizing a different number of instances. In that
case, it is often advantageous to have all of the RAC backup images stored under a single client name. For adhoc operations
this can be accomplished by creating a new netbackup/db/image directory using a temporary virtual name to house a temporary
copy of the backup image files which are then copied from the original client directories. A more flexible permanent solution is to
create a linked image directory structure ahead of time so that the standard backups are distinct from the Oracle backups and
all the RAC images are aggregated into a single directory but accessible from any instance. cd /usr/openv/netbackup/db/images
# Create the image directories for the standard backups of both hosts. mkdir client1 client2 # Create an image directory to
house all the RAC images. mkdir rac # Create links to redirect the vips to the RAC directory. ln -s rac vip1 ln -s rac vip2 The
same structure and functionality is available in NTFS, since Windows 2000, via junctions.
http://www.microsoft.com/technet/sysinternals/FileAndDisk/Junction.mspx Once the above image db structure exists, catalog
maintenance and alternate client restores can occur using NB_ORA_CLIENT=rac. However, the requesting peernames must be
allowed access to the new client name. Best Practices for protecting Oracle RAC with NetBackup Page 13 Issue date: 29th July
2009
cd /usr/openv/netbackup/db/altnames echo "rac" >> peername1 echo "rac" >> peername2 echo "rac" >>
peername_for_alternate_client Best Practices for protecting Oracle RAC with NetBackup Page 14 Issue date: 29th July 2009
COMMENTS