D50311GC11 sg2

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
Oracle Database
@ sic de11g: nt RAC
e r de Stu
Administration
v i lav e this
ic i o_ Volume
o usII • Student Guide
r
ab nse t
( f
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
D50311GC11
Edition 1.1
September 2008
D56242
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2009, Oracle and/or its affiliatesฺ
Author Copyright © 2008, Oracle. All rights reserved.
James Womack Disclaimer
This document contains proprietary information and is protected by copyright and

Technical Contributors other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered in
and Reviewers any way. Except where your use constitutes "fair use" under copyright law, you may
Christopher Andrews not use, share, download, upload, copy, print, display, perform, reproduce, publish,
license, post, transmit, or distribute this document in whole or in part without the
Troy Anthony express authorization of Oracle.
Lothar Auert
The information contained in this document is subject to change without notice. If you
Bruce Carter find any problems in the document, please report them in writing to: Oracle University,
500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
Michael Cebulla warranted to be error-free.
Carol Colrain
Restricted Rights Notice
Jonathan Creighton
Joel Goodman If this documentation is delivered to the United States Government or anyone using
the documentation on behalf of the United States Government, the following notice is
Arturo Gutierrez applicable:
s a
Lutz Hartmann
)h a
ฺbr
U.S. GOVERNMENT RIGHTS
Pete Jones The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or
m
o deฺ
disclose these training materials are restricted by the terms of the applicable Oracle
c
David Kirby
i ฺ
d Gui
license agreement and/or the applicable U.S. Government contract.
Roland Knapp r e
Miroslav Lorenc
Trademark Notice
@ sic dent
Barb Lundhild
e r de Stu
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other
lav e this
names may be trademarks of their respective owners.
Roderick Manalac
v i
Sabiha Miri
ic i o_ o us
Philip Newlan
( f a br se t
Roman Niehoff
r d e licen
Erik Peterson
V e ble
Stefan Pommerenk
V a
il fera
r i c i rans
MarshalloPresser
FabSrinivas
Rick Pulliam
o n -t
n Putrevu
Roy Rossebo
Ira Singer
Linda Smalley
Ranbir Singh
Harald van Breederode
Michael Zoll
Editors
Raj Kumar
Richard Wallis
Nita Pavitran
Amitha Narayan
Graphic Designer
Satish Bettegowda
Publishers
Nita Brozowski
Unauthorized
Veenareproduction
Narasimhan or distribution prohibitedฺ Copyright© 2009, Oracle and/or its affiliatesฺ
Contents
I Introduction
Overview I-2
Course Objectives I-3
Typical Schedule I-4
A History of Innovation I-5
What Is a Cluster? I-6
Oracle Real Application Clusters I-7
s a
Benefits of Using RAC I-8
)h a
Clusters and Scalability I-9
m ฺbr
Levels of Scalability I-10 i ฺ c o deฺ
Scaleup and Speedup I-11 r e d Gui
Speedup/Scaleup and Workloads I-12 @ sic dent
I/O Throughput Balanced: Example I-13
e r de Stu
i lav e this
Performance of Typical Components I-14
v
i o_ o us
Complete Integrated Clusterware I-15
ic
( f abr nse t
Necessity of Global Resources I-16
rde le lice
Global Resources Coordination I-17
e
l a V ab
Global Cache Coordination: Example I-18
i
V sfer
Write to Disk Coordination: Example I-19
i o
ic -tran
Dynamic Reconfiguration I-20
b r
Fa non Object Affinity and Dynamic Remastering I-21
Global Dynamic Performance Views I-22
Additional Memory Requirement for RAC I-23
Efficient Internode Row-Level Locking I-24
Parallel Execution with RAC I-25
RAC Software Principles I-26
RAC Software Storage Principles I-27
RAC Database Storage Principles I-28
RAC and Shared Storage Technologies I-29
Oracle Cluster File System I-31
Automatic Storage Management I-32
CFS or Raw? I-33
Typical Cluster Stack with RAC I-34
RAC Certification Matrix I-35
iii
RAC and Services I-36
Available Demonstrations I-37
1 Oracle Clusterware Installation and Configuration

Objectives 1-2
Oracle RAC 11g Installation 1-3
Oracle RAC 11g Installation: Outline 1-5
Windows and UNIX Installation Differences 1-6
Preinstallation Tasks 1-7
Hardware Requirements 1-8
Network Requirements 1-9
Virtual IP Addresses and RAC 1-10 a
RAC Network Software Requirements 1-11 as
Package Requirements 1-12 ฺbr )h
Required UNIX Groups and Users 1-13 c m
o deฺ
i ฺ
d Gui
oracle User Environment 1-14 r e
User Shell Limits 1-15
@ sic dent
Configuring for Remote Installation 1-16
e r de Stu
v i lav e this
Required Directories for the Oracle Database Software 1-18
ic i o_ o us
Linux Operating System Parameters 1-20
Cluster Setup Tasks 1-22
( f abr nse t
Verifying Cluster Setup with cluvfy 1-23
e rde le lice
Installing Oracle Clusterware 1-24
l a V ab
i
V sfer
Specifying the Inventory Directory 1-25
i o
ic -tran
Specify Home Details 1-26
b r
Fa nonProduct-Specific Prerequisite Checks 1-27
Cluster Configuration 1-28
Private Interconnect Enforcement 1-29
Oracle Cluster Registry File 1-30
Voting Disk File 1-31
Summary and Install 1-32
Run Configuration Scripts on All Nodes 1-33
End of Installation 1-34
Verifying the Oracle Clusterware Installation 1-35
Summary 1-37
Practice 1: Overview 1-38
2 RAC Software Installation

Objectives 2-2
Installing Automatic Storage Management 2-3
Installation Type 2-4
iv
Install Location 2-5
Hardware Cluster Installation Mode 2-6
Product-Specific Prerequisite Checks 2-7
Select Configuration Option 2-8
Privileged Operating System Groups 2-9
Summary 2-10
Execute Configuration Scripts 2-11
End of Installation 2-12
ASM Configuration 2-13
Creating ASM Disk Groups 2-17
Select a Product to Install 2-19
Select Installation Type 2-20
s a
Install Location 2-21
)h a
Specify Cluster Installation 2-22
m ฺbr
Products Prerequisite Check 2-23
i ฺ c o deฺ
Select Configuration Option 2-24
r e d Gui
Privileged Operating System Groups 2-25
@ sic dent
Check Summary 2-26
e r de Stu
root.sh Script 2-27
v i lav e this
ic i o_ o us
Required Tasks Prior to Database Creation 2-28
Summary 2-31 ( f abr nse t

Checks Before Database Creation 2-29
e rde le lice
l a V ab
i
V Database f er Creation
i
3 o RAC
ic Objectives s
an 3-2
b r - t r
Fa non Management Agent Installation: Specify Installation Type 3-3
Specify Installation Location 3-4
Specify Cluster Installation Mode 3-5
Prerequisite Check and OMS Location 3-6
Agent Registration Password 3-7
Management Agent Installation Finish 3-8
Executing the root.sh Script 3-9
Creating the Cluster Database 3-10
Node Selection 3-11
Select Database Type 3-12
Database Identification 3-13
Cluster Database Management Method 3-14
Passwords for Database Schema Owners 3-15
Storage Options for Database Files 3-16
ASM Disk Groups 3-18
v
Database File Locations 3-19
Recovery Configuration 3-20
Database Content 3-21
Initialization Parameters 3-22
Security Settings and Maintenance Tasks 3-23
Database Storage Options 3-24
Create the Database 3-25
Monitor Progress 3-26
Postinstallation Tasks 3-27
Check Managed Targets 3-28
Single Instance to RAC Conversion 3-29
Single-Instance Conversion Using the DBCA 3-30
s a
Conversion Steps 3-31
)h a
Single-Instance Conversion Using rconfig 3-34
m ฺbr
Single-Instance Conversion Using Grid Control 3-36
i ฺ c o deฺ
Summary 3-38 r e d Gui
@ sic dent
e r de Stu
4 RAC Database Administration
v i lav e this
Objectives 4-2
ic i o_ o us
Cluster Database Home Page b4-3
a r et
Cluster Database Instance e ( f
Home Page ns 4-5
d li c e
r lAdministration
Cluster Database e Instance e Page 4-6
a V a b
Cluster Home
V il Pagefer4-7
The
i o a s
ciConfiguration
n Section 4-8
b r t r
- Viewer 4-10
Fa Topology
non
Enterprise Manager Alerts and RAC 4-11
Enterprise Manager Metrics and RAC 4-12
Enterprise Manager Alert History and RAC 4-14
Enterprise Manager Blackouts and RAC 4-15
Redo Log Files and RAC 4-16
Automatic Undo Management and RAC 4-17
Starting and Stopping RAC Instances 4-18
Starting and Stopping RAC Instances with SQL*Plus 4-19
Starting and Stopping RAC Instances with SRVCTL 4-20
Switch Between the Automatic and Manual Policies 4-21
RAC Initialization Parameter Files 4-22
SPFILE Parameter Values and RAC 4-23
EM and SPFILE Parameter Values 4-24
RAC Initialization Parameters 4-26
Parameters That Require Identical Settings 4-28
vi
Parameters That Require Unique Settings 4-29
Quiescing RAC Databases 4-30
Terminating Sessions on a Specific Instance 4-31
How SQL*Plus Commands Affect Instances 4-32
Transparent Data Encryption and Wallets in RAC 4-33
ASM: General Architecture 4-34
ASM Instance and Crash Recovery in RAC 4-36
ASM Instance Initialization Parameters and RAC 4-37
ASM and SRVCTL with RAC 4-38
ASM and SRVCTL with RAC: Examples 4-39
ASM Disk Groups with EM in RAC 4-40
Disk Group Performance Page and RAC 4-41 a
Summary 4-42 as
)h
ฺbr
c m
o deฺ
i ฺ
d Gui
5 Managing Backup and Recovery in RAC r e
Objectives 5-2
@ sic dent
Protecting Against Media Failure 5-3
e r de Stu
Archived Log File Configurations 5-4
v i lav e this
RAC and the Flash RecoveryiArea
ic o_ 5-5 o us
r t
(fab Settings
RAC Backup and Recovery Using EM 5-6
Configure RAC e Recovery e n se with EM 5-7
e
Archived Redo e ic
rdFilelConventions
l in RAC 5-8
a V b
V il RAC
Configure
f e raBackup Settings with EM 5-9
o
ici Oracle a s
nRecovery Manager 5-10
b r - t r
Fa nonConfigure RMAN Snapshot Control File Location 5-11
Configure Control File and SPFILE Autobackup 5-12
Crosschecking on Multiple RAC Clusters Nodes 5-13
Channel Connections to Cluster Instances 5-14
RMAN Channel Support for the Grid 5-15
RMAN Default Autolocation 5-16
Distribution of Backups 5-17
One Local Drive CFS Backup Scheme 5-18
Multiple Drives CFS Backup Scheme 5-19
Non-CFS Backup Scheme 5-20
Restoring and Recovering 5-21
Summary 5-22
vii
6 RAC Performance Tuning
Objectives 6-2
CPU and Wait Time Tuning Dimensions 6-3
RAC-Specific Tuning 6-4
RAC and Instance or Crash Recovery 6-5
Instance Recovery and Database Availability 6-7
Instance Recovery and RAC 6-8
Analyzing Cache Fusion Impact in RAC 6-10
Typical Latencies for RAC Operations 6-11
Wait Events for RAC 6-12
Wait Event Views 6-13
Global Cache Wait Events: Overview 6-14 a
2-way Block Request: Example 6-16 as
3-way Block Request: Example 6-17 ฺbr )h
2-way Grant: Example 6-18 c m
o deฺ
i ฺ
d Gui
Global Enqueue Waits: Overview 6-19 r e
Session and System Statistics 6-20
@ sic dent
Most Common RAC Tuning Tips 6-21
e r de Stu
v i lav e this
Index Block Contention: Considerations 6-23
ic i o_ o us
Oracle Sequences and Index Contention 6-24
( f abr nse t
Undo Block Considerations 6-25
rde le lice
High-Water Mark Considerations 6-26
e
Concurrent Cross-Instance Calls: Considerations 6-27
V ab
i l a
V sfer
Monitoring RAC Database and Cluster Performance 6-28
i o
ic -tran
Cluster Database Performance Page 6-29
b r
Fa nonDetermining Cluster Host Load Average 6-30
Determining Global Cache Block Access Latency 6-31
Determining Average Active Sessions 6-32
Determining Database Throughput 6-33
Accessing the Cluster Cache Coherency Page 6-35
Viewing Cluster Interconnects Page 6-37
Viewing the Database Locks Page 6-39
AWR Snapshots in RAC 6-40
AWR Reports and RAC: Overview 6-41
Automatic Database Diagnostic Monitor for RAC 6-43
What Does ADDM Diagnose for RAC? 6-45
EM Support for ADDM for RAC 6-46
Summary 6-47
viii
7 Services
Objectives 7-2
Traditional Workload Dispatching 7-3
Grid Workload Dispatching 7-4
Data Warehouse: Example 7-5
RAC and Data Warehouse: An Optimal Solution 7-6
Next Step 7-7
What Is a Service? 7-8
High Availability of Services in RAC 7-9
Possible Service Configuration with RAC 7-10
Service Attributes 7-11
Service Types 7-12 a
Service Goodness 7-13 as
Create Services with Enterprise Manager 7-14 ฺbr )h
Create Services with SRVCTL 7-15 c m
o deฺ
i ฺ
d Gui
Preferred and Available Instances 7-16 r e
sic dent
Modify Services with the DBMS_SERVICE Package 7-17
@
Everything Switches to Services 7-18
e r de Stu
v i lav e this
Use Services with Client Applications 7-19
ic i o_ o us
Use Services with the Resource Manager 7-20
( f abr nse t
Services and Resource Manager with EM 7-21
Services and the Resource Manager: Example 7-22
e rde le lice
Use Services with the Scheduler 7-23
l a V ab
i
V sfer
Services and the Scheduler with EM 7-24
i o
ic -tran
Services and the Scheduler: Example 7-26
b r
Fa non Use Services with Parallel Operations 7-27
Use Services with Metric Thresholds 7-28
Change Service Thresholds by Using EM 7-29
Services and Metric Thresholds: Example 7-30
Service Aggregation and Tracing 7-31
Top Services Performance Page 7-32
Service Aggregation Configuration 7-33
Service Aggregation: Example 7-34
trcsess Utility 7-35
Service Performance Views 7-36
Generalized Trace Enabling 7-37
Manage Services 7-38
Manage Services with Enterprise Manager 7-40
Manage Services with EM 7-41
Manage Services: Example 7-42
Manage Services: Scenario 7-43
ix
Using Distributed Transactions with RAC 7-44
Restricted Session and Services 7-46
Summary 7-47
8 High Availability of Connections

Objectives 8-2
Types of Workload Distribution 8-3
Client-Side Connect-Time Load Balancing 8-4
Client-Side Connect-Time Failover 8-5
Server-Side Connect-Time Load Balancing 8-6
Fast Application Notification: Overview 8-7 a
Fast Application Notification: Benefits 8-8 as
FAN-Supported Event Types 8-9 ฺbr )h
FAN Event Status 8-10 c m
o deฺ
i ฺ
d Gui
FAN Event Reasons 8-11 r e
FAN Event Format 8-12
@ sic dent
Load Balancing Advisory: FAN Event 8-13
e r de Stu
v i lav e this
Server-Side Callouts Implementation 8-14
ic i o_ o us
Server-Side Callout Parse: Example 8-15
( f abr nse t
Server-Side Callout Filter: Example 8-16
Configuring the Server-Side ONS 8-17
e rde le lice
Optionally Configure the Client-Side ONS 8-18
l a V ab
i
V sfer
JDBC Fast Connection Failover: Overview 8-19
i o
ic -tran
Using Oracle Streams Advanced Queuing for FAN 8-20
b r
Fa nonJDBC/ODP.NET FCF Benefits 8-21
Load Balancing Advisory 8-22
JDBC/ODP.NET Runtime Connection Load Balancing: Overview 8-23
Connection Load Balancing in RAC 8-24
Load Balancing Advisory: Summary 8-25
Monitor LBA FAN Events 8-26
FAN Release Map 8-27
Transparent Application Failover: Overview 8-28
TAF Basic Configuration Without FAN: Example 8-29
TAF Basic Configuration with FAN: Example 8-30
TAF Preconnect Configuration: Example 8-31
TAF Verification 8-32
FAN Connection Pools and TAF Considerations 8-33
Summary 8-34
x
9 Oracle Clusterware Administration
Objectives 9-2
Oracle Clusterware: Overview 9-3
Oracle Clusterware Run-Time View 9-4
Manually Control Oracle Clusterware Stack 9-6
CRS Resources 9-7
RAC Resources 9-8
Resource Attributes: Example 9-9
Main Voting Disk Function 9-11
Important CSS Parameters 9-13
Multiplexing Voting Disks 9-14
Change Voting Disk Configuration 9-15 a
Back Up and Recover Your Voting Disks 9-16 as
OCR Architecture 9-17 ฺbr )h
OCR Contents and Organization 9-19 c m
o deฺ
i ฺ
d Gui
Managing OCR Files and Locations: Overview 9-20
r e
Automatic OCR Backups 9-21
@ sic dent
Back Up OCR Manually 9-22
e r de Stu
v i lav e this
Recover OCR Using Physical Backups 9-23
ic i o_ o us
Recover OCR Using Logical Backups 9-24
( f abr nse t
Replace an OCR Mirror: Example 9-25
Repair OCR Configuration: Example 9-26
e rde le lice
OCR Considerations 9-27
l a V ab
i
V sfer
Change VIP Addresses 9-28
i o
ic -tran
Change Public/Interconnect IP Subnet Configuration: Example 9-30
b r
Fa non Third-Party Application Protection: Overview 9-31
Application VIP and RAC VIP Differences 9-32
Use CRS Framework: Overview 9-33
Use CRS Framework: Example 9-35
Summary 9-38
10 Diagnosing Oracle Clusterware and RAC Components

Objectives 10-2
One Golden Rule in RAC Debugging 10-3
Oracle Clusterware Main Log Files 10-4
Diagnostics Collection Script 10-5
Managing Diagnostic Data in RAC 10-6
Cluster Verify: Overview 10-7
Cluster Verify Stages 10-8
Cluster Verify Components 10-9
xi
Cluster Verify Locations 10-10
Cluster Verify Configuration File 10-11
Cluster Verify: Examples 10-13
Cluster Verify Output: Example 10-15
Summary 10-16
11 Node Addition and Removal

Objectives 11-2
Add and Delete Nodes and Instances: Overview 11-3
Main Steps to Add a Node to a RAC Cluster 11-4
Check Prerequisites Before Oracle Clusterware Installation 11-5 a
Add Oracle Clusterware to the New Node 11-6 as
Configure the New ONS 11-9 ฺbr )h
Add ASM Home to the New Node 11-10 c m
o deฺ
i ฺ
d Gui
Add RAC Home to the New Node 11-11 r e
Add a Listener to the New Node 11-12
@ sic dent
Add an ASM Instance to the New Node 11-13
e r de Stu
v i lav e this
Add a Database Instance to the New Node 11-15
ic i o_ o us
Add an Instance to Your RAC Database Using EM 11-18
( f abr nse t
Main Steps to Delete a Node from a RAC Cluster 11-21
Delete the Instance on the Node to Be Deleted 11-22
e rde le lice
Clean Up the ASM Instance 11-24
l a V ab
i
V sfer
Remove the Listener from the Node to Be Deleted 11-25
i o
ic -tran
Remove the Node from the Database 11-26
b r
Fa non
Remove the Node from ASM 11-27
Remove the Node from the Oracle Clusterware 11-28
Node Addition and Deletion and the SYSAUX Tablespace 11-30
Summary 11-31
12 Design for High Availability

Objectives 12-2
Causes of Unplanned Down Time 12-3
Causes of Planned Down Time 12-4
Oracle’s Solution to Down Time 12-5
RAC and Data Guard Complementarity 12-6
Maximum Availability Architecture 12-7
RAC and Data Guard Topologies 12-8
RAC and Data Guard Architecture 12-9
Data Guard Broker (DGB) and Oracle Clusterware (OC) Integration 12-11
xii
Fast-Start Failover: Overview 12-12
Data Guard Broker Configuration Files 12-14
Real-Time Query Physical Standby Database 12-15
Hardware Assisted Resilient Data 12-16
Oracle Clusterware Rolling Upgrade 12-17
Clustered ASM Rolling Upgrade 12-18
Patches and the RAC Environment 12-20
Inventory List Locks 12-21
OPatch Support for RAC: Overview 12-22
Rolling Patch Upgrade Using RAC 12-23
Download and Install Patch Updates 12-24
Rolling Release Upgrade Using SQL Apply 12-26
s a
Database High Availability: Best Practices 12-27
)h a
How Many ASM Disk Groups per Database 12-28
m ฺbr
Database Storage Consolidation 12-29
i ฺ c o deฺ
e d Gui
Which RAID Configuration for Best Availability? 12-30
r
Should You Use RAID 1 or RAID 5? 12-31
@ sic dent
e r de Stu
Should You Use ASM Mirroring Protection? 12-32
v i lav e this
What Type of Striping Works Best? 12-33
ASM Striping Only 12-34
ic i o_ o us
( f abr nse t
Hardware RAID–Striped LUNs 12-35
Hardware RAID–Striped LUNs HA 12-36
e rde le lice
It Is Real Simple 12-37
l a V ab
i
V sfer
Extended RAC: Overview 12-38
i o
ic -tran
Extended RAC Connectivity 12-39
b r
Fa non Extended RAC Disk Mirroring 12-40
Achieving Quorum with Extended RAC 12-41
ASM Preferred Mirror Read: Overview 12-42
ASM Preferred Mirror Read: Setup 12-43
Enterprise Manager ASM Configuration Page 12-44
ASM Preferred Mirror Read: Best Practice 12-45
Additional Data Guard Benefits 12-46
Using a Test Environment 12-47
Summary 12-48
Appendix A: Practices and Solutions
Appendix B: Miscellaneous Topics
Appendix C: Practice Environment Scripts
xiii
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Node Addition and Removal
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l i ce© 2008, Oracle. All rights reserved.
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Objectives
After completing this lesson, you should be able to:

• Add a new node to your cluster database
• Remove a node from your cluster database
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Oracle Database 11g: RAC Administration 11 - 2
Add and Delete Nodes and Instances: Overview
Three main methods:

• Silent cloning procedures
• Enterprise Manager Grid Control cloning and adding
instance
• addNode.sh/rootdeletenode.sh and DBCA
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
VNodessfander Instances: Overview
i
Add and Deleteo anhow to add and delete nodes and instances in Oracle Real Application Clusters
b ric describes
Thisalesson - t r
F non
(RAC) databases. There are mainly three methods you can use to add and delete nodes in a RAC
environment:
• Silent cloning procedures. Cloning enables you to copy images of Oracle Clusterware and RAC
software onto the other nodes that have identical hardware and software.
• Enterprise Manager Grid Control. This is basically a GUI interface to cloning procedures.
• Interactive or silent procedures using addNode.sh/rootdeletenode.sh and the Database
Configuration Assistant (DBCA)
The preferred method to add multiple nodes and instances to RAC databases is to use the cloning
procedures. This is especially relevant when you are massively deploying software across your
enterprise. Refer to the Oracle Real Application Clusters Administration and Deployment Guide as
well as the Oracle Clusterware Administration and Deployment Guide for more information about
cloning procedures.
However, in this lesson you are going to see how you can directly use Oracle Universal Installer
(OUI) and DBCA to add one node to and delete one node from your cluster.
Main Steps to Add a Node to a RAC Cluster
1. Install and configure OS and hardware for new node.

2. Add Oracle Clusterware to the new node.
3. Configure ONS for the new node.
[4. Add ASM home to the new node.]
5. Add RAC home to the new node.
6. Add a listener to the new node.
7. Add a database instance to the new node. s a
)h a
m ฺbr
i ฺ c o deฺ
ORACLE_HOME ORACLE_HOME
r e d Gui
CRS HOME CRS HOME
@ sic dent
e r de Stu
v i lav e this
ic i o_Storage
o us
r
ab nse t
( f
e rde lCopyright
l a V ab e
i er to a RAC Cluster
Main Stepsio to VAdd a s f
Node
b riclists the
Theaslide - t r an steps you need to follow to add a new node to your RAC cluster. Basically,
main
F non
you are going to use OUI to copy the Oracle Clusterware software as well as the RAC software to the
new node. For each main step, you have to do some manual configurations.
Note: For all the add node and delete node procedures for UNIX-based systems, temporary
directories such as /tmp, $TEMP, or $TMP should not be shared directories. If your temporary
directories are shared, set your temporary environment variable, such as $TEMP, to a nonshared
location on a local node. In addition, use a directory that exists on all the nodes.
Check Prerequisites Before
Oracle Clusterware Installation
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sBeforef er Oracle Clusterware Installation
i o
Check Prerequisites
an with the Oracle Clusterware installation on the node you want to add to your
ic can-proceed
b r t r
Fa non
Before you
RAC cluster, you must make sure that all operating system and hardware prerequisites are met.
Because installation and configuration of your operating system is not the scope of this lesson, refer
to the first lessons of this course for more information.
After this is done, you can verify that the system has been configured properly for Oracle
Clusterware by using the following Cluster Verify command from one of the nodes that is already
part of your cluster: \
cluvfy stage -pre crsinst -n <list of all nodes> -r 11gR1
The example shown in the slide assumes that you have only one node currently as part of your
cluster, and you want to add a new one called VX0313. If any errors are reported during the
preceding verification, fix them before proceeding to the next step.
Add Oracle Clusterware to the New Node
Execute <Oracle Clusterware home>/oui/bin/addNode.sh.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfetor the New Node
Add Oracleio Clusterware
b
Logain rasicthe oracle
- t r an user and execute the addNode.sh script located in your Oracle Clusterware
F non
home directory on the first node. This script runs the Oracle Universal Installer.
On the Welcome screen, click Next.
On the Specify Cluster Nodes to Add to Installation screen, OUI recognizes the existing nodes and
asks you to enter the “short” public node name of the host you want to add to your cluster. That
should automatically populate the corresponding Private Node Name and “Virtual host name” fields.
Make sure that those three names are correct and click Next.
You next see the Cluster Node Addition Summary screen, where you can review the list of products
to be installed. Click Install.
Note: It is also possible to execute OUI in silent mode. Here is a possible example where you want to
add a new node called newnode:
addNode.sh -silent -responseFile myinstallresponsefile
Here, myinstallresponsefile contains the following:
CLUSTER_NEW_NODES = {"newnode"}
CLUSTER_NEW_PRIVATE_NODE_NAMES = {"newnode-priv"} CLUSTER_NEW_VIRTUAL_HOSTNAMES =
{"newnode-vip"}
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfetor the New Node (continued)
Add Oracleio
Clusterware
b c tran
rinow
Youacan - the installation progression from the Cluster Node Addition Progress screen.
follow
F non
The OUI copies the Oracle Clusterware software to the new node, and then asks you to run few
scripts as the root user on both nodes. Make sure that you run the scripts on the correct node as
specified one after another.
You have to execute the rootaddnode.sh script on the first node. Basically, this script adds the
nodeapps of the new node to the OCR configuration.
After this is done, you have to execute the root.sh script from the new node. This script starts the
Oracle Clusterware stack on the new node and then uses VIPCA (Virtual IP Configuration Assistant)
in silent mode for configuring nodeapps.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfetor the New Node (continued)
Add Oracleio Clusterware
b r icscripts
- t r anexecuted successfully, you can check your Oracle Cluster Registry (OCR)
Fa non
After both are
configuration as shown in the slide. At this point, the crs_stat command reports three new
resources on the new node. These resources correspond to nodeapps.
Click OK on the Execute Configuration scripts screen to reach the end of the Oracle Clusterware
installation.
On the End of Installation screen, click Exit.
Configure the New ONS
Use the racgons add_config command to add new node

ONS configuration information to OCR.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
V i fer
i
Configure theo New n s
ONS
b ricneed to-tradd
Youanow
a the new node ONS (Oracle Notification Server) configuration information to
F no n
the shared ONS configuration information stored in OCR. From the first node, and looking at the
ons.config file located in the <Oracle Clusterware home>/opmn/conf directory, you
can determine the ONS remote port to be used (6251 in the slide). You need to use this port in the
racgons add_config command as shown in the slide to make sure that the ONS on the first
node can communicate with the ONS on the new node.
Add ASM Home to the New Node
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V to thesfNewer Node
Add ASM Home i o n
b ricis needed
Thisastep - t r aonly if you use a specific home directory to host ASM. If you run ASM and your
F non
RAC database out of the same Oracle Home, you can skip this step.
From the first node, you need to execute the addNode.sh script from the ASM home directory as
shown in the slide. The scenario is identical to the one shown for the Oracle Clusterware installation.
However, in the case of an Oracle Home, you just need to select the name of the node you want to
add on the Specify Cluster Nodes to Add to Installation screen, and then run the root.sh script
from the new node after OUI has copied the database software.
Add RAC Home to the New Node
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V to thesfNewer Node
Add RAC Home i o
icfirst node, n
ayou
Fromb r
the - t r need to execute the addNode.sh script from the RAC home directory as
Fa non
shown in the slide. The scenario is identical to the one shown for the ASM home installation.
Add a Listener to the New Node
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
Vto thesNewf er Node
i
Add a Listenero
icnew node, n
ayou
b r - t r
Fa non
From the need to add a listener. In this example, you are adding a listener from the
ASM Home. You need to use NETCA (NETwork Configuration Assistant) for that.
On the Configuration screen, select “Cluster configuration” and click Next.
On the Active Nodes screen, select the name of the new node and click Next.
On the Welcome screen, select “Listener configuration” and click Next.
On the Listener screen, select Add and click Next.
One the Listener Name screen, enter LISTENER in the “Listener name” field.
On the Select Protocols screen, select TCP and click Next.
On the TCP/IP Protocol screen, select “Use the standard port number of 1521,” and click Next.
Continue to click Next until you exit from NETCA.
The steps above add a listener on the new node with the name LISTENER_<New node name>.
Add an ASM Instance to the New Node
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sto f r
ethe
Add an ASM
i c i oInstance
a n New Node
r t r
Fab younocann-add your database instance to the new node, you need to add an ASM instance to the
Before
new node. To do so, use DBCA from your ASM home as shown on the above slide.
On the Welcome screen, click Next.
On the Operations screen, select Configure Automatic Storage Management and click Next.
On the Node Selection screen, select the node you want to add, and click Next.
Note: By default if your ASM instance is running out of the same oracle home, then DBCA
automatically extends ASM to the new node when you use DBCA to extend your database instance
to the new node.
Add an ASM Instance to the New Node
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sto f r
ethe
Add an ASM
i c i oInstance
a n New Node (continued)
r r
-t prompts you for ASM instance addition on your second node. Click Yes.
Faba while,
After
n o nDBCA
DBCA then asks you to enter the password for the ASM administrator. Enter your password and
click OK.
On the ASM Disk Groups screen, you should see all your disk groups mounted on all nodes of your
cluster.
Click Finish.
Add a Database Instance to the New Node
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Instance f erto the New Node
i o
Add a Database s
ana database instance to your RAC database. You can do so by using the DBCA
b ricneed to-tradd
Youanow
F non
from the first node.
On the Welcome screen, select “Oracle Real Application Clusters database” and click Next.
On the Operations screen, select Instance Management and click Next.
On the Instance Management screen, select “Add an instance” and click Next.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Instance f erto the New Node (continued)
i
Add a Databaseo s
an databases” screen, select your RAC database and enter SYS credentials. Then,
ic of -cluster
On a b
the r“List t r
F non
click Next.
On the “List of cluster database instances” screen, click Next.
On the “Instance naming and node selection” screen, select the node name on which you want to add
the instance, and specify the name of that instance. When done, click Next.
On the Instance Storage screen, click Finish.
On the Summary screen, check the various parameters and click OK.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Instance f erto the New Node (continued)
i
Add a Database o
ic if-you n
aares
b r t r
Fa non
At this point, using ASM for your database storage, and there is currently no running ASM
instance on your new node, DBCA detects the need for an ASM instance creation on the new node.
This must be done before the DBCA can create the database instance on that node. Click Yes.
The assistant is now adding your instance to your RAC database on the new node. It will also start
that instance at the end of the operation.
Add an Instance to Your RAC Database Using EM
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V to Your f erRAC Database Using EM
i o
Add an Instance
c tran s
b rialso
Youacan - a new instance to your RAC database by using the Add Instance Wizard. From the
add
F non
Cluster Database page, click the Server tab. On the Server tabbed page, click Add Instance in the
Change Database section of the page.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V to Your f erRAC Database Using EM (continued)
i o
Add an Instance
c tran s
b rinow
Youaare on- the Cluster Credentials page where you specify the cluster and ASM credentials. The
F non
wizard automatically adds the ASM instance before adding the database instance if it is not already
created.
When done, click Next to go to the Host page where you specify on which host you want to add the
database instance. Select the node in question and click Next.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V to Your f erRAC Database Using EM (continued)
i o
Add an Instance
ic -page, s
anclick Submit to start the job’s execution.
On a b
the
rReview t r
F non
On the Confirmation page, click “View job” to see the job’s log. After some refreshes of that page,
you should get a succeeded status.
Main Steps to Delete a Node from a RAC Cluster
1. Delete the instance on the node to be deleted.

2. Clean up the ASM instance.
3. Remove the listener from the node to be deleted.
4. Remove the node from the database.
[5. Remove the node from ASM. ]
6. Remove ONS configuration from the node to be deleted.
a
7. Remove the node from the clusterware. has r)
ฺ b
m eฺ
ฺ c o
ORACLE_HOME ORACLE_HOME ORACLE_HOME
r e di Guid
CRS HOME CRS HOME CRS HOME
@ sic dent
e r de Stu
v i lav e this
ic i o_Storage
o us
r
ab nse t
( f
e rde lCopyright
l a V ab e
i er from a RAC Cluster
Main Stepsio to V
Delete n s
a fNode
b r icall of-the
t r aadd
Fa non
Note: For node and delete node procedures for UNIX-based systems, temporary
directories such as /tmp, $TEMP, or $TMP, should not be shared directories. If your temporary
directories are shared, then set your temporary environment variable, such as $TEMP, to a nonshared
location on a local node. In addition, use a directory that exists on all the nodes.
Delete the Instance on the Node to Be Deleted
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V onsfthe er Node to Be Deleted
i o
Delete the Instance
c tran
b ristep
Theafirst is-to remove the database instance from the node that you want to delete. For that, you
F non
use the DBCA from the node you want to delete.
On the Welcome screen, select “Oracle Real Application Clusters database” and click Next.
On the Operations screen, select Instance Management and click Next.
On the Instance Management screen, select “Delete an instance” and click Next.
On the “List of cluster databases” screen, select the RAC database from which you want to delete an
instance, enter SYS and its password, and click Next.
Delete the Instance on the Node to Be Deleted
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V onsfthe er Node to Be Deleted (continued)
i o
Delete the Instance
an database instances” screen, select the instance that you want to delete and
ic of -cluster
On a b
the
r“List t r
F non
click Finish.
In the Database Configuration Assistant dialog box, click OK to validate your choice.
This triggers the remove instance process. When completed, your instance is removed from your
cluster database.
Clean Up the ASM Instance
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Instance f er
Clean Up thei o ASM
icdatabase n s
ainstance
b r - t r
Fa non
After your
instance.
is removed from the node, you can clean up the corresponding ASM
To do this, you need to use SRVCTL to first stop the ASM instance currently running on the node
that you want to remove, and then remove that ASM instance from the same node. The two
commands are illustrated in the screenshot shown in the slide.
Then, you need to manually remove the initialization parameter file of that ASM instance. As shown
in the slide, you can remove files containing the ASM string from the <ASM home>/dbs directory.
After this is done, you can also remove all the log files of that ASM instance. These files are
generally located in the $ORACLE_BASE/admin directory.
The last thing you can do is to remove the associated ASM entry from the /etc/oratab file.
Remove the Listener from the Node to Be Deleted
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfrom f er the Node to Be Deleted
Remove theioListener
b c tran
rinow
Youacan - the listener from the node that you want to delete. This listener can be from
remove
F non
either the ASM home or the database home depending on when it was created.
To remove the listener, you can use NETCA as shown in the slide.
On the Configuration screen, select “Cluster configuration” and click Next.
On the Active Nodes screen, select the node from which you want to remove the listener and click
Next.
On the Welcome screen, select “Listener configuration” and click Next.
On the Listener screen, select Delete and click Next.
On the “Select listener” screen, select the corresponding listener, normally called LISTENER, and
click Next.
Follow the rest of the screens until the listener is removed from the node.
Remove the Node from the Database
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V from f r
ethe
Remove theioNode
i c a n s Database
r t r
Fab younocann-use the Oracle Universal Installer to remove the database software installation, you
Before
need to update the inventory on the node to be deleted by executing the following command (also
shown in the slide):
./runInstaller –updateNodeList ORACLE_HOME=<Database home> "CLUSTER_NODES=<node to be
removed>" –local
You need to execute this command from the oui/bin subdirectory in the database home.
After this command is executed, you can start OUI from the same directory, and click “Deinstall
products” on the Welcome screen. Then, select the database home and click Remove. This will
remove the database home from the node to be deleted.
You now need to update the corresponding inventory on the remaining nodes. You can use the
following command from the first node:
./runInstaller –updateNodeList ORACLE_HOME=<Database home> "CLUSTER_NODES=<remaining
nodes>"
This command needs to be executed from the oui/bin subdirectory of the database home.
Remove the Node from ASM
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V from f r
eASM
Remove theioNode
ic can-use n
athes
b r t r
Fa non
Before you Oracle Universal Installer to remove the ASM software installation, you need
to update the inventory on the node to be deleted by executing the following command (also shown
in the slide):
./runInstaller –updateNodeList ORACLE_HOME=<ASM home> "CLUSTER_NODES=<node to be
removed>" –local
You need to execute this command from the oui/bin subdirectory in the ASM home.
After this command is executed, you can start OUI from the same directory, and click “Deinstall
products” on the Welcome screen. Then, select the ASM home and click Remove. This will remove
the ASM home from the node to be deleted.
You now need to update the corresponding inventory on the remaining nodes. You can use the
following command from the first node:
./runInstaller –updateNodeList ORACLE_HOME=<ASM home> "CLUSTER_NODES=<remaining
nodes>"
This command needs to be executed from the oui/bin subdirectory of the ASM home.
Note: This step is not needed if you are not using a separate home directory for ASM.
Remove the Node from the Oracle Clusterware
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V from f r
ethe
Remove theioNode
i c a n s Oracle Clusterware
r t r
Fab younocann-use OUI to remove the Oracle Clusterware software installation from the node to be
Before
deleted, you need to perform the following commands:
• The following needs to be done from the first node: <Oracle Clusterware home>/bin/racgons
remove_config <Node to be removed>:6251 (Replace port 6251 in the command above with the
port number that you get in the remoteport section of the ons.config file found in the <Oracle
Clusterware home>/opmn/conf directory.)
• The following needs to be done from the node to be removed as the root user: <Oracle Clusterware
home>/install/rootdelete.sh
• The following needs to be done from the first node as the root user: Determine the node number
to be deleted using <Oracle Clusterware home>/bin/olsnodes –n. Then execute <Oracle Clusterware
home>/install/rootdeletenode.sh <node name to be deleted>,<node number to be deleted>.
These three steps are illustrated in the slide.
Remove the Node from the Oracle Clusterware
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V from f r
ethe
Remove the
i c i oNode
a n s Oracle Clusterware (continued)
r r
b neednto-t update the inventory from the node to be deleted by executing: <Oracle Clusterware
Youanow
F no
home>/oui/bin/runInstaller -updateNodeList ORACLE_HOME=<Oracle Clusterware home>
"CLUSTER_NODES=<Node to be deleted>" CRS=TRUE -local
When done, run OUI from the same directory and choose “Deinstall products” and remove the
Oracle Clusterware installation on the node to be deleted as illustrated in the slide.
You can now update the inventory from the first node by executing the following command: <Oracle
Clusterware home>/oui/bin/runInstaller -updateNodeList ORACLE_HOME=<Oracle Clusterware
home> "CLUSTER_NODES=<Remaining nodes>" CRS=TRUE
To verify the removal of the node from the cluster, run the following commands from the first node:
• srvctl status nodeapps -n <Deleted node> should get a message saying Invalid
node.
• crs_stat | grep -i <Deleted node> should not get any output.
• olsnodes –n should get all the present nodes list without the deleted node.
Note: You should also remove all corresponding oracle homes after this step.
Node Addition and Deletion
and the SYSAUX Tablespace
• The SYSAUX tablespace combines the storage needs for

the following tablespaces:
– DRSYS
– CWMLITE
– XDB
– ODM
s a
– OEM-REPO
)h a
• Use this formula to size m ฺbr
i ฺ c o deฺ
the SYSAUX tablespace:
r e d Gui
s ic ent
300M + (250M * number_of_nodes)
d @
e Stud
r
ve this
i l a
i o _v use
a b ric e to
e (f ens
e d
e l ic © 2008, Oracle. All rights reserved.
r lCopyright
l a V ab
i
Vand Deletion
f er and the SYSAUX Tablespace
Node Addition
i o
ic -tran s
b r
Fa non system-managed tablespace called SYSAUX contains performance data and
A new auxiliary,
combines content that was stored in different tablespaces (some of which are no longer required) in
earlier releases of the Oracle database. This is a required tablespace for which you must plan disk
space. The SYSAUX system tablespace now contains the DRSYS (contains data for OracleText),
CWMLITE (contains the OLAP schemas), XDB (for XML features), ODM (for Oracle Data Mining),
and OEM-REPO tablespaces.
If you add nodes to your RAC database environment, then you may need to increase the size of the
SYSAUX tablespace. Conversely, if you remove nodes from your cluster database, then you may be
able to reduce the size of your SYSAUX tablespace and thus save valuable disk space. The following
is a formula that you can use to properly size the SYSAUX tablespace:
300 megabytes + (250 megabytes * number_of_nodes)
If you apply this formula to a four-node cluster, then you find that the SYSAUX tablespace is sized
around 1,300 megabytes as shown below:
300 + (250 * 4) = 1300
Summary
In this lesson, you should have learned how to:

• Add a new node to your cluster database
• Remove a node from your cluster database
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Practice 11: Overview
This practice covers the following topics:

• Removing the second node of your cluster
• Adding it back again
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i
Important Note o
c tran
b ristrongly
Youaare - advised to follow directly the solution appendix for this lab.
F non
Design for High Availability
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Objectives
After completing this lesson, you should be able to:

• Design a Maximum Availability Architecture in your
environment
• Determine the best RAC and Data Guard topologies for
your environment
• Configure the Data Guard Broker configuration files in a
a RAC environment h as
ฺ b r)
• Decide on the best ASM configuration to use om ฺ
i ฺ c i d e
red Gu
• Patch your RAC system in a rolling fashion
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Causes of Unplanned Down Time
Unplanned Down time
Software failures Hardware failures Human errors Disasters
Operating system CPU Operator error Fire
Database Memory User error Flood
Middleware Power supply DBA Earthquake

s a
Application Bus System admin. Power failure
)h a
Network Disk Sabotage Bombing m ฺbr
i ฺ c o deฺ
Tape
r e d Gui
Controllers @ sic dent
e r de Stu
Network
v i lav e this
Powercio
_ us
b i
r e to
a
(f ens
d e ic © 2008, Oracle. All rights reserved.
r lCopyright
l
e
V ab e
i l
V sDown a er Time
i o
Causes of Unplanned f
b
Oneaof t r an in designing a highly available solution is examining and addressing all
ic true -challenges
rthe
F non
the possible causes of down time. It is important to consider causes of both unplanned and planned
down time. The schema shown in the slide, which is a taxonomy of unplanned failures, classifies
failures as software failures, hardware failures, human error, and disasters. Under each category
heading is a list of possible causes of failures related to that category.
Software failures include operating system, database, middleware, application, and network failures.
A failure of any one of these components can cause a system fault.
Hardware failures include system, peripheral, network, and power failures.
Human error, which is a leading cause of failures, includes errors by an operator, user, database
administrator, or system administrator. Another type of human error that can cause unplanned down
time is sabotage.
The final category is disasters. Although infrequent, these can have extreme impacts on enterprises,
because of their prolonged effect on operations. Possible causes of disasters include fires, floods,
earthquakes, power failures, and bombings. A well-designed high-availability solution accounts for
all these factors in preventing unplanned down time.
Causes of Planned Down Time
Planned down time
Routine operations Periodic maintenance New deployments
Backups Storage maintenance HW upgrade
Performance mgmt Initialization parameters OS upgrades
Security mgmt Software patches DB upgrades

s a
Batches Schema management MidW upgrades
)h a
Operating system App upgradesm ฺbr
i ฺ c o deฺ
Middleware
r e i
d NetGupgrades
u
Network @ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Down f erTime
i o
Causes of Planned
c tran
idown s
b r - can be just as disruptive to operations, especially in global enterprises that
Fa nontime
Planned
support users in multiple time zones, up to 24 hours per day. In these cases, it is important to design a
system to minimize planned interruptions. As shown by the schema in the slide, causes of planned
down time include routine operations, periodic maintenance, and new deployments. Routine
operations are frequent maintenance tasks that include backups, performance management, user and
security management, and batch operations.
Periodic maintenance, such as installing a patch or reconfiguring the system, is occasionally
necessary to update the database, application, operating system middleware, or network.
New deployments describe major upgrades to the hardware, operating system, database, application,
middleware, or network. It is important to consider not only the time to perform the upgrade, but also
the effect the changes may have on the overall application.
Oracle’s Solution to Down Time
RMAN backup/recovery
Fast-start (Data Recovery Advisor)
RAC
Fault Recovery
Data Guard ASM
Streams
System
failures Flashback
Unplanned
down time
Data HARD
failures
Data Guard s a
& )h a
Rolling upgrades/ Streams
m ฺbr
System Online patching
i ฺ c o deฺ
changes r e d Gui
Planned @ sic dent
Dynamic provisioning
down time
e r de Stu
Data i lavOnline t is
hredefinition
changes io_
v e
r ic t o us
( f ab nse
e rde lCopyright
l a V ab e
i
V to Downf er Time
i o
Oracle’s Solution
ic -tran s
b r
Fa nodown
Unplanned n time is primarily the result of computer failures or data failures. Planned down time
is primarily due to data changes or system changes:
• RAC provides optimal performance, scalability, and availability gains.
• Fast-Start Fault Recovery enables you to bound the database crash/recovery time. The database
self-tunes checkpoint processing to safeguard the desired recovery time objective.
• ASM provides a higher level of availability using online provisioning of database storage.
• Flashback provides a quick resolution to human errors.
• Oracle Hardware Assisted Resilient Data (HARD) is a comprehensive program designed to
prevent data corruptions before they happen.
• Recovery Manager (RMAN) automates database backup and recovery. Data Recovery Advisor
(not supported for RAC) diagnoses data failures and presents repair options.
• Data Guard must be the foundation of any Oracle database disaster-recovery plan.
• The increased flexibility and capability of Streams over Data Guard with SQL Apply requires
more investment and expertise to maintain an integrated high availability solution.
• With online redefinition, the Oracle database supports many maintenance operations without
disrupting database operations, or users updating or accessing data.
• The Oracle database continues to broaden support for dynamic reconfiguration, enabling it to
adapt to changes in demand and hardware with no disruption of service.
• The Oracle database supports the application of patches to the nodes of a RAC system, as well
as database software upgrades, in a rolling fashion.
RAC and Data Guard Complementarity
Resource Cause Protection
Nodes RAC
Component
Instances failure RAC
Software failure s a
)h a
Human
error Data Guard m ฺbr
Data & i ฺ c o deฺ
Environment
r e
Flashback
d Gui
@ sic dent
e r de Stu Data Guard
Site v i lav e this &
_
io o us Streams
r ic t
( f ab nse
e rde lCopyright
l a V ab e
i er
VGuardsComplementarity
f
RAC and Datai o
icData Guard antogether provide the benefits of system-level, site-level, and data-level
b r - t r
Fa non
RAC and
protection, resulting in high levels of availability and disaster recovery without loss of data:
• RAC addresses system failures by providing rapid and automatic recovery from failures, such as
node failures and instance crashes.
• Data Guard addresses site failures and data protection through transactionally consistent primary
and standby databases that do not share disks, enabling recovery from site disasters and data
corruption.
Note: Unlike Data Guard using SQL Apply, Oracle Streams enables updates on the replica and
provides support for heterogeneous platforms with different database releases. Therefore, Oracle
Streams may provide the fastest approach for database upgrades and platform migration.
Maximum Availability Architecture
Real-time query
Clients
Oracle Oracle
Application Application
Server Server
WAN Traffic
Manager
Real-time query
s a
)h a
Primary Secondary
m ฺbr
site Data Guard site
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
RAC
v i lav e this RAC databases:
database
ic i o_ o us Phys&log standby
r
ab nse t
( f
e rde lCopyright
l a V ab e
i er
V sArchitecture
f
i i o
MaximumcAvailability
a n (MAA)
r r
-t provide the basis of the database MAA solution. MAA provides the most
FabandnData
RAC
onGuard
comprehensive architecture for reducing down time for scheduled outages and preventing, detecting,
and recovering from unscheduled outages. The recommended MAA has two identical sites. The
primary site contains the RAC database, and the secondary site contains both a physical standby
database and a logical standby database on RAC. Identical site configuration is recommended to
ensure that performance is not sacrificed after a failover or switchover. Symmetric sites also enable
processes and procedures to be kept the same between sites, making operational tasks easier to
maintain and execute.
The graphic illustrates identically configured sites. Each site consists of redundant components and
redundant routing mechanisms, so that requests are always serviceable even in the event of a failure.
Most outages are resolved locally. Client requests are always routed to the site playing the production
role.
After a failover or switchover operation occurs due to a serious outage, client requests are routed to
another site that assumes the production role. Each site contains a set of application servers or mid-
tier servers. The site playing the production role contains a production database using RAC to protect
from host and instance failures. The site playing the standby role contains one standby database, and
one logical standby database managed by Data Guard. Data Guard switchover and failover functions
allow the roles to be traded between sites.
Note: For more information, see the following Web site:
http://otn.oracle.com/deploy/availability/htdocs/maa.htm
RAC and Data Guard Topologies
• Symmetric configuration with RAC at all sites:

– Same number of instances
– Same service preferences
• Asymmetric configuration with RAC at all sites:
– Different number of instances
– Different service preferences a
• Asymmetric configuration with mixture of RAC and ) h as
single instance: m ฺbr
i ฺ c o deฺ
– All sites running under Oracle Clusterware
r e d Gui
– Some single-instance sites not running @ sic under
e n tOracle
Clusterware r d e Stud
i l a ve this
i o _v use
a b ric e to
e (f ens
e d
r lCopyright
l a V ab
i er
VGuardsTopologies
f
RAC and Datai o
c tran
b riconfigure
Youacan
F n o n- a standby database to protect a primary database in a RAC environment.
Basically, all kinds of combinations are supported. For example, it is possible to have your primary
database running under RAC and your standby database running as a single-instance database. It is
also possible to have both the primary and standby databases running under RAC.
The slide explains the distinction between symmetric environments and asymmetric ones.
If you want to create a symmetric environment running RAC, then all databases need to have the
same number of instances and the same service preferences. As the DBA, you need to make sure that
this is the case by manually configuring them in a symmetric way.
However, if you want to benefit from the tight integration of Oracle Clusterware and Data Guard
Broker, make sure that both the primary site and the secondary site are running under Oracle
Clusterware and that both sites have the same services defined.
Note: Beginning with Oracle Database release 11g, the primary and standby systems in a Data Guard
configuration can have different CPU architectures, operating systems (for example, Windows and
Linux for physical standby database only with no EM support for this combination), operating
system binaries (32-bit and 64-bit), and Oracle database binaries (32-bit and 64-bit). For the latest
capabilities and restrictions, see Metalink note 413484.1 at https://metalink.oracle.com/.
RAC and Data Guard Architecture
Primary instance A Standby receiving instance C
ARCn
ARCn LGWR RFS
Flash
Primary recovery
database Standby area
Online redo
redo files s a
files Standby )h a
Flash
recovery databasemฺ
br
LGWR RFS i ฺ c o deฺ
area
r e d Gui
@ sic dApply
e nt
ARCn
e r de ARCn S tu
v i lav e this
Primary instance B
ic i o_ o usStandby apply instance D
( f abr nse t
e rde lCopyright
l a V ab e
i er
VGuardsArchitecture
f
RAC and Data i o an possible to use a “RAC to single-instance Data Guard (DG)” configuration,
icit is perfectly
b r - t r
Fa nonthe possibility to use a RAC-to-RAC DG configuration. In this mode, although
Although
you also have
multiple standby instances can receive redo from the primary database, only one standby instance
can apply the redo stream generated by the primary instances.
A RAC-to-RAC DG configuration can be set up in different ways, and the slide shows you one
possibility with a symmetric configuration where each primary instance sends its redo stream to a
corresponding standby instance using standby redo log files. It is also possible for each primary
instance to send its redo stream to only one standby instance that can also apply this stream to the
standby database. However, you can get performance benefits by using the configuration shown in
the slide. For example, assume that the redo generation rate on the primary is too great for a single
receiving instance on the standby side to handle. Suppose further that the primary database is using
the SYNC redo transport mode. If a single receiving instance on the standby cannot keep up with the
primary, then the primary’s progress is going to be throttled by the standby. If the load is spread
across multiple receiving instances on the standby, then this is less likely to occur.
If the standby can keep up with the primary, another approach is to use only one standby instance to
receive and apply the complete redo stream. For example, you can set up the primary instances to
remotely archive to the same Oracle Net service name.
RAC and Data Guard Architecture (continued)
You can then configure one of the standby nodes to handle that service. This instance then both
receives and applies redo from the primary. If you need to do maintenance on that node, then you can
stop the service on that node and start it on another node. This approach allows for the primary
instances to be more independent of the standby configuration because they are not configured to
send redo to a particular instance.
Note: For more information, refer to the Oracle Data Guard Concepts and Administration guide.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Data Guard Broker (DGB) and
Oracle Clusterware (OC) Integration
• OC manages intrasite HA operations.

• OC manages intrasite planned HA operations.
• OC notifies when manual intervention is required.
• DBA receives notification.
• DBA decides to switch over or fail over using DGB.
• DGB manages intersite planned HA operations. s a
h a
• DGB takes over from OC for intersite failover, ฺbr)
switchover, and protection mode changes: ฺcom eฺ
– DMON notifies OC to stop and disableic
di
u all
re site,t leaving
the G
id
or one instance. @ s den
e r de Stu
– DMON notifies OC to enable
i l a v tstart
and
h is the site according
to the DG site role. o_v se
b r ici to u
e (fa ense
e e l ic © 2008, Oracle. All rights reserved.
rd lCopyright
l a V ab
i
V (DGB) f er and Oracle Clusterware (OC) Integration
Data Guardio Broker s
an with Oracle Clusterware. Oracle Clusterware manages individual instances
ic -integrated
b r t r
Fa non
DGB is tightly
to provide unattended high availability of a given clustered database. DGB manages individual
databases (clustered or otherwise) in a Data Guard configuration to provide disaster recovery in the
event that Oracle Clusterware is unable to maintain availability of the primary database.
For example, Oracle Clusterware posts NOT_RESTARTING events for the database group and
service groups that cannot be recovered. These events are available through Enterprise Manager,
ONS, and server-side callouts. As a DBA, when you receive those events, you might decide to repair
and restart the primary site, or to invoke DGB to fail over if not using Fast-start Failover.
DGB and Oracle Clusterware work together to temporarily suspend service availability on the
primary database, accomplish the actual role change for both databases during which Oracle
Clusterware works with the DGB to properly restart the instances as necessary, and then to resume
service availability on the new primary database. The broker manages the underlying Data Guard
configuration and its database roles whereas Oracle Clusterware manages service availability that
depends upon those roles. Applications that rely upon Oracle Clusterware for managing service
availability will see only a temporary suspension of service as the role change occurs within the Data
Guard configuration.
Fast-Start Failover: Overview
• Fast-Start Failover implements automatic failover to a

standby database:
– Triggered by failure of site, hosts, storage, data file
offline immediate, or network
– Works with and supplements RAC server failover
• Failover occurs in seconds (< 20 seconds).
– Comparable to cluster failover s a
)h a
• Original production site automatically rejoins theฺbr
configuration after recovery. c o m eฺ
ฺ
di Guid
r e
sic process:
• Automatically monitored by an Observer nt @ e
– Locate it on a distinct server on
r d ea distinct
S t uddata center
e is failure
lav eitthon
– Enterprise Manager canvirestart
_ us
cio tClient
– Installed throughriOracle o Administrator
( f ab nse
e rde lCopyright
l a V ab e
i
V Overview f er
i o
Fast-Start Failover:
icFailover s
aisnan Oracle Data Guard feature that automatically, quickly, and reliably fails over
b r - t r
Fa non synchronized standby database in the event of loss of the primary database, without
Fast-Start
to a designated,
requiring manual intervention to execute the failover. In addition, following a fast-start failover, the
original primary database is automatically reconfigured as a new standby database upon reconnection
to the configuration. This enables Data Guard to restore disaster protection in the configuration as
soon as possible.
Fast-Start Failover is used in a Data Guard configuration under the control of the Data Guard Broker,
and may be managed using either DGMGRL or Oracle Enterprise Manager 10g Grid Control. There
are three essential participants in a Fast-Start Failover configuration:
• The primary database, which can be a RAC database
• A target standby database, which becomes the new primary database following a fast-start
failover.
• The Fast-Start Failover Observer, which is a separate process incorporated into the DGMGRL
client that continuously monitors the primary database and the target standby database for
possible failure conditions. The underlying rule is that out of these three participants, whichever
two can communicate with each other will determine the outcome of the fast-start failover. In
addition, a fast-start failover can occur only if there is a guarantee that no data will be lost.
Fast-Start Failover: Overview (continued)
For disaster recovery requirements, install the Observer in a location separate from the primary and
standby data centers. If the designated Observer fails, Enterprise Manager can detect the failure and
can be configured to automatically restart the Observer on the same host.
You can install the Observer by installing the Oracle Client Administrator (choose the Administrator
option from the Oracle Universal Installer). Installing the Oracle Client Administrator results in a
small footprint because an Oracle instance is not included on the Observer system. If Enterprise
Manager is used, also install the Enterprise Manager Agent on the Observer system.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Data Guard Broker Configuration Files
*.DG_BROKER_CONFIG_FILE1=+DG1/RACDB/dr1config.dat
*.DG_BROKER_CONFIG_FILE2=+DG1/RACDB/dr2config.dat
RAC01 RAC02
s a
)h a
m ฺbr
i ฺ c o deฺ
Shared storage r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
V Configuration
f
Data Guardio
i c Broker
a n s Files
r
b ofnthe r
-t Data Guard Broker (DGB) configuration files are maintained for each database so
Facopies
Two
n o
as to always have a record of the last known valid state of the configuration. When the broker is
started for the first time, the configuration files are automatically created and named using a default
path name and file name that is operating system specific.
When using a RAC environment, the DGB configuration files must be shared by all instances of the
same database. You can override the default path name and file name by setting the following
initialization parameters for that database: DG_BROKER_CONFIG_FILE1,
DG_BROKER_CONFIG_FILE2.
You have three possible options to share those files:
• Cluster file system
• Raw devices
• ASM
The example in the slide illustrates a case where those files are stored in an ASM disk group called
DG1. It is assumed that you have already created a directory called RACDB in DG1.
Real-Time Query Physical Standby Database
Client Client Client Client

Read/Write Read only
Primary Standby
Redo apply
cluster cluster
s a
Redo apply
instance
)h a
m ฺbr
RAC diฺc
o deฺ
RAC
r e u i
database
sic dent
database G
@
de Stu
e r
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Physical f er Standby Database
i o
Real-Time Query
ic Redo s
an (physical standby database) has proven to be a popular solution for disaster
b r - t r
Fa non
Data Guard Apply
recovery due to its relative simplicity, high performance, and superior level of data protection.
Beginning with Oracle Database 11g, a physical standby database can be open read-only while redo
apply is active. This means that you can run queries and reports against an up-to-date physical
standby database without compromising data protection or extending recovery time in the event a
failover is required. This makes every physical standby database able to support productive uses even
while in standby role. To enable real-time query, open the database in read-only mode and then issue
the ALTER DATABASE RECOVER MANAGED STANDBY statement. Real-time query provides an
ultimate high availability solution because it:
• Is totally transparent to applications
• Supports Oracle RAC on the primary and standby databases. Although Redo Apply can be
running on only one Oracle RAC instance, you can have all of the instances running in read-only
mode while Redo Apply is running on one instance.
• Enables queries to return transactionally consistent results that are very close to being up-to-date
with the primary database
• Enables you to use fast-start failover to allow for automatic fast failover if the primary database
fails
Note: The COMPATIBLE parameter must be set to 11.0.0 on both the primary and physical standby
databases.
Hardware Assisted Resilient Data
Blocks validated and

Oracle database
protection information added to blocks
[DB_BLOCK_CHECKSUM=TRUE] Vol Man/ASM
Operating system
• Prevents corruption introduced Device driver
in I/O path Host Bus Adapter
• Is supported by major storage vendors:
s a
– EMC, Fujitsu, Hitachi, HP, NEC, SAN )h a
– Network Appliance &mฺb
r
– Sun Microsystems iฺco ideฺ
Virtualization
i c red t Gu
• All file types and block sizes checked @ s den
Protection information validated e r de StuSAN interface
by storage device when enabled v i lav e this Storage device
symchksum –type Oracle ic i o_ enable
o us
r
ab nse t
( f
e rde lCopyright
l a V ab e
i
V Resilientf er Data
i o
Hardware Assisted s
ancause lengthy outages is data corruption. Today, the primary means for
b ric that
Oneaproblem - t rcan
F
detecting n
on
corruptions caused by hardware or software outside of the Oracle database, such as an I/O
subsystem, is the Oracle database checksum. However, after a block is passed to the operating
system, through the volume manager and out to disk, the Oracle database itself can no longer provide
any checking that the block being written is still correct.
With disk technologies expanding in complexity, and with configurations such as Storage Area
Networks (SANs) becoming more popular, the number of layers between the host processor and the
physical spindle continues to increase. With more layers, the chance of any problem increases. With
the HARD initiative, it is possible to enable the verification of database block checksum information
by the storage device. Verifying that the block is still the same at the end of the write as it was in the
beginning gives you an additional level of security.
By default, the Oracle database automatically adds checksum information to its blocks. These
checksums can be verified by the storage device if you enable this possibility. In case a block is
found to be corrupted by the storage device, the device logs an I/O corruption, or it cancels the I/O
and reports the error back to the instance.
Note: The way you enable the checksum validation at the storage device side is vendor specific. The
example given in the slide was used with EMC Symmetrix storage.
Oracle Clusterware Rolling Upgrade
1. unzip p….zip
2. runInstaller from Disk1 directory:
– Choose your Oracle Clusterware home installation.
– Choose all nodes.
– Install.
3. Repeat on each node, one after the other:
– crsctl stop crs
– <crs home>/install/root….sh s a
)h a
Node n Node n Node n
m ฺbr
i ฺ c d ฺ
o Clusterware
e
Oracle Clusterware
x
Oracle Clusterware
x
Oracle
red t Gux+1 i
stop i c
root102
s denOracle RAC/ASM
Oracle RAC/ASM
y
Oracle RAC/ASM
y
d @
e Stu y
r
ve this
i l a
_v use
RDBA1 CRS +ASM1 RDBA1 CRS +ASM1
i o
a b ric e to
e (f ens
e d
r lCopyright
l a V ab
i
V Rolling f er Upgrade
i o
Oracle Clusterware s
an Clusterware can be performed in a rolling fashion. The slide illustrates how to
Alla b ric to-Oracle
upgrades t r
F
do this.non
Once you installed the new software, you need to execute the two commands described in step 3 (in
sequence, one node at a time). The diagram shows you the configuration evolution after stopping and
running the corresponding root….sh script.
Clustered ASM Rolling Upgrade
1. Upgrade Oracle Clusterware

2. From one ASM instance:
ALTER SYSTEM START ROLLING MIGRATION TO
ALTER SYSTEM START ROLLING MIGRATION TO 11.2.0.0.0;
11.2.0.0.0;
3. Repeat on each node, one after the other:
a. Shut down database instance.
s a
b. Shut down ASM instance. h a
c. Upgrade ASM software using OUI. ฺ b r)
c o m eฺ
d. ฺ
di Guid
Restart Oracle Clusterware and all corresponding
r e
resources. sic nt
4. From one ASM instance: r d e@ Stude
i l a ve this
ALTER SYSTEM STOP ROLLING i o _v MIGRATION;
u s e
a b ric e to
e (f ens
e d
r lCopyright
l a V ab
i
VRolling f r
eUpgrade
Clustered ASMi o n s
Alla b
ASM
ricupgrades
- t r astarting with Oracle Database 11g (and later releases) can be performed in a
F n o n
rolling fashion. ASM rolling upgrades enable you to independently upgrade or patch clustered ASM
nodes without affecting database availability. The slide describes how to perform a hypothetical
ASM rolling upgrade from 11.1.0.6 to 11.2.0.0.
To perform a rolling upgrade, your environment must be prepared. If you are using Oracle
Clusterware, your Oracle Clusterware must be fully upgraded to the next patch or release version
before you start the ASM rolling upgrade.
Before you patch or upgrade the ASM software on a node, you must place the ASM cluster into
rolling upgrade mode. This enables you to begin an upgrade and operate your environment in
multiversion software mode. This is illustrated by step 2. You can run this statement from any ASM
instance part of your cluster. The instance from which you run this statement verifies whether the
value that you specified for the target version is compatible with the current installed version of your
software. When the upgrade begins, the behavior of the clustered ASM environment changes, and
only the following operations are permitted on the ASM instance:
• Disk group mount and dismount
• Database file open, close, resize, and delete
• Limited access to fixed views and fixed packages
Oracle disables all global views when a clustered ASM environment is in rolling upgrade mode.
Clustered ASM Rolling Upgrade (continued)
You can use the following SQL function to query the state of a clustered ASM environment:
SELECT SYS_CONTEXT('sys_cluster_properties', 'cluster_state') FROM DUAL;
After the rolling upgrade has started, you can do the following on each node, one after the other:
• Shut down each ASM instance.
• Perform the software upgrade using OUI.
• On node start up, the updated ASM instance can rejoin the cluster.
When you have migrated all of the nodes in your clustered ASM environment to the latest software
version, you can end the rolling upgrade mode from one ASM instance as shown in step 4 on the
slide.
If all of the instances in a clustered ASM environment stop running, the restarted instance will not be
in rolling upgrade mode when any of the ASM instances restart. To perform the upgrade after your
instances restart, you must rerun the commands to restart the rolling upgrade operation.
s a
Note: If a disk goes offline when the ASM instance is in rolling upgrade mode, the disk remains
h a
r) the ASM
offline until the rolling upgrade ends. Also, the timer for dropping the disk is stoppedbuntil
ฺ
cluster is out of rolling upgrade mode.
iฺc uide om ฺ
e d
s icr ent G
r d e@ Stud
i l a ve this
i o _v use
a b ric e to
e (f ens
e r le lic
d
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Patches and the RAC Environment
ex0043 ex0044 ex0045

/u01/app/oracle /u01/app/oracle /u01/app/oracle
/product/db_1 /product/db_1 /product/db_1
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
Apply a patchset to
e r de Stu
lav e this
/u01/app/oracle
/product/db_1 on v i
all nodes.
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
V RACsEnvironment
f
Patches and i othe
icpatches anyour RAC installation is a simple process with the OUI. The OUI can keep track
b r - t r
Fa non
Applying to
of multiple ORACLE_HOME deployments. This intelligence prevents potentially destructive or
conflicting patchsets from being applied.
In the example in the slide, a patchset is applied to the /u01/app/oracle/product/db_1
Oracle Home on all the three nodes of your cluster database. Although you execute the installation
on ex0043, you can choose any of the nodes to perform this task. The steps that you must perform
to add a patchset through the OUI are essentially the same as those to install a new release. You must
change directory to $ORACLE_HOME/bin. After starting the OUI, perform the following steps:
1. Select “Installation from a stage location,” and enter the appropriate patchset source on the
Welcome screen.
2. Select the nodes on the Node Selection screen, where you need to add the patch, and ensure that
they are all available. In this example, this should be all three of the nodes because
/u01/app/oracle/product/db_1 is installed on all of them.
3. Check the Summary screen to confirm that space requirements are met for each node.
4. Continue with the installation and monitor the progress as usual.
The OUI automatically manages the installation progress, including the copying of files to remote
nodes, just as it does with the Oracle Clusterware and database binary installations.
Inventory List Locks
• The OUI employs a timed lock on the inventory list

stored on a node.
• The lock prevents an installation from changing a list
being used concurrently by another installation.
• If a conflict is detected, the second installation is
suspended and the following message appears:
s a
)h a
ฺbr
"Unable to acquire a writer lock on nodes ex0044.
m
Restart the install after verifying thatiฺc o dis
there e ฺ
d Gu i
icrenodes."
no OUI session on any of the selected t s den
@
de Stu
e r
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
V i f er
Inventory Listi o Locks s
an in the OUI is that it prevents potentially destructive concurrent
ic improvements
b
Oneaof
rthe - t r
F non
installations. The mechanism involves a timed lock on the inventory list stored on a node. When you
start multiple concurrent installations, the OUI displays an error message that is similar to the one
shown in the slide. You must cancel the installation and wait until the conflicting installation
completes, before retrying it.
Although this mechanism works with all types of installations, see how it can function if you attempt
concurrent patchset installations in the sample cluster. Use the same configuration as in the previous
scenario for your starting point.
Assume that you start a patchset installation on ex0043 to update ORACLE_HOME2 on nodes
ex0044 and ex0045. While this is still running, you start another patchset installation on ex0044
to update ORACLE_HOME3 on that node. Will these installations succeed? As long as there are no
other problems, such as a down node or interconnect, these processes have no conflicts with each
other and should succeed. However, what if you start your patchset installation on ex0044 to update
ORACLE_HOME3 and then start a concurrent patchset installation for ORACLE_HOME2 (using either
ex0044 or ex0043) on all nodes where this Oracle Home is installed? In this case, the second
installation should fail with the error shown because the inventory on ex0044 is already locked by
the patchset installation for ORACLE_HOME3.
OPatch Support for RAC: Overview
• OPatch supports four different methods:

– All-node patch: Stop all/Patch all/Start all
– Minimize down time: Stop/Patch all but one, Stop
last, Start all down, Patch last/Start last
– Rolling patch: Stop/Patch/Start one at a time
– Local patch: Stop/Patch/Start only one
• How does OPatch select which method to use: a
as
If (users specify -local | -local_node) )h
ฺbr
patching mechanism = Local
c m
o deฺ
else if (users specify -minimize_downtime) i ฺ
d Gui
r e
patching mechanism = Min. Downtime
@ sic dent
else if (patch is a rolling patch)
e r de Stu
i lav e this
patching mechanism = Rolling
v
i o_ o us
else patching mechanism = All-node
ic
( f abr nse t
e rde lCopyright
l a V ab e
i
VUsingsRAC:
f er Overview
i
Patch Upgradeo
ic -tfour andifferent patch methods on a RAC environment:
b r r
Fa non
OPatch supports
• Patching RAC as a single instance (all-node patch): In this mode, OPatch applies the patch to the
local node first, then propagates the patch to all other nodes, and finally updates the inventory.
All instances are down during the entire patching process.
• Patching RAC using a minimum down-time strategy (min. downtime patch): In this mode,
OPatch patches the local node, asks users for a subset of nodes, which will be the first nodes to
be patched. After the initial subset of nodes are patched, OPatch propagates the patch to the
other nodes and finally updates the inventory. The down time happens between the shutdown of
the second subset of nodes and the startup of the initial subset of nodes patched.
• Patching RAC using a rolling strategy (rolling patch): With this method, there is no down time.
Each node is patched and brought up while all the other nodes are up and running, resulting in
no disruption of the system.
• The OPatch strategies discussed above presume that all nodes are patched at the same time.
Additionally, each node can be patched individually, at different times, using the –local, or –
local_node key words, which patch only the local node, or the remote node.
When executing the opatch apply command, the slide shows you which method is used.
Note: Currently, OPatch treats a shared file system as a single-instance patch. This means that
OPatch cannot take advantage of a rolling patch in this case because all nodes must be down.
Rolling Patch Upgrade Using RAC
Clients 1 Clients 2
Oracle
A B Patch patch
upgrades
Operating
Initial RAC configuration Clients on A , patch B system
upgrades a
as
4 Clients Clients 3
ฺbr )h
c o m eฺ
Hardware
ฺ
di upgrades
uid
Patch
s icre ent G
r d e@ Stud
i l a ve this
Upgrade complete cio_ Clients
v seon B , patch A
r i t o u
b
(fa ense
e ic © 2008, Oracle. All rights reserved.
rd lCopyright
l
e
V ab e
i l a
V sfeUsing r
Rolling Patch
i c i o Upgrade
a n RAC
r
b supported
Thisais r
-t only for single patches that are marked as rolling upgrade compatible.
F o n
n patching allows the interoperation of a patched node and an unpatched node
Rolling RAC
simultaneously. This means only one node is out of commission while it is patched.
Using the OPATCH tool to apply a rolling RAC patch, you are prompted to stop the instances on the
node to be patched. First, the local node is patched, and then you are asked for the next node to patch
from a list. As each node is patched, the user is prompted when it is safe to restart the patched node.
The cycle of prompting for a node, of stopping the instances on the node, of patching the node, and
of restarting the instances continues until you stop the cycle or until all nodes are patched. After you
download the patch to your node, you need to unzip it before you can apply it. You can determine
whether the patch is flagged as rolling upgradable by checking the
Patch_number/etc/config/inventory file. Near the end of that file, you must see the
following mark: <online_rac_installable>true</online_rac_installable>
It is important to stress that although rolling patch upgrade allows you to test the patch before
propagating it to the other nodes, it is preferable to test patches in a test environment rather than
directly on your production system.
Note: Some components cannot be changed one node at a time. The classic example is the data
dictionary. Because there is only a single data dictionary, all instances need to be shut down. In these
cases, Oracle Data Guard and physical standby databases are the recommended solutions. Of course,
using Online Patching is the recommended solution for avoiding downtime when an
online patch is available.
Download and Install Patch Updates
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er Updates
VInstallsPatch
f
Download and i o
c tran
ithe
b r
Refer to n-
Fa noOracleMetaLink Web site for required patch updates for your installation.
To download the required patch updates:
1. Use a Web browser to log in to the OracleMetaLink Web site: http://metalink.oracle.com.
2. On the main OracleMetaLink page, click Patches and Updates.
3. On the Patches & Updates page, click Advanced Search.
4. On the Advanced Search page, click the search icon next to the Product or Product Family field.
5. In the Search field, enter RDBMS Server and click Go. Select RDBMS Server under the
Results heading and click Select. RDBMS Server appears in the Product or Product Family
field. The current release appears in the Release field.
Download and Install Patch Updates
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er Updates (continued)
VInstallsPatch
f
i
Download ando an from the list in the Platform or Language field, and click Go. Any
6.ab
ric -trplatform
Select your
F non
available patch updates appear under the Results heading.
7. Click the number of the patch that you want to download.
8. On the Patchset page, click View README and read the page that appears. The README
page contains information about the patchset and how to apply the patches to your installation.
9. Return to the Patchset page, click Download, and save the file on your system.
10. Use the unzip utility provided with Oracle Database 11g to uncompress the Oracle patch
updates that you downloaded from OracleMetaLink. The unzip utility is located in the
$ORACLE_HOME/bin directory.
Rolling Release Upgrade Using SQL Apply
Clients 1 Clients Logs 2

queue
Logs
ship Patch set
upgrades
Version n Version n Version n Version n+1 Major

Initial SQL Apply setup Upgrade standby site release
upgrades a
as
)h
4 Clients Clients 3
Logs Logs
m ฺ
Cluster
br
ship ship
i ฺ c o d e ฺ
red t Gand u i
software
i c
s den hardware
@
de Stu
e r upgrades
Version n+1 Version n+1
v is
ilaVersion tnh Version n+1
_ v e
Switchover, upgrade standby
r ic io oRun
t us mixed to test
( f ab nse
e rde lCopyright
l a V ab e
i
V Upgrade f erUsing SQL Apply
i o
Rolling Release s
an Database 10.1.0.3 and continuing with later versions, a logical standby
icwith-Oracle
b r t r
Fa nonoperate at a later Oracle release than on the primary when using SQL Apply to execute
Beginning
database can
a rolling database upgrade. For example, using SQL Apply and logical standby databases, you are
able to upgrade the Oracle database software from patchset release 10.2.0.n to the next database
10.2.0.(n+1) patchset release. Step 1 in the slide shows the Data Guard configuration before the
upgrade begins, with the primary and logical standby databases both running the same Oracle
software version. In step 2, you stop SQL Apply and upgrade the Oracle database software on the
logical standby database to version n+1. During the upgrade, redo data accumulates on the primary
system. In step 3, you restart SQL Apply, and the redo data that was accumulating on the primary
system is automatically transmitted and applied on the newly upgraded logical standby database. The
Data Guard configuration can run the mixed versions for an arbitrary period. In step 4, you perform a
switchover. Then you activate the user applications and services on the new primary database. Before
you can enable SQL Apply again, you need to upgrade the new standby site. This is because the new
standby site does not understand new redo information. Finally, you raise the compatibility level on
each database.
Note: SQL Apply does not support all data types. This can prevent you from using this method. A
possible alternative is to use Oracle Streams in this case.
Database High Availability: Best Practices
Use SPFILE. Create two or Set Multiplex

more control CONTROL_FILE_RECO production and
files. RD_KEEP_TIME long standby redo
enough. logs
Log checkpoints Use auto-tune Enable ARCHIVELOG Enable
to the alert log. checkpointing. mode and use a flash Flashback
recovery area. Database.
Enable block Use Automatic Use locally managed Use Automatic s a

checking. Undo tablespaces. )h
Segment Space
a
Management. Management. m ฺbr
i ฺ c o deฺ
Use resumable Use Database Register all instances r e u i
d UseGtemporary
space allocation. Resource with remote listeners.
@ sic dentablespaces.
t
Manager.
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er Best Practices
VAvailability:
f
i
Database High o s
angives you a short summary of the recommended practices that apply to single-
b ricin the-slide
Theatable t r
F non
instance databases, RAC databases, and Data Guard standby databases.
These practices affect the performance, availability, and mean time to recover (MTTR) of your
system. Some of these practices may reduce performance, but they are necessary to reduce or avoid
outages. The minimal performance impact is outweighed by the reduced risk of corruption or the
performance improvement for recovery.
Note: For more information about how to set up the features listed in the slide, refer to the following
documents:
• Administrator’s Guide
• Data Guard Concepts and Administration
• Net Services Administrator’s Guide
How Many ASM Disk Groups per Database
• Two disk groups are recommended.

– Leverage maximum of LUNs. Data DG FRA DG
– Backups can be stored on one ERP DB
FRA disk group. CRM DB
– Lower performance may be HR DB
used for FRA (or inner tracks).

• Exceptions: s a
)h a
– Additional disk groups for different capacity or ฺbr
performance characteristics c o m eฺ
ฺ
di Guid
– Different ILM storage tiers ic r e t
@ s den
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er per Database
V DisksGroups
f
How Many iASM o
c tran
ithe
b r
Fa notime,
Most of n- only two disk groups are enough to share the storage between multiple databases.
That way you can maximize the number of Logical Unit Numbers (LUNs) used as ASM disks, which
gives you the best performance, especially if these LUNs are carved on the outer edge of your disks.
Using a second disk group allows you to have a backup of your data by using it as your common
flash recovery area (FRA). You can put the corresponding LUNs on the inner edge of your disks
because less performance is necessary.
The two noticeable exceptions to this rule are whenever you are using disks with different capacity or
performance characteristics, or when you want to archive your data on lower-end disks for
Information Lifecycle Management (ILM) purposes.
Database Storage Consolidation
• Shared storage across several databases:

– RAC and single-instance databases can use the same
ASM instance.
• Benefits:
– Simplified and centralized management
– Higher storage utilization
s a
– Higher performance
)h a
m ฺbr
Payroll GL i ฺ
Payroll c o& GLdeฺ
r e d Gui
@ sic dent
… …
e r de Stu …
10 ° 50 GB 10 ° 50 GB _vi
lav e this 10 ° 100 GB
r ic io o us
( f ab nse t
e rde lCopyright
l a V ab e
i er
V Consolidation
f
i o
Database Storage
icDatabase n
a10gs
b r - t r
Fa non
In Oracle Release 2 and later, Oracle Clusterware does not require an Oracle Real
Application Clusters license. Oracle Clusterware is now available with ASM and single-instance
Oracle Database 11g allowing support for a shared clustered pool of storage for RAC and single-
instance Oracle databases.
This allows you to optimize your storage utilization by eliminating wasted, over-provisioned storage.
This is illustrated in the slide, where instead of having various pools of disks used for different
databases, you consolidate all that in one single pool shared by all your databases.
By doing this, you can reduce the number of LUNs to manage by increasing their sizes, which gives
you a higher storage utilization as well as a higher performance.
Note: RAC and single-instance databases could not be managed by the same ASM instance in Oracle
Database 10g Release 1.
Which RAID Configuration for Best Availability?
A. ASM mirroring
B. Hardware RAID 1 (mirroring)
C. Hardware RAID 5 (parity protection)
D. Both ASM mirroring and hardware RAID
Depends on business requirement and budget s a

Answer: h a
ฺbr)
(cost, availability, performance, and utilization)
m eฺ
ฺ c o
r e di Guid
@ sic dent
ASM leverages hardware e r de RAID. S tu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer for Best Availability?
Which RAID i oConfiguration
anyou have multiple choices as shown in the slide.
To a b ricavailability,
favor - t r
F n on
You could just use ASM mirroring capabilities, or hardware RAID 1 (Redundant Array of
Inexpensive Disks) which is a hardware mirroring technique, or hardware RAID 5. The last possible
answer, which is definitely not recommended, is to use both ASM mirroring and hardware mirroring.
Oracle recommends the use of external redundancy disk groups when using hardware mirroring
techniques to avoid an unnecessary overhead.
Therefore, between A, B, and C, it depends on your business requirements and budget.
RAID 1 has the best performance but requires twice the storage capacity. RAID 5 is a much more
economical solution but with a performance penalty essentially for write-intensive workloads.
Should You Use RAID 1 or RAID 5?
RAID 1 (Mirroring) RAID 5 (Parity)

• Recommended by Oracle • DSS and moderate OLTP
• Most demanding applications
Pros: Pros:
• Best redundancy • Requires less capacity
• Best performance Cons:
• Low recovery overhead • Less redundancy
Cons: s a
• Less performance h a
• Requires higher capacity ฺbr)
• High recovery overhead
m eฺ
ฺ c o
r e di Guid
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V RAIDsf1eorr RAID 5?
Should YouioUse
b r - t r an technique. Mirroring involves taking all writes issued to a given disk and
iisca mirroring
Fa nothen write to another disk. In this way, if there is a failure of the first disk, the second disk,
RAID 1
duplicating
or mirror, can take over without any data loss.
The goal of the RAID-5 design is to provide a reliable, high-performance array of disks with the
minimum amount of redundant hardware. RAID 5 is based on the use of parity protection across
several drives in order to provide protection against disk failure. The RAID-5 configuration is
essentially a striped configuration, like RAID 0, with an additional disk added to cater to the
additional storage needed for the parity information. With the data striped across the drives in this
way, the read performance of RAID 5 is comparable to that of RAID 0. RAID-5 writes, on the other
hand, are almost legendary for their poor performance.
The slide lists the pros and cons of using both techniques, and although Oracle recommends using
RAID 1, you need to take into account that you have to double the number of your disks to store the
same amount of data. The general rule of thumb is to deploy RAID 5 where cost of storage is critical
and performance is not the primary goal, and for applications with primary read operations such as
data warehouse applications. The Flash Recovery Area disk group can be another good use of RAID
5, where the storage capacity requirement is the highest and predominantly sequential I/O.
Note: The ORION tools (http://www.oracle.com/technology/software/index.html#util) can be used to
test and determine the pros and cons of storage arrays for your application.
Should You Use ASM Mirroring Protection?
• Best choice for low-cost storage

• Enables extended clustering solutions
• No hardware mirroring
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
V ASMsMirroring
f
Should YouioUse n Protection?
r i c t r a
Fab nleverage
Basically,
on- the storage array hardware RAID-1 mirroring protection when possible to offload
the mirroring overhead from the server. Use ASM mirroring in the absence of a hardware RAID
capability.
However hardware RAID 1 in most Advanced Technology Attachment (ATA) storage technologies
is inefficient and degrades the performance of the array even more. Using ASM redundancy has
proven to deliver much better performance in ATA arrays.
Because the storage cost can grow very rapidly whenever you want to achieve extended clustering
solutions, ASM mirroring should be used as an alternative to hardware mirroring for low-cost storage
solutions.
Note: For more information about the Oracle Resilient Low-cost Storage Initiative, see the Web site
at: http://www.oracle.com/technology/deploy/availability/htdocs/lowcoststorage.html.
What Type of Striping Works Best?
A. ASM only striping (no RAID 0)

B. RAID 0 and ASM striping
C. Use LVM
D. No striping
s a
Answer: A and B )h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
ASM and RAID stripingvare e rdecomplementary.
S tu
v i la e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfWorks er Best?
o
What Type iof Striping
anyou can use ASM striping only, or you can use ASM striping in combination
As a b ricin the-tslide,
shown r
F non
with RAID 0.
With RAID 0, multiple disks are configured together as a set, or a bank, and data from any one data
file is spread, or striped, across all the disks in the bank.
Combining both ASM striping and RAID striping is called stripe-on-stripe. This combination offers
good performance too.
However, there is no longer a need to use a Logical Volume Manager (LVM) for your database files,
nor it is recommended to not use any striping at all.
ASM Striping Only
Pros: Cons:
• Drives evenly distributed for Data & FRA • Not well balanced across
• Higher bandwidth ALL disks
• Allows small incremental growth (73 GB) • LUN size limited to disk size
• No drive contention
Oracle DB size: 1 TB
Data DG FRA DG
Storage configuration:
1 TB 16°73 GB 8°arrays with 2 TB a
32°73 GB
12°73 GB disks per array as
LUNs
)hLUNs
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
@ sic dent
RAID 1
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
ASM Striping i o Only
ic shown anthis slide, you want to store a one-terabyte database with a corresponding two-
b r - t r
Fa non
In the case in
terabyte flash recovery area. You use RAID 1 to mirror each disk. In total, you have eight arrays of
twelve disks, with each disk being 73 GB. ASM mirroring and hardware RAID 0 are not used.
In addition, each ASM disk is represented by one entire LUN of 73 GB. This means that the Data
disk group (DG) is allocated 16 LUNs of 73 GB each.
On the other side, the Flash Recovery Area disk group is assigned 32 LUNs of 73 GB each.
This configuration enables you to evenly distribute disks for your data and backups, achieving good
performance and allowing you to manage your storage in small incremental chunks.
However, using a restricted number of disks in your pool does not balance your data well across all
your disks. In addition, you have many LUNs to manage at the storage level.
Hardware RAID–Striped LUNs
Pros: Cons:
• Fastest region for Data DG • Large incremental growth
• Balanced data distribution • Data & FRA “contention”
• Fewer LUNs to manage while max
spindles
Data DG FRA DG
1 TB 4°250 GB 8°arrays with 2 TB a
4°500 GB
LUNs
)hLUNs
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
sic dent
RAID 0+1
@
de Stu
e r
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfeLUNs r
i o
Hardware RAID–Striped
ic shown anthis slide, you want to store a one-terabyte database with a corresponding two-
b r - t r
Fa non
In the case in
terabyte flash recovery area. You use RAID 0+1, which is a combination of hardware striping and
mirroring to mirror and stripe each disk. In total, you have eight arrays of twelve disks, with each
disk being 73 GB. ASM mirroring is not used.
Here, you can define bigger LUNs not restricted to the size of one of your disk. This allows you to
put the Data LUNs on the fastest region of your disks, and the backup LUNs on slower parts. By
doing this, you achieve a better data distribution across all your disks, and you end up managing a
significantly less number of LUNs.
However, you must manipulate your storage in much larger chunks than in the previous
configuration.
Note: The hardware stripe size you choose is also very important because you want 1 MB alignment
as much as possible to keep in synch with ASM AUs. Therefore, selecting power-of-two stripe sizes
(128 KB or 256 KB) is better than selecting odd numbers. Storage vendors typically do not offer
many flexible choices depending on their storage array RAID technology and can create unnecessary
I/O bottlenecks if not carefully considered.
Hardware RAID–Striped LUNs HA
Pros: Cons:
• Fastest region for Data DG • Large incremental growth
• Balanced data distribution • Might waste space
• Fewer LUNs to manage
• More high available
Data DG FRA DG
1 TB 2°500 GB 8°arrays with 1.6 TB a
2°800 GB
LUNs
)h
LUNs
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
sic dent
RAID 0+1
@
de Stu
e r
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfeLUNs r
i c i o
Hardware RAID–Striped
a n HA
r t r
Fabcasenshown
In the
on- in this slide, you want to store a one-terabyte database with a corresponding 1.6-
TB flash recovery area. You use RAID 0+1, which is a combination of hardware striping and
mirroring to mirror and stripe each disk. In total, you have eight arrays of twelve disks, with each
disk being 73 GB. ASM mirroring is not used.
Compared to the previous slide, you use bigger LUNs for both the Data disk group and the Flash
Recovery Area disk group. However, the presented solution is more highly available than the
previous architecture because you separate the data from the backups into different arrays and
controllers to reduce the risk of down time in case one array fails.
By doing this, you still have a good distribution of data across your disks, although not as much as in
the previous configuration. You still end up managing a significantly less number of LUNs than in
the first case.
However, you might end up losing more space than in the previous configuration. Here, you are
using the same size and number of arrays to be consistent with the previous example.
It Is Real Simple
• Use external RAID protection when possible.

• Create LUNs by using:
– Outside half of disk drives for highest performance
– Small disk, high rpm (that is, 73 GB/15k rpm)
• Use LUNs with the same performance characteristics.
• Use LUNs with the same capacity. a
h as
• Maximize the number of spindles in your disk group.
r)
m ฺb
i ฺ c o deฺ
r e d Gui
s ic ent
Oracle Database 11g and ASM d e do
@ t uthe
d rest!
r
ve this S
i l a
i o _v use
a b ric e to
e (f ens
e d
r lCopyright
l a V ab
i
V sfer
i o
It Is Real Simple
an11g ASM for volume and file management to equalize the workload across
b ric Database
UseaOracle - t r
F non
disks and eliminate hot spots. The following are simple guidelines and best practices when
configuring ASM disk groups:
• Use external RAID protection when possible.
• Create LUNs using:
- Outside half of disk drives for highest performance
- Small disk with high rpm (for example, 73 GB with 15k rpm). The reason why spindle
(platter) speed is so important is that it directly impacts both positioning time and data
transfer. This means that faster spindle speed drives have improved performance regardless
of whether they are used for many small, random accesses, or for streaming large
contiguous blocks from the disk. The stack of platters in a disk rotates at a constant speed.
The drive head, while positioned close to the center of the disk, reads from a surface that is
passing by more slowly than the surface at the outer edges.
• Maximize the number of spindles in your disk group.
• LUNs provisioned to ASM disk groups should have the same storage performance and
availability characteristics. Configuring mixed speed drives will default to the lowest common
denominator.
• ASM data distribution policy is capacity based. Therefore, LUNs provided to ASM should have
the same capacity for each disk group to avoid imbalance and hot spots.
Extended RAC: Overview
• Full utilization of resources, no matter where they are

located
Site A RAC Site B

database
s a
Clients
)h a
m ฺbr
i ฺ c o deฺ
Site A RAC
r e d GuSite i B
database
@ sic dent
e r de Stu
• Fast recovery from site failure v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
VOverview f er
i
Extended RAC: o s
an share a single set of storage and are located on servers in the same data
icRAC-tdatabases
b r r
Fa non
Typically,
center.
With extended RAC, you can use disk mirroring and Dense Wavelength Division Multiplexing
(DWDM) equipment to extend the reach of the cluster. This configuration allows two data centers,
separated by up to 100 kilometers, to share the same RAC database with multiple RAC instances
spread across the two sites.
As shown in the slide, this RAC topology is very interesting, because the clients’ work gets
distributed automatically across all nodes independently of their location, and in case one site goes
down, the clients’ work continues to be executed on the remaining site. The types of failures that
extended RAC can cover are mainly failures of an entire data center due to a limited geographic
disaster. Fire, flooding, and site power failure are just a few examples of limited geographic disasters
that can result in the failure of an entire data center.
Note: Extended RAC does not use special software other than the normal RAC installation.
Extended RAC Connectivity
• Distances over ten kilometers require dark fiber.

• Set up buffer credits for large distances.
Dark fiber
Site A Site B
DWDM DWDM
device device
s a
)h a
m ฺbr
DBdiฺc
o deฺ
DB
r e u i
copy
sic dent
copy G
@
de network tu
e r S
Clients v i lav e this
Public
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
VConnectivity
f er
Extended RAC i o
itoc extend n
aRACs
b r - t r
Fa non
In order a cluster to another site separated from your data center by more than ten
kilometers, it is required to use DWDM over dark fiber to get good performance results.
DWDM is a technology that uses multiple lasers, and transmits several wavelengths of light
simultaneously over a single optical fiber. DWDM enables the existing infrastructure of a single fiber
cable to be dramatically increased. DWDM systems can support more than 150 wavelengths, each
carrying up to 10 Gbps. Such systems provide more than a terabit per second of data transmission on
one optical strand that is thinner than a human hair.
As shown in the slide, each site should have its own DWDM device connected together by a dark
fiber optical strand. All traffic between the two sites is sent through the DWDM and carried on dark
fiber. This includes mirrored disk writes, network and heartbeat traffic, and memory-to-memory data
passage. Also shown on the graphic are the sets of disks at each site. Each site maintains a copy of
the RAC database.
It is important to note that depending on the site’s distance, you should tune and determine the
minimum value of buffer credits in order to maintain the maximum link bandwidth. Buffer credit is a
mechanism defined by the Fiber Channel standard that establishes the maximum amount of data that
can be sent at any one time.
Note: Dark fiber is a single fiber optic cable or strand mainly sold by telecom providers.
Extended RAC Disk Mirroring
• Need copy of data at each location

• Two options:
– Host-based mirroring
– Remote array-based mirroring
Recommended solution with ASM
Site A Site B Primary Secondary a
as
)hฺbr
c m
o deฺ
i ฺ
d Gui
r e
@ sic dent
DB DB e r deDB Stu DB
l a v h i s
copy copy
o _ vi se t copy copy
b r ici to u
e (fa ense
rd lCopyright
l a V ab
i
VDisk Mirroring
f er
Extended RAC i o s
an one RAC database, each data center has its own set of storage that is
icthere -istronly
b r
Fa non
Although
synchronously mirrored using either a cluster-aware, host-based Logical Volume Manager (LVM)
solution, such as SLVM with MirrorDiskUX, or an array-based mirroring solution, such as EMC
SRDF.
With host-based mirroring (shown on the left of the slide), the disks appear as one set, and all I/Os
get sent to both sets of disks. This solution requires closely integrated clusterware and LVM, and
ASM is the recommended solution.
With array-based mirroring, shown on the right, all I/Os are sent to one site and are then mirrored to
the other. In fact, this solution is like a primary/secondary site setup. If the primary site fails, all
access to primary disks is lost. An outage may be incurred before you can switch to the secondary
site.
Note: With extended RAC, it is critical to design the cluster in a manner that ensures that the cluster
can achieve quorum after a site failure. For more information, refer to the Oracle Technology
Network site.
Achieving Quorum with Extended RAC
Third
site
Voting disk (NFS or iSCSI)
Redundant public network
Redundant private network

s a
)h a
r B
ฺbSite
Site A
c m
o deฺ
Redundant SAN with ASM mirroring i ฺ
d Gui
RAC
r e
ic eRAC t
@ s n
database
r d e Stud database
Database files (ASM failure group)

i l a ve thDatabase is files (ASM failure group)
OCR
o v
_ us e OCR
Voting Disk
r ic i o Voting Disk
b t
e (fa ense
rd lCopyright
l a V ab
i
V with f r
eExtended
i c i o
Achieving Quorum
a n s RAC
r r
t to access strictly more than half of the voting disks at any time; otherwise, that
Fab must
A node
n o be-able
n
node will be evicted from the cluster. Extended clusters are generally implemented with only two
storage systems, one at each site. This means that the site that houses the majority of the voting disks
is a potential single point of failure for the entire cluster. To prevent this potential outage, Oracle
Clusterware supports a third voting disk on an inexpensive low-end standard NFS mounted device
somewhere on the network. It is thus recommended to put this third NFS voting disk on a dedicated
server that is visible from both sites. This situation is illustrated in the slide. The goal is for each site
to run independently of the other when a site failure occurs.
Note: For more information about NFS configuration of the third voting disk, refer to the Oracle
Technology Network site.
ASM Preferred Mirror Read: Overview
Site A Site B
P S
Site A Site B
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
P: Primary AUerd
e Stu
P S
l a v AU this
o _ vi se
S: Secondary
b r ici to u
e (fa ense
rd lCopyright
l a V ab
i
VMirrorsRead:
f er Overview
i
ASM Preferredo
icconfigure anASM failure groups in Oracle Database 11g, ASM always reads the primary
b r - t r
Fa non
When you
copy of a mirrored extent. It may be more efficient for a node to read from a failure group extent that
is closest to the node, even if it is a secondary extent. This is especially true in extended cluster
configurations (when nodes are spread across several sites) where reading from a local copy of an
extent provides improved performance.
With Oracle Database 11g, you can do this by configuring the preferred mirror read by using the new
ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter to specify a list of
preferred mirror read names. The disks in those failure groups become the preferred read disks. Thus,
every node can read from its local disks. This results in higher efficiency and performance as well as
reduced network traffic. The setting for this parameter is instance specific.
ASM Preferred Mirror Read: Setup
Setup
On first instance
ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITEA
On second instance
ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITEB
Monitor s a
)h a
m ฺbr
SELECT preferred_read FROM v$asm_disk;
i ฺ c o deฺ
r e d Gui
SELECT * FROM v$asm_disk_iostat; @ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er Setup
VMirrorsRead:
f
i
ASM Preferred o an set the new ASM_PREFERRED_READ_FAILURE_GROUPS initialization
To a b ric this-trfeature,
configure
F on parameter is a multivalued parameter and should contain a string with a list of failure
nThis
parameter.
group names separated by commas. Each failure group name specified should be prefixed with its
disk group name and a “.” character. This parameter is dynamic and can be modified using the
ALTER SYSTEM command at any time. An example is shown in the slide. However, this
initialization parameter is valid only for ASM instances. With the extended cluster, the failure groups
specified in this parameter should contain only those disks that are local to the corresponding
instance.
The new column PREFERRED_READ has been added to the V$ASM_DISK view. Its format is a
single character. If the disk group to which the disk belongs pertains to a preferred read failure group,
the value of this column is Y.
To identify specific performance issues with the ASM preferred read failure groups, use the
V$ASM_DISK_IOSTAT view. This view displays the disk input/output (I/O) statistics for each
ASM client. If this view is queried from a database instance, only the rows for this instance are
shown.
Enterprise Manager ASM Configuration Page
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V ASM f erConfiguration Page
o
Enterprise iManager
c tran s
b rispecify
Youacan - a set of disks as preferred disks for each ASM instance by using Enterprise
F non
Manager. The preferred read attributes are instance specific. In Oracle Database 11g, the Preferred
Read Failure Groups field (asm_preferred_read_failure_group) is added to the
configuration page.
This parameter takes effect only before the disk group is mounted or when the disk group is created.
It applies only to newly opened files or to a newly loaded extent map for a file.
ASM Preferred Mirror Read: Best Practice
Two sites: Normal redundancy Two sites: High redundancy
P S P S S S S P
Only two failure groups: One for each instance Max four failure groups: Two for each instance
Three sites: High redundancy s a

)h a
m ฺbr
i ฺ c o deฺ
P S S ed
r u i
sic dent G
@
de instance tu
Only three failure groups: One for e r
each S
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
VMirrorsRead:
f er Best Practice
ASM Preferredi o
ic there-tareanonly a limited number of good disk group configurations in an extended cluster.
b r r
Fa non
In practice,
A good configuration takes into account both performance and availability of a disk group in an
extended cluster. Here are some possible examples:
For a two-site extended cluster, a normal redundancy disk group should have only two failure groups;
all disks local to one site should belong to the same failure group. Also, no more than one failure
group should be specified as a preferred read failure group by each instance. If there are more than
two failure groups, ASM may not mirror a virtual extent across both sites. Furthermore, if the site
with more than two failure groups were to go down, it would take the disk group down as well. If the
disk group to be created is a high-redundancy disk group, at most two failure groups should be
created on each site with its local disks, with both local failure groups specified as preferred read
failure groups for the local instance.
For a three-site extended cluster, a high redundancy disk group with three failure groups should be
used. In this way, ASM can guarantee that each virtual extent has a mirror copy local to each site and
that the disk group is protected against a catastrophic disaster on any of the three sites.
Additional Data Guard Benefits
• Greater disaster protection

– Greater distance
– Additional protection against corruptions
• Better for planned maintenance
– Full rolling upgrades
• More performance neutral at large distances a
as
– Option to do asynchronous )h
ฺ br
• If you cannot handle the costs of a DWDM network,
c m
o deฺ
Data Guard still works over cheap, standard i ฺ
d networks.
u i
r e G
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
VGuardsfBenefits
o
Additional iData
ic provides ana greater disaster protection:
b r - t r
Fa non
Data Guard
• Distance over 100 kilometers without performance hit
• Additional protection against corruptions because it uses a separate database
• Optional delay to protect against user errors
Data Guard also provides better planned maintenance capabilities by supporting full rolling upgrades.
Also, if you cannot handle the costs of a DWDM network, Data Guard still works over inexpensive,
standard networks.
Using a Test Environment
• The most common cause of down time is change.

• Test your changes on a separate test cluster before
changing your production environment.
Production Test s a
cluster cluster )h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC
r d e RAC S tu
database e database
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
Using a Testi oEnvironment
icthe most n
alikely
b r - t r
Fa non
Change is cause of down time in a production environment. A proper test
environment can catch more than 90 percent of the changes that could lead to a down time of the
production environment, and is invaluable for quick test and resolution of issues in production.
When your production environment is RAC, your test environment should be a separate RAC cluster
with all the identical software components and versions.
Without a test cluster, your production environment will not be highly available.
Note: Not using a test environment is one of the most common errors seen by Oracle Support
Services.
Summary
In this lesson, you should have learned how to:

• Design a Maximum Availability Architecture in your
environment
• Determine the best RAC and Data Guard topologies for
your environment
• Configure the Data Guard Broker configuration files in a
a RAC environment h as
ฺ b r)
• Decide on the best ASM configuration to use om ฺ
i ฺ c i d e
red Gu
• Patch your RAC system in a rolling fashion
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
Note i o
c tran
ino
b r
Fa nopractice
There is n- for this lesson.
Appendix A
Practices and Solutions
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Table of Contents
Practices for Lesson 1 ......................................................................................................... 4
Practice 1-1: Oracle Clusterware Installation ................................................................. 5
Practices for Lesson 2 ....................................................................................................... 11
Practice 2-1: ASM Installation and Configuration ....................................................... 12
Practice 2-2: Database Software Installation ................................................................ 15
Practice 3-1: Create a Cluster Database........................................................................ 18
Practice 4-1: Add/Remove Redo Log Groups in a RAC Environment ........................ 21
Practice 5-1: Backup and Recovery.............................................................................. 27
a
s
Practice 6-1: ADDM and RAC, Part I .......................................................................... 31 )h a
ฺbr
Practice 6-2: ADDM and RAC, Part II......................................................................... 40
m
c o deฺ
Practice 6-3: ADDM and RAC, Part III........................................................................ 47
i ฺ
e d Gui
r
sic dent
Practice 7-1: Manage Services...................................................................................... 55
@
e r de Stu
Practice 7-2: Monitor Services...................................................................................... 65
lav e this
Practice 7-3: Alert Thresholds and Services................................................................. 71
v i
i o_ o us
Practice 8-1: Create a Server-Side Callout ................................................................... 76
ic
( f abr nse t
Practice 8-2: Use Load Balancing Advisory................................................................. 85
rde le lice
Practice 8-3: Use Transparent Application Failover................................................... 107
e
Practices for Lesson 9 ..................................................................................................... 120
V ab
l a
Practice 9-1: Mirror the OCR ..................................................................................... 121
i
V sfer
i o
Practice 9-2: OCR Backup and Restore...................................................................... 127
ic -tran
b r Practice 9-3: Multiplex Your Voting Disk.................................................................. 135
Fa non Practice 9-4: Protect Xclock with Oracle Clusterware ............................................... 137
Practices for Lesson 10 ................................................................................................... 149
Practice 10-1: Diagnosing Oracle Clusterware Components...................................... 150
Practice 10-2: Fixing Oracle Clusterware issues ........................................................ 159
Practice 11-1: Remove the Second Instance............................................................... 187
Practice 11-2: Clean Up ASM .................................................................................... 188
Practice 11-3: Remove the Listener............................................................................ 191
Practice 11-4: Remove the Database Software from the Second Node...................... 194
Practice 11-5: Remove the ASM Software from the Second Node............................ 198
Practice 11-6: Remove the Second Node from the OCR............................................ 202
Practice 11-7: Remove the Oracle Clusterware Software from the Second Node ..... 203
Practice 11-8: Check Prerequisites Before Oracle Clusterware Installation .............. 211
Practice 11-9: Add Oracle Clusterware to the Second Node...................................... 214
Practice 11-10: Configure ONS for the Second Node ................................................ 218
Practice 11-11: Add ASM Software to the Second Node........................................... 219
Practice 11-12: Add a Listener to the Second Node ................................................... 222
Oracle Database 11g: RAC Administration A - 2

Practice 11-13: Add Database Software to the Second Node..................................... 225
Practice 11-14: Add a Database Instance to the Second Node ................................... 228
Optional Practice: Direct NFS .................................................................................... 233
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practices for Lesson 1
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 1-1: Oracle Clusterware Installation
For most of these practice exercises, you use various Oracle graphical tools (OUI,
DBCA, EMCA, and so on), and telnet/ssh sessions. Your instructor will give you details
about your node names and accounts, as well as your database and your cluster name.
Notes:
• When working on these practices, you must use the provided values to prevent
interference with other students or other classes.
• The solutions provided are not necessarily based on your assigned account. You
should thus substitute your designated user, database, instance, host names, and
related information as appropriate.
In this practice, you set up user equivalence for the oracle user employing Secure Shell
(ssh). The second step entails checking the readiness of the cluster for a Clusterware s a
installation. This is done using the CLUVFY utility. The third and final step of this )h a
ฺbr
practice is the actual installation of the Oracle Clusterware software. Install the software
m
in the /u01/crs11g directory as the oracle user. The install group should be
i ฺ c o deฺ
oinstall. r e d Gui
1) Using a telnet session, connect as user oracle to your @ sicRAC
first d e n
node.
t
You need to
configure secure shell (ssh) on both nodes. To r
dod ethis, navigate
S t u to
a v e i s
$HOME/solutions/less01 and execute
v i l the
e t h
ssh_setup.sh script. When
_
cioto logtoinutosyour second node. You should not be
finished, test the setup by using ssh
prompted for a password. bri
e (fa ense
e rd lesson1]$
e l ic ./ssh_setup.sh
[oracle@vx0301
l a l
V ab of host 'vx0302 (10.216.4.13)' can't be
i
V sfer
The authenticity
i o anfingerprint is
established.
icRSA -key
b r t r
Fa nab:33:1b:a6:89:53:ba:24:77:76:90:e2:7a:1c:79:11.
onAre you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'vx0302,10.216.4.13' (RSA) to the
list of known hosts.
oracle@vx0302's password:
oracle@vx0302's password:
[oracle@vx0301 lesson1]$ ssh vx0302
Last login: Fri Nov 16 14:35:30 2007 from vx0301.us.oracle.com
[oracle@vx0302 ~]$ exit
[oracle@vx0301 lesson1]$
2) Run CLUVFY from /stage/db to make sure that the Clusterware minimum
requirements are met on both nodes before beginning the installation:
[oracle@vx0301 lesson1]$ /stage/db/runcluvfy.sh stage -pre

crsinst -n vx0301,vx0302

Practice 1-1: Oracle Clusterware Installation (continued)
Performing pre-checks for cluster services setup
Checking node reachability...

Node reachability check passed from node "vx0301".
Checking user equivalence...

User equivalence check passed for user "oracle".
Checking administrative privileges...

User existence check passed for "oracle".
Group existence check passed for "oinstall".
Membership check for user "oracle" in group "oinstall" [as
Primary] passed.
Administrative privileges check passed. s a

)h a
Checking node connectivity...
m ฺbr
i ฺ c o withd e ฺ
Node connectivity check passed for subnet "10.216.4.0"
red t Gu i
node(s) vx0302,vx0301. i c
s den
@
de Stu
WARNING:
e r
Make sure IP address "10.216.102.142"
i l a v tis h isup and is a valid IP
address on node "vx0301".
i
Node connectivity checkcpassed o _v for u sesubnet "10.196.28.0" with
node(s) vx0302,vx0301.
a b ri e to
e (f ens
Interfaces e d
r lon
found
e l ic
subnet "10.216.4.0" that are likely
V
rab
ilaeth0:10.216.4.13
candidates for VIP:
V
vx0302
s f e
io raeth0:10.216.4.12
r i cvx0301
t n
Fab nInterfaces
on- found on subnet "10.196.28.0" that are likely
candidates for a private interconnect:
vx0302 eth2:10.196.30.23
vx0301 eth2:10.196.30.22
Checking system requirements for 'crs'...

Total memory check passed.
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for "make-3.80".
Package existence check passed for "binutils-2.15.92.0.2".
Package existence check passed for "gcc-3.4.5".
Package existence check passed for "libaio-0.3.105".
Package existence check passed for "libaio-devel-0.3.105".
Package existence check passed for "libstdc++-3.4.5".

Package existence check passed for "elfutils-libelf-devel-
0.97".
Package existence check passed for "sysstat-5.0.5".
Package existence check passed for "libgcc-3.4.5".
Package existence check passed for "libstdc++-devel-3.4.5".
Package existence check passed for "unixODBC-2.2.11".
Package existence check passed for "unixODBC-devel-2.2.11".
Package existence check passed for "glibc-2.3.4-2.19".
Group existence check passed for "dba".
User existence check passed for "nobody".
System requirement passed for 'crs'
Pre-check for cluster services setup was successful on all the

nodes. s a
)h a
m ฺbr
i ฺ c o deฺ
3) Execute runInstaller from the /stage/db directory and install
r e d Oracle u i
Clusterware. sic dent G
@
denode, setStthe
u value of
a) From a graphical terminal session on your first
e r
ORACLE_BASE=/u01/app/oracleland
i avexecute
t is
h/stage/db/runInstaller .
v
o_ o us e
ic i
[oracle@vx0301 lesson1]$( f abr nexports e t ORACLE_BASE=/u01/app/oracle
[oracle@vx0301
e rdelesson1]$
l i ce /stage/db/runInstaller
l a V able
i
V sfer
i o
icb) On-trtheanSelect a Product to Install screen, choose Oracle Clusterware and click
b r
Fa nonNext.
c) On the Specify Inventory directory and credentials page, ensure the inventory
directory path is /u01/app/oraInventory and the operating system group
name is oinstall. Click Next to continue.
d) On the Specify Home Details page, the Name should be OraCrs11g_home.
Do not accept the default location for the Path field. Enter /u01/crs11g in the
Path field. Click Next.
e) The Product-Specific Prerequisite Page should check the minimum
requirements and report no errors. Click Next to continue.

f) Enter the Cluster Name assigned to you by your instructor in the Cluster Name
field on the Specify Cluster Configuration page. The only node appearing in the
Cluster Nodes window is the installing (local) node. Click the Add button to add
the second node in your cluster. Enter the Public Node Name, the Private Node
Name, and the Virtual Host name, all in fully qualified format. Click OK to return
to the Specify Cluster Configuration page. Review the name information for the
second node and click Next.
g) On the Specify Network Interface Usage page, all network interfaces will be
marked private. The proper interface type designations are: eth0-Public, eth1-Do
Not Use, and eth2-Private. Select eth0 and click Edit. Change the interface type
to Public and click OK. Next, select eth1 (the storage network), and click Edit.
Change the interface type to Do Not Use and click OK. Leave eth2 as the Private
interface and click Next to continue.
s a
h) On the Specify OCR Location page, select External Redundancy and enter
)h a
/dev/sdb1 for the OCR location. Click Next when finished.
m ฺbr
i) Select External Redundancy on the Specify Voting Disk Location
i ฺ c o page dandeฺ
enter /dev/sdb5 for the Voting Disk location and click e
r d Gui
Next.
j) Review the information on the Summary page and @ sicInstall.
click d e n t
Monitor the
progress on the Install page. e
rd s S t u
v e
k) When the Execute Configuration scripts v i la dialog
e t hi appears, open a terminal
box
window to your local node and i _ the
oexecute us
r ic t o
(fab esshntosethe second node and run the
/u01/app/oraInventory/orainstRoot.sh script as root using the sudo
command. Whenefinished,
e rd le lic as root. When finished, exit
a V
/u01/app/oraInventory/orainstRoot.sh
b
V il fera
back to your first node in your terminal window.
r i c i rans ~]$ sudo /u01/app/oraInventory/orainstRoot.sh

o
[oracle@vx0301
Fab nChanging
o n -t permissions of /u01/app/oraInventory to 770.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete
[oracle@vx0301 ~]$
[oracle@vx0301 ~]$ ssh vx0302
Last login: Mon Nov 19 10:03:42 2007 from vx0301.us.oracle.com
[oracle@vx0302 ~]$ sudo /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory to 770.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete
logout
Connection to vx0302 closed.
[oracle@vx0301 ~]$
l) Next, run the /u01/crs11g/root.sh script as root using sudo from the
terminal window on your FIRST node. When it has finished executing, ssh to
your second node and execute the /u01/crs11g/root.sh script as root.
[oracle@vx0301 u01]$ sudo /u01/crs11g/root.sh

Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory

Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name>
<hostname>
node 1: vx0301 vx0301-priv vx0301
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/sdb5
Format of 1 voting devices complete. s a
Startup will be queued to init within 30 seconds.
)h a
Adding daemons to inittab
m ฺbr
Expecting the CRS daemons to be up within 600 seconds.
i ฺ c o deฺ
r e d Gui
Cluster Synchronization Services is active on these nodes.
vx0301
@ sic dent
Cluster Synchronization Services is inactive on these nodes.
vx0302
e r de Stu
v i lav e this
Local node checking complete. Run root.sh on remaining nodes
o_ o us
to start CRS daemons.
[oracle@vx0301 u01]$
ic i
( f abr nse t
[oracle@vx0301 u01]$ ssh vx0302
rde le lice
e
[oracle@vx0302 ~]$ sudo /u01/crs11g/root.sh
V ab
i l
V sfer a
/etc/oracle does not exist. Creating it now.
i o
icSetting anthe permissions on OCR backup directory
b r - t r
Fa nSetting
on up Network socket directories
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
<hostname>
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been
initialized

vx0301
vx0302
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
Creating VIP application resource on (2) nodes...

Creating GSD application resource on (2) nodes...
Creating ONS application resource on (2) nodes...
Starting VIP application resource on (2) nodes...
Starting GSD application resource on (2) nodes...
Starting ONS application resource on (2) nodes...
s a
)h a
Done.
m ฺbr
[oracle@vx0302 ~]$
i ฺ c o deฺ
m) When both orainstRoot.sh and root.sh scripts have r e d runGonuiboth
been
nodes, click OK on the Execute Configuration scripts @ sicdialog e n
box.
t
r d epage S t udthe installer. Click
n) Click the Exit button on the End of Installation
Yes on the dialog box to finish. i l a ve this to exit
o v se
_/u01/crs11g/bin/crs_stat
o) From a terminal session, execute
r ic i o u –t to
b t
check the viability (offa
e
the nodeapps
e n se on both nodes.
[oracle@vx0301 e rd u01]$ e l ic/u01/crs11g/bin/crs_stat -t
Name a V
l a b l
Type Target State Host
i r
i c i o V ansfe application ONLINE ONLINE vx0301
------------------------------------------------------------
a b r ora.vx0301.ons
ora.vx0301.gsd
- tr
F o n application
nora.vx0301.vip application ONLINE ONLINE vx0301
ONLINE ONLINE vx0301
ora.vx0302.gsd application ONLINE ONLINE vx0302

ora.vx0302.ons application ONLINE ONLINE vx0302
ora.vx0302.vip application ONLINE ONLINE vx0302

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 2-1: ASM Installation and Configuration
In this practice, you install and configure ASM
1) Start the Oracle Universal Installer from /stage/db and create a non-shared ORACLE
HOME for ASM.
a) Execute /stage/db/runInstaller to start OUI.
b) Select Oracle Database 11g from the Select a Product to Install page and click
Next.
c) Select Enterprise Edition from the Select Installation Type page and click Next.
d) On the Install Location page, make sure ORACLE_BASE is
/u01/app/oracle. Under Software Location, change the Name to
OraASM11g_home1. Change the Path to
/u01/app/oracle/product/11.1.0/asm_1. s a
)h a
DO NOT ACCEPT THE DEFAULT LOCATION!! Click Next to continue.
m ฺ br
e) On the Specify Hardware Cluster Installation Mode page, make o
i ฺ c sure Cluster
d e ฺ
r d t Gu
Installation is selected. Click the Select All button and theneclick Next toi
continue. i c
s den
@
f) The Product-Specific Prerequisite page should
e rdtoecontinue.
tuminimum
check the
S
v i la e this
requirements and report no errors. ClickvNext
g) Select Install Software Onlyion
ic o_the Select
o u sConfiguration Option page and click
Next.
( f abr nse t
h) On the Privileged e Operating
rdOSASM l i ce System page, select dba for the OSDBA,
OSOPER, V e
and
b l e groups. Click Next to continue.
V a ra by clicking the Install button on the Summary page.
il the installation
e
i) Start
o s f
b r ici Monitor
- t r anthe progress on the Install page.
Fa nj)onWhen the Execute Configuration scripts dialog box appears, open a terminal
window to your local node and execute the
/u01/app/oracle/product/11.1.0/asm_1/root.sh script as root using
the sudo command. When finished, ssh to the second node and run the
/u01/app/oracle/product/11.1.0/asm_1/root.sh as root. When
finished, exit back to your first node in your terminal window.
[oracle@vx0301 ~]$ sudo
/u01/app/oracle/product/11.1.0/asm_1/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:

ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.1.0/asm_1
Enter the full pathname of the local bin directory:

[/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...

Practice 2-1: ASM Installation and Configuration (continued)
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[oracle@vx0301 ~]$
/u01/app/oracle/product/11.1.0/asm_1/root.sh
s a
)h a
m ฺbr
ORACLE_HOME= /u01/app/oracle/product/11.1.0/asm_1
i ฺ c o deฺ
r e d Gui
[/usr/local/bin]: @ sic dent
Copying dbhome to /usr/local/binrd e Stu
...
e
Copying oraenv to /usr/local/bin
v i
...
t his
lav e ...
o_ o us
Copying coraenv to /usr/local/bin
ic i
( f a br se t
r d enthe /etc/oratab file as needed by
e addedlicto
Entries will
Database V
e ble Assistant when a database is created
be
V a
il running ra generic part of root.sh script.
Configuration
e
Finished
o s f
i product-specific
n
b r icNow t r a root actions will be performed.
on-
Fa n[oracle@vx0302
~]$ exit
logout
[oracle@vx0301 ~]$
k) When the root.sh script has been run on both nodes, click OK on the Execute
Configuration scripts dialog box.
l) Click the Exit button on the End of Installation page to exit the installer. Click
Yes on the dialog box to finish.
2) Use DBCA to configure two disk groups, DATA and FRA using block devices
/dev/sd[c-f].
a) Set ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1 and
execute dbca from /u01/app/oracle/product/11.1.0/asm_1/bin.
[oracle@vx0301 ~]$ export
ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1

Practice 2-1: ASM Installation and Configuration (continued)
[oracle@vx0301 ~]$
/u01/app/oracle/product/11.1.0/asm_1/bin/dbca
b) On the Welcome screen, select Oracle Real Application Cluster Database and
click Next.
c) On the Operations screen, select Configure Automatic Storage Management
and click Next.
d) On the Node Selection screen, click the Select All button and click Next.
e) On the Create ASM Instance screen enter oracle1 as the SYS password for
the ASM instances. You must enter it twice. You must also choose the type of
parameter file to use for the ASM instances. Select Create Initialization
Parameter File (IFILE) and accept the default file name and location. Click
Next to continue.
s a
f) A Database Configuration Assistant dialog box appears informing that the ASM
)h a
instances will be started. Click OK to continue.
m ฺbr
g) Another Database Configuration Assistant dialog box appears informing i ฺ c o you d e ฺ
that
no listeners exist and asking if listeners should be started on d nodes.
reyour i
uClick
YES to continue. i c
s den t G
@
de New u
tbutton.
h) On the ASM Disk Groups screen, click the e r
Create S
i) On the Create Disk Group screen,venter i lavDATA
e t hforisthe disk group name. Select
Normal Redundancy. Click ic i o_Change
the
o
s Discovery Path button. Enter
uDisk
/dev/sd[c-f] in the r
abpopup t
e and click OK. In the Show Members area
window
( f
e /dev/sdc n s
of the window, select
r d l i c e and /dev/sdd for the DATA disk group and
click OK. e
V able
i l a er to the ASM Disk Groups page. DATA should now be present,
V are returned
j) You
i o s f
b r ic mounted
- t r an by both instances. Click the Create New button.
Fa nk)onOn the Create Disk Group page, enter FRA for the Disk Group name and select
Normal Redundancy. Select the remaining devices, /dev/sde and /dev/sdf and
click OK.
l) You are returned to the ASM Disk Groups page. Both DATAand FRA disk
groups should now be present, mounted by both instances. Click the Finish button
then click No when prompted to perform another operation.

Practice 2-2: Database Software Installation
In this practice, you install the Oracle Database 11g software.
1) Start the Oracle Universal Installer from /stage/db and create a non-shared
ORACLE HOME for the database.
a) Execute /stage/db/runInstaller to start OUI.
b) Select Oracle Database 11g from the Select a Product to Install page and click
Next.
c) Select Enterprise Edition from the Select Installation Type page and click Next.
d) On the Install Location page, make sure ORACLE_BASE is
/u01/app/oracle. Under Software Location, change the Name to
OraDb11g_home1. Make sure the Path is
/u01/app/oracle/product/11.1.0/db_1. Click Next to continue. s a
)h a
e) On the Specify Hardware Cluster Installation Mode page, make sure Cluster
m ฺbr
i ฺ c o deฺ
Installation is selected. Click the Select All button and then click Next to
continue.
r e d Gui
c minimum
sithe n t
f) The Product-Specific Prerequisite Page should check
@ ud e
requirements and report no errors. Click Next r dtoecontinue.
t
S
veConfiguration
g) Select Install Software Only on theilSelect
v a
e t his Option page and click
Next. _
io o us
r ic
h) On the Privileged Operating
( f ab nSystem
s e t page, select dba for the OSDBA,
r e le ligroups.
OSOPER, anddOSASM
e ce Click Next to continue.
i) Starta theV b by clicking the Install button on the Summary page.
installation
i
V thel r a
feprogress on the Install page.
i c i o Monitor n s
a b r j) When
n - trathe Execute Configuration scripts dialog box appears, open a terminal
F no window to your local node and execute the
/u01/app/oracle/product/11.1.0/db_1/root.sh script as root using the
sudo command. When finished, ssh to the second node and run the
/u01/app/oracle/product/11.1.0/db_1/root.sh as root. When
finished, exit back to your first node in your terminal window.

/u01/app/oracle/product/11.1.0/db_1/root.sh

ORACLE_HOME= /u01/app/oracle/product/11.1.0/db_1

[/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...

Practice 2-2: Database Software Installation (continued)
Copying coraenv to /usr/local/bin ...

[oracle@vx0301 ~]$
s a
/u01/app/oracle/product/11.1.0/db_1/root.sh
)h a
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
ORACLE_HOME= /u01/app/oracle/product/11.1.0/db_1
@ sic dent
r e directory:
dbin S tu
Enter the full pathname of the local
e
[/usr/local/bin]:
v i lav e... t his
Copying dbhome to /usr/local/bin
ic i o_ o us ...
br/usr/local/bin
t
Copying oraenv to /usr/local/bin
Copying coraenv ato
( f n s e ...
e rde le lice
V/etc/oratab
Creating
V i l a
f e rbeabaddedfile...
i c
Entries
i s
o anConfiguration
will to the /etc/oratab file as needed by
b r Database
- t r Assistant when a database is created
Fa nNow on product-specific rootpart
Finished running generic of root.sh script.
actions will be performed.
logout
[oracle@vx0301 ~]$
k) When the root.sh script has been run on both nodes, click OK on the Execute
Configuration scripts dialog box.
l) Click the Exit button on the End of Installation page to exit the installer. Click
Yes on the dialog box to finish.

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 3-1: Create a Cluster Database
In this practice you create a cluster database using DBCA.
1) Create a cluster database using the Global Database Name provided by your
instructor. When the database has been successfully created, log in to Enterprise
Manager to test the configuration.
a) Export ORACLE_HOME as /u01/app/oracle/product/11.1.0/db_1
and execute dbca from ORACLE_HOME/bin.
[oracle@vx0301 ~]$ export
ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
[oracle@vx0301 ~]$ $ORACLE_HOME/bin/dbca
b) On the Welcome screen, select Oracle Real Application Cluster Database and
click Next. a
as
c) On the Operations screen, select Create a Database and click Next.
ฺbr )h
d) On the Node Selection screen, click the Select All button and click Next.
c m
o deฺ
e) On the Database Templates page, select General Purposeeor i ฺ
dTransaction
u i
r G
Processing and click Next to continue.
@ sic dent
f) On the Database Identification page, enter the r e Database
dGlobal S tu Name assigned to
you by your instructor (the SID field will a e
v this
autofill) and click Next.
i l
v scheck e box on the Management
g) Click the Configure Enterprise i o _Manager u
ric eDatabase
Options page. Select Configure
a b to Control for local management and
click Next.
e (f ens
e
h) On the Database
d
r Credentials
e l ic page, select Use the Same Administrative
l a l
V For aAllbAccounts. Enter oracle1 for the password and click Next.
i
V sfer
Password
i o
ici) Select n
aAutomatic Storage Management (ASM) from the Storage Options page
b r - t r
Fa non and click Next to continue.
j) On the ASM Disk Groups page, select the DATA disk group for the database
storage and click Next.
k) Select Oracle Managed Files on the Database File Locations page and enter
+DATA in the Database Area field. Click Next to continue.
l) On the Recovery Configuration page, click the Specify Flash Recovery Area
check box and enter +FRA for the Flash Recovery Area. Accept the default
value of 2048 for the Flash Recovery Area Size and click Next.
m) Click the Sample Schemas check box on the Database Content page and click
Next.
n) On the Initialization Parameters page click the Character Sets folder tab. In the
Database Character Set area select Use Unicode (AL32UTF8) and click Next to
continue.
o) Click the Keep the enhanced 11g security settings … check box on the Security
Settings page and click Next.

Practice 3-1: Create a Cluster Database (continued)
p) On the Automatic Maintenance Tasks page make sure the Enable automatic
maintenance tasks check box is checked and click Next.
q) Click Next on the Database Storage page.
r) On the Database Creation page, click the Create Database check box and click
the Finish button.
s) When the Summary window appears, click OK.
t) A Database Configuration Assistant window appears informing you that the
database creation has completed. Click the Exit button. The Start Cluster
Database dialog box informs you that the instances are being started.
u) Test the Enterprise Manager configuration by opening up a web browser and
entering the following address: https://your_hostname:1158
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 4-1: Add/Remove Redo Log Groups in a RAC
Environment
The goal of this practice is to show you how to add and remove redo log groups in a RAC
database environment.
1) Use Database Control to create two new redo log groups in your database. The two
groups must pertain to the thread number three, and each group must have only one
51200 KB member called redo05.log and redo06.log, respectively.
a) From Database Control Home page click the Server tab. In this example we use
RDB as your database, but make sure you are using the one you were assigned to.
b) On the Cluster Database Server page, click Redo Log Groups in the Storage
section.
s a
c) On the Redo Log Groups page, click Create.
)h a
d) On the Create Redo Log Group page, leave the current value of the Group# field m ฺbr
ฺ c
as is it is (5). Make sure that the File size field is set to 51200 KB. Set the
i o deฺ
r e d Gui
Thread# field to 3. Make sure that the File Directory field in the Redo Log
sic dent
Members section is set to +DATA. When you are finished, click OK.
@
e) This takes you back to the Redo Log Groups r e fromSwhere
dpage tu you should see a
e
Confirmation message giving you successful
v i lav estatus
t hison the new object creation.
i _ Create.
oclick us
f) On the Redo Log Groups page
r ic t o
g) On the Create Redo(fLog abGroup n e leave the current value of the Group# field
spage,
e e
c the File size field is set to 51200 KB. Set the
d surelithat
as is it is (6).rMake
Thread#V e
field to 3. l e
bSee thatWhen
the File Directory field in the Redo Log Members
i
V sfel
section
a is set r
to a
+DATA. you are finished, click OK.
i o
ich) This n you back to the Redo Log Groups page from where you should see a
atakes
b r - t r
Fa nonConfirmation message giving you successful status on the new object creation.
2) Use Database Control to set the RDB2.THREAD initialization parameter to 3 in
SPFILE only.
a) From Database Control Home page click the Server tab.
b) On the Server page, click the Initialization Parameters link in the Database
Configuration section.
c) On the Initialization Parameters page, click the SPFile tab.
d) On the SPFile page, click the Show All button.
e) Make sure that the “Apply changes in SPFile mode to the current running
instance(s). For static parameters, you must restart the database” check box is
deselected.
f) Scroll down the page to the end, and enter 3 in the Value field for the
RDB2.thread field.

Environment (continued)
g) Click the Apply button at the bottom of the page.
h) This takes you back to the Initialization Parameters page on which you should see
an Update Message giving you successful status on your changes.
3) Use the SRVCTL control utility to stop the RDB2 instance, and start it up again.
What happens and why? If necessary, fix the situation.
a) Although it is possible to stop the RDB2 instance, it is not possible to start it up
again because the redo thread number three is not enabled yet. To enable the redo
thread number three, you can connect to the first instance and issue the ALTER
DATABASE ENABLE THREAD 3 command. Then you can start up the RDB2
instance again. a
as
[oracle@vx0306 less04]$ cat sol_04_01_03.sh
#!/bin/bash ฺbr )h
#
c m
o deฺ
# sol_04_01_03.sh i ฺ
d Gui
r e
#
# Must be executed on NODE1 !!!!!!!!!! @ sic dent
#
e r de Stu
v i lav e this
y=`cat /home/oracle/nodeinfo
ic i o_ |osed us-n '1,1p'`
z=`cat /home/oracle/nodeinfo r t
ab nse | sed -n '2,2p'`
( f
DBNAME=`ps e rde| lgrep
-ef
e l i e
cdbw0_RDB | grep -v grep | grep -v
callout1
l a V
| awk
a b
'{ print $8 }' | sed 's/1/''/' | sed
i r
i c i o V ansfe
's/ora_dbw0_/''/'`
a b r I1NAME=$DBNAME"1"
- tr
F no n
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl stop instance -d $DBNAME -i $I2NAME
/u01/crs11g/bin/srvctl start instance -d $DBNAME -i $I2NAME
/u01/crs11g/bin/crs_stat -t
export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
export ORACLE_SID=$I1NAME
$ORACLE_HOME/bin/sqlplus -s /NOLOG <<EOF
connect / as sysdba
set echo on
ALTER DATABASE ENABLE THREAD 3;
EOF

[oracle@vx0306 less04]$
[oracle@vx0306 less04]$ ./sol_04_01_03.sh

PRKP-1001 : Error starting instance RDB2 on node vx0313
CRS-0215: Could not start resource 'ora.RDB.RDB2.inst'.
Name Type Target State Host
------------------------------------------------------------
ora....B1.inst application ONLINE ONLINE vx0306
ora....B2.inst application ONLINE OFFLINE
s a
ora.RDB.db application ONLINE ONLINE vx0313
)h a
ฺbr
ora....SM1.asm application ONLINE ONLINE vx0306
ora....06.lsnr application ONLINE ONLINE vx0306
c m
o deฺ
i ฺ
d Gui
ora.vx0306.ons application ONLINE ONLINE e
vx0306
r
ora.vx0306.vip application ONLINE ONLINE
@ sic dent
vx0306
de Stu
ora....13.lsnr application ONLINE
e r
ONLINE vx0313
ora.vx0313.gsd application
i
ONLINE
v lav e this
ONLINE vx0313
ora.vx0313.ons application
ic i o_ o us
abr nse t
( f
rde le lice
Database altered.
e
Name la V b
V i f e raType Target State Host
r i c io rans application ONLINE ONLINE vx0306

------------------------------------------------------------
-t
ora....B1.inst
Fab nora....B2.inst
o n
ora.RDB.db
application
application
ONLINE
ONLINE
ONLINE
ONLINE
vx0313
vx0313
4) Revert to the original situation where RDB2 was using the redo thread two, and
destroy redo thread number three. Make sure that in the end both instances are up and
running and managed by Oracle Clusterware.

#!/bin/bash
#
# sol_04_01_04.sh
#
# Must be executed on NODE1 !!!!!!!!!!
#
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'`

z=`cat /home/oracle/nodeinfo | sed -n '2,2p'`
DBNAME=`ps -ef | grep dbw0_RDB | grep -v grep | grep -v

callout1 | awk '{ print $8 }' | sed 's/1/''/' | sed
s a
's/ora_dbw0_/''/'`
)h a
I1NAME=$DBNAME"1" m ฺbr
i ฺ c o deฺ
I2NAME=$DBNAME"2"
r e d Gui
@ sic dent
export ORACLE_SID=$I1NAME e r de Stu
v i l av this
echo "Reset thread to 2 for
i o _ secondu s einstance ..."
a b ric -se/NOLOG
to <<EOF
(f ens
$ORACLE_HOME/bin/sqlplus
e
connect V/e as
d
r sysdba
l e lic
ilaSYSTEM
ALTER b thread = 2 SCOPE=SPFILE SID='$I2NAME';
raSET
io V s f e
r i cEOF tran
Fab nechoon- "Stop second instance ..."
/u01/crs11g/bin/srvctl stop instance -d $DBNAME -i $I2NAME
echo "Restart second instance ..."
echo "Removing thread 3 from database ..."
connect / as sysdba
alter database disable thread 3;
alter database drop logfile group 5;
alter database drop logfile group 6;
EOF


Reset thread to 2 for second instance ...
System altered.
Stop second instance ...

Restart second instance ...
Removing thread 3 from database ...
s a
Database altered.
)h a
m ฺbr
Database altered.
i ฺ c o deฺ
r e d Gui
@ sic dent
de Stu
Database altered.
e r s
Name Type
v i av thiState
lTarget Host
o_ oONLINE
us e
------------------------------------------------------------
i
ora....B1.inst application
b r ic t ONLINE vx0306
e (fa ense ONLINE

ora....B2.inst application ONLINE vx0313
ora.RDB.db
e l i c
rd application
application ONLINE ONLINE vx0313
a V
ora....SM1.asm
b l e ONLINE ONLINE vx0306
V il feraapplication ONLINE ONLINE vx0306

ora....06.lsnr application ONLINE ONLINE
i c i o ans application ONLINE ONLINE vx0306

ora.vx0306.gsd
b r t r
ora.vx0306.ons
-
vx0306
Fa nora....SM2.asm
on
ora.vx0306.vip application
application
ONLINE
ONLINE
ONLINE
ONLINE
vx0306
vx0313

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 5-1: Backup and Recovery
The following short exercises allow you to set backup and recovery options, configure a
Recovery Manager Repository, schedule a backup, and review RMAN reports using
Enterprise Manager.
1) Use Enterprise Manager to place the database in ARCHIVELOG mode. Enable
Flashback Recovery and set LOG_ARCHIVE_DEST_1 to the Flash Recovery
Diskgroup (FRA). Adjust the Flash Recovery Area to 3 gigabytes and enable
Flashback Database.
a) Log in to Enterprise Manager as sys/oracle1 and connect as SYSDBA.
b) From your database home page, click the Availability folder tab then click
Recovery Settings under Backup/Recovery. Click the ARCHIVELOG Mode
check box. Add your database name to the beginning of the archive log file name
format string, %t_%s_%r.dbf , for example RDB%t_%s_%r.dbf. Enter +FRA s a
)h a
for the value of LOG_ARCHIVE_DEST_1.
m ฺbr
c) Scroll down to the Flash Recovery section and click the Enable Flashback
i ฺ c o deฺ
Database… button, and set the Flash Recovery Area size e
r tod
3 GB. Clicku i the
Apply button. sic dent G
d) The Confirmation page asks if it is OK to restart
@
de the database.
tu Click Yes.
e r S
e) On the Startup/Shutdown page supply
v i e his and password to log in to
lavthe username
t
the cluster nodes, oracle/oracle
i c i o_ oanduthe s database user, sys/oracle1
connecting as SYSDBA. r Continue.
abClick e t
( f s
r d e click
f) After a few moments,
l i c ethen Refresh button on the Request in Process page to
check onVthee progress
b l eof the database restart.
V a
ilback into
e a
rEnterprise
g) Log
i o s f Manager. When all instances have been restarted,
b r ic navigate
- t r an back to Recovery Settings and verify that your changes have been
Fa non processed.
2) Using Enterprise Manager, configure backup settings in support of RAC. Set disk
parallelism to two, the disk backup location to the Flash Recovery Area, and the
backup type to compressed. The backup policy should include autobackups of the
control file and spfile for every backup, unchanged files should be skipped and block
change tracking should be enabled. Set the point-in-time window to be 31 days.
a) Return to the Availability folder and click on Backup Settings. Under Backup
Settings, click on the Device folder tab. Under Disk Settings, change the
Parallelism value to 2, the Disk Backup location to +FRA, and click the
Compressed Backup Set button under Disk Backup Type. Include the host login,
oracle/oracle under Host Credentials.

Practice 5-1: Backup and Recovery (continued)
b) Next, click on the Policy folder tab. Under Backup Policy, select Automatically
backup the control file… . Enable skipping of unchanged files and block change
tracking. Under Retention Policy, select the Retain at least the specified
number of full backups for each datafile option, specify the Recovery Window
value to be 31 days, and set the Redundancy value to 2. When finished click the
OK button. Return to Backup Settings and verify the changes.
3) Configure an RMAN recovery catalog and set persistent RMAN configuration
parameters that are RAC friendly. When the parameters have been saved, initiate a
full cluster database backup.
a) Execute the sol_05_01_03a.sh script on the first node to create a recovery
catalog and owner.
$ $HOME/solutions/less05/sol_05_01_03a.sh
SQL> CREATE USER rman IDENTIFIED BY rman
s a
TEMPORARY TABLESPACE temp
)h a
DEFAULT TABLESPACE USERS
m ฺbr
QUOTA UNLIMITED ON USERS;
GRANT recovery_catalog_owner TO rman; 2 co ideฺ
3 diฺ4
User created.
i c re t Gu
@ s den
SQL> exit
e r de Stu
2 exit
3 ; v i lav e this
ic i o_ o us
( f abr nse t TO rman;
SQL> GRANT recovery_catalog_owner
e
Grant succeeded. rde le lice
l a V ab
i
V Manager: f er Release 11.1.0.6.0 - Production on Tue Nov
i o s
an 2007
ic20 15:46:45
Recovery
b r - t r
Fa non
Copyright (c) 1982, 2007, Oracle. All rights reserved.
CONNECT CATALOG rman/rman
connected to recovery catalog database
CREATE CATALOG;
recovery catalog created
RMAN> exit
Recovery Manager complete.
b) Return to the Availability folder and click Recovery Catalog Settings under
Backup/Recovery.
c) Next, click the Add Recovery Catalog button.

Practice 5-1: Backup and Recovery (continued)
d) On the Add a Recovery Catalog page, enter the hostname of your first node in
the HOST field and 1521 for PORT. Enter the SID on the specified host (RDB1
for example) Enter rman/rman for the recovery catalog user and password.
Click Next.
e) Click Finish on the Review page.
f) Return to the Recovery Catalog Settings page. Select Use Recovery Catalog
option and select your database from the pull down list. Provide the host
credentials, oracle/oracle. Click the OK button when finished.
g) The next screen will inform you that your database has been successfully
registered with the recovery catalog.
4) Using Enterprise Manager, perform a one-time, full database backup. When the
backup is finished, use the RMAN reporting functionality in Enterprise Manager to a
view the backup details. as
a) Go to the Availability folder tab. Click on Schedule Backup under Manage.
)h
ฺbr
c m
ohave deฺ
b) Select Whole Database under Customized Backup. Make sureiฺyou
provided the host login credentials (oracle/oracle)cthen r e ui
dclick theGSchedule
Customized Backup button. @ si dent
c) Select Full Backup under Backup Type, e r deOnlineSBackup
select tu for Backup
i
Mode, and select also back up all archivedlav logs t on
s under Advanced. Click
hidisk
v
o_ o us e
Next when finished.
ic i
r that eyourt backup destination is Disk. Click Next to
d) On the next page, make
( f absure ns
continue. d e
r le li c e
e) Accept V
the
edefault
binformation; Job Name, Job Description, and One Time
i l a r a
V sfe on the Schedule page and click Next.
i c i o (Immediately)
anReview page, click Submit Job.
b r f) On - t r
the
Fa ng)onThe next page will inform you that your job has been successfully submitted. To
monitor the backup progress you can click the View Job button and occasionally
refreshing your browser.
h) When the backup is finished, return to the Availability folder and click Manage
Current Backups under Manage. Click the backup name components that
correspond to your recently completed database backup.

This practice shows you how to discover performance problems in your RAC
environment. In this practice, you identify performance issues using Enterprise Manager,
and you fix those issues in three different steps. At each step, you generate the same
workload to make sure you are making progress in your resolution.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 6-1: ADDM and RAC, Part I
The goal of this practice is to show you how to manually discover performance issues by
using the Enterprise Manager performance pages as well as ADDM. Part I generates a
workload that uses a bad RAC application design.
1) Execute the setupseq1.sh script to set up the necessary configuration for this practice.
[oracle@vx0306 less06]$ cat setupseq1.sh
#!/bin/bash
#
# setupseq1.sh
#
#
s a
)h a
m ฺbr
DBNAME=`ps -ef | grep dbw0_RDB | grep -v grep | dgrep i ฺco-vuideฺ
callout1 | awk '{ print $8 }' | sed 's/1/''/' i c re| sedt G
's/ora_dbw0_/''/'` @ s d e n
e r de Stu
I1NAME=$DBNAME"1"
v i lav e this
I2NAME=$DBNAME"2"
ic i o_ o us
( f a br se t
r d e licen
e ble
V
V a
il fera
i o
ic connect s
an/ as sysdba
b r - t r
Fa noexec n
dbms_advisor.set_default_task_parameter('ADDM','DB_ACTIVITY_MI
N',30);
exec
dbms_workload_repository.modify_snapshot_settings(interval=>60
0);
drop user jfv cascade;
drop tablespace seq including contents and datafiles;
create tablespace seq extent management local autoallocate
segment space management auto;
create user jfv identified by jfv default tablespace seq
temporary tablespace temp;
grant connect,resource,dba to jfv;
connect jfv/jfv
drop sequence s;
drop table s purge;
drop table t purge;
create table s(sn number);
create table t(c number,d varchar2(20));

Practice 6-1: ADDM and RAC, Part I (continued)
create index it on t(c);
insert into s values(1);
commit;
begin
for i in 1..90000 loop
insert into t values (i,'initial');
end loop;
end;
/
EOF
[oracle@vx0306 less06]$ ./setupseq1.sh
PL/SQL procedure successfully completed.
s a
)h a
m ฺ br
drop user jfv cascade
i ฺ c o deฺ
*
r e d Gui
ERROR at line 1:
ORA-01918: user 'JFV' does not exist @ sic dent
e r de Stu
i v this
lacontents
drop tablespace seq including
o _ v s e and datafiles
*
b r ici to u
ERROR at line 1:
e (fa 'SEQ'
ORA-00959: tablespace
e n sedoes not exist
e rd le lic
l a V ab
i er
V sfcreated.
i o
ic -tran
Tablespace
b r
Fa non
User created.
Grant succeeded.
drop sequence s
*
ERROR at line 1:
ORA-02289: sequence does not exist
drop table s purge

*
ERROR at line 1:
ORA-00942: table or view does not exist
drop table t purge

*
ERROR at line 1:
Table created.
Table created.
Index created.
1 row created. s a
)h a
m ฺ br
Commit complete.
i ฺ c o deฺ
r e d Gui
PL/SQL procedure successfully completed. @ sic dent
e r de Stu
v i lav e this
ic i o_ o us
2) Using Database Control,fa
( brconnected
and s e tas user SYS, navigate to the Performance
page of your Cluster
r d eDatabase.
l i c en
e ble tab on the Cluster Database Home page.
a) ClickatheVPerformance
V il fera
i cb)o On s Database Performance page, make sure Real Time: 15 Seconds
i rtheanCluster
r t is selected from the View Data dropdown list.
Fab non-
Refresh
3) Use PL/SQL to create a new AWR snapshot.
#!/bin/bash
#
# sol_06_01_03.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"

I2NAME=$DBNAME"2"
connect / as sysdba
exec dbms_workload_repository.create_snapshot
EOF

s a
)h a
m ฺ br
i ฺ c o deฺ
r e d Gui
4) Execute the startseq1.sh script to generate a workload on s ic instances
both e n t of your
d
cluster. Do not wait; instead, proceed with the next @
estep. Stud
r
ve this
[oracle@vx0306 less06]$ cat startseq1.shi l a
#!/bin/bash
i o _v use
#
a b ric e to
# startseq1.sh
e (f ens
er bleonlicNODE1 !!!!!!!!!!
d
#
# Must beVexecuted
#
V ila fera
r i c io rans
t
Fab ny=`cat
on- /home/oracle/nodeinfo | sed -n '1,1p'`

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
$ORACLE_HOME/bin/sqlplus -s /nolog @runseq1 $I1NAME &

[oracle@vx0306 less06]$ cat runseq1.sql

set echo on
connect jfv/jfv@&1
declare
v number;
begin
lock table s in exclusive mode;
select sn into v from s;
insert into t values(v,'&1');
update s set sn=sn+1;
commit;
end loop;
end; s a
/
)h a
m ฺ br
exit;
i ฺ c o deฺ
r e d Gui
[oracle@vx0306 less06]$ ./startseq1.sh
@ sic dent
[oracle@vx0306 less06]$ old 7:
e r de into
insert
S tu t
values(v,'&1');
v i lav e this
new 7:
_ us
insert into t values(v,'RDB1');
insert into c t io
old 7:
b i values(v,'&1');
to
r t values(v,'RDB2');
new 7:
a
insert into
(f ens e
d e
r after l ic point.
… Do not wait
e
V ab l e this
i l a
V sfer successfully completed.
PL/SQL procedure
i o
ic -tran
b r
Fa nPL/SQL
on procedure successfully completed.
5) Using Database Control, determine the list of blocking locks in your database.
a) Still on the Performance page, click the Database Locks link in the Additional
Monitoring Links section of the page.
b) On the Database Locks page, make sure that Blocking Locks is selected from the
View dropdown list.
c) If you do not see any locks, refresh the page by clicking Refresh. Perform this
until you see locks. When you see a session lock, you should also see that the
other session is waiting for that same lock. By clicking Refresh several times, you
should see that both sessions are alternately waiting for the other to release the
exclusive lock held on table S.

6) While the scripts are still executing, look at the Average Active Sessions graphic.
Then drill down to the Cluster wait class for the first node. What are your
conclusions?
a) By using the drilldown method of Enterprise Manager, you can quickly identify
the top waiting SQL statements and the top waiting sessions on both instances.
Here it appears that an UPDATE statement on table S is causing most of the waits
for the Cluster wait class.
b) Click the Cluster Database locator link at the top of the page to return to the
Cluster Database Performance page.
c) From there you can now see the Average Active Sessions graph. Make sure that
the View Data field is set to Real Time: 15 Seconds Refresh. After a few seconds,
the graphic should clearly show that the Cluster and Application wait classes are
causing most waits. Using the Throughput tabbed page graph underneath the
s a
Average Active Sessions graph, you should also notice that the transaction rate is
)h a
about 300 per second.
m ฺbr
d) In the Average Active Sessions graph, click the Cluster link on ithe ฺ c oright. This
d e ฺ
takes you to the Active Sessions By Instance: Cluster page. red t Gu i
i c
sshoulddeseenthat the number
e) On the Active Sessions By Instance: Cluster page, @you
e Click
dnodes. tuthe first instance’s link
of active sessions is almost the same on bothe r S
(instance number 1). This takes you itolathev Activet is Waiting: Cluster page
hSessions
for the corresponding instance.
v
o_ o us e
ic i
r eCluster
f) On the Active Sessions abWaiting: t page, you can see the most important
( f s
n waits in the Cluster wait class on the first
e mostlicofethe
wait events causing
r d
instance.VInethe TopleSQL: Cluster section, click the SQL identifier that uses most
a b
raThis takes you to the SQL Details page for the corresponding
il resources.
of the
o V
i ran
statement.s f e
You should see that the script running on the first instance is executing
r i c t
Fab7) nUsing
on- Database Control look at the Cluster Cache Coherency page. What are your
a SELECT/UPDATE statement on table S that causes most of the Cluster waits.
conclusions?
a) On the Cluster Database Home page, click the Performance tab.
b) On the Performance page, click the Cluster Cache Coherency link in the
Additional Monitoring Links section.
c) The Cluster Cache Coherency page clearly shows that there are lots of blocks
transferred per second on the system. This represents more than 17% of the total
logical reads. This is reflected in both the Global Cache Block Transfer Rate and
the Global Cache Block Transfers and Physical Reads (vs. Logical Reads)
graphics.
d) On the Cluster Cache Coherency page, you can also click Interconnects in the
Additional Links section of the page to get more information about your private
interconnect.

8) While the scripts are still executing, look at the Average Active Sessions graph. Then
drill down to the Application wait class for the first instance. What are your
conclusions?
a) By using the drilldown method of Enterprise Manager, you can quickly identify
Here it appears that a LOCK statement on table S is causing most of the waits for
the Application wait class.
b) Go back to the Cluster Database Home page by clicking the Database tab located
on the top right-end corner. On the Cluster Database Home page, click the
Performance tab.
c) On the Performance page, make sure that the View Data field is set to Real Time:
15 Seconds Refresh. After a few seconds, the graphic should clearly show that the
Cluster and Application wait classes are causing most waits. You should also
s a
notice that the transaction rate is about 100 per second.
)h a
d) In the Average Active Sessions graph, click the Application link on the right. This
m ฺbr
takes you to the Active Sessions By Instance: Application page. i ฺ c o deฺ
r e d Gui
e) On the Active Sessions By Instance: Application page, iyou c should
snodes. n tsee that the
number of active sessions is almost the same oneboth @ u e
d the link for the
Click
first instance (number 1) on the Summary e r d
Chart graph.S t
This takes you to the
l a v h i s
Active Sessions Waiting: Application
o _ vi se t
page of the first instance.
f) On the Active Sessions Waiting:
b r ici most t o u page, you can see the most
Application
important wait events
( f a nse
causing of the waits in the Application wait class on
eIn thelTop
the first instance.
iceSQL: Application section, click the SQL identifier
erdof btheleresources.
that usesVmost This takes you to the SQL Details page for the
ila ferstatement.
corresponding
V a You should see that the script running on the first
i c o
i ran
instance s
is executing a LOCK statement on table S that causes most of the
r t
Fab non-
Application waits.
g) After a while, you can see that both scripts are executed by looking at the Average
Active Sessions graph as well as the Database Throughput graphics again. You
should see the number of transactions per second going down.
9) After the workload finishes, use PL/SQL to create a new AWR snapshot.
#!/bin/bash
#
# sol_06_01_09.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
connect / as sysdba
s a
EOF
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
PL/SQL procedure successfully completed. @ sic dent
e r de Stu
v i lav e this
10) Using Database Control, review the
ic i o_latestoADDM
us run. What are your conclusions?
r
abHomenspage, t
e click the Advisor Central link.
a) On the Cluster Database
e ( f
b) On the Advisor rd Central l e
ic make sure that the Advisory Type field is set to All
page,
Types,
e l e
V thatatheb Advisor Runs field is set to Last Run. Click Go.
and
i l a
Vthe Results ertable, select the latest ADDM run corresponding to Instance All.
i
c)o In s f
n View Result. This takes you to the Automatic Database Diagnostic
b r ic Then- t r aclick
Fa nonMonitor (ADDM) page.
d) On the Automatic Database Diagnostic Monitor (ADDM) page, the ADDM
Performance Analysis table shows you the consolidation of ADDM reports from
all instances running in your cluster. This is your first entry point before drilling
down to specific instances. From there, investigate the Top SQL by DB Time,
Table Locks, and Top SQL by Cluster Wait findings.
e) The Top SQL by DB Time should reveal a LOCK TABLE S command as a
possible problem to investigate.
f) The Table Lock finding, which affects both instances, reveals that you should
investigate your application logic regarding the JFV.S object.
g) The Top SQL by Cluster Wait finding again reveals the LOCK TABLE S
command embedded in a PL/SQL block.

h) Back on the Automatic Database Diagnostic Monitor (ADDM) page, you now
have the possibility to drill down to each instance by using the links located in the
Affected Instances table. Click the link corresponding to the most affected
instance (although both should be equally affected).
i) On the corresponding ADDM Database Diagnostic Monitor (ADDM) instance
page, you should retrieve exactly the same top findings that you previously saw at
the cluster level.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 6-2: ADDM and RAC, Part II
The goal of Practice 6 is to show you how to manually discover performance issues by
using the Enterprise Manager performance pages as well as ADDM. In Part II of the
practice, you correct the previously found issue by creating a sequence number instead of
using a table.
1) Execute the setupseq2.sh script to create the necessary objects used for the rest of this
practice.
#!/bin/bash
#
# setupseq2.sh
#
s a
#
)h a
m ฺbr
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'` diฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` co ideฺ
i c re t Gu
@ s
grep |de n -v
DBNAME=`ps -ef | grep dbw0_RDB | grep -v
callout1 | awk '{ print $8 }' | sedrd e Stu| sed grep
e 's/1/''/'
's/ora_dbw0_/''/'`
v i lav e this
ic i o_ o us
br se t
I1NAME=$DBNAME"1"
( f a
I2NAME=$DBNAME"2"
r d e licen
Ve able
exportlaORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
V i ORACLE_SID=$I1NAME
f er
i o
export
an s
ic$ORACLE_HOME/bin/sqlplus
b r - t r
Fa non -s /NOLOG <<EOF
connect / as sysdba
exec
N',30);
exec
0);
connect jfv/jfv
drop table s purge;
drop sequence s;
drop table t purge;

Practice 6-2: ADDM and RAC, Part II (continued)
create sequence s start with 1 increment by 1 nomaxvalue
nocache;
begin
end loop;
end;
/
EOF
[oracle@vx0306 less06]$ ./setupseq2.sh s a

)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
User dropped.
v i lav e this
ic i o_ o us
Tablespace dropped. br
( f a nse t
e rde le lice
V created.
l a rab
Tablespace
V i f e
r i c io rans
User tcreated.
Fab non-
Grant succeeded.
drop table s purge

*
ERROR at line 1:
drop sequence s
*
ERROR at line 1:
drop table t purge

*
ERROR at line 1:

Table created.
Index created.
Sequence created.
[oracle@vx0306 less06]$ s a
)h a
2) Using Database Control, and connected as user SYS, navigate to the Performance
page of your Cluster Database. m ฺbr
i ฺ c o deฺ
e
a) Click the Performance tab on the Cluster Database Home rpage.d Gui
c
siReal n t15 Seconds
b) On the Cluster Database Performance page, make@ sure
e list.Stud e
Time:
r d
Refresh is selected from the View Data dropdown
3) Use PL/SQL to create a new AWR snapshot. i l a ve this
i o _v use
a b riccatesol_06_02_03.sh
to
#!/bin/bash
e (f ens
#
e r le lic
d
l a V ab
# sol_06_02_03.sh
# i er
V be sexecuted
#oMust f
i c i n on NODE1 !!!!!!!!!!
a b r #
n - tra
F no

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
connect / as sysdba

EOF
4) Execute the startseq2.sh script to generate a workload on both instances of your

cluster. Do not wait; instead, proceed with the next step.
[oracle@vx0306 less06]$ cat startseq2.sh
#!/bin/bash
s a
#
)h a
ฺbr
# startseq2.sh
#
c m
o deฺ
# Must be executed on NODE1 !!!!!!!!!! i ฺ
d Gui
# r e
@ sic dent
y=`cat /home/oracle/nodeinfo | seder-n de'1,1p'`
S tu
z=`cat /home/oracle/nodeinfo |
v i e his
v -n t'2,2p'`
lased
ic io_ o|ugrep
DBNAME=`ps -ef | grep rdbw0_RDB
s
callout1 | awk '{(fprint
t
ab n$8se}' | sed 's/1/''/' | sed
-v grep | grep -v
rde le lice
's/ora_dbw0_/''/'`
e
l a V ab
i
V sfer
I1NAME=$DBNAME"1"
i o an
icI2NAME=$DBNAME"2"
b r - t r
Fa non export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1

set echo on
connect jfv/jfv@&1
declare
v number;
begin

commit;
end loop;
end;
/
exit;

[oracle@vx0306 less06]$ old 7: insert into t
values(v,'&1');
new 7: insert into t values(v,'RDB1');
old 7: insert into t values(v,'&1'); s a
)h a
m ฺ br
… Do not wait after this point.
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
5) While the l a V areabstill executing, look at the Average Active Sessions graphic.
scripts
i
Vdrill down f etor the Cluster wait class for the first node. What are your
i o
Then
an
icconclusions? s
b r - t r
Fa na)onBy using the drilldown method of Enterprise Manager, you can quickly identify
Here it appears that an INSERT statement on table T is causing most of the waits
for the Cluster wait class.
c) From there you can now see the Average Active Sessions graph. Make sure that
the View Data field is set to Real Time: 15 Seconds Refresh. After a few seconds,
the graphic should clearly show that the Cluster and Application wait classes are
causing the most waits. In the Throughput tabbed page graph underneath the
Average Active Sessions graph, you should also notice that the transaction rate is
about 400 per second (a better rate than in the previous practice).
d) In the Average Active Sessions graph, click the Cluster link on the right. This
takes you to the Active Sessions By Instance: Cluster page.

e) On the Active Sessions By Instance: Cluster page, you should see that the number
of active sessions is almost the same on both nodes. Click the first instance’s link
(instance number 1). This takes you to the Active Sessions Waiting: Cluster page
for the corresponding instance.
f) On the Active Sessions Waiting: Cluster page, you can see the most important
wait events that cause most of the waits in the Cluster wait class on the first
instance. In the Top SQL: Cluster section, click the SQL identifier that uses most
of the resources. This takes you to the SQL Details page for the corresponding
statement. You should see that the script running on the first instance is executing
an INSERT statement on table T that causes most of the Cluster waits.
g) After a while, you can see that both scripts are executed by looking at the Average
Active Sessions graphic again. The Database Throughput graphic tells you that
the number of transactions per second is now a bit higher than in the previous a
practice for the same workload. Using the sequence number was a bit better in as
this case.
ฺbr )h
c m
o deฺ
i ฺ
d Gui
[oracle@vx0306 less06]$ cat sol_06_02_06.sh re
#!/bin/bash
@ sic dent
#
e r de Stu
lav e this
# sol_06_02_06.sh
# v i
# Must be executed on NODE1
ic i o_ !!!!!!!!!!
o us
# r
ab nse t
( f
e r de lice
l a V able
V i
z=`cat
f e r
/home/oracle/nodeinfo | sed -n '2,2p'`
o
i ran
icDBNAME=`ps s
b r t
on- | awk '{ print $8 }' | sed 's/1/''/' | sed
Fa ncallout1
-ef | grep dbw0_RDB | grep -v grep | grep -v
's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
connect / as sysdba
EOF

7) Using Database Control, review the latest ADDM run. What are your conclusions?
a) On the Cluster Database Home page, click the Advisor Central link.
b) On the Advisor Central page, make sure that the Advisory Type field is set to All
Types and that the Advisor Runs field is set to Last Run. Click Go.
c) In the Results table, select the latest ADDM run corresponding to Instance All.
Then click View Result. This takes you to the Automatic Database Diagnostic
Monitor (ADDM) page.
d) On the Automatic Database Diagnostic Monitor (ADDM) page, the ADDM a
Performance Analysis table shows you the consolidation of ADDM reports from as
)h
ฺbr
down to specific instances. From there, investigate the Top SQL by DB Time,
c m
o deฺ
Top SQL by Cluster Wait, and Sequence Usage findings. i ฺ
d Gui
e) The Top SQL by DB Time should reveal an INSERTsINTO
r e
ic Tecommand t using
@ n
sequence S as a possible problem to investigate.e
r d S t ud
f) The Top SQL by Cluster Wait findingareveals
i l ve the t is statement as above: an
hsame
INSERT INTO T command using v
o_ o us
sequence eS.
ic i
g) The Sequence Usage finding,
( f a br which s e t affects both instances, reveals a high
frequency of sequence cache n
r d e lice misses.
e bDatabase
h) On the Automatic
V le Diagnostic Monitor (ADDM) page, you now have the
i l a
possibility
V r a
to drill
e down to each instance using the links located in the Affected
o
i ran
Instances s f
table. Click the link corresponding to the most affected instance
r i c t
Fab non-
(although both should be equally affected).
8) On the corresponding ADDM Database Diagnostic Monitor (ADDM) instance page,
you should retrieve exactly the same top findings that you previously saw at the
cluster level.

Practice 6-3: ADDM and RAC, Part III
The goal of Practice 6 is to show you how to manually discover performance issues by
using the Enterprise Manager performance pages as well as ADDM. Part III generates the
same workload as in Part II but uses more cache entries for sequence number S.
1) Execute the setupseq3.sh script to create the necessary objects used for the rest of this
practice.
#!/bin/bash
#
# setupseq3.sh
#
#
s a
)h a
m ฺbr
i ฺ c o deฺ
r e dgrep G u i
DBNAME=`ps -ef | grep dbw0_RDB | grep -v grep
@
callout1 | awk '{ print $8 }' | sed 's/1/''/' sic |desed
|
nt -v
's/ora_dbw0_/''/'`
e r de Stu
v i lav e this
I1NAME=$DBNAME"1"
ic i o_ o us
I2NAME=$DBNAME"2" abr e t
( f
e licen s
r d
e ble
V
export aORACLE_SID=$I1NAME
V il fera
r i c i rans
o
b - t
Fa noconnectn / as sysdba
exec
N',30);
exec
0);
connect jfv/jfv
drop table s purge;
drop sequence s;
drop table t purge;

Practice 6-3: ADDM and RAC, Part III (continued)
create sequence s start with 1 increment by 1 nomaxvalue
cache 90000;
begin
end loop;
end;
/
EOF
[oracle@vx0306 less06]$ ./setupseq3.sh

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
User dropped.
e r de Stu
v i lav e this
Tablespace dropped.
ic i o_ o us
( f abr nse t
Tablespace e rde le lice
created.
l a V ab
i
Vcreated.f er
i o
ic -tran
User s
b r
Fa non
Grant succeeded.
drop table s purge

*
ERROR at line 1:
drop sequence s
*
ERROR at line 1:
drop table t purge

*
ERROR at line 1:

Table created.
Index created.
Sequence created.
s a
2) Using Database Control, and connected as user SYS, navigate to the Performance
)h a
page of your Cluster Database.
m ฺbr
a) Click the Performance tab on the Cluster Database Home page.iฺco
d e ฺ
reTime: u i
d 15GSeconds
b) On the Cluster Database Performance page, make sure iRealc t
Refresh is selected from the View Data dropdown@ list.s d e n
3) Use PL/SQL to create a new AWR snapshot.ver
de Stu
i l a t h is
v
[oracle@vx0306 less06]$ cat_ sol_06_03_03.sh e
r ic io o us
ab nse t
#!/bin/bash
#
( f
# sol_06_03_03.sh
e rde le lice
#
l a V ab on NODE1 !!!!!!!!!!
# Must
#o V
i er
be executed
f
i
ic -tran s
b r
Fa ny=`cat
on /home/oracle/nodeinfo | sed -n '1,1p'`

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
connect / as sysdba

EOF
4) Execute the startseq2.sh script to generate the same workload on both instances of
your cluster as for the previous practice. Do not wait; instead, proceed with the next
step.
[oracle@vx0306 less06]$ cat startseq2.sh
#!/bin/bash
s a
#
)h a
# startseq2.sh
m ฺbr
#
i ฺ c o deฺ
# r e d Gui
@ sic dent
y=`cat /home/oracle/nodeinfo | seder-n de'1,1p'`
S tu
z=`cat /home/oracle/nodeinfo |
v i e his
v -n t'2,2p'`
lased
ic io_ o|ugrep
DBNAME=`ps -ef | grep rdbw0_RDB
s
a b e t -v grep | grep -v
callout1 | awk '{(fprint $8s }' | sed 's/1/''/' | sed
r d e licen
's/ora_dbw0_/''/'`
V e ble
V a
il fera
I1NAME=$DBNAME"1"
r i c i rans
o
t
Fab non-
I2NAME=$DBNAME"2"

set echo on
connect jfv/jfv@&1
declare
v number;
begin

commit;
end loop;
end;
/
exit;

[oracle@vx0306 less06]$ old 7: insert into t
values(v,'&1');
old 7: insert into t values(v,'&1'); s a
)h a
m ฺ br
… Do not wait after this point.
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
i c i o_ o us
( f abr nse t
e rde le lice
5) Until the l a V areaexecuted,
scripts b look at the Sessions: Waiting and Working graphic.
V i f e r
i c io arerayour
What
n s conclusions?
r -t time, looking at the Sessions: Waiting and Working graphic, it is clear that
Fab na)onthere
This
are no significant waits. The sequence has a big enough cache value to avoid
the most significant waits.
c) On the Performance page, make sure that the View Data field is set to Real Time:
15 Seconds Refresh. After both scripts finish execution, the Average Active
Sessions graph should clearly show that there are no significant waits on your
cluster. You should also notice that the transaction rate is now around 1,000 per
second.
#!/bin/bash
#
# sol_06_03_06.sh
#

#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
export ORACLE_SID=$I1NAME s a
)h a
m ฺbr
i ฺ c o deฺ
connect / as sysdba
r e d Gui
@ sic dent
EOF
e r de Stu
v i lav e this
i o_ o us
[oracle@vx0306 less06]$c./sol_06_03_06.sh
i
( f a br se t
r d e licen
V e less06]$b le
i l a
[oracle@vx0306
VDatabase r a
eControl,
i
7) Usingo n s f review the latest ADDM run. What are your conclusions?
r i c t r a
Fab na)onOn- the Cluster Database Home page, click the Advisor Central link.
b) On the Advisor Central page, make sure that the Advisory Type field is set to All
Types and that the Advisor Runs field is set to Last Run. Click Go.
c) In the Results table, select the latest ADDM run corresponding to Instance All.
Then click View Result. This takes you to the Automatic Database Diagnostic
Monitor (ADDM) page.
d) On the Automatic Database Diagnostic Monitor (ADDM) page, the ADDM
Performance Analysis table shows you the consolidation of ADDM reports from
down to specific instances. From there, investigate the “Top SQL by DB Time”
and “Top SQL by Cluster Wait” findings. You should no longer see the Sequence
Usage or the specific instances affected.
e) The “Top SQL by DB Time” finding should reveal an INSERT INTO T
command using sequence S as a possible problem to investigate.
f) The “Top SQL by Cluster Wait” finding reveals the same statement as above: an
INSERT INTO T command using sequence S.

g) On the Automatic Database Diagnostic Monitor (ADDM) page, you now have the
ability to drill down to each instance by using the links located in the Affected
Instances table. Click the link corresponding to the most affected instance.
8) On the corresponding ADDM Database Diagnostic Monitor (ADDM) instance page,
you should retrieve exactly the same top findings that you previously saw at the
cluster level.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

In this practice, you create a service, manipulate it, and monitor it.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 7-1: Manage Services
In this practice, you use Database Control to create one service called SERV1. You then
observe what happens to your service when you terminate the instance on which it is
running.
1) Use Database Control to create the SERV1 service. Make sure that you define your
first instance (RDB1 in this example) as preferred, and the second instance (RDB2 in
this example) as available.
a) Connect as user SYS in Database Control.
b) On the Cluster Home page, click the Availability tab.
c) On the Availability page, click the Cluster Managed Database Services link in the
Services section.
d) On the Cluster and Database Login page, enter oracle in both Username and s a
Password fields in the Cluster Credentials section. In the Database Credentials )h a
section, enter sys in the Username field, and oracle1 in the Password field.
m ฺbr
Make sure you select the “Save as preferred credential” check box. When you are
i ฺ c o deฺ
finished, click Continue. r e d Gui
c
sithe n t
e) On the Cluster Managed Database Services page, @ click
e Stud e
Create Service button.
f) On the Create Service page, enter SERV1 r d
ein the Service Name field. Make sure
l a v h i s
t Set Service Policy to
the Start service after creation check
o _ vibox isschecked.
e
Preferred for RDB1, and Available
r ic i for
o u
RDB2. Leave all other fields to their
defaults. ab nse t
( f
e lclick
g) When you are
e rdfinished, i ce OK.
2) After you a V created
have b e
lSERV1, make sure it is taken into account by Oracle
V i l
Clusterware. f
Usee a
rthe crs_stat command from one of the nodes, and then use the
i c i o n s
a b r SRVCTL
n - tracommand.
F o
n[oracle@vx0306 less07]$ cat sol_07_01_02.sh
#!/bin/ksh
#
# sol_07_01_02.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

Practice 7-1: Manage Services (continued)
/u01/crs11g/bin/crs_stat
/u01/crs11g/bin/srvctl status service -d $DBNAME -s SERV1


NAME=ora.RDB.RDB1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vx0306
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vx0313 s a
)h a
NAME=ora.RDB.SERV1.RDB1.srv
m ฺbr
TYPE=application
i ฺ c o deฺ
TARGET=ONLINE
r e d Gui
@ sic dent
NAME=ora.RDB.SERV1.cs
e r de Stu
TYPE=application
v i lav e this
TARGET=ONLINE
STATE=ONLINE on vx0306 cio
_ us
b i
r e to
NAME=ora.RDB.db (f
a ns
d e c e
er ble li
TYPE=application
V
V ila ferona vx0313
TARGET=ONLINE
STATE=ONLINE
r i c io rans
t
Fab nTYPE=application
on-
NAME=ora.vx0306.ASM1.asm
TARGET=ONLINE
NAME=ora.vx0306.LISTENER_VX0306.lsnr
TYPE=application
TARGET=ONLINE
NAME=ora.vx0306.gsd
TYPE=application
TARGET=ONLINE
NAME=ora.vx0306.ons
TYPE=application
TARGET=ONLINE

NAME=ora.vx0306.vip
TYPE=application
TARGET=ONLINE
TYPE=application
TARGET=ONLINE
TYPE=application
TARGET=ONLINE
NAME=ora.vx0313.gsd s a
TYPE=application
)h a
TARGET=ONLINE
m ฺbr
i ฺ c o deฺ
r e d Gui
NAME=ora.vx0313.ons
TYPE=application @ sic dent
TARGET=ONLINE
e r de Stu
v i lav e this
NAME=ora.vx0313.vip
ic i o_ o us
TYPE=application
( f a br se t
TARGET=ONLINE
r d e vx0313l i c en
V e ble
STATE=ONLINE on
V a
il SERV1e rais running on instance(s) RDB1
Service
o
i ran s f
b r ic[oracle@vx0306
t
less07]$
Fa non-
3) Add SERV1 to the tnsnames.ora files on both nodes to both oracle homes. When you
are finished, make sure listeners are aware of its existence.
#!/bin/ksh
#
# sol_07_01_03.sh
#
#


's/ora_dbw0_/''/'`
sed 's/NODE1/'$y'/'
/home/oracle/solutions/less07/wrong_tnsserv1.ora >
/home/oracle/solutions/less07/w1tnsserv1.ora
sed 's/NODE2/'$z'/'
/home/oracle/solutions/less07/w1tnsserv1.ora >
/home/oracle/solutions/less07/tnsserv1.ora
cp
/u01/app/oracle/product/11.1.0/db_1/network/admin/tnsnames.ora
.bak1 s a
cat /home/oracle/solutions/less07/tnsserv1.ora >>
)h a
m ฺbr
i ฺ c o deฺ
cp
r e d Gui
a @ sic dent
/u01/app/oracle/product/11.1.0/asm_1/network/admin/tnsnames.or
r de Stu
e
a.bak2
v i lav e this
o _
cat /home/oracle/solutions/less07/tnsserv1.ora
s >>
b r ici to u
(fa ense
a
ssh $z cp erd
e l ic
a V ab l e
i l
V sfer
i o
ic -tran
.bak1
b r
Fa nssh on $z cp
a
a.bak2
scp /home/oracle/solutions/less07/tnsserv1.ora
$z:/home/oracle/solutions/less07
ssh $z "cat /home/oracle/solutions/less07/tnsserv1.ora >>

"
ssh $z "cat /home/oracle/solutions/less07/tnsserv1.ora >>

a"
export ORACLE_HOME=/u01/app/oracle/product/11.1.0/asm_1

/u01/app/oracle/product/11.1.0/asm_1/bin/lsnrctl services |
grep SERV1
[oracle@vx0306 less07]$ cat wrong_tnsserv1.ora
SERV1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = NODE1-vip)(PORT =
1521))
1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA = s a
(SERVER = DEDICATED)
)h a
(SERVICE_NAME = SERV1)
m ฺbr
)
i ฺ c o deฺ
)
r e d Gui
[oracle@vx0306 less07]$ @ sic dent
e r de Stu
v i lav e this
o_ o us
tnsserv1.ora
r ic i 100% 268
b t
0.3KB/s 00:00
e
Service "SERV1" has
e n se
(fa 1 instance(s).
e rd le lic
V aless07]$
l a r b
[oracle@vx0306
V i f e
r i c io rans
t
Fab non-
4) Connect as SYSTEM under each instance and look at the current value of the
SERVICE_NAMES initialization parameter, and check that it is set correctly.
#!/bin/ksh
#
# sol_07_01_04.sh
#
#


's/ora_dbw0_/''/'`

I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
set echo on
connect system/oracle1@$I1NAME
select instance_name from v\$instance;
show parameter service
show parameter service s a
)h a
EOF
m ฺbr
i ฺ c o deฺ
[oracle@vx0306 less07]$ ./sol_07_01_04.sh icre
d Gui
@ s dent
INSTANCE_NAME
e r de Stu
----------------
v i lav e this
o_ o us
RDB1
ic i
( f a br se t
NAME
r d e licen TYPE VALUE
V e ble
------------------------------------ ----------- -------------
ila fera
-----------------
V
service_names string SERV1, RDB
i o an
icINSTANCE_NAMEs
b r - t r
Fa n----------------
on
RDB2
NAME TYPE VALUE

------------------------------------ ----------- -------------
-----------------
service_names string RDB
5) Using Grid Control, how can you check that SERV1 service is currently running as
expected?
a) If you are still connected to your previous Enterprise Manager session, you should
see it from the Cluster Managed Database Services page.

b) If not, connect as user SYS in Database Control. On the Cluster Home page, click
the Availability tab. On the Availability page, click the Cluster Managed
Database Services link in the Services section. On the Cluster and Database Login
page, click Continue.
c) On the Cluster Managed Database Services page, you should see SERV1 up and
running on the first instance only.
d) On the Cluster Managed Database Services page, click the SERV1 link.
e) On the SERV1 page, you should see that SERV1 is running on the first instance
and is down on the second instance.
6) Using a telnet session connected as user oracle to the first node, execute the
sol_07_01_06_a.sh script. This script monitors events happening inside Oracle
Clusterware. From a second terminal window as user oracle, kill the SMON
s a
process of the first instance (RDB1 in this example). Observe the sequence of events
)h a
in the first session. When the first instance is back, look at the SERVICE_NAMES
m
initialization parameter values on both instances. What do you observe? When you ฺbr
i ฺ c o deฺ
r e d Gui
are finished, enter Ctrl + C in the first window to stop event monitoring.
ic entautomatically
a) Immediately after the SMON process is killed, OraclesClusterware
moves the SERV1 service to its available instance @
de SERV1
RDB2 (in dthis example).
tudoes
Although Oracle Clusterware also restartse
v r
RDB1, s S not fall back to
l a h i
RDB1.
o _ vi se t
b) Here is what you should see
b r icini the first
t o u
session:
e
[oracle@vx0306 less07]$(fa encat sesol_07_01_06_a.sh
#!/bin/ksh rd
e e l ic
#
l a V ab l
i
V sfer
# sol_07_01_06_a.sh
i o
ic# Must
#
aben executed on NODE1 !!!!!!!!!!
b r - t r
Fa n#on
/u01/crs11g/bin/evmwatch -A -t "@timestamp @@"
[oracle@vx0306 less07]$ ./sol_07_01_06_a.sh

"14-Nov-2007 09:41:42 RAC: ora.vx0306.ASM1.asm: imcheck: "
"14-Nov-2007 09:41:42 RAC: ora.vx0306.ASM1.asm: imup: "
"14-Nov-2007 09:43:19 RAC: ora.RDB.RDB1.inst: imcheck: "
"14-Nov-2007 09:43:19 RAC: ora.RDB.RDB1.inst: imup: "
"14-Nov-2007 09:44:59 CRS is requested to perform action fail
on resource ora.RDB.RDB1.inst by Instance Monitor"
"14-Nov-2007 09:44:59 CRS ora.RDB.RDB1.inst is transitioning
from state ONLINE to state OFFLINE on member vx0306"
"14-Nov-2007 09:44:59 RAC: ora.RDB.RDB1.inst: down: "
"14-Nov-2007 09:44:59 CRS is requested to perform action fail
on resource ora.RDB.SERV1.RDB1.srv by ora.RDB.RDB1.inst"

"14-Nov-2007 09:44:59 CRS ora.RDB.SERV1.RDB1.srv is
transitioning from state ONLINE to state OFFLINE on member
vx0306"
"14-Nov-2007 09:44:59 RAC: ora.RDB.SERV1.RDB1.srv: down: "
"14-Nov-2007 09:44:59 CRS ora.RDB.SERV1.RDB1.srv stopped"
"14-Nov-2007 09:44:59 CRS ora.RDB.SERV1.RDB1.srv is
transitioning from state OFFLINE to state ONLINE on member
vx0313"
"14-Nov-2007 09:44:05 RAC: ora.RDB.SERV1.RDB1.srv: up: "
"14-Nov-2007 09:45:01 CRS ora.RDB.SERV1.RDB1.srv started on
member vx0313"
"14-Nov-2007 09:44:05 CRS ora.RDB.SERV1.RDB1.srv was modified"
"14-Nov-2007 09:45:06 RAC: ora.RDB.RDB1.inst: imstop: "
"14-Nov-2007 09:45:06 CRS ora.RDB.RDB1.inst stopped"
"14-Nov-2007 09:45:06 CRS ora.RDB.RDB1.inst is transitioning
from state OFFLINE to state ONLINE on member vx0306" s a
"14-Nov-2007 09:45:06 CRS is requested to perform action check
)h a
on resource ora.RDB.db by ora.RDB.RDB1.inst"
m ฺbr
c o deฺ
"14-Nov-2007 09:45:14 CRS is requested to perform action check
i ฺ
r e d Gui
on resource ora.RDB.SERV1.cs by ora.RDB.SERV1.RDB1.srv"
@ sic dent
r de Stu
e
v i lav e this
"14-Nov-2007 09:45:45 RAC: ora.RDB.RDB1.inst: up: "
o_ o us
"14-Nov-2007 09:45:45 RAC: ora.RDB.RDB1.inst: imstop: "
ic i
( f abr nse t
rde le lice
"14-Nov-2007 09:45:45 CRS ora.RDB.RDB1.inst started on member
vx0306" e
V ab
i l a
V sfer
i o
ic -tran
b r
Fa non
c) Here is what you should see in the second session:

[oracle@vx0306 less07]$ cat sol_07_01_06_b.sh
#!/bin/ksh
#
# sol_07_01_06_b.sh
#
#


's/ora_dbw0_/''/'`

I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl status database -d $DBNAME
ps -ef | grep "ora_smon_$I1NAME" | grep -v grep | grep -v

killsmon | awk '{print "kill -9 " $2 }' >
/home/oracle/solutions/less07/z.sh
chmod 777 /home/oracle/solutions/less07/z.sh
s a
echo "Killing smon..."
)h a
m ฺ br
i ฺ c o deฺ
r e d Gui
echo "waiting for instance back ..."
@ sic dent
sleep 120
e r de Stu
v i lav e this
o_ o us
ic i
( f a br se t
$ORACLE_HOME/bin/sqlplusn-s /NOLOG <<EOF
e rde le lice
connect V
l a rab
system/oracle1@$I1NAME
V
selecti f e
instance_name from v\$instance;
i o
ic connect
show s
ansystem/oracle1@$I2NAME
parameter service
b r - t r
Fa noselectn instance_name from v\$instance;
EOF
[oracle@vx0306 less07]$ ./sol_07_01_06_b.sh

Instance RDB1 is running on node vx0306
Instance RDB2 is running on node vx0313
Killing smon...
waiting for instance back ...
INSTANCE_NAME
----------------
RDB1
NAME TYPE VALUE

------------------------------------ ----------- -------------
-----------------
service_names string RDB
INSTANCE_NAME
----------------
RDB2
NAME TYPE VALUE

------------------------------------ ----------- -------------
-----------------
service_names string SERV1, RDB
s a
7) Using Grid Control, check that SERV1 is running on RDB2. )h a
m ฺ br
a) Connect as user SYS in Database Control. On the Cluster Home page, click theฺ
o Database
d i
Availability tab. On the Availability page, click the Cluster Managed
e
ฺ c
u i de
r
Services link in the Services section. On the Cluster andicDatabase t G page,
Login
s n
click Continue.
r d e@ Stude
b) On the Cluster Managed Database Services
i l a vepage,thyou
is should see SERV1 up and
running on the second instance only
i o _v use
with a warning.
c) On Cluster Managed Database
a b ric Services
e to page, click the SERV1 link.
d) On the SERV1 page,e (f you should
e ns see that SERV1 is stopped on the preferred
d
erup onbthe
instance and li c
available instance.
a V l e
V il fera
r i c i rans
o
t
Fab non-

Practice 7-2: Monitor Services
The goal of this practice is to use Database Control to determine the amount of resources
used by sessions executing under a particular service. You also use Database Control to
relocate a service to another instance.
1) Execute the createjfv.sh script. This script creates a new user called JFV identified by
the password JFV. The default tablespace of this user is USERS, and its temporary
tablespace is TEMP. This new user has the CONNECT, RESOURCE, and DBA
roles.
[oracle@vx0306 less07]$ cat createjfv.sh
#!/bin/ksh
#
# createjfv.sh
#
s a
)h a
ฺbr
#
c m
o deฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` ed i ฺ u i
r G
@ sic dent
DBNAME=`ps -ef | grep dbw0_RDB | grep
e r de-v grep
S tu| grep -v
callout1 | awk '{ print $8 }' l|av
v i t his | sed
sed 's/1/''/'
e
's/ora_dbw0_/''/'`
ic i o_ o us
I1NAME=$DBNAME"1" fab
r et
d e (
c e ns
V er ble li
I2NAME=$DBNAME"2"
V ilaORACLE_SID=$I1NAME
f e ra
o
export
icexport n s
i raORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
b r - t
Fa non $ORACLE_HOME/bin/sqlplus -s /NOLOG <<EOF
connect / as sysdba
drop user JFV cascade;
create user JFV identified by jfv default tablespace users
grant connect, resource, dba to JFV;
EOF
[oracle@vx0306 less07]$ ./createjfv.sh
User dropped.
User created.

Practice 7-2: Monitor Services (continued)
Grant succeeded.
2) From a terminal session connected to node1, using SQL*Plus, connect to SERV1
with user JFV. When connected, determine the instance on which your session is
currently running. Then execute the following query: select count(*) from
dba_objects,dba_objects,dba_objects. Do not wait; instead, proceed with
the next step.
#!/bin/ksh
#
# sol_07_02_02.sh
#
# Must be executed on NODE1 !!!!!!!!!! s a
#
)h a
m ฺbr
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` diฺc
o deฺ
r e u i
sic dent G
@
e-v grep tu| grep -v
e rd's/1/''/'
S
callout1 | awk '{ print $8 }' | v
i l a sed
t h is | sed
's/ora_dbw0_/''/'`
i o _v use
I1NAME=$DBNAME"1"
a b ric e to
e (f ens
e r le lic
I2NAME=$DBNAME"2"d
l a V ab
i
export
f er
V ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
ORACLE_SID=$I1NAME
i o
ic -tran
export s
b r
Fa n$ORACLE_HOME/bin/sqlplus
on -s /NOLOG <<EOF
connect jfv/jfv@SERV1
select count(*) from dba_objects,dba_objects,dba_objects;
EOF
INSTANCE_NAME
----------------
RDB2

3) After a while, go to the Database Control Top Consumers page from the Cluster
Database page. Connect as user SYS. Then check that SERV1 is using more and
more resources.
a) From the Cluster Database Home page, click the Performance tab.
b) On the Performance page, click the Top Consumers link in the Additional
Monitoring Links section.
c) This takes you to the Top Consumers page with the Overview tab selected.
d) On the Overview page, you can see the Top Services pie chart.
e) Make sure the View Data dropdown list is set to Real Time: 15 Second Refresh.
Wait for the page to be refreshed a couple of times. Little by little, SERV1 is
consuming almost all the resources (up to 100%).
f) To have more details, click Top Services tab on the Top Consumers page. s a
)h a
m ฺbr
g) Make sure the View Data dropdown list is set to Real Time: 15 Second Refresh,
and View dropdown list is set to Active Services. You can click the “+” icon on
i ฺ c o deฺ
r e d Gui
the left of the SERV1 link to expand the service. This shows you the list of
@ sic dent
instances currently running the service. You can also click the SERV1 link itself
to look at the detailed Statistics of the corresponding service.
e r de fromSatSQL*Plus
u
i l a v this
4) Check statistics on your service with gv$service_stats session
connected as SYSDBA.
i o _v use
to
#!/bin/ksh
e (f ens
#
e r le lic
d
l a V ab
# sol_07_02_04.sh
# i er
V be sexecuted
#ioMust f
r i c t r a n on NODE1 !!!!!!!!!!
Fab non-
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

select stat_name, sum(value) from gv\$service_stats where
service_name = 'SERV1' group by stat_name;
EOF
INSTANCE_NAME
----------------
RDB1
STAT_NAME
SUM(VALUE) s a
--------------------------------------------------------------
)h a
-- ----------
m ฺbr
user calls
i ฺ c o deฺ
21
r e d Gui
DB CPU
212910000 @ sic dent
redo size
e r de Stu
1076
v i lav e this
o_ o us
db block changes
6
ic i
DB time
( f abr nse t
rde le lice
216690394
e
user rollbacks
V ab
0
i l a
V sfer
gc cr blocks received
i o
ic -tran
8
b r
Fa non
gc cr block receive time
0
gc current blocks received
6
opened cursors cumulative
42
workarea executions - multipass
0
STAT_NAME
SUM(VALUE)
--------------------------------------------------------------
-- ----------
session cursor cache hits
8
user I/O wait time
10798
parse count (total)
41
physical reads 2

gc current block receive time 0
workarea executions - optimal 11
concurrency wait time 7522
parse time elapsed 78594
physical writes 0
workarea executions - onepass 0
execute count 41
STAT_NAME SUM(VALUE)
--------------------------------------------------------------
-- ----------
session logical reads 3377
cluster wait time 7896
application wait time 0
logons cumulative 1
sql execute elapsed time 216621181 s a
user commits 0
)h a
m ฺ br
28 rows selected.
i ฺ c o deฺ
r e d Gui
@ sic dent
5) Using Database Control, relocate SERV1 to its preferred
r d e instance.
S tu
a) Click the Cluster Database link at theila
e
topvof your is Database Control page.
v e t hcurrent
i o_ o page
b) This takes you back to the Performance
ic us for your database. Click the
Availability tab. ab nser t
( f
e rde lpage,
c) On the Availability l i e Cluster Managed Database Services.
cclick
e Database Services: Cluster and Database Login page,
d) On the l a V aManaged
Cluster b
i
V Continue.
click f er
i o s
ice) On-trtheanCluster Managed Database Services page, click SERV1.
b r
Fa non f) On the Cluster Managed Database Services: SERV1 page, select the Available
instance (RDB2 in this example) and click Relocate.
g) On the Relocate Service from instance page, make sure the Preferred instance
(RDB1 in this example) is selected, and click OK.
h) On the Cluster Managed Database Service: SERV1 page, you should see a
successful Service Relocate status at the top of the page. Also, the Instances table
should indicate that SERV1 is now running on its preferred instance.
6) What happens to your already connected SERV1 session running on the second
instance?
a) You can see that although the service has been switched to RDB1, your session is
still executing under SERV1 on RDB2. So, if you want to manually relocate a
service, you should make sure that no session is currently connected. From a
terminal window, execute the following.
#!/bin/ksh

#
# sol_07_02_06.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
s a
I2NAME=$DBNAME"2"
)h a
m ฺ br
i ฺ c o deฺ
r e d Gui
$ORACLE_HOME/bin/sqlplus -s /NOLOG <<EOF @ sic dent
connect system/oracle1@$I1NAME ve
rde s Stu
v i la e thifrom gv\$session where
o_ o us
select inst_id, username, service_name
username = 'JFV';
ic i
( f a br se t
EOF
r d eless07]$
l i c en./sol_07_02_06.sh
e ble
[oracle@vx0306
V
V a
INST_ID e ra
il fUSERNAME
o s
i ran ------------------------------
ic----------
b r t
on-
Fa n--------------------------------------------------------------
SERVICE_NAME
--
2 JFV
SERV1
b) If your first session is still executing the query, stop its execution by pressing
[Ctrl] + [C].

Practice 7-3: Alert Thresholds and Services
The goal of this practice is to set thresholds to service SERV1, and use Database Control
to monitor the response time metric for this service. In this practice, you create the
following configuration:
Service Name Usage Preferred Available Response Time (sec)–

Instances Instances Warning/Critical
SERV1 Client service RDBA1 RDBA2 0.4 1.0
1) Set alert thresholds for your service SERV1 by using Database Control. Specify the
values defined above.
a) From the Cluster Database Home page, click the link corresponding to your first
instance in the Instances table. This is the instance currently running SERV1.
s a
b) On the Database Instance page, click Metric and Policy settings in the Related
)h a
Links section at the bottom of the page.
m ฺbr
c) On the Metric and Policy Settings page, select All metrics fromithe ฺ c oView deฺ
dropdown list. r e d Gui
ic ent
syou
d) Scroll down the Metric and Policy Settings pageeuntil
d @ t d the Service
find
u
Response Time (per user call) (microseconds)
v r
e metric. s S
l a h i
t icon in the last column (Edit
o _ vi multi-pens
e) On the same line, click the corresponding
s e
column).
b r ici to u
f) On the Edit Advanced (faSettings:n e Response Time (per user call)
sService
(microseconds) e
rd page,leclick
l e
ic Add.
e
V aObjects
V a
g) TheilMonitored
f e r b table should now show two entries.
i ch) s in the Service Name field, 40000000 in the Warning Threshold
io EnterraSERV1
n
r -t and 100000000 in the Critical Threshold field. Make sure the corresponding
Fab nonfield,
line is selected, and click Continue.
i) On the Metric and Policy Settings page, you should see an Information warning
explaining that your settings have been modified but not saved. Click OK to save
the new settings.
j) On the Confirmation page, you can see an Update succeeded message. Click OK.
k) This takes you back to the Database Instance page.
2) Use Database Control to print the Service Response Time Metric Value graphic for
SERV1.
a) From the Database Instance page, click All Metrics in the Related Links section at
the bottom of the page.
b) On the All Metrics page, expand the Database Services link. On the All Metrics
page, click the Service Response Time (per user call) (microseconds) link.
c) On the Service Response Time (per user call) (microseconds) page, click the
SERV1 link in the Service Name column.

Practice 7-3: Alert Thresholds and Services (continued)
d) On the Service Response Time (per user call) (microseconds): Service Name
SERV1: Last 24 hours page, select Real Time: 30 Second Refresh from the View
Data drop-down list.
e) You should now see the Service Response Time (per user call) (microseconds):
Service Name SERV1 page with your warning and critical thresholds set
correctly.
3) Execute the sol_07_03_03.sh script to generate workload on your database. Looking
at the Service Response time graphic for SERV1, what do you observe?
#!/bin/ksh
#
# sol_07_03_03.sh
#
s a
)h a
ฺbr
#
c m
o deฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` ed
i ฺ u i
r G
@ sic dent
e r de-v grep
S tu| grep -v
his | sed
callout1 | awk '{ print $8 }' il|av t
sed 's/1/''/'
v
o_ o us e
's/ora_dbw0_/''/'`
ic
r eti
I1NAME=$DBNAME"1"(fab
d e c e ns
V er ble li
I2NAME=$DBNAME"2"
V i la fera
i c n s
io raORACLE_SID=$I1NAME
export
r t
Fab non-
connect jfv/jfv@SERV1
DECLARE
t number;
BEGIN
select count(*) into t from dba_objects;
end loop;
END;
/
EOF

… Do not wait!
a) Still looking at the Service Response Time (per user call) (microseconds): Service
Name SERV1 page on your first session, you should see the graphic crossing the
warning threshold after few minutes. This will trigger a warning alert soon after
the warning threshold is crossed.
b) You can see this alert propagated to your Database Instance Home page, and
Cluster Database Home page.
c) To go back to your Database Instance Home page, click the Database Instance a
locator link on the Service Response Time page. as
d) You should see the warning raised in the Alerts section of the Database Instance
)h ฺbr
page. c m
o deฺ
i ฺ
d linkGofuithe
e) On the Database Instance page, click the Cluster Database r e
locator
page.
@ sic dent
f) You should see the warning alert in the Problem e r deServices
S tuline in the High
Availability section of the page. Clicking i lavthiselink
t is you to the Cluster Home
htakes
v
_ SERV1
page. From there you can click
Managed Database Services: r ic i othe
SERV1 t o usafter
link to directly go to the Cluster
Login page. The SERV1 ( f abpagenshowss e you the alertyouwith

page clicked Continue on the
its details.
d e c e
li its execution, you should not see the corresponding
g) Soon afterV ether script
l efinishes
alert
V rab Database Home page anymore. You can go to the Alert
ilaon yourfeCluster
i c io History
a n s on the first instance to look at the alert history for your services. You
page
r r
-t go
Fab nonany
can to the Database Instance Home page using the locator links at the top of
pages. From the Database Instance Home page, scroll down to the bottom of
the page, and click Alert History in the Related Links section.
4) Use Database Control to remove the thresholds that you specified during this practice.
a) From the Cluster Database Home page, click the link corresponding to the first
instance of your cluster in the Instances section at the bottom of the page.
b) On the Database Instance page, scroll down to the bottom of the page. Click
Metric and Policy Settings in the Related Links section.
c) On the Metric and Policy Settings page, scroll down the page until you see
SERV1 in the Metric Thresholds table.
d) On the line corresponding to the SERV1 entry, remove both the Warning
Threshold and Critical Threshold values.
e) Click OK.
f) On the Confirmation page, you should see an Update succeeded message. Click
OK.

5) Execute sol_07_03_05.sh to cleanup your environment.
#!/bin/ksh
#
# sol_07_03_05.sh
#
#


s a
's/ora_dbw0_/''/'`
)h a
m ฺbr
I1NAME=$DBNAME"1"
i ฺ c o deฺ
r e d Gui
sic dent
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl stop service d @

-de $DBNAME
t u -s SERV1 -f
r
ve t-d S
/u01/crs11g/bin/srvctl remove service
i l a h is$DBNAME -s SERV1
export ORACLE_SID=$I1NAMEio_
v se
r ic t o u
b
(fa ense
e
rd le lic -s /NOLOG <<EOF
$ORACLE_HOME/bin/sqlplus
e
V ab
V i
connect l a er cascade;
system/oracle1
i o s f
b r ic execute
drop
- t r andbms_service.delete_service('SERV1');
user jfv
Fa non
EOF

SERV1 PREF: RDB1 AVAIL: RDB2
Remove service SERV1 from the database RDB? (y/[n]) y
…

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 8-1: Create a Server-Side Callout
The goal of this practice is to create a server-side callout program to trap various database
events.
1) Write a shell script that is able to trap FAN events generated by CRS. The events that
need to be trapped must be for your database (RDB in this example), and the event
types that need to be trapped are SERVICEMEMBER (up/down), SERVICE
(up/down), and INSTANCE (up/down). The callout script should create a file that
logs the events that are trapped. The log file must be located in the
$ORA_CRS_HOME/racg/log directory. Make sure that the callout script is deployed
on both nodes.
#!/bin/ksh
#
s a
# sol_08_01_01.sh
)h a
ฺbr
#
c m
o deฺ
#
i ฺ
d Gui
r e
@ sic dent
de'2,2p'` tu
z=`cat /home/oracle/nodeinfo | sed r-n S
e
v i la|vgrep
e t hisgrep | grep -v
ic i o_}' o| sed
DBNAME=`ps -ef | grep dbw0_RDB
us 's/1/''/' | sed
-v
r
callout1 | awk '{ print
's/ora_dbw0_/''/'`fab
$8
t
e ( e n se
e rd le lic
l a V ab
sed 's/RACDATABASE/'$DBNAME'/'
V i f e r
/home/oracle/solutions/less08/wrong_callout1.sh >
r i c io rans
/home/oracle/solutions/less08/callout1.sh
t
Fab ncat on-/home/oracle/solutions/less08/callout1.sh
cp /home/oracle/solutions/less08/callout1.sh
/u01/crs11g/racg/usrco/
chmod 777 /u01/crs11g/racg/usrco/callout1.sh
scp /u01/crs11g/racg/usrco/callout1.sh
$z:/u01/crs11g/racg/usrco/
ssh $z chmod 777 /u01/crs11g/racg/usrco/callout1.sh

[oracle@vx0306 less08]$ cat wrong_callout1.sh

#!/bin/sh
NOTIFY_EVENTTYPE=$1
AWK=/usr/bin/awk
for ARGS in $*; do
PROPERTY=ècho $ARGS | $AWK -F"=" '{print $1}'`

Practice 8-1: Create a Server-Side Callout (continued)
VALUE=ècho $ARGS | $AWK -F"=" '{print $2}'`
case $PROPERTY in
VERSION|version) NOTIFY_VERSION=$VALUE ;;
SERVICE|service) NOTIFY_SERVICE=$VALUE ;;
DATABASE|database) NOTIFY_DATABASE=$VALUE ;;
INSTANCE|instance) NOTIFY_INSTANCE=$VALUE ;;
HOST|host) NOTIFY_HOST=$VALUE ;;
STATUS|status) NOTIFY_STATUS=$VALUE ;;
REASON|reason) NOTIFY_REASON=$VALUE ;;
CARD|card) NOTIFY_CARDINALITY=$VALUE ;;
TIMESTAMP|timestamp) NOTIFY_LOGDATE=$VALUE ;;
??:??:??) NOTIFY_LOGTIME=$PROPERTY ;;
esac
done
if ((( [ $NOTIFY_EVENTTYPE = "SERVICEMEMBER" ] || s a

[ $NOTIFY_EVENTTYPE = "SERVICE" ] ||
)h a
[ $NOTIFY_EVENTTYPE = "INSTANCE" ] \
m ฺbr
) &&
i
\
ฺ c o deฺ
( [ $NOTIFY_STATUS = "up" ] ||
r e d Gui
\
[ $NOTIFY_STATUS
)) &&
= "down" ]
@ sic dent \
\
( [ $NOTIFY_DATABASE = "RACDATABASE"
e r de Stu ] \
))
v i lav e this
o_ o us
then
ic i
echo $* >> /u01/crs11g/racg/log/crsevtco.log
fi
( f abr nse t
rde le lice
e
V aless08]$
b
i l a
[oracle@vx0306
r
./sol_08_01_01.sh
i c i o V ansfe
#!/bin/sh
a b r NOTIFY_EVENTTYPE=$1
- tr
F o n
nAWK=/usr/bin/awk
for ARGS in $*; do
PROPERTY=ècho $ARGS | $AWK -F"=" '{print $1}'`
VALUE=ècho $ARGS | $AWK -F"=" '{print $2}'`
case $PROPERTY in
VERSION|version) NOTIFY_VERSION=$VALUE ;;
SERVICE|service) NOTIFY_SERVICE=$VALUE ;;
DATABASE|database) NOTIFY_DATABASE=$VALUE ;;
INSTANCE|instance) NOTIFY_INSTANCE=$VALUE ;;
HOST|host) NOTIFY_HOST=$VALUE ;;
STATUS|status) NOTIFY_STATUS=$VALUE ;;
REASON|reason) NOTIFY_REASON=$VALUE ;;
CARD|card) NOTIFY_CARDINALITY=$VALUE ;;
TIMESTAMP|timestamp) NOTIFY_LOGDATE=$VALUE ;;
??:??:??) NOTIFY_LOGTIME=$PROPERTY ;;
esac
done
if ((( [ $NOTIFY_EVENTTYPE = "SERVICEMEMBER" ] ||

[ $NOTIFY_EVENTTYPE = "SERVICE" ] ||
[ $NOTIFY_EVENTTYPE = "INSTANCE" ] \
) && \
( [ $NOTIFY_STATUS = "up" ] || \
[ $NOTIFY_STATUS = "down" ] \
)) && \
( [ $NOTIFY_DATABASE = "RDB" ] \
))
then
echo $* >> /u01/crs11g/racg/log/crsevtco.log
fi
callout1.sh
100% 1210 1.2KB/s 00:00
2) Create two new services. The first one is called SCO1 and has RDB1 as its preferred
s a
)h
instance, and RDB2 as its available instance. The second is called SCO2 and has a
RDB2 as its preferred instance, and RDB1 as its available instance.
m ฺbr
i ฺ c o deฺ
#!/bin/ksh r e d Gui
#
@ sic dent
# sol_08_01_02.sh
e r de Stu
lav e this
#
v i
#
ic i o_ o us
( f abr nse t
r d e lice
e
V ab l e
i l
V sf-efa er | grep dbw0_RDB | grep -v grep | grep -v
i o
DBNAME=`ps
iccallout1 an| awk '{ print $8 }' | sed 's/1/''/' | sed
b r - t r
Fa n's/ora_dbw0_/''/'`
on
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl add service -d $DBNAME -s SCO1 -r

$I1NAME -a $I2NAME
/u01/crs11g/bin/srvctl add service -d $DBNAME -s SCO2 -r

$I2NAME -a $I1NAME

3) Run the following command on one terminal window on the first node. We call that
window TW1N1: evmwatch -A -t "@timestamp @@"
Do not wait; proceed with the next step.

#!/bin/ksh
#
# sol_08_01_03.sh
#
#
/u01/crs11g/bin/evmwatch -A -t "@timestamp @@"

4) Run the following command on one terminal window on the second node. We call s a
that window TW1N2: evmwatch -A -t "@timestamp @@"
)h a
Do not wait; proceed with the next step.
m ฺbr
i ฺ c o deฺ
r e d Gui
sic dent
#!/bin/ksh
#
# sol_08_01_04.sh @
de Stu
e r
lav e this
#
# Must be executed on NODE2 v i
!!!!!!!!!!
#
ic i o_ o us
( f abr nse t
r de lice -A -t "@timestamp @@"
/u01/crs11g/bin/evmwatch
e
[oracle@vx0313
l a le
V aless08]$
b
i
V sfer less08]$ ./sol_08_01_04.sh
i o
ic -tran
[oracle@vx0313
b r
Fa 5) nFrom
on another terminal window connected to node1 (TW2N1), start the SCO1 service
using srvctl. When finished, look at terminal windows TW1N1 and TW1N2, and then
look at the generated log files in the /u01/crs1020/racg/log directory by using
TW2N1. What do you observe?
a) You should see that only the log on the first instance contains the start of SCO1.
b) On TW2N1 you can see:
#!/bin/ksh
#
# sol_08_01_05.sh
#
#


's/ora_dbw0_/''/'`
/u01/crs11g/bin/srvctl start service -d $DBNAME -s SCO1
sleep 10
cat /u01/crs11g/racg/log/crsevtco.log
ssh $z cat /u01/crs11g/racg/log/crsevtco.log

[oracle@vx0306 less08]$ ./sol_08_01_05.sh s a

SERVICE VERSION=1.0 service=SCO1 database=RDB instance=
)h a
host=vx0306 status=up reason=user timestamp=15-Nov-2007
m ฺbr
03:11:27
i ฺ c o deฺ
SERVICEMEMBER VERSION=1.0 service=SCO1 database=RDB
r e d Gui
instance=RDB1 host=vx0306 status=up reason=user card=1
timestamp=15-Nov-2007 03:11:27 @ sic dent
r de Stu
cat: /u01/crs11g/racg/log/crsevtco.log: No such file or
e
directory
v i lav e this
o_ o us
ic i
c) On TW1N1 you can see:
( f abr nse t
[oracle@vx0306 eless08]$
rd03:11:27 l i e ./sol_08_01_03.sh
cCRS
"15-Nov-2007
V e b l eto stateora.RDB.SCO1.cs is transitioning
V a a
l fer03:11:27
from istate OFFLINE ONLINE on member vx0306"
i c i o ans from state OFFLINE to state ONLINEison member

"15-Nov-2007 CRS ora.RDB.SCO1.RDB1.srv
a b r vx0306" - tr
transitioning
F no n
"15-Nov-2007 03:11:27 RAC: ora.RDB.SCO1.RDB1.srv: up: "
"15-Nov-2007 03:11:27 RAC: ora.RDB.SCO1.cs: up: "
"15-Nov-2007 03:11:27 CRS ora.RDB.SCO1.RDB1.srv started on
member vx0306"
"15-Nov-2007 03:11:27 CRS ora.RDB.SCO1.RDB1.srv was modified"
"15-Nov-2007 03:11:28 CRS ora.RDB.SCO1.cs started on member
vx0306"
d) On TW1N2 you can see:

"15-Nov-2007 03:11:27 CRS ora.RDB.SCO1.cs is transitioning
from state OFFLINE to state ONLINE on member vx0306"

"15-Nov-2007 03:11:27 CRS ora.RDB.SCO1.RDB1.srv is
vx0306"
member vx0306"
vx0306"
"15-Nov-2007 03:11:16 RAC: ora.vx0313.ASM2.asm: imup: " s a
)h a
m ฺbr
i ฺ c o deฺ
6) From TW2N1, start the SCO2 service using srvctl. When finished,
r e d look G u i
at terminal
windows TW1N1 and TW1N2, and then look at the generated
@ sic loge n
files t in the
r d e doSyou
/u01/crs1020/racg/log directory by using TW2N1. What
t udobserve?
i l ve instance
a) You should see that only the log on theasecond
t his contains the start of
SCO2. v
o_ o us e
ic i
b) On TW2N1 you can see:
( f abr nse t
[oracle@vx0306
e rdeless08]$l i ce cat sol_08_01_06.sh
#!/bin/ksh
l a V able
# i
V sfer
#osol_08_01_06.sh
r i
ic# -tran
b
Fa n##oMustn be executed on NODE1 !!!!!!!!!!


's/ora_dbw0_/''/'`
/u01/crs11g/bin/srvctl start service -d $DBNAME -s SCO2
sleep 10
cat /u01/crs11g/racg/log/crsevtco.log
ssh $z cat /u01/crs11g/racg/log/crsevtco.log


03:11:27
timestamp=15-Nov-2007 03:11:27
03:13:52
timestamp=15-Nov-2007 03:13:52
s a
c) On TW1N1 you can see:
)h a
"15-Nov-2007 03:14:50 CRS ora.RDB.SCO2.cs is transitioning
m ฺbr
from state OFFLINE to state ONLINE on member vx0313"
i ฺ c o deฺ
r e d Gui
sic dent
@
vx0313"
e r de Stu
i lav e this
v
ic i o_ o us
abr nse t
vx0313"
( f
member vx0313"
e rde le lice
l a V ab
i
V TW1N2
d) On f r can see:
eyou
i o s
an 03:14:50 CRS ora.RDB.SCO2.cs is transitioning
ic"15-Nov-2007
b r - t r
Fa nfrom on state OFFLINE to state ONLINE on member vx0313"
vx0313"
vx0313"
member vx0313"
7) From TW2N1, stop both services using srvctl, and then remove both services. When
you are finished, remove callout1.sh script from the /u01/crs11g/racg/usrco directory
as well as /u01/crs11g/racg/log/crsevtco.log from both nodes. When you are finished,
stop both evmwatch sessions with Ctrl-C.

#!/bin/ksh
#
# sol_08_01_07.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1" s a
)h a
I2NAME=$DBNAME"2"
m ฺbr
i ฺ c o deฺ
/u01/crs11g/bin/srvctl stop service -d $DBNAME -s
r e d Gui
SCO1
/u01/crs11g/bin/srvctl stop service -d @ sic d-senSCO2

$DBNAME
t
e r de Stu
/u01/crs11g/bin/srvctl remove service
i l a v t-d h is$DBNAME -s SCO1
/u01/crs11g/bin/srvctl c i o _v service
u se -d $DBNAME -s SCO2
a b ri e to
remove
e (f ens
rm /u01/crs11g/racg/usrco/callout1.sh
e r le lic
d
l a V ab
rm /u01/crs11g/racg/log/crsevtco.log
i er
V$z rm s/u01/crs11g/racg/usrco/callout1.sh
f
i o
ic -tran
ssh
b r
Fa nssh on $z rm /u01/crs11g/racg/log/crsevtco.log
connect / as sysdba
exec DBMS_SERVICE.DELETE_SERVICE('SCO1');
exec DBMS_SERVICE.DELETE_SERVICE('SCO2');
EOF

SCO1 PREF: RDB1 AVAIL: RDB2
Remove service SCO1 from the database RDB? (y/[n]) y
SCO2 PREF: RDB2 AVAIL: RDB1
Remove service SCO2 from the database RDB? (y/[n]) y

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 8-2: Use Load Balancing Advisory
The goal of this practice is to test the Load Balancing Advisory on the listener side only
to determine how connections are spread across your instances while running an OLTP-
type of workload.
1) Create two new services using Database Control:
First one is called SNOLBA and should be defined with the following parameters:
goal => DBMS_SERVICE.GOAL_NONE, clb_goal =>
DBMS_SERVICE.CLB_GOAL_LONG.
Second one is called SLBA and should be defined with the following parameters:
goal => DBMS_SERVICE.GOAL_SERVICE_TIME, clb_goal =>
DBMS_SERVICE.CLB_GOAL_SHORT.
Basically, SNOLBA does not use the Load Balancing Advisory, while SLBA uses it.
Make sure that both services have both instances as preferred, and start them both.
s a
a) Connect to your Database Control console as user SYS.
)h a
b) On the Cluster Database Home page, click the Availability tab. m ฺbr
i
c) On the Availability page, click the Cluster Managed DatabasedServices ฺco ulink i d eฺthe
in
Services section.
s icre ent G
d) On the Cluster and Database Login page, enter e all@ udcredentials if not
necessary
t
r d S
done already, and click Continue.
i l a ve this
e) On the Cluster Managed Database
i o _vServices
u s e click the Create Service button.
page,
b
f) On the Create Serviceapage, ricentereSNOLBA
to in the Service Name field, and select
e (f Load
Long as the Connection e s
nBalancing Goal option. Then click OK.
r d
enow see l i c
g) You should
l a V a b lea Create Service: Successful status on the Cluster Managed
i
Database er On this page, you should also see that SNOLBA is started on
Services.
V instances.
f
i o
ic -tran
both s
b r
Fa nh)onOn the Cluster Managed Database Services page, click Create Service.
i) On the Create Service page, enter SLBA in the Service Name field, and select
Short as the Connection Load Balancing Goal option.
j) In the Notification Properties section, check Enable Load Balancing Advisory,
and make sure Service Time option is selected. Then click OK.
k) You should now see a Create Service: Successful status on the Cluster Managed
Database Services. On this page, you should also see that SLBA is started on both
instances.
2) Add corresponding service names to your tnsnames.ora files on both nodes. We
recommend that you add them to both tnsnames.ora files: the one residing in your
ASM home and the one residing in your Database home.
#!/bin/ksh
#
# sol_08_02_02.sh
#

Practice 8-2: Use Load Balancing Advisory (continued)
#


's/ora_dbw0_/''/'`
sed 's/NODE1/'$y'/'
/home/oracle/solutions/less08/wrong_tnslba.ora >
/home/oracle/solutions/less08/w1tnslba.ora
sed 's/NODE2/'$z'/' /home/oracle/solutions/less08/w1tnslba.ora s a

> /home/oracle/solutions/less08/tnslba.ora
)h a
m ฺ br
cp
i ฺ c o deฺ
r e d Gui
.bak1 @ sic dent

r de Stu>>
cat /home/oracle/solutions/less08/tnslba.ora
e
v i lav e this
cp
ic i o_ o us
a br se t
( f
a
r d e licen
e ble
a.bak2 V
cat V
a
il fera
/home/oracle/solutions/less08/tnslba.ora >>
i o
ica -tran s
b r
Fa non
ssh $z cp
.bak1
ssh $z cp
a
a.bak2
scp /home/oracle/solutions/less08/tnslba.ora
ssh $z "cat /home/oracle/solutions/less08/tnslba.ora >>

"

ssh $z "cat /home/oracle/solutions/less08/tnslba.ora >>
a"
[oracle@vx0306 less08]$ cat wrong_tnslba.ora
SNOLBA =
(DESCRIPTION =
1521))
1521))
(CONNECT_DATA = s a
)h a
(SERVICE_NAME = SNOLBA)
m ฺbr
)
i ฺ c o deฺ
)
r e d Gui
SLBA = @ sic dent
(DESCRIPTION =
e r de Stu
(ADDRESS = (PROTOCOL = TCP)(HOST
i l a is
v =thNODE1-vip)(PORT =
1521))
o v se
_TCP)(HOST
(ADDRESS = (PROTOCOL
r ic i =
o u = NODE2-vip)(PORT =
1521))
a b e t
(LOAD_BALANCE(f= yes)ns
e
rd le= lic e
e
(CONNECT_DATA
V a= bDEDICATED)
i l a
(SERVER
V) sfer
(SERVICE_NAME = SLBA)
i o
ic ) -tran
b r
Fa n[oracle@vx0306
on less08]$

tnslba.ora 100% 534
0.5KB/s 00:00
3) From a terminal window, execute the createfan.sh script. This script creates a simple
table used by the following scripts in this practice. This script is located in your
$HOME/solutions/less08 directory.
[oracle@vx0306 less08]$ cat createfan.sh
#!/bin/ksh
#
# createfan.sh
#
#



's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
s a
)h a
m ฺ br
connect / as sysdba
i ฺ c o deฺ
drop user JFV cascade;
r e d Gusers u i
temporary tablespace temp; @ s c dent
create user JFV identified by jfv default itablespace
grant connect, resource, dba to JFV;

e r de Stu
connect jfv/jfv
create table fan(c number);vil
av this
i o _ use
EOF
a b ric e to
e (f ens
e r le lic
d
a
[oracle@vx0306
l V aless08]$
b ./createfan.sh
i fe r
i c i o Vdropped.n s
tra
User
a b r -
F no n
User created.
Grant succeeded.
Table created.
4) Make sure you keep your terminal window (referred to as TW1) open. You use it
later.
5) Create a new terminal window from where you will start the workload. This terminal
window is referred to as TW2.

6) Create a new terminal window from where you will execute the following SQL
statement as SYSDBA under SQL*Plus: select inst_id,count(*) from
gv$session where username='JFV' group by inst_id order by
inst_id;
This statement count the number of sessions connected as user JFV on both instances.
This terminal window is referred to as TW3.
#!/bin/ksh
#
# sol_08_02_03.sh
#
#
s a
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` )h a
m ฺbr
callout1 | awk '{ print $8 }' | sed 's/1/''/'
s i cre| sedn t G
's/ora_dbw0_/''/'`
r d e@ Stude
I1NAME=$DBNAME"1"
i l a ve this
i o _v use
I2NAME=$DBNAME"2"
a b ric e to
e (f ens
e r le lic
d
l a V ab
i
V sfer
$ORACLE_HOME/bin/sqlplus /NOLOG @sol_08_02_06_00.sql
i o
ic -tran
b r
Fa non
[oracle@vx0306 less08]$ cat sol_08_02_06_00.sql

connect / as sysdba
host echo "execute sol_08_02_06.sql from here"

connect / as sysdba
[oracle@vx0306 less08]$ cat sol_08_02_06.sql
select inst_id,count(*) from gv$session where username='JFV'

group by inst_id order by inst_id;

SQL*Plus: Release 11.1.0.6.0 - Production on Thu Nov 15
05:02:47 2007
Connected.
execute sol_08_02_06.sql from here
SQL> @sol_08_02_06
no rows selected
SQL>
s a
7) Create two additional terminal windows from where you will run the primes
)h a
ฺbr
executable to generate CPU load on your first node. These terminal windows are
m
called TW4 and TW5, respectively.
i ฺ c o deฺ
e
8) In TW2, execute the startfanload.sh script using SNOLBA as rthe u i to the
dfirst argument
command. sic dent G
@
de Stu
[oracle@vx0306 less08]$ cat startfanload.sh
e r
/home/oracle/solutions/less08/fan.sh
v i lav e $1 t hi&&s
ic i o_ o us $1 &
/home/oracle/solutions/less08/fan.sh $1
abr nse t
( f
e
$1 &
rd le lic e
/home/oracle/solutions/less08/fan.sh $1 &
V e
b
$1 &
a
il fera
V
$1 &
i c i o ans
b r t r
-
$1 &
Fa n/home/oracle/solutions/less08/fan.sh
on
$1 &
[oracle@vx0306 less08]$ cat fan.sh

#!/bin/ksh
#
# fan.sh
#
users=120
x=1
y=$users

UNPW="jfv/jfv@"$1
ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
export ORACLE_HOME
while [ $x -le $y ]
do
/u01/app/oracle/product/11.1.0/db_1/bin/sqlplus -s $UNPW
@fan.sql
done
[oracle@vx0306 less08]$ cat fan.sql
begin
insert into fan select sid from v$mystat where rownum<2;
commit; s a
delete fan where c in (select sid from v$mystat where
)h a
rownum<2);
m ฺbr
commit;
i ฺ c o deฺ
end loop;
r e d Gui
end;
/ @ sic dent
exit;
e r de Stu
v i lav e this
ic i o_ o us
( f a br se t
[oracle@vx0306 less08]$ n
r d eless08]$
l i c e ./startfanload.sh SNOLBA
V e ble
[oracle@vx0306 begin
*
V a
ilat linee ra1:
ERROR
o s f
i ran deadlock detected while waiting for resource
b r icORA-00060:
t
Fa non-
ORA-06512: at line 5
begin
*
ERROR at line 1:
ORA-00060: deadlock detected while waiting for resource
begin
*
ERROR at line 1:
begin
*
ERROR at line 1:


s a
begin
)h a
*
m ฺbr
ERROR at line 1:
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f
9) Start the primes executableabinrbothnsTW4
e t and TW5.
e
a) In TW4, you
desee: lice
rcan
l a V able
i
[oracle@vx0306f er less08]$
VGot primestore ./primes
i o an s
icDBG:-tInitialized.
DBG: ptr is 0xb1ba7008
b r r
Fa nnext_prime=0xb1ba7010,cur_prime=0xb1ba700c,*next_prime=0,*cur_
on
prime=3
pp_10K=1700
pp_10K=1422
pp_10K=1353
pp_10K=1312
pp_10K=1275
pp_10K=1258
pp_10K=1218
pp_10K=1222
pp_10K=1199
pp_10K=1203 pp_100K=13162
pp_10K=1178
pp_10K=1165
b) In TW5, you can see:

[oracle@vx0306 less08]$ ./primes
DBG: Got primestore ptr is 0xb1bbf008

DBG: Initialized.
next_prime=0xb1bbf010,cur_prime=0xb1bbf00c,*next_prime=0,*cur_
prime=3
pp_10K=1700
pp_10K=1422
pp_10K=1353
pp_10K=1312
pp_10K=1275
pp_10K=1258
pp_10K=1218
pp_10K=1222
pp_10K=1199
pp_10K=1203 pp_100K=13162
pp_10K=1178
pp_10K=1165
pp_10K=1177 s a
pp_10K=1151
)h a
pp_10K=1149
m ฺbr
10) Monitor TW3 by repeating the execution of the SQL statement. What i ฺ c o
do you ide
ฺ
observe?
i c red t Gu
a) You should see approximately the same numbereof @ s deon
connections
n both nodes—
e
perhaps a bit more on the second node because r tu has much more load
d the firstSnode
i lav before
than the second. Wait at least threevminutes t s figures for your analysis.
hiusing
i
This is the time necessary before o_the LBA e
usalgorithm takes place.
r ic t o
b) In TW3, you can see:fab se
e ( e n
[oracle@vx0306 e rd less08]$
e l ic ./sol_08_02_06.sh
l a V ab l
i
V 2007
SQL*Plus: f er 11.1.0.6.0 - Production on Thu Nov 15
Release
i o
ic -tran
05:02:47 s
b r
Fa nCopyright
on (c) 1982, 2007, Oracle. All rights reserved.
Connected.
execute sol_08_02_06.sql from here
SQL> @sol_08_02_06
no rows selected
SQL> /
no rows selected
SQL> /
INST_ID COUNT(*)
---------- ----------
1 9
2 9

SQL> /
INST_ID COUNT(*)
---------- ----------
1 8
2 10
SQL> /
INST_ID COUNT(*)
---------- ----------
1 8
2 10
SQL> / s a
)h a
INST_ID COUNT(*)
m ฺbr
---------- ----------
i ฺ c o deฺ
1 8
r e d Gui
2 10
@ sic dent
SQL> /
e r de Stu
v i lav e this
INST_ID COUNT(*)
---------- ---------- cio
_ us
9r
b i to
1
a
(f 9 ens e
2
e
r le lic
d
SQL> / V
e
V ila ferab
i c io INST_ID
a n s COUNT(*)
r t r
Fab non- 1
---------- ----------
8
2 10
SQL> /
INST_ID COUNT(*)
---------- ----------
1 9
2 9
SQL> /
INST_ID COUNT(*)
---------- ----------
1 8
2 10
SQL>

11) Stop primes generation by pressing [Ctrl] + [C] in both corresponding terminal
windows: TW4 and TW5.
a) In TW4, you can see
pp_10K=786 pp_100K=8091
pp_10K=799
pp_10K=767
pp_10K=792
pp_10K=838
pp_10K=802
pp_10K=806
pp_10K=813
pp_10K=818
pp_10K=796
pp_10K=790 pp_100K=8021
s a
pp_10K=816
)h a
ฺbr
pp_10K=819
pp_10K=768
c m
o deฺ
pp_10K=796
i ฺ
d Gui
r e
@ sic dent
e r de Stu
pp_10K=817 v i lav e this
pp_10K=806
ic i o_ o us
pp_10K=818
pp_10K=793 ( f abr nse t
pp_10K=804 rde l i ce
pp_10K=805 e le
V pp_100K=8042
b
i l
V sfer
pp_10K=798 a a
i o
icpp_10K=803
pp_10K=791
an
b r - t r
Fa npp_10K=789
on
pp_10K=801
pp_10K=833
pp_10K=812
pp_10K=792
pp_10K=783
pp_10K=822 pp_100K=8024
pp_10K=811
pp_10K=817
pp_10K=809
pp_10K=790
12) In TW4, execute stopfanload.sh script to stop the workload.
[oracle@vx0306 less08]$ cat stopfanload.sh
#!/bin/ksh
#
# stopfanload.sh

#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
s a
)h a
m ฺ br
ps -ef | grep "fan.sh" | awk '{print "kill -9 " $2 }'
i ฺ c o > deฺ
/home/oracle/solutions/less08/x.sh
r e d Gui
chmod 777 /home/oracle/solutions/less08/x.sh @ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f a
ps -ef | grep "fan.sql"br | sawke t '{print "kill -9 " $2 }' >
r d e licen
e ble
chmod 777V/home/oracle/solutions/less08/x.sh
V a
il fera
r i c i rans
o
t
Fab n$ORACLE_HOME/bin/sqlplus
on- system/oracle1@$I1NAME
@/home/oracle/solutions/less08/kill_fan.sql
$ORACLE_HOME/bin/sqlplus system/oracle1@$I1NAME
@/tmp/drop_fan.sql
rm /tmp/drop_fan.sql
@/home/oracle/solutions/less08/kill_fan.sql
@/tmp/drop_fan.sql
rm /tmp/drop_fan.sql
[oracle@vx0306 less08]$ cat kill_fan.sql

set head off
set timing off
set feedback off;
set pagesize 0
set verify off
spool /tmp/drop_fan.sql;
select 'alter system kill session ''' || sid || ',' || serial#

|| ''';'
from v$session
where username='JFV';
select 'exit;' from dual;
spool off s a
)h a
exit;
m ฺbr
i ฺ c o deฺ
r e d Gui
[oracle@vx0306 less08]$ ./stopfanload.sh s
ic ent
/home/oracle/solutions/less08/x.sh:rd
@
e 19: t d (17917) -
ukill:
line
ve this S
No such process
i l a
i o _v- Production
u s e
SQL*Plus: Release 11.1.0.6.0
b r ic t o on Thu Nov 15
(fa ense
05:11:55 2007
e
rd1982, l ic Oracle. All rights reserved.
Copyright (c) e
V ab l e 2007,
i l
V sto: a er
i o n f
b r icOracle
Connected
- t r aDatabase
Fa nProduction
on
11g Enterprise Edition Release 11.1.0.6.0 -
With the Partitioning, Real Application Clusters, OLAP, Data

Mining
and Real Application Testing options
alter system kill session '96,12499';

exit;
Disconnected from Oracle Database 11g Enterprise Edition
Release 11.1.0.6.0 - Production
Mining

05:11:55 2007
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 -
Production
Mining
System altered. s a
)h a
m ฺ br
System altered.
i ฺ c o deฺ
r e d Gui
System altered. @ sic dent
e r de Stu
v i lav e this
o_ o us
System altered.
ic i
( f a br se t
System altered.
r d e licen
V e ble
V a
il altered.
e ra
System
o
i ran s f
b r icalter
t system kill session '111,13128'
Fa n*on-
ERROR at line 1:
ORA-00030: User session ID does not exist.
alter system kill session '114,27665'

*
ERROR at line 1:
ORA-00030: User session ID does not exist.

Mining

13) Repeat steps 8-9-10-11-12 using the command “startfanload SLBA” in step 8. What
do you observe this time? Wait for at least five minutes before doing your analysis.
This leaves enough time for the LBA to take place.
a) You should see that the listener distributes more connections to the second
instance.
SQL> /
INST_ID COUNT(*)
---------- ----------
1 4
2 11
s a
)h a
SQL> /
m ฺbr
i ฺ c o deฺ
INST_ID COUNT(*)
r e d Gui
sic dent
---------- ----------
1 11
@
de Stu
2 7
e r
SQL> / v i lav e this
ic i o_ o us
INST_ID COUNT(*)
( f abr nse t
1 rde 9ice
---------- ----------
e
V2 able 8 l
i l a
V/ sfer
i o
ic -tran
SQL>
b r
Fa nonINST_ID COUNT(*)
---------- ----------
1 7
2 10
SQL> /
INST_ID COUNT(*)
---------- ----------
1 4
2 14
SQL> /
INST_ID COUNT(*)
---------- ----------
1 4
2 13

SQL> /
INST_ID COUNT(*)
---------- ----------
1 4
2 15
SQL>
14) In TW1, look at the generated LBA FAN events that were sent during the second
period while using the SLBA service. What do you observe?
a) You should see that percentages are different for both instances after you started
using the SLBA workload.
s a
)h a
ฺbr
#!/bin/ksh
#
c m
o deฺ
# sol_08_02_03.sh
i ฺ
d Gui
#
r e
@ sic dent
de Stu
#
e r
v this
v i lased e-n '1,1p'`
i o _ | sed
|
u s
r
z=`cat /home/oracle/nodeinfo
b ic t o -n '2,2p'`
fa nse | grep -v grep | grep -v
DBNAME=`ps -ef e| (grep dbw0_RDB
callout1 | e awk l ice $8 }' | sed 's/1/''/' | sed
rd '{leprint
l a V ab
i
's/ora_dbw0_/''/'`
V sfer
i o an
icI1NAME=$DBNAME"1"
b r - t r
Fa nI2NAME=$DBNAME"2"
on
$ORACLE_HOME/bin/sqlplus /NOLOG @sol_08_02_14.sql

connect / as sysdba
SELECT TO_CHAR(enq_time,'HH:MI:SS') Enq_time, user_data FROM

sys.sys$service_metrics_tab ORDER BY 1 ;
exit;


05:54:24 2007
Connected.
ENQ_TIME
--------
USER_DATA(SRV, PAYLOAD)
--------------------------------------------------------------
------------------ s a
02:02:28
)h a
SYS$RLBTYP('SLBA', 'VERSION=1.0 database=RDB service=SLBA {
m ฺbr
{instance=RDB2 perce
i ฺ c o deฺ
nt=50 flag=UNKNOWN aff=FALSE}{instance=RDB1 percent=50
r e d Gui
flag=UNKNOWN aff=FALSE} }
timestamp=2007-11-15 04:02:28') @ sic dent
e r de Stu
02:02:58
v i e t his service=SLBA {
lavdatabase=RDB
o_ o us
SYS$RLBTYP('SLBA', 'VERSION=1.0
ic i
( f a br se t
flag=UNKNOWN aff=FALSE} n
rde le lic04:02:58')
e }
e
timestamp=2007-11-15
V ab
i l
V sfera
i o
ic -tran
...
b r
Fa non
ENQ_TIME
--------
--------------------------------------------------------------
------------------
timestamp=2007-11-15 05:21:33')
03:22:04
timestamp=2007-11-15 05:22:04')
03:22:34

ENQ_TIME
--------
--------------------------------------------------------------
------------------
timestamp=2007-11-15 05:22:34')
03:23:04
nt=74 flag=GOOD aff=FALSE}{instance=RDB1 percent=26 flag=GOOD
aff=TRUE} } timest s a
amp=2007-11-15 05:23:04')
)h a
m ฺbr
i ฺ c o deฺ
ENQ_TIME
r e d Gui
--------
USER_DATA(SRV, PAYLOAD) @ sic dent
r de Stu
--------------------------------------------------------------
e
------------------
v i lav e this
o_ o database=RDB
03:23:34
ic
SYS$RLBTYP('SLBA', 'VERSION=1.0
r i
t us service=SLBA {
{instance=RDB2 perceb
e (fa ense
nt=78 flag=GOOD aff=TRUE}{instance=RDB1 percent=22 flag=GOOD
aff=FALSE} e timest lic
}rd
l a
amp=2007-11-15 le
V a05:23:34')
b
i
V sfer
i o an
icSYS$RLBTYP('SLBA',
03:24:04
b r - t r
Fa n{instance=RDB2
on
'VERSION=1.0 database=RDB service=SLBA {
perce
aff=TRUE} } timest
amp=2007-11-15 05:24:04')
ENQ_TIME
--------
--------------------------------------------------------------
------------------
03:24:35
aff=FALSE} } timest
amp=2007-11-15 05:24:35')
03:25:05

aff=TRUE} } timest
ENQ_TIME
--------
--------------------------------------------------------------
------------------
amp=2007-11-15 05:25:05')
03:25:35
nt=41 flag=GOOD aff=TRUE}{instance=RDB1 percent=59 flag=GOOD s a
aff=FALSE} } timest
)h a
amp=2007-11-15 05:25:35')
m ฺbr
i ฺ c o deฺ
03:26:07
r e d Gui {
SYS$RLBTYP('SLBA', 'VERSION=1.0 database=RDB
{instance=RDB2 perce @ sic dent
service=SLBA
e r de Stu
ENQ_TIME
v i lav e this
--------
USER_DATA(SRV, PAYLOAD)cio
_ us
b i
r e to
a
--------------------------------------------------------------
(f ens
d e
------------------
ic
r aff=TRUE}{instance=RDB1
l
e
nt=38 flag=GOOD
l
aff=TRUE}V} timesta
b e percent=62 flag=GOOD
V ila fera05:26:07')
mp=2007-11-15
r i c io rans
t
Fab nSYS$RLBTYP('SLBA',
on-
03:26:38
'VERSION=1.0 database=RDB service=SLBA {
aff=FALSE} } timest
amp=2007-11-15 05:26:38')
03:27:08
ENQ_TIME
--------
--------------------------------------------------------------
------------------
timestamp=2007-11-15 05:27:08')

03:27:38
timestamp=2007-11-15 05:27:38')
ENQ_TIME
--------
--------------------------------------------------------------
------------------
03:28:08
{instance=RDB2 perce s a
)h a
m ฺbr
timestamp=2007-11-15 05:28:08')
i ฺ c o deฺ
r e d Gui
03:28:38
SYS$RLBTYP('SLBA', 'VERSION=1.0 database=RDB @ sic service=SLBA
d e n t {
{instance=RDB2 perce e t u
rd s Spercent=51
v e
v i la e thi
nt=49 flag=UNKNOWN aff=FALSE}{instance=RDB1
o_ o us
ic i
timestamp=2007-11-15 05:28:38')
( f abr nse t
... e rde le lice
l a V ab
i
V sfer
i o
ic--------
ENQ_TIME
an
b r - t r
Fa nUSER_DATA(SRV,
on PAYLOAD)
--------------------------------------------------------------
------------------
03:41:39
timestamp=2007-11-15 05:41:39')
03:42:09
timestamp=2007-11-15 05:42:09')
ENQ_TIME
--------

--------------------------------------------------------------
------------------
200 rows selected.

Mining
15) Stop both services: SNOLBA and SLBA. When you are finished, remove them from
s a
your cluster configuration and your database.
)h a
[oracle@vx0306 less08]$ cat sol_08_02_15.sh m ฺbr
#!/bin/ksh i ฺ c o deฺ
# r e d Gui
# sol_08_02_15.sh
@ sic dent
#
e r de Stu
lav e this
#
v i
ic i o_ o us
( f abr nse |t sed -n '1,1p'`
e rde le lice
l a V -efa|b grep dbw0_RDB | grep -v grep | grep -v

i
DBNAME=`ps
V |sfawk er '{ print $8 }' | sed 's/1/''/' | sed
i o
callout1
an
ic's/ora_dbw0_/''/'`
b r - t r
Fa non I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl stop service -d $DBNAME -s SNOLBA
/u01/crs11g/bin/srvctl stop service -d $DBNAME -s SLBA
/u01/crs11g/bin/srvctl remove service -d $DBNAME -s SNOLBA
/u01/crs11g/bin/srvctl remove service -d $DBNAME -s SLBA
connect / as sysdba

exec DBMS_SERVICE.DELETE_SERVICE ('SNOLBA');
exec DBMS_SERVICE.DELETE_SERVICE ('SLBA');
EOF

SNOLBA PREF: RDB2 RDB1 AVAIL:
Remove service SNOLBA from the database RDB? (y/[n]) y
SLBA PREF: RDB2 RDB1 AVAIL:
Remove service SLBA from the database RDB? (y/[n]) y
PL/SQL procedure successfully completed. s a

)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 8-3: Use Transparent Application Failover
The goal of this practice is to create a service called TAFB that uses the new FAN
propagation system through Advance Queuing to support Transparent Application
Failover.
1) Using Database Control and PL/SQL, create a new service called TAFB that is
configured to do BASIC SESSION TAF.
a) Connect to your Database Control console as user SYS.
b) On the Cluster Database Home page, click the Availability tab.
c) On the Availability page, click the Cluster Managed Database Services link in the
Services section.
d) On the Cluster and Database Login page, enter all necessary credentials if not
done already, and click Continue. s a
)h a
ฺbr
e) On the Cluster Managed Database Services page, click the Create Service button.
m
f) On the Create Service page, enter “TAFB” in the Service Name field.
i ฺ c o deฺ
g) Select Long as the Connection Load Balancing Goal option. r e d Gui
h) Click OK. @ sic dent
e r de Stu
i l a is on the Cluster Managed
v thstatus
i) You should now see a Create Service: Successful
i o _v see that
Database Services. You should also
u s eTAFB is now started on both
instances.
a b ric e to
(f useethe
j) When TAFB is created,
e nsDBMS_SERVICE.MODIFY_SERVICE
d c
e ble li service attributes:
procedure torset the remaining
V
la feraless08]$ cat sol_08_03_01_j.sh
i
[oracle@vx0306
V
r i c io rans
#!/bin/ksh
n-t
Fab n##osol_08_03_01_12.sh
#

#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl add service -d $DBNAME -s TAFB -r

$I1NAME,$I2NAME

Practice 8-3: Use Transparent Application Failover (continued)
/u01/crs11g/bin/srvctl start service -d $DBNAME -s TAFB
connect / as sysdba
execute dbms_service.modify_service (
-
'TAFB' ,
-
aq_ha_notifications => true ,
-
failover_method => dbms_service.failover_method_basic , s a
-
)h a
failover_type => dbms_service.failover_type_session ,
m ฺ br
-
i ฺ c o deฺ ,
failover_retries => 180
r e d Gui
-
failover_delay => 5 @ sic dent ,
- e
rd s S t u
v e
clb_goal
la e thi
=> dbms_service.clb_goal_long);
v i
EOF
ic i o_ o us
( f abr nse t
[oracle@vx0306
e rdeless08]$
le l i ce ./sol_08_03_01_j.sh
PRKP-1083V: The bservice
i
V sfe
PRKP-1062 l a : r a
Service TAFB
TAFB already exists.
is already running.
i o
icPL/SQL n
aprocedure
b r - t r
Fa non
successfully completed.
2) To connect to your database by using the TAFB service, what must you do next?
a) You should add the corresponding TNS entry in your
$ORACLE_HOME/network/admin/tnsnames.ora file. We highly recommend that
you add that entry in both your ASM and Database homes:
#!/bin/ksh
#
# sol_08_03_02.sh
#
#


's/ora_dbw0_/''/'`
sed 's/NODE1/'$y'/'
/home/oracle/solutions/less08/wrong_tnstafb.ora >
/home/oracle/solutions/less08/w1tnstafb.ora
sed 's/NODE2/'$z'/'
/home/oracle/solutions/less08/w1tnstafb.ora >
/home/oracle/solutions/less08/tnstafb.ora
cp s a
)h a
m ฺbr
.bak3
i ฺ c o deฺ
cat /home/oracle/solutions/less08/tnstafb.ora >>
r e d Gui
sic dent
@
cp
e r de Stu
v i lav e this
o_ o us
a
ic i
a.bak4
( f a br se t
r d e licen
cat /home/oracle/solutions/less08/tnstafb.ora >>
e ble
V
a
V a
il fera
r i c i $zracpns
o
ssh
t
Fab n/u01/app/oracle/product/11.1.0/db_1/network/admin/tnsnames.ora
on-
.bak3
ssh $z cp
a
a.bak4
scp /home/oracle/solutions/less08/tnstafb.ora
ssh $z "cat /home/oracle/solutions/less08/tnstafb.ora >>

"
ssh $z "cat /home/oracle/solutions/less08/tnstafb.ora >>

a"

[oracle@vx0306 less08]$ cat wrong_tnstafb.ora
TAFB =
(DESCRIPTION =
1521))
1521))
(CONNECT_DATA =
(SERVICE_NAME = TAFB)
)
) s a
)h a
m ฺ br
i ฺ c o265 deฺ
tnstafb.ora
r d Gui
100%
e
0.3KB/s 00:00
e rdthee TAFB
S tu is started on both
la e this
3) Using SQL*Plus from your first node, verifyvthat service
instances. v i
ic i o_ sol_08_03_03.sh
o us
[oracle@vx0306 less08]$ r
ab nse
cat t
#!/bin/ksh ( f
#
e rde le lice
l a V ab
# sol_08_03_03.sh
# i er
V be sexecuted
#ioMust f
r i c t r a n on NODE1 !!!!!!!!!!
Fab non-
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

EOF
NAME TYPE VALUE

------------------------------------ ----------- -------------
-----------------
service_names string RDB, TAFB
NAME TYPE VALUE

------------------------------------ ----------- ------------- s a
-----------------
)h a
service_names string RDB, TAFB
m ฺbr
i ฺ c o deฺ
r e dto connect.u i Check
4) Connect to the database by using the TAFB service. Use user JFV
c TAF
siwhich n G
tpolicy you are
which instance you are currently connected to. Also check
@ e
udTW1.
forethis steptas
using. We refer to the telnet session that you userd
ve this S
i l a
_v use
[oracle@vx0306 less08]$ cat vi sol_08_03_04.sh
cat: vi: No such file or io directory
#!/bin/ksh
a b ric e to
#
e (f ens
# sol_08_02_04.sh
e r le lic
d
#
l a V ab on NODE1 !!!!!!!!!!
i er
#io V
# Must be executed
s f
b r ic -tran
Fa ny=`cat

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
$ORACLE_HOME/bin/sqlplus /NOLOG @sol_08_03_04_00.sql

connect / as sysdba

connect / as sysdba
connect jfv/jfv@TAFB
select instance_name from v$instance;
col service_name format a10
select failover_type,failover_method,failed_over from
v$session where username='JFV';
s a
)h a
m ฺbr
SQL*Plus: Release 11.1.0.6.0 - Production on ThudiNov ฺco15uideฺ
08:39:30 2007
s icre ent G
Copyright (c) 1982, 2007, Oracle. rAll d e@ S t
rights
d
ureserved.
i l a ve this
Connected.
i o _v here u s e
execute sol_08_03_04.sql
a b ric e to
from
e
SQL> @sol_08_03_04(f ens
Connected. er d
e l ic
l a V ab l
i
V sfer
INSTANCE_NAME
i o
icRDB1-tran
----------------
b r
Fa non
FAILOVER_TYPE FAILOVER_M FAI
------------- ---------- ---
SESSION BASIC NO
SQL>
5) Create a new terminal window, referred to as TW2, and determine how many
sessions are started on each instance to support your connection on the first terminal
session?
a) Only one session is created to support your connection. Note that this session can
be started on either RDB1 or RDB2. In this example, it is started on RDB2, and
no corresponding session on RDB1 exists
#!/bin/ksh
#

# sol_08_03_05.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2" s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
av this
col service_name format a10vil
i o _ use
b r ic
select failover_type,failover_method,failed_over
t o from
(fa ense
v\$session where username='JFV';
e
rd le licfrom v\$instance;
e
select instance_name
V ab format a10
l a
col service_name
i
V sfwhere
select er
failover_type,failover_method,failed_over from
i o
ic -tran
v\$session username='JFV';
b r
Fa nEOF on
INSTANCE_NAME
----------------
RDB1

------------- ---------- ---
SESSION BASIC NO
INSTANCE_NAME
----------------
RDB2

no rows selected
6) Still connected as user JFV from your first session, insert a row into the FAN table
and commit your modification.
insert into fan values(1);
commit;
SQL> @sol_08_03_06
s a
1 row created.
)h a
m ฺbr
Commit complete. i ฺ c o deฺ
r e d Gui
SQL>
@ sic dent
7) Kill the OS process corresponding to the oracle r e Ssession
dshadow tu that is connected
e s that.
v i lav escript
using TAFB service. You can use the killtafb.sh t hifor
a) As an indication, you can use ic i o_following
the
o usSQL statement to help you kill the
r t
correct process:
select 'kille-9 (fa'b||espid
n se
e
from v$process rd le lic
a V
addr inb(select paddr
where
V i l f e ra from v$session where username='JFV');
o
ici You s
n run this statement on both nodes.
ashould
b r - t r
Fa n[oracle@vx0306
on less08]$ cat killtafb.sh
#!/bin/ksh
#
# killtafb.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

@/home/oracle/solutions/less08/kill_tafb.sql
chmod 777 /tmp/drop_tafb.sh
/tmp/drop_tafb.sh
rm /tmp/drop_tafb.sh
@/home/oracle/solutions/less08/kill_tafb.sql
s a
scp /tmp/drop_tafb.sh $z:/tmp/drop_tafb.sh
)h a
m ฺ br
ssh $z chmod 777 /tmp/drop_tafb.sh
i ฺ c o deฺ
r e d Gui
ssh $z /tmp/drop_tafb.sh
@ sic dent
ssh $z rm /tmp/drop_tafb.sh
e r de Stu
v i lav e this
rm /tmp/drop_tafb.sh
[oracle@vx0306 less08]$cio
_ us
b i
r e to
a
(f ens
d e
r less08]$
l ic cat kill_tafb.sql
[oracle@vx0306
V e l e
set V ila offferab
head
i c io timing
set
a n s off
r t r
Fab nset
set -feedback
on pagesize 0
off;
set verify off
spool /tmp/drop_tafb.sh;
select 'kill -9 ' || spid

from v$process
where addr in (select paddr from v$session where
username='JFV');
spool off
exit;
[oracle@vx0306 less08]$ ./killtafb.sh

08:43:33 2007
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 -
Production
Mining
kill -9 1517
Release 11.1.0.6.0 - Production s a
)h a
Mining
m ฺbr
i ฺ c o deฺ
r e d Nov G u i
SQL*Plus: Release 11.1.0.6.0 - Production on
08:43:33 2007 @ sic dent
Thu 15
e r de Stu
Copyright (c) 1982, 2007, Oracle.
i l a All is reserved.
v thrights
i o _v use
Connected to:
a b ric e to
Oracle Database 11g
e (f Enterprise
e ns Edition Release 11.1.0.6.0 -
Production er d li c
With the V
a b l e
i l fe r a
Partitioning, Real Application Clusters, OLAP, Data
i c i
ando VReal
Mining
a n s
Application Testing options
b r t r
on-
Fa nDisconnected from Oracle Database 11g Enterprise Edition
Mining
drop_tafb.sh 100% 0
0.0KB/s 00:00
8) Insert again one row in FAN from your first session. What do you observe?
a) Because your server process has been killed, you are no longer able to correctly
execute a SQL statement from your session.
SQL> @sol_08_03_08
insert into fan values(1)

*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 5805
Session ID: 114 Serial number: 39333
SQL>
9) Try again. What happens?

a) This time it is possible to again execute a SQL command because you have been
automatically failed over to an instance that offers the TAFB service. Note that
this instance can be either RDB1 or RDB2.
s a
)h a
m ฺbr
commit;
i ฺ c o deฺ
r e d Gui
sic dent
SQL> @sol_08_03_09 @
de Stu
e r
1 row created. v i lav e this
ic i o_ o us
Commit complete. (fa
br se t
r d e licen
V e ble
V a
il fera
INSTANCE_NAME
r i c i rans
o
----------------
RDB1 t
Fab non-
SQL>
10) Verify that you automatically failed over.
select failover_type,failover_method,failed_over from

v$session where username='JFV';
exit;
SQL> @sol_08_03_10
INSTANCE_NAME
----------------
RDB1


------------- ---------- ---
SESSION BASIC YES

Mining
11) Remove TAFB from both the cluster configuration and the database.
s a
)h a
ฺbr
#!/bin/ksh
#
c m
o deฺ
# sol_08_03_11.sh
i ฺ
d Gui
#
r e
@ sic dent
#
e r de Stu
v i l av this
i o _ | sed u s e-n '1,1p'`
a b ric e |tosed -n '2,2p'`
(f ens | grep -v grep | grep -v
DBNAME=`ps -ef e| grepcdbw0_RDB
callout1 | e
d li $8 }' | sed 's/1/''/' | sed
r '{leprint
V awk
ila ferab
's/ora_dbw0_/''/'`
V
r i c io rans
t
I1NAME=$DBNAME"1"
Fab nI2NAME=$DBNAME"2"
on-
/u01/crs11g/bin/srvctl stop service -d $DBNAME -s TAFB
/u01/crs11g/bin/srvctl remove service -d $DBNAME -s TAFB
connect / as sysdba
exec dbms_service.delete_service('TAFB');
EOF

TAFB PREF: RDB2 RDB1 AVAIL:

Remove service TAFB from the database RDB? (y/[n]) y
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 9-1: Mirror the OCR
The goal of this practice is to mirror your OCR file, and replace a corrupted OCR file.
1) Check your OCR configuration.
#!/bin/bash
#
# sol_09_01_01.sh
#
#

s a
)h a
m ฺbr
's/ora_dbw0_/''/'`
i ฺ c o deฺ
r e d Gui
sic dent
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2" @
de Stu
e r
/u01/crs11g/bin/ocrcheck v i lav e this
ic i o_ o us
more /etc/oracle/ocr.loc
(
[oracle@vx0306 less09]$ f abr nse t
e r de lice
V aless09]$
[oracle@vx0306
l a b le ./sol_09_01_01.sh
V i
Status of
f e r
Oracle Cluster Registry is as follows :
o s
ici -tranVersion : 2
b r
Fa non
Total space (kbytes) : 616652
Used space (kbytes) : 3832
Available space (kbytes) : 612820
ID : 992212279
Device/File Name : /dev/sdb1
Device/File integrity
check succeeded
Device/File not configured
Cluster registry integrity check succeeded
ocrconfig_loc=/dev/sdb1
local_only=FALSE
2) Mirror your OCR using /dev/sdb2 as the new mirror.

#!/bin/bash
#

Practice 9-1: Mirror the OCR (continued)
# sol_09_01_02.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
s a
sudo /u01/crs11g/bin/ocrconfig -replace ocrmirror /dev/sdb2
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
3) Check again your OCR configuration.
e r de Stu
i l av this
v
#!/bin/bash
i o _ use
#
# sol_09_01_03.sh fab
ric e to
d e (
c e ns
er bleonli NODE1 !!!!!!!!!!
#
V
# Must be executed
#
V ila fera
i c io n s
y=`catra/home/oracle/nodeinfo
b r - t | sed -n '1,1p'`
Fa nz=`caton /home/oracle/nodeinfo | sed -n '2,2p'`
's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/ocrcheck
more /etc/oracle/ocr.loc

Status of Oracle Cluster Registry is as follows :
Version : 2

ID : 992212279
check succeeded
check succeeded
#Device/file getting replaced by device /dev/sdb2

ocrconfig_loc=/dev/sdb1
ocrmirrorconfig_loc=/dev/sdb2
local_only=false
)h a
4) Corrupt your OCR mirror using the dd command.
m ฺbr
i ฺ c o deฺ
#!/bin/bash
r e d Gui
#
@ sic dent
# sol_09_01_04.sh
# e r de Stu
i
v l av this
#
i o _ use
(f ens | sed -n '2,2p'`
e
r le lic
d
e
i
DBNAME=`ps
i o
callout1
an
b r - t r
on
I2NAME=$DBNAME"2"
sudo dd if=/dev/zero of=/dev/sdb2 bs=1024k count=200


200+0 records in
200+0 records out
5) Check the integrity of your OCR configuration again.

#!/bin/bash
#
# sol_09_01_05.sh

#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/ocrcheck s a
)h a
m ฺ br
[oracle@vx0308 less09]$ /u01/crs11g/bin/ocrcheckdiฺc
-- What we get now! o deฺ
r e u i
PROT-602: Failed to retrieve data from the icluster
s c dent G
registry
[oracle@vx0308 less09]$ @
de Stu
e r
v i lav e this
o_ o us
-- What we should be getting!
i
[oracle@vx0306 less09]$c./sol_09_01_05.sh
i
Status of Oracle Cluster
( f a br Registry
s e t is as follows :
Version
r d en
espacelic(kbytes) : 2
V e ble (kbytes)
Total : 616652
i l a r a
Used space : 3832
i c i o V anID sfe
: 992212279
b r t r
Fa non-
check succeeded
Device/File needs to be
synchronized with the other device
6) Replace your OCR mirror using /dev/sdb1
#!/bin/bash
#
# sol_09_01_06.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
sudo /u01/crs11g/bin/ocrconfig -replace ocrmirror /dev/sdb2


)h a
7) Check the integrity of your OCR configuration again.
m ฺbr
i ฺ c o deฺ
#!/bin/bash
r e d Gui
#
@ sic dent
# sol_09_01_07.sh
# e r de Stu
i
v l av this
#
i o _ use
(f ens | sed -n '2,2p'`
e
r le lic
d
e
i
DBNAME=`ps
i o
callout1
an
b r - t r
on
I2NAME=$DBNAME"2"
/u01/crs11g/bin/ocrcheck

Version : 2
ID : 992212279
check succeeded

check succeeded
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 9-2: OCR Backup and Restore
The goal of this practice is to back up and restore an OCR file.
1) For security purposes, generate a logical OCR backup file.
#!/bin/bash
#
# sol_09_02_01.sh
#
#

s a
)h a
m ฺbr
's/ora_dbw0_/''/'`
i ฺ c o deฺ
r e d Gui
sic dent
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2" @
de Stu
e r
v this
v i la-export
sudo /u01/crs11g/bin/ocrconfig
i o _ use
ric e to
/home/oracle/solutions/less09/logicalocrbak
a b
(f ens
ls -l /home/oracle/solutions/less09/logicalocrbak
[oracle@vx0306 d e
r le lic
less09]$
e
V ab
V i l a
[oracle@vx0306 e r less09]$ ./sol_09_02_01.sh
o s f
i ran 1 root root 86550 Nov 16 2007
b r ic-rw-r--r--
t
on-
Fa n[oracle@vx0306
/home/oracle/solutions/less09/logicalocrbak
less09]$
2) Locate a physical backup of your OCR.
#!/bin/bash
#
# sol_09_02_02.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"

Practice 9-2: OCR Backup and Restore (continued)
I2NAME=$DBNAME"2"
HNAME=`/u01/crs11g/bin/ocrconfig -showbackup | grep -m 1 vx |

awk '{ print $1 }'`
BNAME=`/u01/crs11g/bin/ocrconfig -showbackup | grep -m 1 vx |

awk '{ print $4 }'`
echo "backups on $y"

/u01/crs11g/bin/ocrconfig -showbackup
echo "backups on $HNAME"

ssh $HNAME ls -l $BNAME
s a
)h a
backups on vx0306
m ฺbr
i ฺ c o deฺ
vx0313 2007/11/16 02:15:43
/u01/crs11g/cdata/vx_cluster02/backup00.ocricre
d Gui
@ s dent
vx0313 2007/11/15 22:15:43
e r de Stu
v i lav e this
/u01/crs11g/cdata/vx_cluster02/backup01.ocr
vx0313
ic i o_ o us
2007/11/15 18:15:43
( f abr nse t
vx0306 e rde le lic01:58:23

2007/11/15
e
l a V ab
/u01/crs11g/cdata/vx_cluster02/day.ocr
i er
V sf2007/11/13
i o n
ic/u01/crs11g/cdata/vx_cluster02/week.ocr
vx0306
a 05:58:16
b r t r
onon vx0313
Fa nbackups
-rw-r--r-- 1 root root 4603904 Nov 16 02:15
3) Stop CRS resources on both nodes. When you are finished, verify that they are all
stopped.
#!/bin/bash
#
# sol_09_02_03.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl stop nodeapps -n $z
/u01/crs11g/bin/srvctl stop nodeapps -n $y
/u01/crs11g/bin/srvctl stop database -d $DBNAME
/u01/crs11g/bin/srvctl stop asm -n $y

/u01/crs11g/bin/srvctl stop asm -n $z s a
)h a
m ฺ br
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
Name Type
i l a
Target s
v thiState Host
i o _v OFFLINE
u s e OFFLINE
------------------------------------------------------------
a b ric e toOFFLINE OFFLINE
ora.RDB.db
e (f ens OFFLINE OFFLINE
application
ora....SM1.asme d
r application
e l ic OFFLINE OFFLINE
l a V
ora....06.lsnr
a b l
application OFFLINE OFFLINE
i
ora.vx0306.gsd r
V sfe application OFFLINE OFFLINE
application OFFLINE
i o n
icora.vx0306.vip
a
ora.vx0306.ons OFFLINE
b r t r
on-
Fa nora....SM2.asm
ora....13.lsnr application OFFLINE OFFLINE
ora.vx0313.gsd application OFFLINE OFFLINE
ora.vx0313.ons application OFFLINE OFFLINE
ora.vx0313.vip application OFFLINE OFFLINE
4) Stop CRS on both nodes.
#!/bin/bash
#
# sol_09_02_04.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
echo "Stopping Oracle Clusterware on $z"

ssh $z sudo /u01/crs11g/bin/crsctl stop crs
sleep 60
echo "Stopping Oracle Clusterware on $y"

sudo /u01/crs11g/bin/crsctl stop crs
s a
sleep 60
)h a
m ฺbr
i ฺ c o deฺ
[oracle@vx0306 less09]$ ./sol_09_02_04.sh icre
d Gui
Stopping Oracle Clusterware on vx0313 @s e n t
Stopping resources. r d e Stud
This could take several minutes.ve s
Successfully stopped Oracle v i la e thiresources
o_ o uServices.
Clusterware
c i s
r i
Stopping Cluster Synchronization
t
f ab nsSynchronization
Shutting down the Cluster
Shutdown request (successfully
e issued. Services daemon.
e
Stopping Oraclerde Clusterware
le l i ce on vx0306
Stopping V b
This i l a a
resources.
er several minutes.
Vcouldsftake
i o
icStopping a n stopped Oracle Clusterware resources
Successfully
b r t r
on- down the Cluster Synchronization Services daemon.
Fa nShutting
Cluster Synchronization Services.
Shutdown request successfully issued.

5) Restore your OCR by using the backup identified in step 2.
#!/bin/bash
#
# sol_09_02_05.sh
#
#


's/ora_dbw0_/''/'`

I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
HNAME=`/u01/crs11g/bin/ocrconfig -showbackup | grep -m 1 vx |

awk '{ print $1 }'`
BNAME=`/u01/crs11g/bin/ocrconfig -showbackup | grep -m 1 vx |

awk '{ print $4 }'`
ssh $HNAME sudo /u01/crs11g/bin/ocrconfig -restore $BNAME


)h a
6) Restart CRS on both nodes.
m ฺbr
i ฺ c o deฺ
#!/bin/bash
r e d Gui
#
@ sic dent
# sol_09_02_06.sh
# e r de Stu
i
v l av this
#
i o _ use
(f ens | sed -n '2,2p'`
e
r le lic
d
e
i
DBNAME=`ps
i o
callout1
an
b r - t r
on
I2NAME=$DBNAME"2"
sudo /u01/crs11g/bin/crsctl start crs
sleep 300
ssh $z sudo /u01/crs11g/bin/crsctl start crs
sleep 300

Attempting to start Oracle Clusterware stack

The CRS stack will be started shortly
------------------------------------------------------------
ora....B2.inst application ONLINE OFFLINE
ora....SM2.asm application ONLINE OFFLINE
ora....13.lsnr application ONLINE OFFLINE
ora.vx0313.gsd application ONLINE OFFLINE
ora.vx0313.ons application ONLINE OFFLINE
ora.vx0313.vip application ONLINE ONLINE vx0306 s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
------------------------------------------------------------
ONLINE
ONLINE
ONLINE
@
ONLINE sic dent
vx0306
vx0313
ora.RDB.db application ONLINE
e r de Stu
ONLINE vx0306
ora....SM1.asm application ONLINE
v i lav e this
ONLINE vx0306
o_ o us
i
ora.vx0306.gsd applicationc i ONLINE ONLINE vx0306
( f abr nse t
rde le lice
e
ora....SM2.asm application
V ab
i l a
ora....13.lsnr application
V sfer
ONLINE
ONLINE
ONLINE
ONLINE
vx0313
vx0313
i o
ic -tran
b r
Fa non
7) Check your OCR integrity from both nodes.

#!/bin/bash
#
# sol_09_02_07.sh
#
#


's/ora_dbw0_/''/'`

I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/cluvfy comp ocr -n all

Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...

All nodes free of non-clustered, local-only configurations. s a
)h a
Uniqueness check for OCR device passed.
m ฺbr
i ฺ c o deฺ
Checking the version of OCR...
r e d Gui
OCR of correct Version "2" exists.
@ sic dent
Checking data integrity of OCR... rde tu
e S
Data integrity check for OCR passed.
v i lav e this
OCR integrity check passed.
ic i o_ o us
( f a br se t
r d eless09]$
Verification of OCR integrity
l i c en was successful.
e ble
[oracle@vx0306
V
8) Do aV a
il backup
manual e ra of your OCR.
o
i ran s f
b r ic[oracle@vx0306
- t less09]$ cat sol_09_02_08.sh
Fa n#!/bin/bash
on
#
# sol_09_02_08.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
sudo /u01/crs11g/bin/ocrconfig -manualbackup

/u01/crs11g/bin/ocrconfig -showbackup
echo "and on $z"

ssh $z /u01/crs11g/bin/ocrconfig -showbackup
vx0306 2007/11/16 03:58:37

/u01/crs11g/cdata/vx_cluster02/backup_20071116_035837.ocr
vx0313 2007/11/15 22:15:43

s a
vx0313 2007/11/15 18:15:43
)h a
m ฺbr
i ฺ c o deฺ
vx0313 2007/11/15 14:15:43
/u01/crs11g/cdata/vx_cluster02/backup02.ocricre
d Gui
@ s dent
vx0306 2007/11/14 01:58:19
e r de Stu
v i lav e this
vx0306
ic i o_ o us
2007/11/13 05:58:16
( f abr nse t
/u01/crs11g/cdata/vx_cluster02/week.ocr
vx0306 e rde le lic03:58:37

2007/11/16
e
V ab
and V
on ilavx0313
f er
i o
icvx0313 s
an 2007/11/15 22:15:43
b r - t r
Fa n/u01/crs11g/cdata/vx_cluster02/backup00.ocr
on
vx0313 2007/11/15 18:15:43
vx0313 2007/11/15 14:15:43

vx0306 2007/11/14 01:58:19

vx0306 2007/11/13 05:58:16

/u01/crs11g/cdata/vx_cluster02/week.ocr
vx0306 2007/11/16 03:58:37


Practice 9-3: Multiplex Your Voting Disk
The goal of this practice is to add two mirrors to your voting disk configuration.
1) Determine your current voting disk configuration.
#!/bin/bash
#
# sol_09_03_01.sh
#
#

s a
)h a
m ฺbr
's/ora_dbw0_/''/'`
i ฺ c o deฺ
r e d Gui
sic dent
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2" @
de Stu
e r
/u01/crs11g/bin/crsctl queryvcss i lavvotedisk
e t his
[oracle@vx0306 less09]$ io_ us
r ic t o
(
[oracle@vx0306 less09]$f ab n./sol_09_03_01.sh
se
e lic e
0. 0
erd/dev/sdb5
Located 1Vvoting b l e
disk(s).
V ila feraless09]$
[oracle@vx0306
i c io tworanew
n s
a b r
2) Add
- t members to your voting disk configuration using both /dev/sdb6, and
F no n
/dev/sdb7.
#!/bin/bash
#
# sol_09_03_04.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

Practice 9-3: Multiplex Your Voting Disk (continued)
sudo /u01/crs11g/bin/crsctl add css votedisk /dev/sdb6
sudo /u01/crs11g/bin/crsctl add css votedisk /dev/sdb7


Now formatting voting disk: /dev/sdb6.
Successful addition of voting disk /dev/sdb6.
3) Determine your current voting disk configuration.

s a
)h a
ฺbr
#!/bin/bash
#
c m
o deฺ
# sol_09_03_03.sh
i ฺ
d Gui
#
r e
@ sic dent
#
e r de Stu
y=`cat /home/oracle/nodeinfo | i l h is
v -n t'1,1p'`
ased
z=`cat /home/oracle/nodeinfo v
o_ |osed e
s-n '2,2p'`
ic i u
( f
DBNAME=`ps -ef | grep abrdbw0_RDBs e t | grep -v grep | grep -v
callout1 | awk e'{ print
r d l i c en$8 }' | sed 's/1/''/' | sed
V e ble
's/ora_dbw0_/''/'`
V a
il fera
r i c i rans
o
I1NAME=$DBNAME"1"
t
Fab nI2NAME=$DBNAME"2"
on-
/u01/crs11g/bin/crsctl query css votedisk

0. 0 /dev/sdb5
1. 0 /dev/sdb6
2. 0 /dev/sdb7
Located 3 voting disk(s).

Practice 9-4: Protect Xclock with Oracle Clusterware
Important note: Before you start the following practices, make sure that VNC is started
on both nodes as user oracle on port 5802.
You execute all of your commands for this practice under VNC terminal sessions from
the first and second nodes.
The goal of this practice is to protect the Xclock application with Oracle Clusterware so
that it is automatically restarted each time you kill it.
1) Make sure that both nodes are added to the xhost access control list. Do this on both
nodes.
#!/bin/ksh
#
# sol_09_04_01.sh
s a
#
)h a
ฺbr
#
c m
o deฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'`red
i ฺ u i
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'` sic dent G
@
e Stu
cd /home/oracle/solutions/less09 erd
v i lav e this
> _displayenv.sh
echo export DISPLAY=:2.0 o
r ic i
t o usdisplayenv.sh
ab ns+e$z >> displayenv.sh
echo /usr/X11R6/bin/xhost + $y >>
( f
echo /usr/X11R6/bin/xhost
e e
erd ble lic
chmod 777Vdisplayenv.sh
V i la fera
r i c io rans
./displayenv.sh
t
Fab nscp o n-displayenv.sh $z:/home/oracle/solutions/less09
ssh $z "chmod 777 /home/oracle/solutions/less09/displayenv.sh"
ssh $z /home/oracle/solutions/less09/displayenv.sh

vx0306 being added to access control list
displayenv.sh 100% 80
0.1KB/s 00:00
2) Create an action script for CRS to monitor the xclock application. You need to make
sure that the clock is always displayed on the VNC session attached to your first
node. When you are finished, copy that script in the default CRS action scripts
location on both nodes. Call your action script crsclock_action.scr.

Practice 9-4: Protect Xclock with Oracle Clusterware (continued)
#!/bin/ksh
#
# sol_09_04_02.sh
#
#

cd /home/oracle/solutions/less09
sed 's/NODE1/'$y'/' wrong_crsclock_action.scr >

crsclock_action.scr
s a
)h a
ฺbr
sudo cp /home/oracle/solutions/less09/crsclock_action.scr
/u01/crs11g/crs/script/
c m
o deฺ
i ฺ
d Gui
e
sudo chmod 777 /u01/crs11g/crs/script/crsclock_action.scr
r
@ sic dent
scp crsclock_action.scr $z:/home/oracle/solutions/less09
e r de Stu
ssh $z "sudo cp
v i lav e this
ic i o_ o us
/home/oracle/solutions/less09/crsclock_action.scr
br se t
/u01/crs11g/crs/script/"
( f a
r d e 777
ssh $z "sudo chmod
l i c en
e ble
/u01/crs11g/crs/script/crsclock_action.scr"
V
cat V
a
il fera
r i c i rans
o /u01/crs11g/crs/script/crsclock_action.scr
t
Fab n[oracle@vx0306
on- less09]$
[oracle@vx0306 less09]$ cat wrong_crsclock_action.scr

#!/bin/bash
# start/stop/check script for xclock example
# To test this change BIN_DIR to the directory where xclock is
based
# and set the DISPLAY variable to a server within your
network.
BIN_DIR=/usr/X11R6/bin
LOG_DIR=/tmp
BIN_NAME=xclock
DISPLAY=NODE1:2.0
export DISPLAY
if [ ! -d $BIN_DIR ]
then

echo "start failed"
exit 2
fi
PID1=`ps -ef | grep $BIN_NAME | grep -v grep | grep -v

xclock_app | awk '{ print $2 }'`
case $1 in
'start')
if [ "$PID1" != "" ]
then
status_p1="running"
else
if [ -x $BIN_DIR/$BIN_NAME ]
then
umask 002 s a
${BIN_DIR}/${BIN_NAME} &
)h a
2>${LOG_DIR}/${BIN1_NAME}.log
m ฺbr
status_p1="started"
i ฺ c o deฺ
else
r e d Gui
found" @ sic dent
echo `basename $0`": $BIN_NAME: Executable not
fi
e r de Stu
fi
v i lav e this
i _ us
o$status_p1"
echo "$BIN_NAME:
b ic
r e to
;;
a
(f ens
e
r le lic
d
'stop') e
V [ "${PID1}"
b
i l a if
e r a != "" ]
i c i o V ansfkill
then
-9 ${PID1} && echo "$BIN_NAME daemon killed"
b r t r
Fa non-
else
echo "$BIN_NAME: no running Process!"
fi
;;
'check')
if [ "$PID1" != "" ]
then
echo "running"
exit 0
else
echo "not running"
exit 1
fi
;;
*)
echo "Usage: "`basename $0`" {start|stop|check}"
;;
esac


crsclock_action.scr 100% 1339
1.3KB/s 00:00
#!/bin/bash
# start/stop/check script for xclock example
# To test this change BIN_DIR to the directory where xclock is
based
# and set the DISPLAY variable to a server within your
network.
BIN_DIR=/usr/X11R6/bin
LOG_DIR=/tmp
BIN_NAME=xclock s a
DISPLAY=vx0306:2.0
)h a
export DISPLAY
m ฺbr
i ฺ c o deฺ
if [ ! -d $BIN_DIR ]
r e d Gui
then
echo "start failed" @ sic dent
exit 2
e r de Stu
fi
v i lav e this
PID1=`ps -ef | grep $BIN_NAME
ic i o_ o| grep
us -v grep | grep -v
xclock_app | awk '{ b r t
a printns$2e }'`
( f
e lice
case $1 in erd
'start') V
l a a b le
i
V if f[er"$PID1" != "" ]
i o s
ic -tran status_p1="running"
then
b r
Fa non else
if [ -x $BIN_DIR/$BIN_NAME ]
then
umask 002
${BIN_DIR}/${BIN_NAME} &
2>${LOG_DIR}/${BIN1_NAME}.log
status_p1="started"
else
echo `basename $0`": $BIN_NAME: Executable not
found"
fi
fi
echo "$BIN_NAME: $status_p1"

;;
'stop')
if [ "${PID1}" != "" ]
then

kill -9 ${PID1} && echo "$BIN_NAME daemon killed"
else
echo "$BIN_NAME: no running Process!"
fi
;;
'check')
if [ "$PID1" != "" ]
then
echo "running"
exit 0
else
echo "not running"
exit 1
fi
;;
*) s a
echo "Usage: "`basename $0`" {start|stop|check}"
)h a
;;
m ฺ br
esac
i ฺ c o deฺ
r e d Gui
3) Create a new CRS application resource profile called myClock
@ sic using e n t action script
the
d e p is S
you just created. Make sure that the resource attribute tod
settu favored, h is set to
the list corresponding to your two nodes with e r
v this
the second one first, ci is set to 5 and ra
i l a
is set to 2.
i o _v use
to
#!/bin/ksh
e (f ens
#
e r le lic
d
l a V ab
# sol_09_04_03.sh
# i
V sexecutedf er
#ioMust be n
r i c t r a on NODE1 !!!!!!!!!!
Fab non-
#

sudo /u01/crs11g/bin/crs_profile -create myClock -t

application -a crsclock_action.scr -p favored -h "$z $y" -o
ci=5,ra=2
more /u01/crs11g/crs/profile/myClock.cap

NAME=myClock
TYPE=application
ACTION_SCRIPT=crsclock_action.scr
ACTIVE_PLACEMENT=0
AUTO_START=restore
CHECK_INTERVAL=5

DESCRIPTION=myClock
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=vx0313 vx0306
OPTIONAL_RESOURCES=
PLACEMENT=favored
REQUIRED_RESOURCES=
RESTART_ATTEMPTS=2
SCRIPT_TIMEOUT=60
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba s a
USR_ORA_DEBUG=0
)h a
USR_ORA_DISCONNECT=false
m ฺbr
USR_ORA_FLAGS=
i ฺ c o deฺ
USR_ORA_IF=
r e d Gui
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG= @ sic dent
USR_ORA_NETMASK=
e r de Stu
USR_ORA_OPEN_MODE=
v i lav e this
o_ o us
USR_ORA_OPI=false
USR_ORA_PFILE=
ic i
( f abr nse t
USR_ORA_PRECONNECT=none
rde le lice
USR_ORA_SRV=
e
USR_ORA_START_TIMEOUT=0
V ab
i l a
USR_ORA_STOP_MODE=immediate
V sfer
USR_ORA_STOP_TIMEOUT=0
i o
ic -tran
USR_ORA_VIP=
b r
Fa non [oracle@vx0306 less09]$
4) Register the myClock resource with CRS, and make sure user oracle can manage that
resource.
#!/bin/ksh
#
# sol_09_04_04.sh
#
#
/u01/crs11g/bin/crs_stat myClock
sudo /u01/crs11g/bin/crs_register myClock
sudo /u01/crs11g/bin/crs_setperm myClock -u user:oracle:r-x
/u01/crs11g/bin/crs_stat myClock


CRS-0210: Could not find resource 'myClock'.
NAME=myClock
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

------------------------------------------------------------
myClock application OFFLINE OFFLINE s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
ONLINE
ONLINE
ONLINE
ONLINE @ sic dent
vx0306
vx0306
ora.vx0306.ons application ONLINE r
ONLINE
e de Stu
vx0306
ora.vx0306.vip application ONLINE
v i lav e this
ONLINE vx0306
o_ o us
ic i
( f abr nse t
rde le lice
e
V ab
i l a
Vsure you er a VNC session under user oracle opened on your first node
i
5) Makeo s fhave
n This is necessary to see the clock. Now start the myClock resource
b r icterminal
- t r awindow.
Fa nasousern oracle, and determine the location and status of this resource after startup.
Look at the VNC session on first node; you should see the clock.
#!/bin/ksh
#
# sol_09_04_05.sh
#
#
/u01/crs11g/bin/crs_start myClock
sleep 30

Attempting to start ‘myClock’ on member ‘vx0313’
Start of ‘myClock’ on member ‘vx0313’ succeeded.
------------------------------------------------------------
myClock application ONLINE ONLINE vx0313
ora....SM2.asm application ONLINE ONLINE vx0313 s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
6) Kill the xclock application on the node on which d it e uWhat do you observe?
was started.
t
r
ve session S
a) You should see the clock restarting on
v i l aVNC
e t hison first node.
[oracle@vx0306 less09]$ccat
i i o_ sol_09_04_06.sh
o us
#!/bin/ksh r
ab nse t
( f
rde le lice
#
e
# sol_09_04_06.sh
V ab
#
i l a
V sexecuteder
# Must
i o be f on NODE1 !!!!!!!!!!
b r ic -tran
#
Fa ny=`cat
echo "CLOCKPID=\`ps -ef | grep xclock | grep -v grep | grep -v

xclock_app | awk '{ print \$2 }'\`" >
/home/oracle/solutions/less09/killclockpid.sh
echo "sudo kill -9 \$CLOCKPID" >>
chmod 777 /home/oracle/solutions/less09/killclockpid.sh
scp /home/oracle/solutions/less09/killclockpid.sh
$z:/home/oracle/solutions/less09/
ssh $z /home/oracle/solutions/less09/killclockpid.sh


usage: kill [ -s signal | -p ] [ -a ] pid ...
kill -l [ signal ]
killclockpid.sh 100% 112
0.1KB/s 00:00
7) Determine the node on which myClock is currently running.
#!/bin/ksh
#
# sol_09_04_07.sh
#
s a
)h a
ฺbr
#
c m
o deฺ
i ฺ
d Gui
r e
@ sic dent
[oracle@vx0306 less09]$ ./sol_09_04_07.sh e r de Stu
Name Type i lTarget s
av thiState Host
v
o_ oONLINE e
myClock r
application ic i
t us
------------------------------------------------------------
ONLINE vx0313
a b e
(f ens ONLINE ONLINE vx0306
ora....B1.inst application ONLINE ONLINE
ora....B2.inst eapplication
c
d i
ora.RDB.db er application
l vx0313
a V b l e ONLINE ONLINE vx0306
V il feraapplication ONLINE ONLINE vx0306

ora....SM1.asm application ONLINE ONLINE
i c i o ans application ONLINE ONLINE vx0306

ora....06.lsnr
b r t r
ora.vx0306.gsd
- vx0306
Fa nora.vx0306.ons
on application
ONLINE
ONLINE
ONLINE
ONLINE
vx0306
vx0306
8) Repeat the following sequence twice: kill the xclock application and wait for it to be
restarted. After the last restart, what do you observe?
a) You should see that after three kills, myClock is relocated to the other node. This
is due to the ra attribute that you defined for myClock.
#!/bin/ksh
#
# sol_09_04_08.sh
#

#

cd /home/oracle/solutions/less09
echo "CLOCKPID=\`ps -ef | grep xclock | grep -v grep | grep -v

xclock_app | awk '{ print \$2 }'\`" > killclockpid.sh
echo "sudo kill -9 \$CLOCKPID" >> killclockpid.sh
chmod 777 killclockpid.sh
./killclockpid.sh
s a
scp killclockpid.sh $z:/home/oracle/solutions/less09
)h a
ssh $z "chmod 777
m ฺbr
/home/oracle/solutions/less09/killclockpid.sh"
i ฺ c o deฺ
r e d Gui
sleep 20 @ sic dent
e r de Stu
./killclockpid.sh
v i lav e this
ic i o_ o us
( f a br se t
sleep 20
r d e licen
V e ble
l a a
i r
-t
i c i o V ansfe
a b r - tr
F o n
n[oracle@vx0306 less09]$ ./sol_09_04_08.sh
kill -l [ signal ]
killclockpid.sh 100% 112
0.1KB/s 00:00
kill -l [ signal ]
------------------------------------------------------------
myClock application ONLINE ONLINE vx0306

9) Stop, unregister myClock resource, and remove the .scr and .cap files that were
created.
#!/bin/ksh
#
# sol_09_04_09.sh
#
#
s a
)h a
ฺbr
c m
o deฺ
i ฺ
d Gui
/u01/crs11g/bin/crs_stop myClock r e
ic ent
@ s
r d e Stud
sudo /u01/crs11g/bin/crs_unregister i l a ve myClock
t his
v
o-t_ o us e
ic i
abr nse t
( f
rde le lice
sudo rm /u01/crs11g/crs/profile/myClock.cap
e
l a V ab
sudo rm /u01/crs11g/crs/script/crsclock_action.scr
i
V$z "sudo f errm /u01/crs11g/crs/profile/myClock.cap"
i o
ssh
icssh -$z s
n rm /u01/crs11g/crs/script/crsclock_action.scr"
a"sudo
b r t r
Fa n[oracle@vx0306
on less09]$

Attempting to stop `myClock` on member `vx0306`
Stop of `myClock` on member `vx0306` succeeded.
------------------------------------------------------------
myClock application OFFLINE OFFLINE

------------------------------------------------------------
ora....13.lsnr application ONLINE ONLINE vx0313 s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
rm: cannot remove `/u01/crs11g/crs/profile/myClock.cap': No
such file or directory
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 10-1: Diagnosing Oracle Clusterware Components
The following short exercises are intended to make you familiar with various aspects of
inspecting and diagnosing your Oracle Clusterware components including log file
locations and assorted utilities provided to aid in data collection activities.
1) In this exercise you stop the crsd process uncleanly and inspect the logs that are
generated. Run the script cleanlog.sh as root. Using the ps command find the process
ID for the crsd process and kill it with a signal 9. Wait a few moments and change
directory to /u01/crs11g/hostname and inspect the various log files that are generated.
a) Change directory to $HOME/solutions/less10 and run the cleanlog.sh using sudo.
[oracle@vx0306 less10]$ cat sol_10_01_01a.sh
# sol_10_01_01a.sh
cd $HOME/solutions/less10
s a
)h a
sudo ./cleanlog.sh
m ฺbr
i ฺ c o deฺ
r e d Gui
[oracle@vx0306 less10]$ cat cleanlog.sh s
ic ent
d @
e Stud
# cleanlog.sh
r
ve -n t'1,1p'`
y=`cat /home/oracle/nodeinfov| i lased e his
ic i o_ |osedus-n '2,2p'`
r
a-cb 1-11` t
e
HOST=`hostname|cut ( f
e n se
e rd le lic
a V ab
cd /u01/crs11g/log/$y
l
find i
V sferf -exec
. -type rm {} \;
i o
ic -tran
b r
Fa non
[oracle@vx0306 less10]$ ./sol_10_01_01a.sh
b) Using the ps command find the process ID of the crsd process. Use the kill
command with sudo to kill the process since it is owned by root.
[oracle@vx0306 less10]$ cat sol_10_01_01b.sh
# sol_10_01_01b.sh
ps -ef|grep -i crsd
ps -ef | grep -i "crsd" | grep -v grep | grep -v killcrs | awk

'{print "sudo kill -9 " $2 }' >
chmod 777 /home/oracle/solutions/less10/z.sh
echo "Killing crsd..."

(continued)
[oracle@vx0306 less10]$ ./sol_10_01_01b.sh

root 3220 1 0 Nov16 ? 00:00:00 /bin/sh
/etc/init.d/init.crsd run
root 7658 3220 0 Nov16 ? 00:07:17
/u01/crs11g/bin/crsd.bin reboot
oracle 26952 26950 0 02:54 pts/1 00:00:00 grep -i crsd
Killing crsd...
s a
c) Change directory to /u01/crs11g/log/<hostname> and find the log files generated) h a
by the termination of the crsd process. ฺ b r
c o m e ฺ
i ฺ i d
red t Gu
[oracle@vx0306 less10]$ cat sol_10_01_01c.sh
# sol_10_01_01c.sh
i c
s den
y=`cat /home/oracle/nodeinfo | sed -n @
de'1,1p'`tu
e r S
lav e this
v i
HOST=`hostname|cut -c 1-11`
ic i o_ o us
find . -type f -print ( f abr nse t
[oracle@vx0306
e rdeless10]$l i ce
l a V able
i
V sfer less10]$ ./sol_10_01_01c.sh
i o an
ic./crsd/crsdOUT.log
[oracle@vx0306
b r - t r
Fa n./crsd/crsd.log
on
./racg/ora.RDB.db.log
./alertvx0306.log
d) Inspect the various files that were generated, starting with the Clusterware Alert
log.
[oracle@vx0306 less10]$ cat sol_10_01_01d.sh
# sol_10_01_01d.sh

cat ./alert$y.log
cat ./crsd/crsd.log
cat ./client/ocr*
cat ./client/css*

(continued)
cat ./racg/ora*
[oracle@vx0306 less10]$ ./sol_10_01_01d.sh

Oracle Database 11g CRS Release 11.1.0.6.0 - Production
Copyright 1996, 2007 Oracle. All rights reserved.
2007-11-19 02:55:02.303
[crsd(27280)]CRS-1012:The OCR service started on node vx0306.
2007-11-19 02:55:14.234
[crsd(27280)]CRS-1201:CRSD started on node vx0306.
s a
2007-11-19 02:57:00.698: [ default][3056539872]prlsndmain:
)h a
ฺbr
olsnodes successful!!
c m
o deฺ
i ฺ
d Gui
2007-11-19 02:55:11.535: [ COMMCRS][3054398368]clsc_connect:
r e
(0x816b678) no listener at
@ sic dent
de Stu
(ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
e r
2007-11-19 02:55:11.535: [
v i e his crs_qstat error
avclsrcqryapi:
lRACG][3055859392]
t
[27501][3055859392][ora.RDB.db]:
ic i o_ o us
(
[oracle@vx0306 less10]$ f abr nvisesol_10_01_01d.sh
t
[oracle@vx0306 eless10]$e ./sol_10_01_01d.sh
e rd 11g e l ic Release 11.1.0.6.0 - Production
l a l
V1996,ab2007 Oracle. All rights reserved.
Oracle Database CRS
i
Copyright
er
V sf02:55:02.303
i o
2007-11-19
an
ic[crsd(27280)]CRS-1012:The
b r - t r OCR service started on node vx0306.
Fa n[crsd(27280)]CRS-1201:CRSD
on
2007-11-19 02:55:14.234
started on node vx0306.
2007-11-19 02:55:01.532: [ default][3055683264] CRS Daemon
Starting
2007-11-19 02:55:01.533: [ CRSMAIN][3055683264] Checking the
OCR device
2007-11-19 02:55:01.566: [ CRSMAIN][3055683264] Connecting to
the CSS Daemon
2007-11-19 02:55:01.636: [ CRSMAIN][3055683264] CRSD running
as the Privileged user
2007-11-19 02:55:02.005: [ CLSVER][3055683264] Active Version

from OCR:11.1.0.6.0
2007-11-19 02:55:02.005: [ CLSVER][3055683264] Active Version
and Software Version are same
2007-11-19 02:55:02.005: [ CRSMAIN][3055683264] Initializing
OCR

(continued)
2007-11-19 02:55:02.076: [ OCRRAW][3055683264]proprioo: for
disk 0 (/dev/sdb1), id match (1), my id set
(125597460,2105976786) total id sets (1), 1st set
(125597460,2105976786), 2nd set (0,0) my votes (1), total
votes (2)
2007-11-19 02:55:02.076: [ OCRRAW][3055683264]proprioo: for
disk 1 (/dev/sdb2), id match (1), my id set
(125597460,2105976786) total id sets (1), 1st set
(125597460,2105976786), 2nd set (0,0) my votes (1), total
votes (2)
2007-11-19 02:55:02.319: [ CRSD][3055683264] ENV Logging
level for Module: allcomp 0
2007-11-19 02:55:02.321: [ CRSD][3055683264] ENV Logging
level for Module: default 0
s a
2007-11-19 02:55:02.326: [ CRSD][3055683264] ENV Logging
)h a
ฺbr
level for Module: OCRRAW 0
2007-11-19 02:55:02.338: [ CRSD][3055683264] ENV Logging
c m
o deฺ
level for Module: OCROSD 0
i ฺ
d Gui
2007-11-19 02:55:02.346: [ e
CRSD][3055683264] ENV Logging
r
level for Module: OCRCAC 0
@ sic dent
de Stu
2007-11-19 02:55:02.348: [ CRSD][3055683264] ENV Logging
level for Module: COMMCRS 0
e r
2007-11-19 02:55:02.350: [
i lav e this
CRSD][3055683264] ENV Logging
v
level for Module: COMMNS 0
ic i o_ o us
abr nse t
2007-11-19 02:55:02.352: [ CRSD][3055683264] ENV Logging
( f
level for Module: CRSUI 0
e rde le lice
2007-11-19 02:55:02.353: [ CRSD][3055683264] ENV Logging
l a V ab
level for Module: CRSCOMM 0
i
V sfer
2007-11-19 02:55:02.356: [ CRSD][3055683264] ENV Logging
i o
level for Module: CRSRTI 0
ic -tran
b r 2007-11-19 02:55:02.357: [ CRSD][3055683264] ENV Logging
Fa non level for Module: CRSMAIN 0
2007-11-19 02:55:02.359: [ CRSD][3055683264] ENV Logging
level for Module: CRSPLACE 0
2007-11-19 02:55:02.360: [ CRSD][3055683264] ENV Logging
level for Module: CRSAPP 0
2007-11-19 02:55:02.367: [ CRSD][3055683264] ENV Logging
level for Module: CRSRES 0
2007-11-19 02:55:02.376: [ CRSD][3055683264] ENV Logging
level for Module: CRSOCR 0
2007-11-19 02:55:02.377: [ CRSD][3055683264] ENV Logging
level for Module: CRSTIMER 0
2007-11-19 02:55:02.379: [ CRSD][3055683264] ENV Logging
level for Module: CRSEVT 0
2007-11-19 02:55:02.389: [ CRSD][3055683264] ENV Logging
level for Module: CRSD 0
2007-11-19 02:55:02.396: [ CRSD][3055683264] ENV Logging
level for Module: CLUCLS 0
2007-11-19 02:55:02.398: [ CRSD][3055683264] ENV Logging
level for Module: CLSVER 0

(continued)
2007-11-19 02:55:02.400: [ CRSD][3055683264] ENV Logging
level for Module: CSSCLNT 0
2007-11-19 02:55:02.405: [ CRSD][3055683264] ENV Logging
level for Module: OCRAPI 0
2007-11-19 02:55:02.416: [ CRSD][3055683264] ENV Logging
level for Module: OCRUTL 0
2007-11-19 02:55:02.418: [ CRSD][3055683264] ENV Logging
level for Module: OCRMSG 0
2007-11-19 02:55:02.419: [ CRSD][3055683264] ENV Logging
level for Module: OCRCLI 0
2007-11-19 02:55:02.426: [ CRSD][3055683264] ENV Logging
level for Module: OCRSRV 0
2007-11-19 02:55:02.436: [ CRSD][3055683264] ENV Logging
level for Module: OCRMAS 0
s a
2007-11-19 02:55:02.436: [ CRSMAIN][3055683264] Filename is
)h a
ฺbr
/u01/crs11g/crs/init/vx0306.pid
[ clsdmt][2800208800]Listening to
c m
o deฺ
(ADDRESS=(PROTOCOL=ipc)(KEY=vx0306DBG_CRSD))
i ฺ
d Gui
e
2007-11-19 02:55:02.549: [ CRSMAIN][3055683264] Using
r
Authorizer location: /u01/crs11g/crs/auth/
@ sic dent
de Stu
RTI
e r
i lav e this
v
EVMMgr
ic i o_ o us
abr nse t
2007-11-19 02:55:02.866: [CRSTIMER][2783157152] Timer Thread
Starting.
( f
rde le lice
2007-11-19 02:55:03.761: [ CRSMAIN][3055683264] CRSD locked
e
l a V ab
during state recovery, please wait.
i
V sfer
2007-11-19 02:55:05.318: [ CRSRES][3055683264] ora.vx0306.vip
i o
check shows ONLINE
ic -tran
b r 2007-11-19 02:55:06.517: [ CRSRES][3055683264] ora.vx0306.gsd
Fa non check shows ONLINE
2007-11-19 02:55:07.727: [ CRSRES][3055683264] ora.vx0306.ons
check shows ONLINE
2007-11-19 02:55:08.967: [ CRSRES][3055683264]
ora.vx0306.LISTENER_VX0306.lsnr check shows ONLINE
2007-11-19 02:55:10.307: [ CRSRES][3055683264]
ora.vx0306.ASM1.asm check shows ONLINE
2007-11-19 02:55:11.598: [ CRSRES][3055683264] ora.RDB.db
check shows ONLINE
2007-11-19 02:55:14.038: [ CRSRES][3055683264]
ora.RDB.RDB1.inst check shows ONLINE
2007-11-19 02:55:14.137: [ CRSMAIN][3055683264] CRSD
recovered, unlocked.
2007-11-19 02:55:14.138: [ CRSMAIN][3055683264] QS socket on:
2007-11-19 02:55:14.207: [ CRSMAIN][3055683264] CRSD UI socket
on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
2007-11-19 02:55:14.233: [ CRSMAIN][3055683264] E2E socket on:
(ADDRESS=(PROTOCOL=tcp)(HOST=vx0306-priv)(PORT=49896))

(continued)
2007-11-19 02:55:14.234: [ CRSMAIN][3055683264] Starting
Threads
2007-11-19 02:55:14.234: [ CRSMAIN][3055683264] CRS Daemon
Started.
runCommandServer for (UI = 1, E2E = 0). 0
runCommandServer for (UI = 1, E2E = 0). 1
cat: ./client/ocr*: No such file or directory
s a
)h a
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
2007-11-19 02:55:11.535: [ COMMCRS][3054398368]clsc_connect:
@ sic dent
de Stu
(0x816b678) no listener at
e r
ilav e this
2007-11-19 02:55:11.535: [ _v RACG][3055859392]
c s
io o uclsrcqryapi:
r
[27501][3055859392][ora.RDB.db]:
b i t crs_qstat error
[oracle@vx0306 eless10]$ (fa ense

e rd le lic
2) Use the crsctl
l a Vcommand
a b to check the health of your clusterware. Use the CLUVFY
i r
fe the viability of the cluster nodeapps.
i c i o V atonscheck
command
a b r a) Run - trthe crsctl command from the /u01/crs11g/bin directory as shown below.
F no n
[oracle@vx0306 less10]$ cat sol_10_01_02a.sh
# sol_10_01_02a.sh
cd /u01/crs11g/bin
./crsctl check crs
[oracle@vx0306 less10]$ ./sol_10_01_02a.sh

Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy
b) Change directory to /stage/10gR2/rdbms/clusterware/cluvfy and run CLUVFY

using the runcluvfy.sh script as shown below to check the nodeapps.
[oracle@vx0306 less10]$ cat sol_10_01_02b.sh
# sol_10_01_02b.sh

(continued)
/u01/crs11g/bin/cluvfy comp nodeapp -n all -verbose
[oracle@vx0306 less10]$ ./sol_10_01_02b.sh
Verifying node application existence
Checking node application existence...
Checking existence of VIP node application

Node Name Required Status
s a
Comment
)h a
ฺbr
------------ ------------------------ --------------------
---- ----------
c m
o deฺ
vx0306 yes exists
i ฺ
d Gui
passed
r e
vx0313 yes exists
@ sic dent
de Stu
passed
Result: Check passed.
e r
v this
v i laapplication
Checking existence of ONS node
i o _ use
Node Name r
Required
b ic t o Status
Comment
(fa ense
------------ e ------------------------
e rd le lic --------------------
l a V abno
---- ----------
V i
vx0306
f e r exists
i c iovx0313
passed
a n s
b r - t r no exists
Fa nResult:
on Check passed.
passed
Checking existence of GSD node application

Node Name Required Status
Comment
------------ ------------------------ --------------------
---- ----------
vx0306 no exists
passed
vx0313 no exists
passed
Result: Check passed.
Verification of node application existence was successful.

3) Using the crs_stat command find all crs configuration data for the VIP resource
located on your second node.

(continued)
# sol_10_01_03.sh

cd /u01/crs11g/bin
./crs_stat -p ora.${z}.vip

s a
NAME=ora.vx0313.vip
)h a
ฺbr
TYPE=application
ACTION_SCRIPT=/u01/crs11g/bin/racgwrap
c m
o deฺ
ACTIVE_PLACEMENT=1
i ฺ
d Gui
AUTO_START=1
r e
CHECK_INTERVAL=15
@ sic dent
FAILOVER_DELAY=0 e r de Stu
DESCRIPTION=CRS application for VIP on a node
FAILURE_INTERVAL=0
v i lav e this
FAILURE_THRESHOLD=0
ic i o_ o us
abr nse t
HOSTING_MEMBERS=vx0313
(
OPTIONAL_RESOURCES= f
rde le lice
PLACEMENT=favored
e
l a V ab
REQUIRED_RESOURCES=
i
V sfer
RESTART_ATTEMPTS=0
i o
SCRIPT_TIMEOUT=60
ic -tran
b r START_TIMEOUT=0
Fa non STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=eth0
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=255.255.252.0
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate

(continued)
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=10.216.4.75
4) Determine the file(s) the OCR is using. Determine the total space available and what
is currently being used.
a) Run the ocrcheck command from the /u01/crs11g/bin directory as shown below.
# sol_10_01_04.sh
cd /u01/crs11g/bin
./ocrcheck
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
Status of Oracle Cluster Registry is as follows
: e@
sic 2 de: nt
Version
Total space (kbytes) e:rd S tu
616652
Used space (kbytes) ilav : this 3848
o _v us:e 612804
Available space (kbytes)
i
ID
a b ric e to :: 992212279
e (f ens
Device/File Name /dev/sdb1
e rd le lic Device/File integrity

check succeeded
VDevice/File
V i l a
f e rab Name : /dev/sdb2
i c i o succeededn s Device/File integrity
a b r check
n - tra
F no Cluster registry integrity check succeeded
b) In this example, the OCR is using the file /dev/sdb1 and /dev/sdb2. The size of the
file is 616652k and of that, 3848k is currently in use.

Practice 10-2: Fixing Oracle Clusterware issues
This practice introduces an error into the cluster synchronization service on your cluster.
From your solutions directory, run the lab10-2-prep.sh script to prepare for the practice.
Then run the breakcss.sh script to introduce the error condition. Observe the results and
inspect your logs to identify and diagnose the problem. Repair the problem and return
your cluster to normal operation.
1) Change your directory to $HOME/solutions/less10 and execute the lab10-2-prep.sh
[oracle@vx0306 less10]$ cat lab10-2-prep.sh
# lab10-2-prep.sh
# voting disk backup

dd if=/dev/sdb5 of=/home/oracle/solutions/less10/vdisk.bak
s a
bs=4k 2>/dev/null
)h a
ฺbr
dd if=/dev/sdb5 of=/home/oracle/solutions/less10/vdisk.bak2
bs=8k 2>/dev/null
c m
o deฺ
i ฺ
d Gui
sudo cp /etc/init.d/init.cssd /home/oracle
r e
sudo sed -i '43i/bin/sleep 30' /etc/init.d/init.cssd
@ sic dent
e r de Stu
v i lav e this
i c i o_ o us
( f abr n./lab10-2-prep.sh
s et
[oracle@vx0306 eless10]$e
e rd le lic
l V ab
2) From thea$HOME/solutions/less10 directory, execute the breakcss.sh script. The
i
scriptVstops thefe r
i c i o anatsthecluster
introduced end
nodeapps and changes the cluster configuration. The error is
of the script execution.
b r - t r
Fa na)onYour nodes will now reboot in reaction to the problem that was introduced. There
is something seriously wrong!
[oracle@vx0306 less10]$ cat breakcss.sh
# breakcss.sh


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
/u01/crs11g/bin/srvctl stop database -d $DBNAME

Practice 10-2: Fixing Oracle Clusterware issues (continued)
/u01/crs11g/bin/srvctl stop asm -n $y
/u01/crs11g/bin/srvctl stop asm -n $z
/u01/crs11g/bin/srvctl stop nodeapps -n $z
/u01/crs11g/bin/srvctl stop nodeapps -n $y
ssh $z sudo /u01/crs11g/bin/crsctl stop crs
sleep 60
sudo /u01/crs11g/bin/crsctl stop crs

s a
sleep 60
)h a
m ฺ br
sudo /u01/crs11g/bin/crsctl delete css votedisk /dev/sdb6
i ฺ c o d-eฺ
force
r e d/dev/sdb7 u i
sudo /u01/crs11g/bin/crsctl delete css votedisk
s ic ent G -
e@ Stud
force
r d
sudo /u01/crs11g/bin/crsctl start
i l a vecrs this
i o _v use
sleep 300
a b ric e to
e
/u01/crs11g/bin/crs_stat (f en-t s
d li c
er/u01/crs11g/bin/crsctl
a V b l e
ssh $z sudo
i l fe r a start crs
i c i o V 300 n s
tra
sleep
a b r -
F o n
n/u01/crs11g/bin/crs_stat -t
while true
do
dd if=/dev/zero of=/dev/sdb5 bs=1M count=20
done
[oracle@vx0306 less10]$ ./breakcss.sh

------------------------------------------------------------
ora....B1.inst application OFFLINE OFFLINE
ora....B2.inst application OFFLINE OFFLINE
ora.RDB.db application OFFLINE OFFLINE
ora....SM1.asm application OFFLINE OFFLINE

ora....SM2.asm application OFFLINE OFFLINE
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Stopping resources.
This could take several minutes. s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
Successful deletion of voting disk /dev/sdb6.
Successful deletion of voting disk /dev/sdb7. @ sic dent
e r de Stu
v i lav e this
Name Type
ic i o_ oTarget
us State Host
b r t
(fa ense ONLINE ONLINE vx0306
------------------------------------------------------------
e
ic
rd application
l
ora....B2.inst
V e b l e ONLINE OFFLINE
i l
ora.RDB.db
a
ora....SM1.asm r aapplication
V sfe application ONLINE ONLINE vx0306
application
ONLINE
ONLINE
ONLINE
ONLINE
vx0306
i o n
icora.vx0306.gsd
a
ora....06.lsnr vx0306
b r t r
on-
Fa nora.vx0306.ons
ora....SM2.asm application ONLINE OFFLINE
ora....13.lsnr application ONLINE OFFLINE
ora.vx0313.gsd application ONLINE OFFLINE
------------------------------------------------------------

20+0 records in
20+0 records out
20+0 records in
20+0 records out
20+0 records in
20+0 records out
20+0 records in
20+0 records out
20+0 records in
20+0 records out
20+0 records in s a
20+0 records out
)h a
20+0 records in
m ฺbr
20+0 records out
i ฺ c o deฺ
20+0 records in
r e d Gui
20+0 records out
-- At this point your session @ sic dent
gets disconnected!
r e Stu
dnode
3) From your classroom PC, attempt to ping the
a e
v thisin your cluster. When you can
first
i l
ping the node successfully, log in.
i o _v use
[oracle@vx0309 ~]$ ping
a b ric10.216.4.19
e to56(84) bytes of data.
f
PING 10.216.4.19 ((10.216.4.19)
e ns icmp_seq=0 ttl=64 time=10.2 ms
64 bytes fromd10.216.4.19:
li c e
64 bytes V er 10.216.4.19:
from
b l e icmp_seq=1 ttl=64 time=0.352 ms
64 bytes
V a
il from e ra10.216.4.19: icmp_seq=2 ttl=64 time=0.323 ms
64o
i ran
bytes s f
from 10.216.4.19: icmp_seq=3 ttl=64 time=0.325 ms
r i c t
Fab n--- on-10.216.4.19 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.323/2.817/10.271/4.303 ms, pipe 2
[oracle@vx0301 ~]$
4) You must now inspect your Oracle Clusterware log files to find the problem.
a) When the node is stable, change your directory to your log directory,
/u01/crs11g/log/<host name>. List the log files. The first file you should look at is
the clusterware alert log. In this example, that file is alerted-.log.
# sol_10_02_04.sh

cd /u01/crs11g/log/$y*
ls -al

total 52
drwxr-xr-t 8 root oinstall 4096 Nov 22 02:33 .
drwxr-xr-x 4 oracle oinstall 4096 Nov 26 02:04 ..
drwxr-x--- 2 oracle oinstall 4096 Nov 22 02:33 admin
-rw-rw-r-- 1 oracle oinstall 18421 Nov 26 02:48
alertvx0308.log
drwxrwx--- 2 oracle oinstall 4096 Nov 26 02:48 client
drwxr-x--- 2 root oinstall 4096 Nov 22 02:34 crsd
drwxr-x--- 4 oracle oinstall 4096 Nov 26 02:54 cssd
drwxr-x--- 2 oracle oinstall 4096 Nov 22 05:32 evmd
drwxrwx--T 5 oracle oinstall 4096 Nov 22 02:42 racg
s a
)h a
m ฺbr
5) Looking at the alert log, what could be the cause of the problem? ฺco
i d e ฺ
d G
reoffline. i
u would
i
s denc
a) You can see that the voting disk appears to be intermittently t This
certainly cause problems with CSS. @
e r de Stu
lav e this
# sol_10_02_05.sh
v i
ic i o_ o us
( f abr nse ||t sed
-n '1,1p'`
sed -n '2,2p'`
e r de lice
l a V able
cd /u01/crs11g/log/$y*
i
Valert*.logf er
i o an
ic[oracle@vx0308
view s
a b r - t r less10]$
F o n
n[oracle@vx0308 less10]$ ./sol_10_02_05.sh
…
[crsd(6213)]CRS-1005:The OCR upgrade was completed. Version
has changed from 185599488 to 185599488. Details in
/u01/crs11g/log/vx0308/crsd/crsd.log.
2007-11-26 01:39:29.697
[cssd(7551)]CRS-1612:node vx0309 (2) at 50% heartbeat fatal,
eviction in 14.080 seconds
2007-11-26 01:39:36.768
2007-11-26 01:39:40.807
2007-11-26 01:39:41.817

2007-11-26 01:39:42.827
2007-11-26 01:39:43.788
[cssd(7551)]CRS-1607:CSSD evicting node vx0309. Details in
/u01/crs11g/log/vx0308/cssd/ocssd.log.
[cssd(7551)]CRS-1601:CSSD Reconfiguration complete. Active
nodes are vx0308 .
2007-11-26 01:39:43.911
[crsd(6213)]CRS-1204:Recovering CRS resources for node vx0309.
[cssd(7551)]CRS-1603:CSSD on node vx0308 shutdown by user.
2007-11-26 01:45:20.142
[cssd(6454)]CRS-1605:CSSD voting file is online: /dev/sdb5.
Details in /u01/crs11g/log/vx0308/cssd/ocssd.log.
nodes are vx0308 . s a
2007-11-26 01:45:27.868
)h a
[evmd(5062)]CRS-1401:EVMD started on node vx0308.
m ฺbr
2007-11-26 01:45:27.996
i ฺ c o deฺ
r e d Gui
[crsd(5835)]CRS-1005:The OCR upgrade was completed. Version
/u01/crs11g/log/vx0308/crsd/crsd.log. @ sic dent

has changed from 185599488 to 185599488. Details in
2007-11-26 01:45:27.997
e r de Stu
v i lav e this
o_ o us
2007-11-26 01:45:28.987
ic i
( f abr nse t
2007-11-26 01:45:29.218
rde le lice
[crsd(5835)]CRS-1205:Auto-start failed for the CRS resource .
e
Details in vx0308.
V ab
i l a
2007-11-26 01:45:29.232
V sfer
[crsd(5835)]CRS-1205:Auto-start failed for the CRS resource .
i o
ic -tran
Details in vx0308.
b r
Fa non
nodes are vx0308 vx0309 .
2007-11-26 01:51:15.029
[cssd(6454)]CRS-1604:CSSD voting file is offline: /dev/sdb5.
2007-11-26 01:51:23.029
2007-11-26 01:51:24.225
2007-11-26 01:51:33.413
2007-11-26 01:51:34.470
2007-11-26 01:51:43.894

2007-11-26 01:51:46.128
2007-11-26 01:51:53.913
2007-11-26 01:51:55.017
2007-11-26 01:52:04.424
2007-11-26 01:52:06.618
2007-11-26 01:52:14.439 s a
)h a
m ฺbr
2007-11-26 01:52:15.597
i ฺ c o deฺ
r e d Gui
2007-11-26 01:52:24.802 @ sic dent

r de Stu
e
v i lav e this
o_ o us
2007-11-26 01:52:25.838
ic i
( f abr nse t
rde le lice
2007-11-26 01:52:35.053
e
V ab
i l a
V sfer
2007-11-26 01:53:39.748
i o
ic -tran
b r
Fa non
2007-11-26 01:53:47.775
2007-11-26 01:53:48.984
2007-11-26 01:53:48.984
2007-11-26 02:02:07.242
2007-11-26 02:02:07.254
nodes are vx0308 vx0309 .
2007-11-26 02:02:08.937

2007-11-26 02:02:08.941
[evmd(6235)]CRS-1401:EVMD started on node vx0308.
2007-11-26 02:02:09.907
2007-11-26 02:02:16.220
…
6) The alert log indicates that further information can be found. Where would you have
to look at?
a) In the /u01/crs11g/log/<host name>/cssd/ocssd.log. The real problem is revealed
here. You can see that the voting disk, /dev/sdb5 is corrupted. This was the cause
of the reboot.
s a
)h a
ฺbr
# sol_10_02_06.sh
c m
o deฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` ed i ฺ u i
r G
@ sic dent
view /u01/crs11g/log/$y*/cssd/ocssd.log
e r de Stu
v i lav e this
i o_ o us
[oracle@vx0308 less10]$ic./sol_10_02_06.sh
( f abr nse t
…
e rde le lic01:39:36.767
e
[
a V
CSSD]2007-11-26
b
[2941873056] >WARNING:
V ra seconds
il infe7.010
clssnmPollingThread: node vx0309 (2) at 75% heartbeat fatal,
i c
eviction
[i a n s
o CSSD]2007-11-26
b r - t r 01:39:40.807 [2941873056] >WARNING:
Fa neviction
on
in 2.970 seconds
[ CSSD]2007-11-26 01:39:41.817 [2941873056] >WARNING:
[ CSSD]2007-11-26 01:39:42.827 [2941873056] >WARNING:
[ CSSD]2007-11-26 01:39:43.787 [2941873056] >TRACE:
clssnmPollingThread: Eviction started for node vx0309 (2),
flags 0x0001, state 3, wt4c 0
[ CSSD]2007-11-26 01:39:43.787 [2925087648] >TRACE:
clssnmDoSyncUpdate: Initiating sync 91161491
[ CSSD]2007-11-26 01:39:43.787 [2925087648] >TRACE:
clssnmDoSyncUpdate: diskTimeout set to (27000)ms
[ CSSD]2007-11-26 01:39:43.787 [2925087648] >TRACE:
clssnmSetupAckWait: Ack message type (11)
[ CSSD]2007-11-26 01:39:43.787 [2925087648] >TRACE:
clssnmSetupAckWait: node(1) is ALIVE

[ CSSD]2007-11-26 01:39:43.787 [2925087648] >TRACE:
clssnmSendSync: syncSeqNo(91161491)
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmWaitForAcks: Ack message type(11), ackCount(1)
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
clssnmHandleSync: diskTimeout set to (27000)ms
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
clssnmHandleSync: Acknowledging sync: src[1] srcName[vx0308]
seq[1] sync[91161491]
[ CSSD]2007-11-26 01:39:43.788 [3056544128] >USER:
NMEVENT_SUSPEND [00][00][00][06]
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmWaitForAcks: done, msg type(11)
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmNeedConfReq: No configuration to change
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE: s a
clssnmDoSyncUpdate: Terminating node 2, vx0309,
)h a
misstime(30010) state(5)
m ฺbr
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
i ฺ c o deฺ
r e d Gui
[
sic dent
CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmSetupAckWait: node(1) is ACTIVE @
[ r de Stu
CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
e
clssnmSendVote: syncSeqNo(91161491)
v i lav e this
o_ o us
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
ic i
[
abr nse t
CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
( f
rde le lice
clssnmSendVoteInfo: node(1) syncSeqNo(91161491)
[ e
CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
V ab
[ i l a
V sfer
CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
i o
ic -tran
clssnmCheckDskInfo: Checking disk info...
b r
Fa non
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmEvict: Start
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmEvict: Evicting node 2, vx0309, birth 91161446, death
91161491, impendingrcfg 1, stateflags 0x1
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmEvict: Fencing node 2, vx0309, with SAGE
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmWaitOnEvictions: Start
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmWaitOnEvictions: node 2, undead 0, reqid 0
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:
clssnmSendUpdate: syncSeqNo(91161491)
[ CSSD]2007-11-26 01:39:43.788 [2925087648] >TRACE:

[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
clssnmUpdateNodeState: node 0, state (0/0) unique (0/0)
prevConuni(0) birth (0/0) (old/new)
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
clssnmUpdateNodeState: node 1, state (3/3) unique
(1195926668/1195926668) prevConuni(0) birth
(91161490/91161490) (old/new)
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
(91161446/91161446) (old/new)
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
clssnmDeactivateNode: node 2 (vx0309) left cluster
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >USER:

clssnmHandleUpdate: SYNC(91161491) from node(1) completed s a
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >USER:
)h a
clssnmHandleUpdate: NODE 1 (vx0308) IS ACTIVE MEMBER OF
m ฺbr
CLUSTER
i ฺ c o deฺ
[ CSSD]2007-11-26 01:39:43.788 [3000888224] >TRACE:
r e d Gui
[ @ sic dent
clssnmHandleUpdate: diskTimeout set to (200000)ms
CSSD]2007-11-26 01:39:43.789 [2925087648] >TRACE:
e r de Stu
[
lav e this
CSSD]2007-11-26 01:39:43.789 [2925087648] >TRACE:
v i
o_ o us
clssnmDoSyncUpdate: Sync 91161491 complete!
[
ic i
CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
( f abr nse t
clssgmReconfigThread: started for reconfig (91161491)
rde le lice
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >USER:
e
NMEVENT_RECONFIG [00][00][00][02]
V ab
[
l a
CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
i
V sfer
clssgmCleanupGrocks: cleaning up grock crs_version type 2
i o
ic -tran
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
b r
Fa non
clssgmCleanupOrphanMembers: cleaning up remote mbr(2)
grock(crs_version) birth(91161446/0)
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmCleanupGrocks: cleaning up grock EVMDMAIN type 2
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmCleanupOrphanMembers: cleaning up remote mbr(2)
grock(EVMDMAIN) birth(91161446/0)
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmCleanupGrocks: cleaning up grock CRSDMAIN type 2
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmCleanupGrocks: cleaning up grock ocr_vx_cluster03 type 2
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_vx0308
type 3
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmCleanupGrocks: cleaning up grock #CSS_CLSSOMON type 2
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmEstablishConnections: 1 nodes in cluster incarn 91161491
[ CSSD]2007-11-26 01:39:43.789 [2950532000] >TRACE:
clssgmPeerListener: connects done (1/1)

[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmEstablishMasterNode: MASTER for 91161491 is node(1)
birth(91161490)
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmChangeMasterNode: requeued 0 RPCs
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmMasterCMSync: Synchronizing group/lock status
[ CSSD]2007-11-26 01:39:43.789 [2899909536] >TRACE:
clssgmMasterSendDBDone: group/lock status synchronization
complete
[ CSSD]CLSS-3000: reconfiguration successful, incarnation
91161491 with 1 nodes
[ CSSD]CLSS-3001: local node number 1, master node number 1
[ CSSD]2007-11-26 01:39:43.848 [2899909536] >TRACE: s a

clssgmReconfigThread: completed for reconfig(91161491), with
)h a
status(1)
m ฺbr
[ CSSD]2007-11-26 01:40:13.684 [2984102816] >WARNING:
i ฺ c o deฺ
r e d Gui
clssgmShutDown: Received explicit shutdown request from
client.
[ @ sic dent
CSSD]2007-11-26 01:40:13.684 [2984102816] >TRACE:
r de Stu
clssgmClientShutdown: Aborting client (0x835f580) proc
e
(0x835ecc8)
v i lav e this
o_ o us
[ CSSD]2007-11-26 01:40:13.684 [2984102816] >TRACE:
ic i
clssgmClientShutdown: Aborting client (0x8364128) proc
(0x835ef20)
( f abr nse t
rde le lice
[ CSSD]2007-11-26 01:40:13.684 [2984102816] >TRACE:
e
clssgmClientShutdown: Aborting client (0x8364580) proc
V ab
[ i l a
(0x835ef20)
V sfer
CSSD]2007-11-26 01:40:13.684 [2984102816] >TRACE:
i o
ic -tran
clssgmClientShutdown: Aborting client (0x8364af8) proc
b r
Fa non
(0x835ef20)
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
clssgmClientShutdown: Aborting client (0x835f320) proc
(0x835fdf8)
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
clssgmClientShutdown: waited 0 seconds on 5 IO capable clients
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
clssgmClientShutdown: I/O capable proc (0x835ecc8), pid (7479)
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
clssgmClientShutdown: I/O capable proc (0x835ef20), pid (6213)
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
clssgmClientShutdown: I/O capable proc (0x835fdf8), pid (6223)

[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.685 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.688 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.688 [2984102816] >TRACE:
[ CSSD]2007-11-26 01:40:13.688 [2984102816] >TRACE:
clssgmClientShutdown: I/O capable proc (0x835ef20), pid (6213) s a
[ CSSD]2007-11-26 01:40:13.688 [2984102816] >TRACE:
)h a
m ฺbr
[ CSSD]2007-11-26 01:40:13.688 [2984102816] >TRACE:
i ฺ c o deฺ
r e d Gui
[
sic dent
CSSD]2007-11-26 01:40:13.709 [2984102816] >TRACE:
@
[ r de Stu
CSSD]2007-11-26 01:40:13.709 [2984102816] >TRACE:
e
v i lav e this
o_ o us
[ CSSD]2007-11-26 01:40:13.742 [2984102816] >TRACE:
ic i
[
abr nse t
CSSD]2007-11-26 01:40:13.764 [2984102816] >WARNING:
( f
rde le lice
clssgmClientShutdown: graceful shutdown completed.
e
[ clsdmt]Listening to
V ab
[ l a
(ADDRESS=(PROTOCOL=ipc)(KEY=vx0308DBG_CSSD))
i
V sfer
CSSD]2007-11-26 01:45:19.974 >USER: Oracle Database
i o
ic -tran
10g CSS Release 11.1.0.6.0 Production Copyright 1996, 2004
b r
Fa non
Oracle. All rights reserved.
[ CSSD]2007-11-26 01:45:19.974 >USER: CSS daemon log for
node vx0308, number 1, in cluster vx_cluster03
[ CSSD]2007-11-26 01:45:19.984 [3055966592] >TRACE:
clssscmain: local-only set to false
[ CSSD]2007-11-26 01:45:20.001 [3055966592] >TRACE:
clssnmReadNodeInfo: added node 1 (vx0308) to cluster
[ CSSD]2007-11-26 01:45:20.016 [3055966592] >TRACE:
[ CSSD]2007-11-26 01:45:20.020 [3055966592] >WARNING:
clssnmReadWallet: Open Wallet returned 28759
[ CSSD]2007-11-26 01:45:20.020 [3055966592] >WARNING:
clssnmInitNMInfo: Node not configured for node kill
[ CSSD]2007-11-26 01:45:20.037 [3028274080] >TRACE:
clssnm_skgxninit: Compatible vendor clusterware not in use
[ CSSD]2007-11-26 01:45:20.037 [3028274080] >TRACE:
clssnm_skgxnmon: skgxn init failed
[ CSSD]2007-11-26 01:45:20.044 [3055966592] >TRACE:
clssnmNMInitialize: Network heartbeat thresholds are:

impending reconfig 15000 ms, reconfig start (misscount) 30000
ms
[ CSSD]2007-11-26 01:45:20.046 [3055966592] >TRACE:
clssnmNMInitialize: Voting file I/O timeouts are: short 27000
ms, long 200000 ms
[ CSSD]2007-11-26 01:45:20.052 [3055966592] >TRACE:
clssnmDiskStateChange: state from 1 to 2 disk (0//dev/sdb5)
[ CSSD]2007-11-26 01:45:20.052 [3028274080] >TRACE:
clssnmvDPT: spawned for disk 0 (/dev/sdb5)
[ CSSD]2007-11-26 01:45:20.119 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:45:20.142 [3055966592] >TRACE:
clssnmFatalInit: fatal mode enabled
[ CSSD]2007-11-26 01:45:20.184 [3019201440] >TRACE:
clssnmvKillBlockThread: spawned for disk 0 (/dev/sdb5) initial
sleep interval (1000)ms s a
[ CSSD]2007-11-26 01:45:20.188 [3028274080] >TRACE:
)h a
clssnmReadDskHeartbeat: node 2, vx0309, has a disk HB, but no
m ฺbr
network HB, DHB has rcfg 91161491, wrtcnt, 338751, LATS
i ฺ c o deฺ
136615880, lastSeqNo 338751, timestamp 1196062746/394546570
r e d Gui
[ CSSD]2007-11-26 01:45:20.200 [3001019296] >TRACE:
clssnmClusterListener: Listening on @ sic dent
r de Stu
e
v i e his
lav [3001019296]
t
[ CSSD]2007-11-26 01:45:20.207
clssnmClusterListener: c i o _ s
u vx0309 (2),
>TRACE:
probcon(0x8282438) br
i Probing t o node
[
e
CSSD]2007-11-26
e n se
(fa 01:45:20.218 [3001019296] >TRACE:
clsc_send_msg: r d
e0) ble l i c
(0x82820a8) NS err (12571, 12560), transport
V
ila fera
(530, 111,
V
i c[io
a n s
CSSD]2007-11-26 01:45:20.218 [3001019296] >TRACE:
b r t r
on-
Fa n(0x8282438),
clssnmDiscHelper: vx0309, node(2) connection failed, con
probe(0x8282438)
[ CSSD]2007-11-26 01:45:20.218 [2992360352] >TRACE:
clssgmDeathChkThread: Spawned
[ CSSD]2007-11-26 01:45:20.268 [2983967648] >TRACE:
clssgmclientlsnr: listening on
(ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_vx_cluster03_1
))
[ CSSD]2007-11-26 01:45:20.268 [2983967648] >TRACE:
(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_vx0308_vx_cluster03))
[ CSSD]2007-11-26 01:45:20.268 [2983967648] >TRACE:
(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_vx0308_))
[ CSSD]2007-11-26 01:45:20.296 [2949995424] >TRACE:
clssgmPeerListener: Listening on
(ADDRESS=(PROTOCOL=tcp)(DEV=22)(HOST=10.196.30.29)(PORT=18702)
)
[ CSSD]2007-11-26 01:45:27.387 [2924551072] >TRACE:
clssnmRcfgMgrThread: Local Join

[ CSSD]2007-11-26 01:45:27.388 [2924551072] >WARNING:
clssnmLocalJoinEvent: takeover succ
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
clssnmSetFirstIncarn: Incarnation 0
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
clssnmSetFirstIncarn: Node 1 incarnation 91161492
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
clssnmSetFirstIncarn: Node 2 incarnation 91161491 s a
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
)h a
clssnmSetFirstIncarn: Incarnation set to 91161493
m ฺbr
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
i ฺ c o deฺ
r e d Gui
[
sic dent
CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
@
[ r de Stu
CSSD]2007-11-26 01:45:27.388 [3001019296] >TRACE:
e
v i lav e this
o_ o us
[ CSSD]2007-11-26 01:45:27.388 [3001019296] >TRACE:
ic i
seq[1] sync[91161493]
( f abr nse t
rde le lice
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
e
V ab
[
l a
CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
i
V sfer
i o
ic -tran
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
b r
Fa non
clssnmDoSyncUpdate: node(1) is transitioning from joining
state to active state
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
clssnmSendVote: syncSeqNo(91161493)
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [3001019296] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
clssnmCheckDskInfo: diskTimeout set to (200000)ms
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
clssnmEvict: Start

[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.388 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.389 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.389 [3001019296] >TRACE:
[ CSSD]2007-11-26 01:45:27.389 [3001019296] >TRACE:
(1196063119/1196063119) prevConuni(0) birth s a
(91161493/91161493) (old/new)
)h a
[ CSSD]2007-11-26 01:45:27.389 [3001019296] >TRACE:
m ฺbr
i ฺ c o deฺ
r e d Gui
[
sic dent
CSSD]2007-11-26 01:45:27.410 [3001019296] >TRACE:
@
clssscInitSAGEFencing: kgzf fence initialization successfully
started
e r de Stu
[
lav e this
CSSD]2007-11-26 01:45:27.411 [3001019296] >USER:
v i
o_ o us
clssnmHandleUpdate: SYNC(91161493) from node(1) completed
[
ic i
CSSD]2007-11-26 01:45:27.411 [3001019296] >USER:
( f abr nse t
rde le lice
CLUSTER
[ e
CSSD]2007-11-26 01:45:27.411 [3001019296] >TRACE:
V ab
[ i l a
V sfer
CSSD]2007-11-26 01:45:27.411 [2924551072] >TRACE:
i o
ic -tran
b r
Fa non
[ CSSD]2007-11-26 01:45:27.411 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:45:27.417 [3055966592] >USER:
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE: KGZF:
context successfully initialized, API version 1.1
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:

clssscSAGEInitFenceCompl: kgzf fence initialization
successfully completed
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >USER:
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:
[ CSSD]2007-11-26 01:45:27.429 [2949995424] >TRACE:
clssgmPeerListener: connects done (1/1)

[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:
birth(91161493)
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:
[ CSSD]2007-11-26 01:45:27.429 [2899372960] >TRACE:
complete
[ CSSD]2007-11-26 01:45:27.430 [2899372960] >TRACE: s a

)h a
status(1)
m ฺbr
[ CSSD]2007-11-26 01:45:27.512 [2983967648] >TRACE:
i ฺ c o deฺ
clssgmCommonAddMember: clsomon joined
r e d Gui
(1/0x1000000/#CSS_CLSSOMON)
[ @ sic dent
CSSD]2007-11-26 01:46:21.593 [3001019296] >TRACE:
r de Stu
clssnmConnComplete: MSGSRC 2, type 5, node 2, flags 0x0001,
e
v i lav e this
con 0x8370ac0, probe (nil), nodekillsz 0
o_ o us
[ CSSD]2007-11-26 01:46:21.593 [3001019296] >TRACE:
ic i
clssnmConnComplete: node 2, vx0309, con(0x8370ac0),
( f abr nse t
probcon((nil)), ninfcon((nil)), node unique 1196063173, prev
rde le lice
unique 0, msg unique 1196063173 node state 0
[ e
CSSD]2007-11-26 01:46:21.593 [3001019296] >TRACE:
V ab
[ i l a
clssnmsendConnAck: node 2, node state 0
V sfer
CSSD]2007-11-26 01:46:21.593 [3001019296] >TRACE:
i o
ic -tran
clssnmSendConnAck: connected to node 2 , ninfcon (0x8370ac0),
b r
Fa non
state (0)
[ CSSD]2007-11-26 01:46:21.593 [3001019296] >TRACE:
clssnmConnComplete: connecting to node 2 (con 0x8370ac0),
ninfcon (0x8370ac0), state (0)
[ CSSD]2007-11-26 01:46:21.593 [3001019296] >TRACE:
clssnmConnComplete: ninf->killinfosz 0
[ CSSD]2007-11-26 01:46:21.941 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.941 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.941 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.941 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.941 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.941 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.941 [3001019296] >TRACE:

[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.942 [3001019296] >TRACE:
seq[5] sync[91161494]
[ CSSD]2007-11-26 01:46:21.942 [3055966592] >USER:
[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
clssnmDoSyncUpdate: node(2) is transitioning from joining
state to active state
[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
clssnmSetupAckWait: Ack message type (13) s a
[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
)h a
m ฺbr
[ CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
i ฺ c o deฺ
r e d Gui
[
sic dent
CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
clssnmSendVote: syncSeqNo(91161494) @
[ r de Stu
CSSD]2007-11-26 01:46:21.942 [2924551072] >TRACE:
e
v i lav e this
o_ o us
[ CSSD]2007-11-26 01:46:21.942 [3001019296] >TRACE:
ic i
[
abr nse t
CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
( f
rde le lice
[ e
CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
V ab
[ i l a
V sfer
CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
i o
ic -tran
clssnmEvict: Start
b r
Fa non
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [3001019296] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [3001019296] >TRACE:
(91161493/91161493) (old/new)

[ CSSD]2007-11-26 01:46:21.943 [3001019296] >TRACE:
(91161494/91161494) (old/new)
[ CSSD]2007-11-26 01:46:21.943 [3001019296] >USER:
[ CSSD]2007-11-26 01:46:21.943 [3001019296] >USER:
CLUSTER
[ CSSD]2007-11-26 01:46:21.943 [3001019296] >USER:
CLUSTER
[ CSSD]2007-11-26 01:46:21.943 [3001019296] >TRACE:
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
clssnmWaitForAcks: done, msg type(15) s a
[ CSSD]2007-11-26 01:46:21.943 [2924551072] >TRACE:
)h a
m ฺbr
[ CSSD]2007-11-26 01:46:21.945 [2899372960] >TRACE:
i ฺ c o deฺ
r e d Gui
[
sic dent
CSSD]2007-11-26 01:46:21.945 [2899372960] >USER:
NMEVENT_RECONFIG [00][00][00][06] @
[ r de Stu
CSSD]2007-11-26 01:46:21.946 [2899372960] >TRACE:
e
v i lav e this
o_ o us
[ CSSD]2007-11-26 01:46:22.023 [2949995424] >TRACE:
ic i
clssgmInitialRecv: (0x83831b0) accepted a new connection from
( f abr nse t
node 2 born at 91161494 active (2, 2), vers (11,1,1,2)
rde le lice
[ CSSD]2007-11-26 01:46:22.023 [2949995424] >TRACE:
e
clssgmInitialRecv: conns done (2/2)
V ab
[
l a
CSSD]2007-11-26 01:46:22.023 [2899372960] >TRACE:
i
V sfer
i o
ic -tran
birth(91161493)
b r
Fa non
[ CSSD]2007-11-26 01:46:22.024 [2899372960] >TRACE:
[ CSSD]2007-11-26 01:46:22.028 [2899372960] >TRACE:
complete
[ CSSD]2007-11-26 01:46:22.030 [2899372960] >TRACE:

status(1)
[ CSSD]2007-11-26 01:51:14.745 [3019201440] >ERROR:
clssnmvDiskKillCheck: voting disk corrupted
(0x00000000,0x00000000) (0//dev/sdb5)
[ CSSD]2007-11-26 01:51:14.745 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:51:23.029 [3028274080] >TRACE:

[ CSSD]2007-11-26 01:51:24.225 [3019201440] >ERROR:
(0x00000000,0x00000000) (0//dev/sdb5)
[ CSSD]2007-11-26 01:51:24.225 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:51:33.413 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:51:34.470 [3019201440] >ERROR:
(0x00000000,0x00000000) (0//dev/sdb5)
[ CSSD]2007-11-26 01:51:34.470 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:51:43.894 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:51:34.470 [3019201440] >ERROR:
clssnmvDiskKillCheck: voting disk corrupted s a
(0x00000000,0x00000000) (0//dev/sdb5)
)h a
[ CSSD]2007-11-26 01:51:34.470 [3019201440] >TRACE:
m ฺbr
i ฺ c o deฺ
[ CSSD]2007-11-26 01:51:43.894 [3028274080] >TRACE:
r e d Gui
[ @ sic dent
CSSD]2007-11-26 01:51:45.548 [3019201440] >ERROR:
r
e de Stu
v i lav e this
(0x00000000,0x00000000) (0//dev/sdb5)
o_ o us
[ CSSD]2007-11-26 01:51:45.548 [3019201440] >TRACE:
ic i
[
abr nse t
CSSD]2007-11-26 01:51:53.913 [3028274080] >TRACE:
( f
rde le lice
[ e
CSSD]2007-11-26 01:51:55.016 [3019201440] >ERROR:
V ab
i l a
V sfer
(0x00000000,0x00000000) (0//dev/sdb5)
i o
ic -tran
[ CSSD]2007-11-26 01:51:55.016 [3019201440] >TRACE:
b r
Fa non
[ CSSD]2007-11-26 01:52:04.424 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:52:06.098 [3019201440] >ERROR:
(0x00000000,0x00000000) (0//dev/sdb5)
[ CSSD]2007-11-26 01:52:06.098 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:52:14.439 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:52:15.597 [3019201440] >ERROR:
(0x00000000,0x00000000) (0//dev/sdb5)
[ CSSD]2007-11-26 01:52:15.597 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:52:24.802 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:52:25.836 [3019201440] >ERROR:
(0x00000000,0x00000000) (0//dev/sdb5)

[ CSSD]2007-11-26 01:52:25.836 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:52:35.053 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:53:39.731 [3019201440] >ERROR:
(0x00000000,0x00000000) (0//dev/sdb5)
[ CSSD]2007-11-26 01:53:39.731 [3019201440] >TRACE:
[ CSSD]2007-11-26 01:53:47.775 [3028274080] >TRACE:
[ CSSD]2007-11-26 01:53:48.984 [3028274080] >ERROR:
Internal Error Information:
Category: 1234
Location: fwrite_faile
Other: fwrite unable to write buffer s a
Dep: 5
)h a
m ฺ br
[ CSSD]2007-11-26 01:53:48.984 [3028274080] >ERROR:
i o deฺ
ฺc/dev/sdb5
clssnmvWriteBlocks: write failed 1 at offset 17 dof
r e u i
[ CSSD]2007-11-26 01:53:48.984 [3028274080] >TRACE:
n G
sic (0//dev/sdb5)
t
clssnmDiskStateChange: state from 4 to @
e Stud>ERROR:
3 disk e
[ r
CSSD]2007-11-26 01:53:48.984 [3019201440] d
clssnmvDiskKillCheck: voting disk
i l a vecorrupted
t his
v
_ us
(0x00000000,0x00000000) (0//dev/sdb5) e
[ clsdmt]Listening to cio
b i
r e to
a
(ADDRESS=(PROTOCOL=ipc)(KEY=vx0308DBG_CSSD))
(f 02:02:06.830
ns
[
r d e
CSSD]2007-11-26
l i c e >USER: Oracle Database
Oracle. V
e ble reserved.
10g CSS Release 11.1.0.6.0 Production Copyright 1996, 2004
la fera
All rights
[ ViCSSD]2007-11-26 02:02:06.830 >USER: CSS daemon log for
i o an s
ic[ -tCSSD]2007-11-26
node vx0308, number 1, in cluster vx_cluster03
b r r
Fa nclssscmain:
on
02:02:06.855 [3056085376] >TRACE:
local-only set to false
[ CSSD]2007-11-26 02:02:06.918 [3056085376] >TRACE:
[ CSSD]2007-11-26 02:02:06.964 [3056085376] >TRACE:
[ CSSD]2007-11-26 02:02:06.968 [3056085376] >WARNING:
clssnmReadWallet: Open Wallet returned 28759
[ CSSD]2007-11-26 02:02:06.968 [3056085376] >WARNING:
clssnmInitNMInfo: Node not configured for node kill
[ CSSD]2007-11-26 02:02:06.997 [3028392864] >TRACE:
clssnm_skgxninit: Compatible vendor clusterware not in use
[ CSSD]2007-11-26 02:02:06.997 [3028392864] >TRACE:
clssnm_skgxnmon: skgxn init failed
[ CSSD]2007-11-26 02:02:07.016 [3056085376] >TRACE:
clssnmNMInitialize: Network heartbeat thresholds are:
impending reconfig 15000 ms, reconfig start (misscount) 30000
ms

[ CSSD]2007-11-26 02:02:07.020 [3056085376] >TRACE:
clssnmNMInitialize: Voting file I/O timeouts are: short 27000
ms, long 200000 ms
[ CSSD]2007-11-26 02:02:07.039 [3056085376] >TRACE:
[ CSSD]2007-11-26 02:02:07.039 [3028392864] >TRACE:
clssnmvDPT: spawned for disk 0 (/dev/sdb5)
[ CSSD]2007-11-26 02:02:07.150 [3028392864] >TRACE:
[ CSSD]2007-11-26 02:02:07.168 [3019320224] >TRACE:
clssnmvKillBlockThread: spawned for disk 0 (/dev/sdb5) initial
sleep interval (1000)ms
[ CSSD]2007-11-26 02:02:07.173 [3056085376] >TRACE:
clssnmFatalInit: fatal mode enabled
[ CSSD]2007-11-26 02:02:07.212 [3019320224] >ERROR:
clssnmvDiskKillCheck: voting disk corrupted s a
(0x00000000,0x00000000) (0//dev/sdb5)
)h a
[ CSSD]2007-11-26 02:02:07.212 [3019320224] >TRACE:
m ฺbr
i ฺ c o deฺ
[ CSSD]2007-11-26 02:02:07.212 [3028392864] >TRACE:
r e d Gui
@ sic dent
clssnmReadDskHeartbeat: node 2, vx0309, has a disk HB, but no
network HB, DHB has rcfg 91161496, wrtcnt, 339634, LATS
r de Stu
184000, lastSeqNo 339634, timestamp 1196064115/395916300
e
[
lav e this
CSSD]2007-11-26 02:02:07.227 [3002399648] >TRACE:
v i
o_ o us
clssnmClusterListener: Listening on
ic i
( f a br se t
[
en
e licSpawned
CSSD]2007-11-26 02:02:07.239
r d
[2992634784] >TRACE:
e ble 02:02:07.254 [3002399648] >TRACE:
clssgmDeathChkThread:
V
[
V a
il fera
CSSD]2007-11-26
clssnmClusterListener: Probing node vx0309 (2),
i o an s
ic[ -tCSSD]2007-11-26
probcon(0xb265f128)
b r r
Fa nclssnmConnComplete:
on
02:02:07.255 [3002399648] >TRACE:
MSGSRC 2, type 6, node 2, flags 0x0001,
con 0xb265f128, probe 0xb265f128, nodekillsz 0
[ CSSD]2007-11-26 02:02:07.255 [3002399648] >TRACE:
clssnmConnComplete: node 2, vx0309, con(0xb265f128),
probcon(0xb265f128), ninfcon((nil)), node unique 1196063173,
prev unique 0, msg unique 1196063173 node state 0
[ CSSD]2007-11-26 02:02:07.255 [3002399648] >TRACE:
clssnmConnComplete: connected to node 2 (con 0xb265f128),
ninfcon (0xb265f128), state (0), flag (1037)
[ CSSD]2007-11-26 02:02:07.255 [3002399648] >TRACE:
clssnmConnComplete: ninf->killinfosz 0
[ CSSD]2007-11-26 02:02:07.293 [2984242080] >TRACE:
(ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_vx_cluster03_1
))
[ CSSD]2007-11-26 02:02:07.293 [2984242080] >TRACE:
(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_vx0308_vx_cluster03))

[ CSSD]2007-11-26 02:02:07.293 [2984242080] >TRACE:
(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_vx0308_))
[ CSSD]2007-11-26 02:02:07.320 [2950134688] >TRACE:
clssgmPeerListener: Listening on
(ADDRESS=(PROTOCOL=tcp)(DEV=22)(HOST=10.196.30.29)(PORT=46494)
)
[ CSSD]2007-11-26 02:02:08.324 [3002399648] >TRACE:
[ CSSD]2007-11-26 02:02:08.324 [3002399648] >TRACE:
seq[5] sync[91161496]
[ CSSD]2007-11-26 02:02:08.324 [2924690336] >TRACE:
clssnmRcfgMgrThread: initial lastleader(2) unique(1196063173)
[ CSSD]2007-11-26 02:02:08.324 [3002399648] >TRACE:
clssnmSendVoteInfo: node(2) syncSeqNo(91161496) s a
[ CSSD]2007-11-26 02:02:08.325 [3002399648] >TRACE:
)h a
clssnmHandleUpdate: setting initial cluster incarnation to
m ฺbr
91161493
i ฺ c o deฺ
[ CSSD]2007-11-26 02:02:08.325 [3002399648] >TRACE:
r e d Gui
prevConuni(0) birth (0/0) (old/new) @ sic dent
[ r de Stu
CSSD]2007-11-26 02:02:08.325 [3002399648] >TRACE:
e
v i lav e this
o_ o us
(1196064126/1196064126) prevConuni(0) birth (0/91161496)
(old/new)
ic i
[
abr nse t
CSSD]2007-11-26 02:02:08.325 [3002399648] >TRACE:
( f
rde le lice
e
(1196063173/1196063173) prevConuni(0) birth (0/91161494)
V ab
[ i l
(old/new)
a
V sfer
CSSD]2007-11-26 02:02:08.347 [3002399648] >TRACE:
i o
ic -tran
clssscInitSAGEFencing: kgzf fence initialization successfully
b r
Fa non
started
[ CSSD]2007-11-26 02:02:08.347 [3002399648] >USER:
[ CSSD]2007-11-26 02:02:08.347 [3002399648] >USER:
CLUSTER
[ CSSD]2007-11-26 02:02:08.347 [3002399648] >USER:
CLUSTER
[ CSSD]2007-11-26 02:02:08.347 [3002399648] >TRACE:
[ CSSD]2007-11-26 02:02:08.428 [3056085376] >USER:
[ CSSD]2007-11-26 02:02:08.438 [2899512224] >TRACE:
[ CSSD]2007-11-26 02:02:08.438 [2899512224] >TRACE: KGZF:
context successfully initialized, API version 1.1

[ CSSD]2007-11-26 02:02:08.438 [2899512224] >TRACE:
clssscSAGEInitFenceCompl: kgzf fence initialization
successfully completed
[ CSSD]2007-11-26 02:02:08.438 [2899512224] >USER:
[ CSSD]2007-11-26 02:02:08.438 [2899512224] >TRACE:
[ CSSD]2007-11-26 02:02:08.439 [2950134688] >TRACE:
clssgmInitialRecv: (0x82f7430) accepted a new connection from
node 2 born at 91161494 active (2, 2), vers (11,1,1,2)
[ CSSD]2007-11-26 02:02:08.439 [2950134688] >TRACE:
clssgmInitialRecv: conns done (2/2)
[ CSSD]2007-11-26 02:02:08.440 [2899512224] >TRACE:
birth(91161494)
[ CSSD]2007-11-26 02:02:08.440 [2899512224] >TRACE: s a
)h a
[ CSSD]2007-11-26 02:02:08.446 [2950134688] >TRACE:
m ฺbr
clssgmHandleDBDone(): src/dest (2/65535) size(68) incarn
i ฺ c o deฺ
91161496
r e d Gui
[
sic dent
CSSD]CLSS-3000: reconfiguration successful, incarnation
91161496 with 2 nodes @
CSSD]CLSS-3001: local node v e rde 1,s master
S tu
[
i l a number
t h i node number 2
i o _v us[2899512224]
e
[
b r iccompleted
CSSD]2007-11-26 02:02:08.447
t o >TRACE:
clssgmReconfigThread:
( f a nse for reconfig(91161496), with
rde le lic02:02:08.557
e
status(1)
[ e
CSSD]2007-11-26
V
[2950134688] >TRACE:
V ila ferab
clssgmCommonAddMember:
(1/0x1000000/#CSS_CLSSOMON)
clsomon joined
i c[io
a n s
CSSD]2007-11-26 02:02:16.220 [3028392864] >TRACE:
b r t r
Fa n[on- CSSD]2007-11-26 02:02:16.222 [3028392864] >ERROR:
clssnmvReadFatal: voting device corrupt

(0x00000000/0x00000000/0//dev/sdb5)
7) Fix the diagnosed problem.

a) Although Oracle Clusterware is able to function in this situation, it is necessary to
recover the voting disk file, /dev/sbd5. A backup was made at the beginning of
this practice by the lab10-2-prep.sh script. It is called vdisk.bak and located in
your solutions directory. After you stop Oracle Clusterware on both nodes, use the
dd command and specify a 4K block size. Execute the command as shown below.
[oracle@vx0308 less10]$ sudo /u01/crs11g/bin/crsctl stop crs
Stopping resources.

[oracle@vx0309 ~]$ sudo /u01/crs11g/bin/crsctl stop crs

Stopping resources.
[oracle@vx0309 ~]$
s a
)h a
# sol_10_02_07.sh
m ฺbr
i ฺ c o deฺ
dd if=/home/oracle/solutions/less10/vdisk.bak of=/dev/sdb5
r e d Gui
bs=4k
e r de Stu
v i lav e this
o_ o us
154224+1 records in
ic i
154224+1 records outbr e t
[oracle@vx0308 less10]$ n( f a s
e r de lice
8) After you fixed
l a b le what should you do?
V theaproblem,
i
V Oracle
a)o Re-start f er Clusterware on both nodes using the crsctl command as root on
i
ic both s
n as shown below.
anodes
b r - t r
Fa n[oracle@vx0306
on less10]$ cat sol_10_02_08.sh
# sol_10_02_08.sh

sudo /u01/crs11g/bin/crsctl start crs
ssh $z sudo /u01/crs11g/bin/crsctl start crs



9) Using the crs_stat command, check the status of your CRS stack and nodeapps. Be
patient, it takes a few minutes for the components to restart.
# sol_10_02_09.sh

TYPE=application
s a
TARGET=ONLINE
)h a
ฺbr
c m
o deฺ
NAME=ora.RDB.RDB2.inst i ฺ
d Gui
TYPE=application r e
ic ent
TARGET=ONLINE
@ s
r d e Stud
NAME=ora.RDB.db i l a ve this
TYPE=application
i o _v use
TARGET=ONLINE
a b ric e to
(f ens
e
e r le lic
d
l a V ab
i
V sfer
TYPE=application
i o an on vx0308
TARGET=ONLINE
icSTATE=ONLINE
b r - t r
Fa non NAME=ora.vx0308.LISTENER_VX0308.lsnr
TYPE=application
TARGET=ONLINE
NAME=ora.vx0308.gsd
TYPE=application
TARGET=ONLINE
NAME=ora.vx0308.ons
TYPE=application
TARGET=ONLINE
NAME=ora.vx0308.vip
TYPE=application
TARGET=ONLINE

TYPE=application
TARGET=ONLINE
TYPE=application
TARGET=ONLINE
NAME=ora.vx0309.gsd
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vx0309 s a
)h a
NAME=ora.vx0309.ons
m ฺbr
TYPE=application
i ฺ c o deฺ
TARGET=ONLINE
r e d Gui
@ sic dent
NAME=ora.vx0309.vip
e r de Stu
TYPE=application
v i lav e this
TARGET=ONLINE
STATE=ONLINE on vx0309 cio
_ us
b i
r e to
a
(f ens
d e
r lwill l ibec the last things that are started and may take several
10) The database V abe
instances e
minutes i l a
V bothsso.
to do erWhat could be the cause of that delay? Because it may take too long
i o f
b r ic -tran
to restart instances, you can manually start them if needed.
Fa na)onTake a look at the database alert log on your first node and see what is happening.
The delay, of course is caused by instance recovery. Remember, that the problem
we introduced at the beginning of the practice caused the Oracle Clusterware
stack to crash and reboot the node and this of course crashed the database also.
#!/bin/bash
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"

I2NAME=$DBNAME"2"
cat /u01/app/diag/rdbms/rdb*/RDB*/trace/alert*
sleep 100
[oracle@vx0308 less10]$ ./sol_10_02_10.sh s a

)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

IMPORTANT NOTE: Before you start the following labs, make sure VNC is started on
both nodes as user oracle on port 5802.
You execute your commands under VNC terminal sessions alternatively on the first and
second node.
Before you start the following labs, execute the sol_11_00_00.sh script located in
your /home/oracle/solutions/less11 directory. You should execute this script under
the VNC terminal session started on the first node as user oracle.

s a
)h a
r
80ฺb
displayenv.sh 100%
c m
o deฺ
0.1KB/s 00:00 i ฺ
d Gui
vx0306 being added to access control list r e
@ sic dent
e r de Stu
i lavnode t is cluster, and add it back
hyour
v
The goal of this practice is to remove the second
o_ o us e of
again using DBCA and OUI. ic i
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-1: Remove the Second Instance
The goal of this practice is to remove the Database instance that exists on the second
node of your cluster. You use dbca for that task.
1) Connected as user oracle on the first node, use DBCA to remove the second instance
of your cluster.
#!/bin/bash
#
# sol_11_01_01.sh
#
#
s a
)h a
m ฺ br
cd $ORACLE_HOME/bin
i ฺ c o deฺ
r e d Gui
./dbca
e r de Stu
a) On the Welcome screen select Oracle v i laReal e t his Clusters Database and
v Application
click Next.
ic i o_ o us
f
b) On the Operations screen,
( abrselect s e t Management, and click Next.
Instance
c) On the Instancer d eManagement
l i c enscreen, select Delete an Instance, and click Next.
V e ble
a a databases screen, select your Database Name. Then enter
l Listfeofrcluster
d) Onithe
V
i c o
i ran
sys in the sUsername field and oracle1 in the password field. Click Next.
r -t the list of cluster database instances screen, select the instance from second
Fab ne)onnode.
On
It should be seen as an active instance. Click Next.
f) When you are finished, on the same screen, click Finish.
g) In the Database Configuration Assistant dialog box that appears, confirm that the
operation is to delete the second instance on the second node and click OK.
h) In the second Database Configuration Assistant dialog box that appears, click OK
to confirm that you want to proceed.
i) Wait until the progress bar goes to 100%. On the Database Configuration
Assistant dialog box that appears, click No to confirm that you do not want to
perform any other operation.

Practice 11-2: Clean Up ASM
The goal of this practice is to remove the ASM instance from the second node of your
cluster. You use srvctl for that task.
1) Connected as user oracle from the first node, stop and remove the ASM instance
running on your second node by using srvctl. When you are finished, check that the
corresponding resource has been removed from your Oracle Clusterware
configuration.
#!/bin/bash
#
# sol_11_02_01.sh
#
s a
#
)h a
m ฺbr
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'` diฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` co ideฺ
i c re t Gu
@ s
grep |de n -v
DBNAME=`ps -ef | grep dbw0_RDB | grep -v
callout1 | awk '{ print $8 }' | sedrd e Stu| sed grep
e 's/1/''/'
's/ora_dbw0_/''/'`
v i lav e this
ic i o_ o us
br se t
I1NAME=$DBNAME"1"
( f a
I2NAME=$DBNAME"2"
r d e licen
V e ble stop asm -n $z
a
il fera
/u01/crs11g/bin/srvctl
V
r i c i rans
o
/u01/crs11g/bin/srvctl remove asm -n $z
b - t
Fa n/u01/crs11g/bin/crs_stat
on -t

------------------------------------------------------------

Practice 11-2: Clean Up ASM (continued)
2) Remove the initialization parameter file of that ASM instance on the second node.
#!/bin/bash
#
# sol_11_02_02.sh
#
#


s a
's/ora_dbw0_/''/'`
)h a
m ฺbr
I1NAME=$DBNAME"1"
i ฺ c o deฺ
r e d Gui
sic dent
I2NAME=$DBNAME"2"
@
de Stu
ssh $z rm -f /u01/app/oracle/product/11.1.0/asm_1/dbs/*ASM*
e r
lav e this
v i
ic i o_ o us
( f
[oracle@vx0306 less11]$abr n./sol_11_02_02.sh
s et
e lice
3) Remove theV erddirectories
ASM
b l e from the second node.
V a
il feraless11]$ cat sol_11_02_03.sh
r i c i rans
[oracle@vx0306
o
#!/bin/bash
n-t
Fab n##osol_11_02_03.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
ssh $z rm -rf /u01/app/oracle/product/11.1.0/asm_1/admin/+ASM

Practice 11-2: Clean Up ASM (continued)
ssh $z rm -rf /u01/app/oracle/admin/+ASM

4) Last thing you can do is to remove the associated ASM entry from the /etc/oratab file
on the second node.
#!/bin/bash
#
# sol_11_02_04.sh
#
s a
#
)h a
m ฺbr
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` i ฺ c o deฺ
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'`red u i
s ic ent G
@
DBNAME=`ps -ef | grep dbw0_RDB | grep e-v grepu|d grep -v
t | sed
d's/1/''/'
e r S
lav e this
callout1 | awk '{ print $8 }' | sed
's/ora_dbw0_/''/'`
v i
ic i o_ o us
abr nse t
I1NAME=$DBNAME"1"
( f
rde le lice
I2NAME=$DBNAME"2"
e
l a V ab /etc/oratab | grep -v +ASM2 >
i
V sfer
ssh $z sudo "cat
i o
ic -tran
/home/oracle/solutions/less11/neworatab"
b r
Fa nssh on $z sudo cp /etc/oratab /etc/jfvbakoratab
ssh $z sudo rm /etc/oratab
ssh $z sudo cp /home/oracle/solutions/less11/neworatab

/etc/oratab
ssh $z sudo chmod 664 /etc/oratab
ssh $z sudo chown oracle /etc/oratab


Practice 11-3: Remove the Listener
The goal of this practice is to use netca to remove the listener from the second node of
your cluster.
1) You can now remove the listener from the node you want to delete. This listener can
be from either the ASM home or the Database home depending when it was created.
However, we suppose that you created it from your ASM instance in this practice. To
remove the listener, you can use NETCA.
#!/bin/bash
#
# sol_11_03_01.sh
#
s a
#
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic ||degrep
nt -v
de Stu
callout1 | awk '{ print $8 }' | sed 's/1/''/' sed
's/ora_dbw0_/''/'`
e r
v i lav e this
I1NAME=$DBNAME"1"
ic i o_ o us
I2NAME=$DBNAME"2" ab
r et
e ( f ns
d
r le li c e
e
scp /home/oracle/solutions/less11/runnetca.sh
V
ila ferab
$z:/home/oracle/solutions/less11/runnetca.sh
V
i c n s 777 /home/oracle/solutions/less11/runnetca.sh
io $zrachmod
r ssh
t
Fab nssh on-$z /home/oracle/solutions/less11/runnetca.sh
[oracle@vx0306 less11]$ cat runnetca.sh

#!/bin/bash
#
# runnetca.sh
#
#

cd $ORACLE_HOME/bin

Practice 11-3: Remove the Listener (continued)
export DISPLAY=$y:2.0
./netca

…
a) On the Configuration screen, select Cluster configuration and click Next.

b) On the Active Nodes screen select only the second node from which you want to
remove the listener and click Next.
c) On the Welcome screen, select Listener configuration and click Next. s a
)h a
d) On the Listener screen, select Delete and click Next.
m ฺbr
e) On the Select listener screen, select the corresponding listener, normally
i ฺ c o calledd e ฺ
LISTENER, and click Next.
red t Gu i
i c
f) On the Oracle Net Configuration Assistant dialog boxsthat appears, n confirm that
@ d e
you want to remove LISTENER by clickingrYes.
e de Stu
lav see
g) On the Listener Deleted screen, you ishould
v e t hisLISTENER was successfully
that
deleted, and click Next.
ic i o_ o us
( f abr nDone
h) On the Listener Configuration s e tscreen, you should see that listener
r isenowlecomplete,
configuration d
e l ice and click Next.
i) On the
l a V abscreen, click Finish.
Welcome
i er and the instance on the second node are no longer part of the
Vsure thesflistener
2) Make
i o an
icOracle-trClusterware
a b r configuration.
F o n
n[oracle@vx0306 less11]$ cat sol_11_03_02.sh
#!/bin/bash
#
# sol_11_03_02.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

Practice 11-3: Remove the Listener (continued)

------------------------------------------------------------
ora.vx0313.gsd application ONLINE ONLINE vx0313 s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-4: Remove the Database Software from the Second
Node
The goal of this practice is to remove the Database software installation from the second
node of your cluster only.
1) On the second node, and connected as user oracle in a VNC session, make sure you
have your ORACLE_HOME set to /u01/app/oracle/product/11.1.0/db_1.
2) On the second node, and connected as user oracle, change your current directory to
$ORACLE_HOME/oui/bin and execute the following command:
./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME
"CLUSTER_NODES=<second node name>" -local

#!/bin/bash
#
s a
# sol_11_04_02.sh
)h a
#
m ฺbr
i ฺ c o deฺ
#
r e d Gui
@ sic dent
z=`cat /home/oracle/nodeinfo | sed -n
e r de'2,2p'`
S tu
DBNAME=`ps -ef | grep dbw0_RDB v i la|vgrep
e t hisgrep | grep -v
-v
c
callout1 | awk '{ print $8
i i o_}' o| sed
us 's/1/''/' | sed
's/ora_dbw0_/''/'` br t
e (fa ense
rd le lic
I1NAME=$DBNAME"1"
e
l a V ab
i
V sfer
I2NAME=$DBNAME"2"
i o an
icscp -/home/oracle/solutions/less11/removedb.sh
b r t r
Fa n$z:/home/oracle/solutions/less11/removedb.sh
on
ssh $z chmod 777 /home/oracle/solutions/less11/removedb.sh
ssh $z "/home/oracle/solutions/less11/removedb.sh $z"

[oracle@vx0306 less11]$ cat removedb.sh

#!/bin/bash
#
# removedb.sh
#
#


Node (continued)
$ORACLE_HOME/oui/bin/runInstaller -updateNodeList
ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=$1" -local

removedb.sh 100% 345 0.3KB/s
00/00
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
s a
‘UpdateNodeList’ was successful.
)h a
ฺbr
Starting Oracle Universal Installer…
c m
o deฺ
Checking swap space: … Passed
i ฺ
d Gui
Checking Monitor: … Passed
r e
@ sic dent
de to remove
3) On the second node, use OUI from the databaserhome
e S tu the database
software.
v i lav e this
[oracle@vx0306 less11]$ cat
ic i o_ sol_11_04_03.sh
o us
#!/bin/bash r
ab nse t
# ( f
# sol_11_04_03.sh
e r de lice
#
l a V able
V
# Musti be
f e r
executed on NODE1 !!!!!!!!!!
o
ic#i -tran s
b r
Fa ny=`cat

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
scp /home/oracle/solutions/less11/removedb2.sh
$z:/home/oracle/solutions/less11/removedb2.sh
ssh $z chmod 777 /home/oracle/solutions/less11/removedb2.sh
ssh $z /home/oracle/solutions/less11/removedb2.sh

Node (continued)
[oracle@vx0306 less11]$ cat removedb2.sh
#!/bin/bash
#
# removedb2.sh
#
#

s a
)h a
$ORACLE_HOME/oui/bin/runInstaller
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
…
v i lav e this
i _ Products.
oDeinstall us
a) On the Welcome screen, click
r ic t o
( f
b) On the Inventory window, ab select
n seyour database home name (should be
e e
c Remove.
rd leandliclick
OraDb11g_home1),
e
V ab window, where you can see the list of products to be
l
c) Onithe a
V sclick
Confirmation
er Yes.
i o removed, f
b r icd) On-trtheanWarning window that appears, click Yes to remove the ORACLE_HOME
Fa nonat the end.
e) Wait for the Deinstall progress bar to complete (100%).
f) On the Inventory window, you should now see that your database home has been
removed. On the Inventory window, click Close.
g) On the Welcome screen, click Cancel.
h) On the Exit dialog box, click Yes to exit OUI.
4) On the first node, make sure you export your ORACLE_HOME environment variable
set to /u01/app/oracle/product/11.1.0/db_1, and change your current directory to
$ORACLE_HOME/oui/bin. Then execute the following command:
"CLUSTER_NODES=<first node name>"

#!/bin/bash
#
# sol_11_04_04.sh

Node (continued)
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
s a
)h a
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
@ sic dent
de Stu
ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=$y"
e r
v i lav e this
i o_ o us
[oracle@vx0306 less11]$ic./sol_11_04_04.sh
(
Starting Oracle Universalf abr nsInstaller…
et
e lice
rdspace:
Checking swap
V e b l e… … Passed
Checking a
il ferapointer isPassed
Monitor:
The V s
i c i o inventory
inventory
a n
located at /etc/oraInst.loc
b r The
- t r is located at /u01/app/oraInventory
Fa n[oracle@vx0306
on
‘UpdateNodeList’ was
less11]$
successful.

Practice 11-5: Remove the ASM Software from the Second Node
The goal of this practice is to remove the ASM software installation from the second
node of your cluster only.
1) On the second node, and connected as user oracle in a VNC session, make sure you
have your ORACLE_HOME set to /u01/app/oracle/product/11.1.0/asm_1.
2) On the second node, and connected as user oracle, change your current directory to
$ORACLE_HOME/oui/bin and execute the following command:
"CLUSTER_NODES=<second node name>" -local

#!/bin/bash
#
# sol_11_05_02.sh
s a
#
)h a
# m ฺbr
i ฺ c o deฺ
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'`red u i
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'` sic dent G
@
de-v greptu| grep -v
e r S
callout1 | awk '{ print $8 }' il|av his | sed
v e t
sed 's/1/''/'
's/ora_dbw0_/''/'`
ic i o_ o us
I1NAME=$DBNAME"1"(fab
r et
d e c e ns
V er ble li
I2NAME=$DBNAME"2"
V i la fera
i cscp
a s
io /home/oracle/solutions/less11/removeasm.sh
n
r t r
Fab non-
$z:/home/oracle/solutions/less11/removeasm.sh
ssh $z chmod 777 /home/oracle/solutions/less11/removeasm.sh
ssh $z "/home/oracle/solutions/less11/removeasm.sh $z"

[oracle@vx0306 less11]$ cat removeasm.sh

#!/bin/bash
#
# removeasm.sh
#
#


(continued)
ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=$1" -local

removeasm.sh 100% 347 0.3KB/s
00/00
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
3) On the second node, use OUI from the ASM home to@ sic thedASM
remove e nt software.
e r de Stu
lav e this
#!/bin/bash
v i
#
ic i o_ o us
# sol_11_05_03.sh
# ( f abr nse t
# Must be executed
e r de onlicNODE1
e !!!!!!!!!!
#
l a V able
i er
V /home/oracle/nodeinfo
f
i c i o
y=`cat
a n s | sed -n '1,1p'`
a b r z=`cat- t r /home/oracle/nodeinfo | sed -n '2,2p'`
F o n
nDBNAME=`ps -ef | grep dbw0_RDB | grep -v grep | grep -v
's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
scp /home/oracle/solutions/less11/removeasm2.sh
$z:/home/oracle/solutions/less11/removeasm2.sh
ssh $z chmod 777 /home/oracle/solutions/less11/removeasm2.sh
ssh $z /home/oracle/solutions/less11/removeasm2.sh
[oracle@vx0306 less11]$ cat removeasm2.sh

#!/bin/bash

(continued)
#
# removeasm2.sh
#
#

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
…
@ sic dent
a) On the Welcome screen, click Deinstall Products.
e r de Stu
b) On the Inventory window, select your i v home
laASM t s (should be
hiname
v
o_Remove. e
OraASM11g_home1), and click
r ic i
t o us
c) On the Confirmationfa
( b where
window, se you can see the list of products to be
e
removed, clickdYes. e n
e r le lic
d) On the
l a V awindow
Warning b that appears, click Yes to remove the ORACLE_HOME
i r
i c i o atVtheaend.
n sfe
a b r e) Wait - tr for the Deinstall progress bar to complete (100%).
F n
nf)o In the Inventory window, you should now see that your ASM home has been
removed. In the Inventory window, click Close.
g) On the Welcome screen, click Cancel.
h) In the Exit dialog box, click Yes to exit OUI.
4) On the first node, make sure you export your ORACLE_HOME environment variable
set to /u01/app/oracle/product/11.1.0/asm_1, and change your current directory to
$ORACLE_HOME/oui/bin. Then execute the following command:
"CLUSTER_NODES=<first node name>"

#!/bin/bash
#
# sol_11_05_04.sh
#
#

(continued)

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
s a
)h a
m ฺbr
ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=$y"
i ฺ c o deฺ
r e d Gui
@ sic dent
e r
[oracle@vx0306 less11]$ ./sol_11_05_04.shde Stu
i lav e this
v
Checking swap space: …ric io_ o us
Passed
Checking Monitor: f…ab Passede t
( s
enlocated at /etc/oraInst.loc
e licis
r d
The inventory pointer
e isbllocated
V
The inventory
a e successful.
at /u01/app/oraInventory
V il feraless11]$
r i c i rans
o
[oracle@vx0306
t
Fab non-

Practice 11-6: Remove the Second Node from the OCR
The goal of this practice is to remove the second node ONS configuration from the OCR.
1) On the first node, use the racgons command tool to remove the configuration of the
ONS of the second node from the OCR.
#!/bin/bash
#
# sol_11_06_01.sh
#
#

s a
)h a
m ฺbr
i ฺ c o deฺ
's/ora_dbw0_/''/'`
r e d Gui
I1NAME=$DBNAME"1"
@ sic dent
e r de Stu
I2NAME=$DBNAME"2"
v i lav e this
ic i o_ o us
cat /u01/crs11g/opmn/conf/ons.config
( f abr nremove_config
s et
d e
/u01/crs11g/bin/racgons
li c e $z
V er less11]$
[oracle@vx0306
l e
V ila ferab
r i c io rans
t
Fab nlocalport=6150
on-
useocr=on
allowgroup=true
usesharedinstall=true
racgons: host vx0313 is deleted from ONS configuration.

Practice 11-7: Remove the Oracle Clusterware Software from the
Second Node
The goal of this practice is to remove the Oracle Clusterware software installation from
the second node of your cluster.
1) From the second node as user root, execute the following command:
<Oracle Clusterware home>/install/rootdelete.sh

#!/bin/bash
#
# sol_11_07_01.sh
#
#
s a
m ฺbr
s i cre| sedn t G
's/ora_dbw0_/''/'`
r d e@ Stude
I1NAME=$DBNAME"1"
i l a ve this
i o _v use
I2NAME=$DBNAME"2"
a b ric e to
e (f ens
ssh $z sudo /u01/crs11g/install/rootdelete.sh
[oracle@vx0306e d
r less11]$
e l ic
l a V ab l
i
V sfer
i o
icGetting a n
b r [oracle@vx0306
t r less11]$ ./sol_11_07_01.sh
Fa nNODEon- = vx0313local node name
Getting local node name

NODE = vx0313
Stopping resources.
Waiting for Cluster Synchronization Services daemon to stop
Cluster Synchronization Services daemon has stopped
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
Cleaning up Network socket directories

Second Node (continued)
2) From the first node as the root user, execute the following command:
<Oracle Clusterware home>/install/rootdeletenode.sh <node name to be
deleted>,<node number to be deleted>

#!/bin/bash
#
# sol_11_07_02.sh
#
#

s a
)h a
m ฺbr
's/ora_dbw0_/''/'`
i ฺ c o deฺ
r e d Gui
sic dent
I1NAME=$DBNAME"1"
@
de Stu
I2NAME=$DBNAME"2"
e r
v this
/u01/crs11g/bin/olsnodes -n vila
i o _ use
b ric e to
sudo /u01/crs11g/install/rootdeletenode.sh
a $z,2
( f
e licen
[oracle@vx0306 less11]$ s
r d
e ble
a V
V il feraless11]$ ./sol_11_07_02.sh
[oracle@vx0306
r i c i ran1 s
o
vx0306
t 2
Fab nCRS-0210:
on-
vx0313
Could not find resource 'ora.vx0313.ons'.
CRS-0210: Could not find resource 'ora.vx0313.vip'.
CRS-0210: Could not find resource 'ora.vx0313.gsd'.
PRKO-2112 : Some or all node applications are not removed
successfully on node: vx0313
CRS-0210: Could not find resource 'ora.vx0313.vip'.CRS-0210:
Could not find resource 'ora.vx0313.ons'.CRS-0210: Could not
find resource 'ora.vx0313.gsd'.
CRS nodeapps are deleted successfully
Value SYSTEM.crs.versions.vx0313 marked for deletion is not
there. Ignoring.
Successfully deleted 15 values from OCR.
Key SYSTEM.css.interfaces.nodevx0313 marked for deletion is
not there. Ignoring.
Key SYSTEM.crs.versions.vx0313 marked for deletion is not
there. Ignoring.
Successfully deleted 13 keys from OCR.

Node deletion operation successful.
'vx0313,2' deleted successfully
3) On the second node, connected as user root, make sure you export the
ORACLE_HOME environment variable set to /u01/crs11g. Then execute the
following command:
/u01/crs11g/oui/bin/runInstaller -updateNodeList
ORACLE_HOME=/u01/crs11g "CLUSTER_NODES=<second node name>" CRS=TRUE
-local

#!/bin/bash
#
# sol_11_07_03.sh
s a
#
)h a
m ฺbr
#
i ฺ c o deฺ
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'`sic dent G
@
de-v grep
tu| grep -v
e r S
v this | sed
callout1 | awk '{ print $8 }' il|a sed 's/1/''/'
o v
_ use
's/ora_dbw0_/''/'` i
ric e to
I1NAME=$DBNAME"1"(f a b
d e c e ns
V er ble li
I2NAME=$DBNAME"2"
V ila fera
i cscp
a s
io /home/oracle/solutions/less11/removecrs.sh
n
r t r
Fab non-
$z:/home/oracle/solutions/less11/removecrs.sh
ssh $z chmod 777 /home/oracle/solutions/less11/removecrs.sh
ssh $z "/home/oracle/solutions/less11/removecrs.sh $z"

[oracle@vx0306 less11]$ cat removecrs.sh

#!/bin/bash
#
# removecrs.sh
#
#


export ORACLE_HOME=/u01/crs11g
ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=$1" CRS=TRUE -local

removecrs.sh 100% 331 0.3KB/s
00/00
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
de Shome
4) On the second node, start OUI from the Oracle Clusterware
e r tu and deinstall
Oracle Clusterware software on the second
i v this
lanode.
o v e
_ sol_11_07_04.sh
s
[oracle@vx0306 less11]$ cat i
ric e to u
#!/bin/bash
# a b
(f ens
# sol_11_07_04.sh d e
r le lic
# e
V ab
V i l a e r
# Must
o
be
s f
executed on NODE1 !!!!!!!!!!
b r ic#i -tran
Fa ny=`cat

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
scp /home/oracle/solutions/less11/removecrs2.sh
$z:/home/oracle/solutions/less11/removecrs2.sh
ssh $z chmod 777 /home/oracle/solutions/less11/removecrs2.sh
ssh $z /home/oracle/solutions/less11/removecrs2.sh

[oracle@vx0306 less11]$ cat removecrs2.sh
#!/bin/bash
#
# removecrs2.sh
#
#

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
de Stu
…
e r
v
a) On the Welcome screen, click Deinstalli lavProducts
e t h. is
i o _ us
b) In the Inventory window,
b ic toOracle Clusterware software home name
rselect your
a
f clicknRemove.
(OraCrs11g_home)(and s e
d e c e
li you should see the list of products to deinstall. Click
V er blwindow,
c) In the confirmation e
Yes.
V ila fera
i c io In the
d) a n s dialog box, make sure you click No to avoid CRS directory to be
Warning
r r
-t at the end of this installation. You do it manually afterward.
Fab nonremoved
e) Wait for the progress bar to finish.
f) In the Inventory window, after deinstallation was successful, you should no
longer see the Oracle Clusterware software home. Click Close.
g) On the Welcome screen, click Cancel to finish this deinstallation.
h) In the Exit window, click Yes to exit from OUI.
5) On the first node as user root, execute the following command:
/u01/crs11g/oui/bin/runInstaller -updateNodeList
ORACLE_HOME=/u01/crs11g "CLUSTER_NODES=<first node name>" CRS=TRUE

#!/bin/bash
#
# sol_11_07_05.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
s a
)h a
m ฺbr
ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=$y" CRS=TRUE
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
Checking swap space: …ric io_ o us
Checking Monitor: f…ab Passed
Passed
e t
(
e licis s
enlocated at /etc/oraInst.loc
r d
The inventory pointer
e isbllocated
V
The inventory
a e successful.
at /u01/app/oraInventory
V il feraless11]$
r i c i rans
o
[oracle@vx0306
n-tyour Oracle Clusterware configuration. What do you observe?

Fab6) nCheck
o
a) You should not see any remaining resources on the second node. Also, make sure
your second node is no longer part of your cluster from a CRS perspective.
#!/bin/bash
#
# sol_11_07_06.sh
#
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"

I2NAME=$DBNAME"2"
/u01/crs11g/bin/olsnodes -n

------------------------------------------------------------
s a
)h a
ฺbr
c m
o deฺ
i ฺ
d Gui
r e
vx0306 1
@ sic dent
de Stu
7) Remove your Oracle Clusterware, ASM, and e r
v thishome directories from your
Database
i l a
second node.
i o _v use
to
#!/bin/bash
e (f ens
#
e r le lic
d
V ab
# sol_11_07_07.sh
# Vila er
#ioMust be s f
b r ic# -tran executed on NODE1 !!!!!!!!!!
Fa non

's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
ssh $z "sudo rm -rf /u01/crs11g"
ssh $z "sudo rm -rf /u01/app/oracle/product/11.1.0/asm_1"
ssh $z "sudo rm -rf /u01/app/oracle/product/11.1.0/db_1"


s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-8: Check Prerequisites Before Oracle Clusterware
Installation
The goal of this practice is to make sure your environment meets the expected
prerequisites before you install the Oracle Clusterware software on the second node
again.
1) Before you can proceed with the Oracle Clusterware installation on the node you
want to add to your RAC cluster, you need to make sure that all operating system and
hardware prerequisites are met. Use Cluster Verify to check Oracle Clusterware pre-
installation.
#!/bin/bash
#
# sol_11_08_01.sh
s a
#
)h a
m ฺbr
#
i ฺ c o deฺ
z=`cat /home/oracle/nodeinfo | sed -n '2,2p'`sic dent G
@
de-v grep
tu| grep -v
e r
v this | sed
DBNAME=`ps -ef | grep dbw0_RDB | grep S
callout1 | awk '{ print $8 }' il|a sed 's/1/''/'
o v
_ use
's/ora_dbw0_/''/'` i
ric e to
I1NAME=$DBNAME"1"(f a b
d e c e ns
V er ble li
I2NAME=$DBNAME"2"
V ila fera
r i c io rans less11]$ stage -pre crsinst -n $y,$z -r 11gR1
/u01/crs11g/bin/cluvfy
t
Fab non-
[oracle@vx0306
Performing pre-checks for cluster services setup
Checking node reachability...

Node reachability check passed from node "vx0306".
Checking user equivalence...

User equivalence check passed for user "oracle".
Checking administrative privileges...

User existence check passed for "oracle".
Membership check for user "oracle" in group "oinstall" [as
Primary] passed.

Installation (continued)
Administrative privileges check passed.
Checking node connectivity...
Node connectivity check passed for subnet "10.216.4.0" with

WARNING:
Make sure IP address "10.216.96.120" is up and is a valid IP
address on node "vx0313".
Node connectivity check failed for subnet "10.216.96.0".
Node connectivity check passed for subnet "10.196.28.0" with
s a
Interfaces found on subnet "10.216.4.0" that are likely
)h a
ฺbr
candidates for VIP:
vx0306 eth0:10.216.4.17 eth0:10.216.4.68
c m
o deฺ
vx0313 eth0:10.216.4.24
i ฺ
d Gui
r e
Interfaces found on subnet "10.196.28.0" that
candidates for a private interconnect: @
sic aree n t
likely
r d e Stud
ve this
vx0306 eth2:10.196.30.27
vx0313 eth2:10.196.30.34
i l a
i o _v use
Node connectivity check
a b ric e to
failed.
e (f ens
e d
e l ic
r requirements
a V check
Checking system
b l for 'crs'...
Total lmemory
V idisk space
f e ra check passed.
passed.
o
Free
i space
icSwap s
an check passed.
b r - t r
Fa nKernel
on architecture
System check passed.
version check passed.
Package existence check passed for "make-3.80".
Package existence check passed for "binutils-2.15.92.0.2".
Package existence check passed for "gcc-3.4.5".
Package existence check passed for "libaio-0.3.105".
Package existence check passed for "libaio-devel-0.3.105".
Package existence check passed for "libstdc++-3.4.5".
Package existence check passed for "elfutils-libelf-devel-
0.97".
Package existence check passed for "sysstat-5.0.5".
Package existence check passed for "libgcc-3.4.5".
Package existence check passed for "libstdc++-devel-3.4.5".
Package existence check passed for "unixODBC-2.2.11".
Package existence check passed for "unixODBC-devel-2.2.11".
Package existence check passed for "glibc-2.3.4-2.19".
Group existence check passed for "dba".
User existence check passed for "nobody".

Installation (continued)
System requirement passed for 'crs'
Pre-check for cluster services setup was unsuccessful on all

the nodes.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-9: Add Oracle Clusterware to the Second Node
The goal of this practice is to install the Oracle Clusterware software on the second node
of your cluster.
1) Login as the oracle user execute the addNode.sh script located in your Oracle
Clusterware home directory on the first node (/u01/crs11g/oui/bin). Make sure your
ORACLE_HOME is exported to /u01/crs11g. This script runs the Oracle Universal
Installer. Then add the Oracle Clusterware software to the new node.
#!/bin/bash
#
# sol_11_09_01.sh
#
s a
#
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
sic ||degrep
@ nt -v
de Stu
callout1 | awk '{ print $8 }' | sed 's/1/''/' sed
's/ora_dbw0_/''/'`
e r
v i lav e this
I1NAME=$DBNAME"1"
ic i o_ o us
I2NAME=$DBNAME"2" ab
r et
e ( f ns
d c e
er ble li
V
V ila fera
r i c io rans
cd $ORACLE_HOME/oui/bin
t
Fab nexport
on- DISPLAY=$y:2.0
./addNode.sh

…
a) On the Welcome screen, click Next.

b) On the Specify Cluster Nodes to Add to Installation screen, enter the public,
private, and virtual host name in the first row of the Specify New Nodes table (use
the same scheme as for the first node in the Existing Nodes table). Click Next.
c) On the Cluster Node Addition Summary screen, click Install.
d) The Cluster Node Addition Progress screen appears. Wait until OUI asks you to
run some scripts.
e) When the Execute Configuration scripts window appears, do the following:

(continued)
f) Execute /u01/crs11g/install/rootaddnode.sh on the first node as root user.
[oracle@vx0306 less11]$ cat sol_11_09_01f.sh
#!/bin/bash
#
# sol_11_09_01f.sh
#
#


callout1 | awk '{ print $8 }' | sed 's/1/''/' | sed s a
's/ora_dbw0_/''/'`
)h a
m ฺ br
I1NAME=$DBNAME"1"
i ฺ c o deฺ
r e d Gui
I2NAME=$DBNAME"2"
@ sic dent
sudo /u01/crs11g/install/rootaddnode.sh
e r de Stu
v i lav e this
ic i o_ o us
( f a br ./sol_11_09_01f.sh
s e t
clscfg: EXISTING
r d l i c enReleaseversion
e 4configuration 4 detected.
V eto add
clscfg: version
b
is 11 1.
le1 new nodes to the configuration
l
Attempting
i a
V ports: a
erCSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
Using
i o n s f
b r ic<hostname>
node
t a
<nodenumber>:
r <nodename> <private interconnect name>
Fa nnodeon- 2: vx0313 vx0313-priv vx0313

Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/u01/crs11g/bin/srvctl add nodeapps -n vx0313 -A vx0313-
vip/255.255.252.0/eth0
g) Execute /u01/crs11g/root.sh on the second node as root user.

[oracle@vx0306 less11]$ cat sol_11_09_01g.sh
#!/bin/bash
#
# sol_11_09_01g.sh
#
#


(continued)
's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"
ssh $z sudo /u01/crs11g/root.sh

[oracle@vx0306 less11]$ ./sol_11_09_01g.sh

s a
)h a
ฺbr
OCR LOCATIONS = /dev/sdb1,/dev/sdb2
OCR backup directory '/u01/crs11g/cdata/vx_cluster02' does not
c m
o deฺ
exist. Creating now
i ฺ
d Gui
r e
@ sic dent
de Stu
e r
v i lav e this
ic i o_ o us
abr nse t
( f
<hostname>
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa nNO onKEYSisWERE
-force
WRITTEN. Supply -force parameter to override.
destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been
initialized
vx0306
vx0313
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
h) When both scripts have been executed in the correct order, go back to the Execute
Configuration scripts screen and click OK.

(continued)
i) On the End of Installation screen, click Exit.
j) In the Exit dialog box, click Yes to exit the installer.
2) Look at the Oracle Clusterware resources, and check that you now have nodeaps
running on the second node.
#!/bin/bash
#
# sol_11_09_02.sh
#
#
s a
)h a
ฺbr
c m
o-v deฺ
DBNAME=`ps -ef | grep dbw0_RDB | grep -v grep | grep
callout1 | awk '{ print $8 }' | sed 's/1/''/' e i
| dsed
ฺ u i
r G
's/ora_dbw0_/''/'`
@ sic dent
I1NAME=$DBNAME"1"
e r de Stu
v i lav e this
I2NAME=$DBNAME"2"
ic i o_ o us
( f abr n-t
/u01/crs11g/bin/crs_stat s et
[oracle@vx0306 eless11]$e
e rd le lic
l a V ab
i
V sfer less11]$ ./sol_11_09_02.sh
i o
[oracle@vx0306
icName-tran
b r Type Target State Host
Fa non ------------------------------------------------------------

Practice 11-10: Configure ONS for the Second Node
The goal of this practice is to configure the OCR to include the second node ONS
existence.
1) Use racgons add_config command to add second node ONS configuration
information to the OCR.
#!/bin/bash
#
# sol_11_10_01.sh
#
#
y=`cat /home/oracle/nodeinfo | sed -n '1,1p'` s a

)h a
m ฺ br
DBNAME=`ps -ef | grep dbw0_RDB | grep -v grep | grepo-v
callout1 | awk '{ print $8 }' | sed 's/1/''/' | dsed i ฺc uideฺ
's/ora_dbw0_/''/'`
s icre ent G
I1NAME=$DBNAME"1" r d e@ Stud
i l a ve this
I2NAME=$DBNAME"2"
i o _v use
a
/u01/crs11g/bin/racgonsb ricadd_config
e to $z:6251
e (f ens
e r le lic
d
l a V ab
i
V sfer less11]$
[oracle@vx0306 ./sol_11_10_01.sh
i o
ic -tran
b r
Fa non

Practice 11-11: Add ASM Software to the Second Node
The goal of this practice is to install the ASM software on the second node of your
cluster.
1) Login as the Oracle user execute the addNode.sh script located in your ASM home
directory on the first node. This script runs the Oracle Universal Installer.
#!/bin/bash
#
# sol_11_11_01.sh
#
#

)h a
m ฺ br
's/ora_dbw0_/''/'`
s icre ent G
i l a ve this
I2NAME=$DBNAME"2"
i o _v use
a b ric e to
e (f ens
e r le lic
d
l a V ab
export i
V sfer DISPLAY=$y:2.0
i o an
ic./addNode.sh
b r - t r
Fa n[oracle@vx0306
on less11]$

…
b) On the Specify Cluster Nodes to Add to Installation screen, check the name of
your second node in the Specify New Nodes table. Click Next.
run some scripts.
f) Execute /u01/app/oracle/product/11.1.0/asm_1/root.sh on the second node as root
user. Just type "return" for all questions ask (default values).

(continued)
#!/bin/bash
#
# sol_11_11_01f.sh
#
#


's/ora_dbw0_/''/'`
s a
)h a
ฺbr
I1NAME=$DBNAME"1"
c m
o deฺ
I2NAME=$DBNAME"2"
i ฺ
d Gui
r e
sic dent
ssh $z sudo /u01/app/oracle/product/11.1.0/asm_1/root.sh
@
e r de Stu
v i l av this
i o _ use
[oracle@vx0306 less11]$ ./sol_11_11_01f.sh
Running Oracle 11g root.sh
a b ric escript...
to
e (f ensvariables are set as:
The followingdenvironment
e ic
r leoracle
l
l a V ab /u01/app/oracle/product/11.1.0/asm_1
ORACLE_OWNER=
i
V sfer
ORACLE_HOME=
i o an full pathname of the local bin directory:
icEnter-trthe
b r
Fa n[/usr/local/bin]:
on
The file "dbhome" already exists in /usr/local/bin. Overwrite
it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite
it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]:

g) When the script has been executed, go back to the Execute Configuration scripts
window and click OK.

(continued)
h) On the End of Installation screen, click Exit.
i) In the Exit dialog box, click Yes to exit the installer.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-12: Add a Listener to the Second Node
The goal of this practice is to configure and start a new listener on the second node of
your cluster.
1) Connected as user oracle on the second node, execute netca from the ASM home you
just installed. Then, using netca, add a new standard listener to the second node.
#!/bin/bash
#
# sol_11_12_01.sh
#
#

)h a
m ฺ br
's/ora_dbw0_/''/'`
s icre ent G
i l a ve this
I2NAME=$DBNAME"2"
i o _v use
a b ric e to
scp /home/oracle/solutions/less11/runnetcaasm.sh
e (f ens
$z:/home/oracle/solutions/less11/runnetcaasm.sh
e r le lic
d
l
ssh $z chmod
a V 777 a b /home/oracle/solutions/less11/runnetcaasm.sh
i r
i c i
ssh n sfe
o V$z a"/home/oracle/solutions/less11/runnetcaasm.sh"
a b r [oracle@vx0306
- tr
F n o n less11]$
[oracle@vx0306 less11]$ cat runnetcaasm.sh

#!/bin/bash
#
# runnetcaasm.sh
#
#
cd $ORACLE_HOME/bin
./netca

Practice 11-12: Add a Listener to the Second Node (continued)

…
a) On the Configuration screen, select Cluster configuration and click Next.
b) On the Active Nodes screen, select ONLY the second node (the one you want to
add) and click Next.
c) On the Welcome screen, select Listener configuration and click Next.
d) On the Listener screen, select Add and click Next.
e) On the Listener Name screen, enter LISTENER (should be the default value
already typed) Click Next.
s a
f) On the Select Protocols screen, select only TCP (should be the default) and click
)h a
Next.
m ฺbr
g) On the TCP/IP Protocol screen, select Use the standard port number i ฺ c oof 1521d e ฺ
and
click Next. red t Gu i
s i c n
h) On the More Listeners screen, select No, and click@ Next. de
e r deshouldSseetu
i) On the Listener Configuration Done screen,
i l a v this
you a Listener
o v se
configuration complete message._Click Next.
j) On the Welcome screen, clickr i
ic Finish.to u
b
(fa enslistener e
2) Check that the Oracle
r d e Clusterware
l i c resource is created on the second node.
[oracle@vx0306V e less11]$
b le cat sol_11_12_02.sh
i l
V sfer
#!/bin/bash a a
#io
b r - t r an
ic# sol_11_12_02.sh
Fa n#on
#


's/ora_dbw0_/''/'`
I1NAME=$DBNAME"1"
I2NAME=$DBNAME"2"

Practice 11-12: Add a Listener to the Second Node (continued)
------------------------------------------------------------
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-13: Add Database Software to the Second Node
The goal of this practice is to install the Database software on the second node of your
cluster.
1) Login as the oracle user execute the addNode.sh script located in your database home
directory on the first node. This script runs the Oracle Universal Installer. Add the
database software to your second node.
#!/bin/bash
#
# sol_11_13_01.sh
#
#
s a
m ฺbr
s i cre| sedn t G
's/ora_dbw0_/''/'`
r d e@ Stude
I1NAME=$DBNAME"1"
i l a ve this
i o _v use
I2NAME=$DBNAME"2"
a b ric e to
e (f ens
e r le lic
d
l a V ab
i
V sfer
i o
icexport n
aDISPLAY=$y:2.0
b r - t r
Fa n./addNode.sh
on

…

b) On the “Specify Cluster Nodes to Add to Installation” screen, check the name of
your second node in the Specify New Nodes table. Click Next.
run some scripts.
f) Execute /u01/app/oracle/product/11.1.0/db_1/root.sh on the second node as root
user. Just type "return" for all questions ask (default values).

(continued)
#!/bin/bash
#
# sol_11_13_01f.sh
#
#


's/ora_dbw0_/''/'`
s a
)h a
ฺbr
I1NAME=$DBNAME"1"
c m
o deฺ
I2NAME=$DBNAME"2"
i ฺ
d Gui
r e
sic dent
ssh $z sudo /u01/app/oracle/product/11.1.0/db_1/root.sh
@
e r de Stu
v i l av this
i o _ use
[oracle@vx0306 less11]$ ./sol_11_13_01f.sh
Running Oracle 11g root.sh
a b ric escript...
to
e (f ensvariables are set as:
The followingdenvironment
e ic
r leoracle
l
l a V ab /u01/app/oracle/product/11.1.0/db_1
ORACLE_OWNER=
i
V sfer
ORACLE_HOME=
i o an full pathname of the local bin directory:
icEnter-trthe
b r
Fa n[/usr/local/bin]:
on
it? (y/n)
[n]:
it? (y/n)
[n]:
Overwrite it? (y/n)
[n]:

g) When the script has been executed, go back to the Execute Configuration scripts
window and click OK.

(continued)
h) On the End of Installation screen, click Exit.
i) In the Exit dialog box, click Yes to exit the installer.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Practice 11-14: Add a Database Instance to the Second Node
The goal of this practice is to add a new instance to your RAC database on the second
node using both DBCA and Enterprise Manager.
1) Use DBCA to add an ASM instance to your second node.
#!/bin/bash
#
# sol_11_14_01.sh
#
#

s a
)h a
m ฺbr
i ฺ c o deฺ
's/ora_dbw0_/''/'`
r e d Gui
I1NAME=$DBNAME"1"
@ sic dent
e r de Stu
I2NAME=$DBNAME"2"
v i lav e this
ic i o_ o us
( f abr nse t
rde le lice
cd $ORACLE_HOME/bin
./dbca V
e
V ila ferab
r i c io rans less11]$
[oracle@vx0306
t
Fab non-
…
a) On the Welcome screen, select Oracle Real Application Clusters database and
click Next.
b) On the Operations screen, select Configure Automatic Storage Management and
click Next.
c) On the Nodes Selection screen, select both nodes, and click Next.
d) In the Database Configuration Assistant dialog box, click Yes to extend your
ASM cluster to the second node.
e) In the ASM Credentials dialog box, enter oracle1 and click OK.
f) On the ASM Disk Groups screen, click Finish.
g) In the Database Configuration Assistant dialog box, click No to exit from DBCA.

(continued)
2) Start your Enterprise Manager Database Control console.
#!/bin/bash
#
# sol_11_14_01.sh
#
#


callout1 | awk '{ print $8 }' | sed 's/1/''/' | sed s a
's/ora_dbw0_/''/'`
)h a
m ฺ br
I1NAME=$DBNAME"1"
i ฺ c o deฺ
r e d Gui
I2NAME=$DBNAME"2"
@ sic dent
r de Stu
e
v i lav e this
o_ o us
ic i
cd $ORACLE_HOME/binabr e t
( f
e licen s
./emctl start r d
e ble
dbconsole
a V
V il feraless11]$
[oracle@vx0306
r i c i rans
o
o n -t
Fab n[oracle@vx0306 less11]$ ./sol_11_14_02.sh
Oracle Enterprise Manager 11g Database Control Release
11.1.0.6.0
Copyright (c) 1996, 2007 Oracle Corporation. All rights
reserved.
https://vx0306.us.oracle.com:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control ......
started.
--------------------------------------------------------------
----
Logs are generated in directory
/u01/app/oracle/product/11.1.0/db_1/vx0306_RDB1/sysman/log
3) Use Enterprise Manager Database Control to add an instance to your existing
database.
a) Open a browser window and connect as SYS to your database control console.
b) When on the Cluster Database Home page, click the Server tab.

(continued)
c) On the Server page, click the Add instance link in the Change Database section.
d) On the Add Instance: Cluster Credentials page, enter both Cluster credentials
(oracle/oracle) and ASM credentials (sys/oracle1). Make sure that you check the
Save as Preferred Credentials check box. When you are finished, click Next.
e) On the Add Instance: Host page, select the second node of your cluster and click
Next.
f) On the Add Instance: Review page, click the Submit Job button.
g) On the Confirmation page, click the View Job button.
h) When on the Execution page, refresh you browser page until the job is finished.
4) At this point check that all resources are created and active on both nodes.
s a
)h a
#!/bin/bash
m ฺbr
#
i ฺ c o deฺ
# sol_11_14_02.sh
r e d Gui
sic dent
#
@
de Stu
#
e r
y=`cat /home/oracle/nodeinfov| i e his
v -n t'1,1p'`
lased
ic i o_ |osedus-n '2,2p'`
r
ab dbw0_RDB t
e f
DBNAME=`ps -ef | (grep
e n se | grep -v grep | grep -v
e
callout1 | awk l ic $8 }' | sed 's/1/''/' | sed
rd '{leprint
a V ab
's/ora_dbw0_/''/'`
l
i
V sfer
i o
ic -tran
I1NAME=$DBNAME"1"
b r
on

------------------------------------------------------------

(continued)
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

There are no practices for this lesson.
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non

Optional Practice: Direct NFS
In this practice, you use Direct NFS to create a new RAC database that resides on NFS
storage.
1) Before you can create the database, you need to install a new Oracle Home on your
system because Direct NFS requires a special Oracle Disk Manager. Use Oracle
Universal Installer to install a new Oracle Home in
/u01/app/oracle/product/11.1.0/dnfs.
a) Start Oracle Universal Installer:
[oracle@vx0308 opt]$ cat sol_opt_01a.sh
/stage/db/runInstaller
[oracle@vx0308 opt]$
s a
[oracle@vx0308 opt]$ ./ sol_opt_01a.sh
)h a
…
m ฺbr
b) On the Select a Product to Install screen, select Oracle Database ฺ11g
i c oand click
d e ฺ
Next. red t Gu i
i c
s Edition nand click Next.
c) On the Select Installation Type screen, select Enterprise
@ d e
e r de Stuin the Oracle Base
d) On the Install Location screen, enter /u01/app/oracle
v i lav e this
field and /u01/app/oracle/product/11.1.0/dnfs in the Path field.
o _ s
Then click Next.
b r ici to u
e (fa Cluster
e) On the Specify Hardware
e n seInstallation Mode screen, select Cluster
e rd click
Installation, and
e ic All button. Then click Next.
Select
l
f) Onithel a l
V ab Prerequisite Checks screen, click Next.
Product-Specific
i o V the Select
s f erConfiguration Option screen, select Install Software Only and click
b r g) On
ic Next.
- t r an
Fa non
h) On the Privileged Operating System Groups screen, click Next.
i) On the Summary screen, click Install.
j) Follow the instructions on the Execute Configuration Scripts screen:
[oracle@vx0308 opt]$ cat sol_opt_01j.sh
#!/bin/ksh
#
# sol_opt_01j.sh
#
#

sudo /u01/app/oracle/product/11.1.0/dnfs/root.sh

Optional Practice: Direct NFS (continued)
ssh $z "sudo /u01/app/oracle/product/11.1.0/dnfs/root.sh"
[oracle@vx0308 opt]$ ./sol_opt_01j.sh


ORACLE_HOME= /u01/app/oracle/product/11.1.0/dnfs

[/usr/local/bin]:
it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite s a
it? (y/n)
)h a
[n]:
m ฺbr
i ฺ c o deฺ
Overwrite it? (y/n)
r e d Gui
[n]:
@ sic dent
Entries will be added to the /etc/oratab
e rdea database
S tuas needed by
file
Database Configuration Assistantvwhen
i l a t h isscript. is created
Finished running generic part
i o _v uswill
of e be performed.
root.sh
Now product-specific root
Finished product-specific
a b ric root
actions
e toactions.
Running Oracle 11g
e (f root.sh
e ns script...
d li
er environment c
a V l e
b oracle variables are set as:
i l
The following
fe r a
i c i o VORACLE_HOME=
ORACLE_OWNER=
a n s /u01/app/oracle/product/11.1.0/dnfs
b r t r
Fa nEnter on- the full pathname of the local bin directory:
[/usr/local/bin]:
it? (y/n)
[n]:
it? (y/n)
[n]:
Overwrite it? (y/n)
[n]:


k) Click OK on the Execute Configuration Scripts screen.
l) On the End of Installation screen, click Exit.
m) On the Exit screen, click Yes.
2) After installing the new Oracle Home, you need to create and mount your NFS
directory on both nodes.
a) On the first node and second node, as root, create a new directory called
/RACDB. Make sure that this directory is owned by oracle and that enough
rights are granted. Then mount your NFS volume to /RACDB. You need to know
the filer name. Ask your instructor.
[oracle@vx0308 opt]$ cat sol_opt_02a.sh
#!/bin/ksh
#
s a
# sol_opt_02a.sh
)h a
#
m ฺbr
i ฺ c o deฺ
#
r e d Gui
@ sic dent
z=`cat /home/oracle/nodeinfo | sed -n
r d e'2,2p'`
S tu
e
v this
# EDIT THIS VARIABLE FIRST!!vila
i o s e
_ NFSuserver:"
ric e to
echo "Enter the name of your
read f
a b
(f ens
sudo mkdir /RACDBd e
r le lic/RACDB
sudo chown e
V ab
oracle:dba
V i l a e r /RACDB
sudo
o
chmod
i mount s f777
n -t nfs -o rw,nfsvers=3 $f:/vol/$y/$y/RACdb /RACDB
b r icsudo t r a
Fa nssh on-$z "sudo mkdir /RACDB"
ssh $z "sudo chown oracle:dba /RACDB"
ssh $z "sudo chmod 777 /RACDB"
ssh $z "sudo mount -t nfs -o rw,nfsvers=3 $f:/vol/$y/$y/RACdb
/RACDB"
[oracle@vx0308 opt]$ ./sol_opt_02a.sh

Enter the name of your NFS server:
f18
b) (Optional) As root, add the following line to /etc/fstab on both nodes:

<nfs server name>:/vol/<first node name>/<first node
name>/RACdb /RACdb nfs rw,nfsvers=3 0 0
3) Change the Oracle Disk Manager for your new Oracle Home on both nodes.

[oracle@vx0308 opt]$ cat sol_opt_03.sh
#!/bin/ksh
#
# sol_opt_03.sh
#
#

cd /u01/app/oracle/product/11.1.0/dnfs/lib
mv libodm11.so libodm11.so_stub
ln -s libnfsodm11.so libodm11.so
s a
) h a
ssh $z "mv /u01/app/oracle/product/11.1.0/dnfs/lib/libodm11.so
/u01/app/oracle/product/11.1.0/dnfs/lib/libodm11.so_stub" ฺ b r
ssh $z "ln -s c o m e ฺ
i ฺ i d
/u01/app/oracle/product/11.1.0/dnfs/lib/libnfsodm11.so
i c red t Gu
/u01/app/oracle/product/11.1.0/dnfs/lib/libodm11.so"
@ s den
[oracle@vx0308 opt]$ ./sol_opt_03.sh
e r de Stu
v i lav e this
4) You can now use DBCA from your i _ Oracle
onew us Home on the first node to create a
r ic
b /RACDB.
new RAC database storedaon t o
e ( f
e n se
a) Start DBCAL:
e rd le lic
V aopt]$
i l a
[oracle@vx0308
V f e r b cat sol_opt_04.sh
r i c io rans
/u01/app/oracle/product/11.1.0/dnfs/bin/dbca
t
Fab n[oracle@vx0308
on- opt]$
[oracle@vx0308 opt]$ ./sol_opt_04.sh

…
b) On the Welcome screen, select Oracle Real Application Clusters database and
click Next.
c) On the Operations screen, select Create a Database and click Next.
d) On the Node Selection screen, click Select All and click Next.
e) On the Database Templates screen, select General Purpose or Transaction
Processing, and then click Next.
f) On the Database Identification screen, enter DNFS in the Global Database Name
field and click Next.
g) On the Management Options screen, uncheck Configure Enterprise Manager and
click Next.

h) On the Database Credentials screen, select “Use the Same Administrative
Password for All Accounts,” enter oracle1 in both credential fields, and then
click Next.
i) On the Storage Option screen, select Cluster File System and click Next.
j) On the Database File Locations screen, select “Use Common Location for All
Database Files,” enter /RACDB in the Database Files Location field, and then
click Next.
k) On the Recovery Configuration screen, select Specify Flash Recovery Area, enter
/RACDB in the Flash Recovery Area field, and then click Next.
l) On the Database Content screen, select Sample Schema and click Next.
m) On the Initialization Parameters screen, click Next.
s a
n) On the Security Settings screen, click Next.
)h a
o) On the Automatic Maintenance Tasks screen, click Next.
m ฺbr
p) On the Database Storage screen, click Next. i ฺ c o deฺ
r e d Gui
@ sic Finish.
q) On the Creation Options, select Create Database and click
e n t
r) On the Summary screen, click OK.
r d e Stud
i l
s) Wait for the progress bar on the Databasea veConfiguration
t his Assistant screen to reach
100%. v
o_ o us e
ic i
abr nsAssistant
t) On the Database Configuration
( f e t screen that appears, click Exit.
deClusterlicDatabase
u) Wait for the rStart
e
e screen to appear.
a V ab l e
i l
V sfer
i o
ic -tran
b r
Fa non

s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Miscellaneous Topics
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Contents
• Additions to Lesson 8
• Additions to Lesson 10
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
i o
ic -tran
b r
Fa non
Oracle Database 11g: RAC Administration B - 2
Load Balancing Advisory Workflow in RAC
ODP.NET/ JDBC
Connection Pool Runtime
OCI pools 6 Manager Connection
Deal work Instances for
Load
ONS 7 per instance aging 7 Balancing
JDBC Connection Pool Connection

5 load
balancing
s a
)h a
Listener
GOAL ≠ NONE
m ฺbr 3a
CLB_GOAL=SHORT
i ฺ c o deฺ
Local
ONS
RACG
MMON r e dMMNL Gui PMON
2sic n t
IMON
4 Master
@ e 2a
3 r d e Stud1
EMON
AQ
Advisory
i l a ve thisAWR service metrics
i o _v use
messages
a b ric e to
Oracle RAC Database
e (f ens
e d
r lCopyright
l a V ab
i
V Workflowf er in RAC
i o
Load Balancing
ic -tran s
b r
Fa non(Manageability MoNitor Light) calculates the service metrics for service goal and
1. MMNL
resource consumption every five seconds. MMNL derives the service goodness from these
data.
2. MMON computes and posts the LBA FAN event to a system queue, and MMNL forwards
the service goodness and delta to PMON.
3. IMON (Instance Monitor) and EMON retrieve the event from the queue, and PMON
forwards the goodness and delta values to the listeners.
4. IMON posts the LBA FAN event to the local ONS daemon, and EMON posts it to AQ
subscribers.
5. The server ONS sends the event to the mid-tier ONS (if used).
6. The mid-tier receives the event and forwards them to subscribers. Each connection pool
subscribes to receive events for its own services. On receipt of each event, the Connection
Pool Manager refreshes the ratio of work to forward to each RAC instance connection part
of the pool. It also ranks the instances to use when aging out connections.
7. Work requests are routed to RAC instances according to the ratios calculated previously.
When there are insufficient idle connections to satisfy the ratios, new connections are added
to the next best instances determined by the listeners.
Closed Workload
BATCH work requests

Set connections in pool:
Service with #max_workers x #nodes
a set maximum of
(Lots of spares)
work requests
allowed
Connection Cache
33%
?
s a
33%
33%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
BATCH is BATCH is v Inst2 se
Inst1
i o _ u
BATCH is Inst3
ric e to
fine. fine. fine.
a b
(f ens
r lCopyright
l
e
V ab e
i l a
V sfer
i o
Closed Workload
ic -tran
b r
Fa non slides describe two possible use cases with both connection load balancing, and
The following
run-time connection load balancing in a RAC environment. The first case, presented on the
slide, shows you what is called a closed workload. In this configuration, you predefine the total
number of connections to your RAC instances, and you want them to be evenly distributed on
each instance. In the slide example, three connections are established on each instance, and this
is the maximum you want to accept. You do so, because you know that you have a predefined
maximum number of work requests that can be handled by your connection pool for your
service. This is typically the case when you control the application workload that runs on your
system. Therefore, the idea is to distribute a maximum number of connections evenly across all
your RAC instances for that particular application or service.
Closed Workload: Steady State
BATCH work requests

Set connection load Set Load Balancing
balancing: Advisory goal:
CLB_GOAL = LONG GOAL ≠ NONE
Connection Cache
33%
?
s a
33%
33%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
BATCH is BATCH is v Inst2 se
Inst1
i o _ u
BATCH is Inst3
ric e to
fine. fine. fine.
a b
(f ens
r lCopyright
l
e
V ab e
i l a
V Steady er State
i o
Closed Workload: s f
b r ic -tran
Fa non this configuration at the RAC level by setting the connection balancing
You implement
algorithm to LONG to make sure that the listeners distribute the connection requests evenly
across all RAC instances. By setting the Load Balancing Advisory to either service time or
throughput, you ensure that your connection pool distributes the work requests inside the pool to
the best instances according to your Load Balancing Advisory goal.
At steady state, all work requests are spread evenly across your RAC instances as shown in the
slide. This is because each RAC instance supports the same amount of work using the same
amount of resources on each node.
Closed Workload: More Work
CLB_GOAL = LONG BATCH work requests GOAL ≠ NONE

Max reached
Non-BATCH (in this case) Non-BATCH
work requests work requests
Connection Cache
33%
?
s a
33%
33%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
BATCH is BATCH is v e BATCH is
Inst1
very busy. fine. io
_ Inst2 u s Inst3
a b ric e to busy.
e (f ens
e d
r lCopyright
l a V ab
i
V More f erWork
i
Closed Workload:o
ic -tran s
In a b rslide,
non
this more work is added to your RAC system by adding more work requests to your
F
BATCH service as well as by adding new services requests. This ends up overloading the first
RAC instance by consuming too many resources on the corresponding node, thereby decreasing
its capacity to serve BATCH in an acceptable way. As a result, only the second node, shown in
the slide, is still capable to serve BATCH requests normally.
In consequence, Load Balancing Advisory FAN events are sent to your connection pool asking
the pool to redistribute most of the BATCH work requests to the second instance, which
currently serves BATCH best.
Closed Workload: Reaction
CLB_GOAL = LONG BATCH work requests GOAL ≠ NONE

Use Load Balancing
Advisory only to distribute
work in the pool.
Connection Cache
10%
?
s a
60%
30%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
BATCH is BATCH is v e BATCH is
Inst1
very busy. fine. io
_ Inst2 u s Inst3
a b ric e to busy.
e (f ens
e d
r lCopyright
l a V ab
i
V Reaction f er
i o
Closed Workload:
an s
In a b ric your
- t r
non
reaction, connection pool redistributes 60 percent of the BATCH work requests to the
F
pool of connections already established on the second instance. Here, only the work requests are
redistributed. The connections are not redistributed because you define them to make sure that
they cannot be saturated by the BATCH work requests.
Open Workload
CLB_GOAL = LONG CRM work requests GOAL ≠ NONE

Service with Set connections in pool:
NO set maximum of reasonable
work requests (based on RAC capacity)
Connection Cache
33%
?
s a
33%
33%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
CRM is CRM is v Inst2 se CRM is
Inst1
fine. i o _ u Inst3
ric e to
fine. fine.
a b
(f ens
r lCopyright
l
e
V ab e
i l
V sfera
Open Workloadi o
ic -tran
In a b rcase,
non
this you do not know in advance how many work requests will have to be handled by
Fyour connection pool. So, you cannot predefine them accordingly. This is typically the case with
shopping cart type of applications.
In consequence, you predefine a reasonable amount of connections in your pool, based on your
RAC capacity to handle work.
Open Workload: Steady State
CRM work requests

Set connection load Set Load Balancing
balancing: Advisory goal:
CLB_GOAL = SHORT GOAL ≠ NONE
Connection Cache
33%
?
s a
33%
33%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
Inst1
fine. i o _ u Inst3
ric e to
fine. fine.
a b
(f ens
r lCopyright
l
e
V ab e
i l a
V Steady erState
i o
Open Workload: n s f
b ric -this
Toaimplement t r aarchitecture
non
on your RAC system, you want it to be able not only to
Fredistribute the work requests in your connection pool, but also for your connections to be
redistributed to the least loaded nodes because you cannot determine in advance how many
connections are needed to avoid their saturation. You do so by setting the connection balancing
algorithm to SHORT as well as setting your Load Balancing Advisory algorithm to either service
time or throughput.
Open Workload: More Work
CLB_GOAL = SHORT CRM work requests GOAL ≠ NONE
Non-CRM Non-CRM
work requests work requests
Connection Cache
33%
?
s a
33%
33%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
RAC RAC erd
e Stu RAC
i l a v t h i s
Inst1
very busy. i o _ u Inst3
ric e to
fine. busy.
a b
(f ens
r lCopyright
l
e
V ab e
i l
V Morea r
eWork
Open Workload:i o s f
b ic -tran
rslide,
In a
non
this more work is added to your RAC system by adding more work requests to your
FCRM service as well as by adding new services requests. This ends up overloading the first RAC
instance by consuming too many resources on the corresponding node, thereby decreasing its
capacity to serve CRM in an acceptable way. As a result, only the second node, shown in the
slide, is still capable to serve CRM requests normally.
In consequence, Load Balancing Advisory FAN events are sent to your connection pool asking
the pool to redistribute most of the CRM work requests to the second instance, which currently
serves CRM best.
Open Workload: Reaction
CLB_GOAL = SHORT CRM work requests GOAL ≠ NONE

Use Load Balancing
Advisory at two levels.
Connection Cache
10%
?
s a
60%
30%
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
Gravitation si
c n t
@
e Stud e
RAC RAC erd RAC
i l a v t h i s
Inst1
very busy. i o _ u Inst3
ric e to
fine. busy.
a b
(f ens
r lCopyright
l
e
V ab e
i l a
V Reaction er
i o
Open Workload: s f
In a b ric to those
- t r anFAN events, your connection pool has to redistribute the work requests for
non
reaction
FCRM to the second instance. In this case, because you did not predefine a sufficient number of
connections to handle CRM work request, your connection pool must create additional
connections to handle the CRM service. Because your connection load balancing algorithm also
uses the Load Balancing Advisory information, listeners are going to send those new connection
requests to the second instance as illustrated in the slide. This is called connections gravitation.
In this example, not only most of the CRM work requests, but also connection requests, are
directed to the second instance.
OCR-Related Tools Debugging
• OCR tools:
– ocrdump
– ocrconfig
– ocrcheck
– srvctl
• Logs generated in a
$ORA_CRS_HOME/log/<hostname>/client/ as
)hฺbr
• Debugging control through m
$ORA_CRS_HOME/srvm/admin/ocrlog.ini diฺco ide
ฺ
i c re t Gu
mesg_logging_level = 5
@ s den
comploglvl="OCRAPI:5 ; OCRSRV:5; OCRCAC:5; OCRMAS:5;
comptrclvl="OCRAPI:5 ; OCRSRV:5; OCRCAC:5;ve
rde OCRCONF:5;
S
OCRMAS:5;s OCRCONF:5;
tu OCRRAW:5"
i l a t h i OCRRAW:5"
i o _v use
a b ric e to
e (f ens
e d
r lCopyright
l a V ab
i er
VTools sDebugging
f
OCR-Related i o
c tran
b rialready
Asayou
F n o n- saw, you can use various tools to manipulate Oracle Cluster Registry (OCR):
ocrdump, ocrconfig, ocrcheck, and srvctl.
These utilities create log files in $ORA_CRS_HOME/log/<hostname>/client/. To change the
amount of logging, edit the $ORA_CRS_HOME/srvm/admin/ocrlog.ini file. The default logging
level is 0, which basically means minimum logging. When mesg_logging_level is set to 0,
which is its default value, only error conditions are logged. You can change this setting to 3 or 5
for detailed logging information.
If that is not enough, you can also change the logging and trace level for each of the components
used to manipulate OCR. To do that, edit the entries containing comploglvl and comptrclvl in
ocrlog.ini.
The slide shows you the three lines you could add to ocrlog.ini to turn on additional
debugging information. A typical example where you might have to change the ocrlog.ini file
is in a situation where you get errors while using either ocrdump or ocrconfig tools.
Note: You should never execute the commands shown on this slide on your production
environment unless explicitly asked by Oracle Support. For additional information about
Oracle Clusterware diagnosis, you can also refer to the Note 353335.1 from Metalink as well as
the Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment
guide. reproduction or distribution prohibitedฺ Copyright© 2009, Oracle and/or its affiliatesฺ
Unauthorized
Enable Resource Debugging
• Change USR_ORA_DEBUG resource attribute to 1 for

specific resources:
# crsctl debug log res "ora.atlhp8.vip:1"
• Set the environment variable _USR_ORA_DEBUG to 1 in

racgwrap scripts to debug for all resources:
s a
# Set _USR_ORA_DEBUG to enable more tracing
)h a
#_USR_ORA_DEBUG=1 && export _USR_ORA_DEBUG
m ฺbr
i ฺ c o deฺ
_USR_ORA_DEBUG=1
r e d Gui
export _USR_ORA_DEBUG
@ sic dent
$ORACLE_HOME/bin/racgmain "$@“ rde S tu
e
status=$?
v i lav e this
exit $status
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V Debugging
f er
i o
Enable Resource
ic -tran s
b r
Fa nbeonrequested to enable tracing to capture additional information for problem resolution
You may
with Oracle Clusterware resources when working with Oracle Support. Because the procedures
described here may adversely affect performance, perform these activities only with the
assistance of Oracle Support.
To generate additional trace information for a particular running resource, as the root user, you
can use crsctl to enable resource debugging using the following syntax:
crsctl debug log res "<resource name>:1"
The example in the slide enables debugging for the ora.atlhp8.vip resource. This has the
effect of setting the environment variable USER_ORA_DEBUG to 1 in its resource’s profile. This
setting is enforced before running the start, stop, or check action scripts for the resource.
If you are asked to enable tracing for all resources, add the following lines to the racgwrap
script in both $ORA_CRS_HOME/bin and $ORACLE_HOME/bin:
_USR_ORA_DEBUG=1
export _USR_ORA_DEBUG
After you capture all trace information, do not forget to either execute corresponding crsctl
debug log res "<resource name>:0" commands, or to remove the added lines from
racgwrap scripts to switch off debugging.
Enable Oracle Clusterware Modules Debugging
crsctl
lsmodule debug
log trace statedump
crs css evm

s a
)h a
ฺbr
$ crsctl lsmodules crs
The following are the CRS modules :
c m
o CRSRESe ฺ
CRSUI CRSCOMM CRSRTI CRSMAIN CRSPLACE CRSAPP i ฺ
d GCSSCLNT
u i d
CRSCOMM CRSOCR CRSTIMER CRSEVT CRSD
i c reCLUCLS t
COMMCRS COMMNS
@ s den
e r de Stu
v i lav e this
# crsctl debug log crs CRSEVT:5,CRSAPP:5,CRSTIMER:5,CRSRES:5
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
VClusterware
f er Modules Debugging
Enable Oracle
i o
ic Oracle an s
The b r
main - t r Clusterware daemons (crsd, cssd, evmd) use various internal modules during
Fa non
their execution. You can use crsctl commands as the root user to enable dynamic debugging
for the Oracle Clusterware modules.
The crsctl lsmodules crs|css|evm commands are used to list the module’s components that
can be used for debugging. The example in the slide lists the ones for crs.
When asked by Oracle Support, you can then use commands such as the following to enable
addition logging:
• crsctl debug log <module name> <component>:<debugging level>,
where <module name> is the name of the module, crs, evm, or css; <component name> is
the name of the corresponding component obtained using the crsctl lsmodules
command; and <debugging level> is a level from 1 to 5
• crsctl debug statedump crs|css|evm dump state information for crs, css, or
evm modules
• crsctl debug trace css|crs|evm dump crs, css, or evm in-memory tracing
caches
The example in the slide shows you how to dynamically enable additional logging (level 5) for
the following CRS components: CRSEVT, CRSAPP, CRSTIMER, and CRSRES.
Note: Never execute these commands on a production system without explicit guidance from
Oracle Support.
Enable Tracing for Java-Based Tools
• To enable tracing for:

– cluvfy – srvctl
– netca – gsdctl
– vipca
• Set SRVM_TRACE to TRUE:
$ export SRVM_TRACE=TRUE
$ srvctl config database –d xwkE > /tmp/srvctl.trc
s a
$ cat /tmp/srvctl.trc
)h a
/u01/app/oracle/product/Crs/jdk/jre/bin/java -classpath
m ฺbr
/u01/app/oracle/product/Crs/jlib/netcfg.jar: …
i ฺ c o deฺ
srvctl.jar -DTRACING.ENABLED=true -DTRACING.LEVEL=2
r e d Gui
sic dent
oracle.ops.opsctl.OPSCTLDriver config database -d xwkE
[main] [9:47:27:454] [OPSCTLDriver.setInternalDebugLevel:165] @
de Stu
tracing is true at level 2 to file null
e r
v i lav e this
[main] [9:47:27:496] [OPSCTLDriver.<init>:95] Security manager
is set …
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
V for Java-Based
f
i c i
Enable Tracing o a n s Tools
r t r
F b non-tools and utilities that are available in RAC are invoked by executing scripts of
AllaJava-based
the same name as the tool or utility. This includes the Cluster Verification Utility (cluvfy), the
Database Configuration Assistant (dbca), the Database Upgrade Assistant (dbua), the Net
Configuration Assistant (netca), the Virtual Internet Protocol Configuration Assistant (vipca),
Server Control (srvctl), and the Global Services Daemon (gsdctl). For example, to run the
Database Configuration Assistant, enter the command dbca.
By default, Oracle enables traces for dbca and dbua. The resulting log files are written to
$ORACLE_HOME/cfgtoollogs/dbca/ and $ORACLE_HOME/cfgtoollogs/dbua/, respectively.
For cluvfy, gsdctl, srvctl, and vipca, you can set the SRVM_TRACE environment variable to
TRUE to make the system generate traces. Traces are written to either log files or standard output.
For example, the system writes traces to log files in $ORA_CRS_HOME/cv/log/ for cluvfy.
However, as shown in the slide, it writes traces directly to the standard output for srvctl.
Oracle Support.
RAC Diagnostic Infrastructure: Overview
Node2
SGA
Trace buffer Pi
Pi
Trace buffer Pj
Pj
DIAG
s a
Online tools Node1 Node2 Node3
)h a
x$trace
x$trace_events SGA SGA SGA
m ฺbr
i ฺ c o deฺ
oradebug
r e d Gui
DIAGsic
alter tracing
alter system set DIAG DIAG
e n t
d @
e Stud
r
ve this Offline tools
i
v sel a Trace navigation
i o _ u Trace loader
r ic t o
( f ab nse
e rde lCopyright
l a V ab e
i er
V Infrastructure:
f
i c i
RAC Diagnostico a n s Overview
r t r
Fabslidenshows
The
on- you the diagnostic architecture of RAC instances in your cluster. This
architecture enables trace processing to incur very low overhead. The DIAG process was
introduced in Oracle9i Database to manage all diagnostics-related activities, acting as a broker
between online debugging tools and regular database processes. All debugging commands are
issued through the DIAG process on the same node to reach their intended targets. This DIAG
process then coordinates with DIAG processes on other nodes of the same cluster to complete
the commands. Activities such as setting trace levels, archiving the in-memory trace logs to
files, taking memory/crash dumps are done by the DIAG processes resulting in very little
overhead to the database server.
By default, minimal tracing is always on for foreground and background processes, and all trace
information is written into in-memory buffers within the System Global Area (SGA) instead of
being written into files directly. Via the online diagnostic tools, you can instruct DIAG to set
trace levels, archive the in-memory trace logs to files, and take memory dumps. This can be
done for one or all processes on all instances. Offline tools then transform the archived logs into
human-readable formats, load them into database for query, or display them with the GUI
interface of the Trace Navigation tool used by Oracle Support. All these trace files have .trw as
their file extension, so they can be distinguished from regular process trace files. Also, these
trace files are circular, similar to the memory buffers to limit the file size.
DIAG Trace Control Interfaces
• Initialization parameter TRACE_ENABLED

• SQL:
– X$TRACE_EVENTS and X$TRACE
– ALTER TRACING ON|OFF|ENABLE|DISABLE|FLUSH
– ALTER SYSTEM SET
• oradebug: a
as
– RAC aware )h ฺbr
– Clusterwide hang analyzer
c m
o deฺ
– System and process state dumps i ฺ
d Gui
r e
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
V sInterfaces
f
DIAG Trace i oControl
ic -tran
b r
Fa non tracing facility provides various controls on its tracing behavior. You can specify
The diagnostic
the controls through either:
• Initialization parameter during instance startup: TRACE_ENABLED, is visible to the users to
enable or disable the tracing mechanism. By default, its value is TRUE, and minimal tracing
takes place. However, you can set it to FALSE to meet some high-end benchmark
requirements.
• SQL statements: At run time of an instance, tracing behavior can also be modified through
SQL statements such as:
- ALTER TRACING to turn on or turn off the tracing mechanism, to flush trace buffers to
disk, and to enable or disable trace events
- ALTER SYSTEM SET to change value of initialization parameters
• Fixed table views: As shown in the slide, there are two fixed table views related to this
tracing mechanism. They are used for online monitoring of tracing characteristics and
contents of trace buffers in SGA.
• oradebug during run time: It is possible to execute oradebug commands on remote
instances from a local instance and get the results back to the local node. You can use the
-r and -g options to do that.
Oracle Support.
DIAG Crash Dumps
• Crash dump (clusterwide) by DIAG processes when

detecting the death of a fatal Oracle process:
– Local system state dump
– Remote SGA buffers dump
• If a RAC instance crashes:
– Second last process to terminate
s a
– Allowed to dump trace
)h a
ฺbon r
• Local DIAG coordinates dumping of trace buffers m
o deฺ
remote nodes. i ฺ
d Guic
r e
• Dumps in the cdmp_<timestamp>@ sic denunder
directory t
e u
erd s St
background dump destination
v i lav e thi
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer
DIAG Crash i o Dumps
ic -tran
a b r
Crash
F dump
n o nis one of the most important features of the DIAG process. During an instance
crash, DIAG sends out a dump message to peer DIAG processes in the cluster and then creates a
system state dump. Each remote DIAG process then flushes its trace information to the disks.
That way a cluster snapshot can be reused later by Oracle Support for better diagnostics.
During the instance cleanup procedure, DIAG is the second last process to be terminated
because it needs to perform trace flushing to the file system. By default, the terminating process,
usually PMON, gives a small amount of time to DIAG for dumping.
Instance freezing is not required in order to obtain the snapshot of traces across all instances. It
is because all traces with execution history required for diagnosis are already stored in the
memory buffers and are dumped to file after a DIAG process receives the crash notification.
Traces for the moment of crash are likely to be in this history.
Note: A dump directory, named cdmp_<timestamp>, is created at the BACKGROUND_DUMP_DEST
location and all trace dump files are placed in this directory on the remote nodes.
DIAG Trace Control Interface: Examples
SQL> alter tracing enable "10425:10:135"; 1
SQL> SELECT trclevel,status,procs FROM x$trace_events

2
2> WHERE event=10425;
SQL> alter tracing disable "ALL"; 3
SQL> alter tracing flush; 4
s a
SQL> oradebug setmypid
)h5a
Statement processed.
m ฺbr
SQL> oradebug setinst all
i ฺ c o deฺ
r e d Gui
SQL> oradebug -g def hanganalyze 3
@ sic dent
Hang Analysis in
e r de Stu
lav e this
/u01/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_diag_11347.trc
SQL>
v i
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i er
V sInterface:
f
DIAG Trace
i c i oControl
a n Examples
r
abslide shows r
-t you some possible examples of use as long as you can connect normally to your
The
F o n
database:n
1. The first example is used from a SQL*Plus session to turn on tracing for event 10425 at
level 10 for process ID 135.
2. You can then query X$TRACE_EVENTS to determine which events are traced (STATUS=1) at
which level. This view contains 1,000 event statuses. You can also query X$TRACE to
retrieve trace records directly from the SGA.
3. This statement is used to disable tracing for all events.
4. This is used to archive trace logs related to your process. Using this command, you can also
flush trace buffers of all other processes. Using some undocumented parameters, it is also
possible to continuously flush trace buffers to disk instead of having to manually flush
them.
5. This example shows you how to use the hang analyzer throughout your RAC cluster. After
you are connected to SQL*Plus as SYSDBA, you attach oradebug to your process. Then you
set the instance list to all instances in your cluster, and execute the hang analyzer at level 3.
This executes a hang analysis clusterwide. When executed, the command returns the name
of the trace file containing the result of this analysis.
Oracle Support.
Hang Analysis with 11gR1: Example
$ sqlplus /nolog
SQL*Plus: Release 11.1.0.6.0 - Production on Fri Nov 2 22:51:24 2007
SQL> set _prelim on

SQL> connect / as sysdba
Prelim connection established
SQL> oradebug setorapname reco
Oracle pid: 19, Unix process pid: 11381, image:
[email protected] (RECO)
SQL> oradebug dump hanganalyze_global 1 s a
)h a
SQL> oradebug dump systemstate_global 267
m ฺbr
i ฺ c o deฺ
SQL> oradebug setorapname diag
r e d Gui
sic dent
Oracle pid: 4, Unix process pid: 11347, image:
[email protected] (DIAG)
@
de Stu
SQL> oradebug tracefile_name
e r
lav e this
/u01/app/oracle/diag/rdbms/racdb1/racdb11/trace/racdb11_diag_11347.trc
SQL>
v i
ic i o_ o us
( f abr nse t
e rde lCopyright
l a V ab e
i
VAnalysis f r
ewith
Getting Hang
i c i o a n s 11gR1 Example
r r
Fab noeven
Sometimes n-t connecting to the instance can hang. In this case there is a slightly unsafe (uses
remote Oradebug commands) but useful mechanism to collect diagnostic dumps.
This mechanism leverages prelim connections that were introduced in Oracle Database 10.1.
Prelim connections create an authenticated foreground that just attaches to the SGA. This is
enough to execute Oradebug commands. At the same time, since no process state object, session
state object, etc. are created, you are guaranteed to be able to connect to the hung instance.
The above example shows you such a case where you cannot access your instance normally.
The cluster-wide hang analysis dump will be in the local diag trace file. The per-instance system
state dump will be in the diag trace file for each instance.
WARNING: REMOTE ORADEBUG COMMANDS ARE UNSAFE. ONLY RUN THESE
COMMAND WITH EXPLICIT GUIDANCE FROM ORACLE SUPPORT.
NOTE: The example specifies the RECO process, not DIAG, because the system will use
RECO to message DIAG to do the global hang analysis and system state dumps.
In addition, Oracle Database 11gR1 introduces the V$WAIT_CHAINS view (shown in Enterprise
Manager) which displays information about blocked sessions. A wait chain comprises sessions
that are blocked by one another. Each row represents a blocked and blocker session pair. If a
wait chain is not a cyclical wait chain, then the last row for the chain does not have a blocker.
New 11gR1 Background Processes
Background Short Description

Process Name
ACMS Ensure distributed SGA memory updates
DBRM DataBase Resource Manager
DIAG Performs diagnostic dumps
DIA0 Hang detection and deadlock resolution
EMNC Event management and notifications coordination

s a
GTXn Provides XA support in RAC )h a
m ฺ br
KATE ASM metafile proxy
i ฺ c o deฺ
MARK Mark AU as stale when disk goes offline r e d Gui
@ sic dent
PING Measure interconnect latency
e r de Stu
VKTM i lav einth
Provides clock information is
RAC
v
o_ o us
ic i
( f abr nse t
e rde lCopyright
l a V ab e
i
V sfer Processes
New 11gR1i oBackground
•ab
ric -tranControlfile to Memory Server per-instance process is an agent that
non
ACMSAtomic
F contributes to ensuring a distributed SGA memory update is either globally committed on
success or globally aborted in the event of a failure in an Oracle RAC environment.
• DBRM: DataBase Resource Manager process is responsible for setting resource plans and
other resource manager related tasks.
• DIAG: DIAGnosability process performs diagnostic dumps and executes global oradebug
commands.
• DIA0: DIAgnosability process 0 (only 0 is currently being used) is responsible for hang
detection and deadlock resolution.
• EMNC: EMoN Coordinator (event monitor coordinator) is the background server process
used for database event management and notifications.
• GTXj: Global Txn process j (0-j) processes provide transparent support for XA global
transactions in an Oracle RAC environment. The database autotunes the number of these
processes based on the workload of XA global transactions. Global transaction processes
are only seen in an Oracle RAC environment.
• KATE: Konductor of ASM Temporary Errands performs proxy I/O to an ASM metafile
when a disk goes offline.
• MARK: Mark Au for Resync Koordinator marks ASM allocation units as stale following a
missed
Unauthorized write to an offline
reproduction disk.
or distribution prohibitedฺ Copyright© 2009, Oracle and/or its affiliatesฺ
New 11gR1 Background Processes (continued)
• PING: Interconnect latency measurement
• VKTM: Virtual Keeper of TiMe process: Is responsible for providing a wall-clock time
(updated every second) and reference-time counter (updated every 20 ms and available only
when running at elevated priority)
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Appendix C
Practice Environment Scripts
s a
)h a
m ฺbr
i ฺ c o deฺ
r e d Gui
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non
Practice Environment Scripts
Included in the course files on the cluster nodes is a collection of scripts that can be used
to set up the complete cluster database environment according to the practice instructions.
In addition, the scripts can be used to catch up to any point in the practices if you
encounter any unforeseen difficulties. The scripts are located in
/home/oracle/PCPS. The directory is initially restricted to prevent inadvertent
execution of the scripts. To remove the access restrictions from the PCPS directory,
execute the /home/oracle/viewdir.sh script.
$ pwd
/home/oracle
$ ./viewdir.sh
$ cd PCPS
s a
The scripts can be used to install and configure one component at a time in parallel with
)h a
ฺbr
the practice instructions or the entire cluster database can be built with a single command
m
including RMAN and Services configuration and OCR and Voting Disk backups.
i ฺ c o deฺ
r e d Gui
Note: These scripts must be run on the first node from the PCPS
@ sicdirectory.
e n t The first node
can be determined by viewing the /home/oracle/nodeinfo
r d e Stud file.
i l a ve this
_v use
$ cat /home/oracle/nodeinfo
eg6608 Í First node i o
eg6610
a b ric e to
e (f ens
e r le lic
d
The following
l a V can
actions
a bbe performed:
i fe r
i c i o V networkn s
a b r command
1.
This
make
n - tra checks whether the RAC network setup is correct by pinging around and
F checking
no /etc/hosts. Note that pinging to the VIPs returns an error when the
clusterware is not installed or if the VIPs are down for any reason such as disabling the
nodeapps.
$ pwd
/home/oracle/PCPS
$ make network
./check_network
Checking public addresses.
Pinging eg6608: Received reply.
Pinging eg6610: Received reply.
Checking private addresses.
Pinging eg6608-priv: Received reply.
Pinging eg6610-priv: Received reply.
Checking VIP addresses.
Pinging eg6608-vip:
Oracle Database 11g: RAC Administration C - 2

Received no reply - Expected if CRS is not yet installed!
Pinging eg6610-vip: Received no reply - Expected if CRS is not
yet installed!
2. make raw
This checks the raw-devices setup by reading from or writing to them. The purpose of
this check is to verify the setup of the shared disks.
Note: Do not perform the “write” test after the clusterware is installed.
$ make raw
./check_raw
Type of test? (read/write): write
WARNING: This will overwrite your raw slices!!
Continue? (yes/no): yes
Writing to /dev/sdb1: OK.
Writing to /dev/sdb6: OK. s a
Writing to /dev/sdb7: OK. )h a
Writing to /dev/sdc: OK.
m ฺbr
Writing to /dev/sdd: OK.
i ฺ c o deฺ
Writing to /dev/sde: OK.
r e d Gui
Writing to /dev/sdf: OK.
@ sic dent
3. make ssh e r de Stu
This configures user equivalency between your
v i his
lavRACenodes.
t
ic i o_ o us
$ make ssh
( f abr nse t
de equivalency.
./configure_ssh
Setting up SSH ruser
e l i ce
l a
oracle@eg6610's
a le oracle@eg6610's password:
V password:
b
Checking i
V SSHsuser
f er equivalency.
i o
ic -tran
eg6608
b r
Fa non
eg6610
4. make crs
This silently installs the clusterware after prompting you for the cluster name and
database name.
$ make crs
./install_crs
Enter the cluster name: clustorl01
Enter the database name: RDBA
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 80 MB. Actual 6573 MB

Passed
Checking swap space: must be greater than 150 MB. Actual 4095
MB Passed

Preparing to launch Oracle Universal Installer from
/tmp/OraInstall2008-08-14_12-36-32PM. Please wait ...
Oracle Universal Installer, Version 11.1.0.6.0 Production
Copyright (C) 1999, 2007, Oracle. All rights reserved.
You can find the log of this install session at:
/u01/app/oraInventory/logs/installActions2008-08-14_12-36-
32PM.log
README.11gR1........................................ 100% Done.
...

<hostname> s a
node 1: eg6608 eg6608-priv eg6608
)h a
node 2: eg6610 eg6610-priv eg6610
m ฺbr
i ฺ c o deฺ
r e d Gui
...
@ sic dent
e r de Stu
Done.
v i lavTarget
e t his State
HA Resource
----------- ic i o_ o u------
s -----
b r
a nse t
ora.eg6608.gsd
ora.eg6608.ons ( f
e lice
ONLINE
ONLINE
ONLINE on eg6608
ONLINE on eg6608
ora.eg6608.vip r d
e ble ONLINE ONLINE on eg6608
l
ora.eg6610.gsda V a ONLINE ONLINE on eg6610
i
V sfe
ora.eg6610.ons r ONLINE ONLINE on eg6610
i o
ic -tran
b r
ora.eg6610.vip ONLINE ONLINE on eg6610
Fa non
5. make asmhome
This silently installs the Oracle database software for the ASM home.
$ make asmhome
./install_asmhome
Checking Temp space: must be greater than 80 MB. Actual 6572 MB

Passed
Checking swap space: must be greater than 150 MB. Actual 4095
MB Passed
/tmp/OraInstall2008-08-14_12-44-38PM. Please wait ...Oracle
Universal Installer, Version 11.1.0.6.0 Production

38PM.log
........................................... 100% Done.
...
The installation of Oracle Database 11g was successful.

Please check '/u01/app/oraInventory/logs/silentInstall2008-08-
14_12-44-38PM.log' for more details.
running root.sh
6. make asm
This silently installs the Oracle software for the ASM home, configures ASM instances,
and creates the +DATA and +FRA disk groups.
$ make asm
./create_asm
creating ASM instances and ASM diskgroups.
Look at the log file
s a
"/u01/app/oracle/cfgtoollogs/dbca/silent.log" for further
)h a
details.
m ฺbr
HA Resource Target State
i ฺ c o deฺ
----------- ------ -----
r e d Gui
sic dent
ora.eg6608.ASM1.asm ONLINE ONLINE on eg6608
ora.eg6610.ASM2.asm ONLINE
@
ONLINE on eg6610
de Stu
e r
v i lav e this
7. make dbhome
ic i o_ o us
$ make dbhome ( f abr nse t
./install_dbhome r d e lice
Starting Oracle e le Installer...
V Universal
b
i l a a
er must be greater than 80 MB. Actual 6571 MB
V Tempsfspace:
i o
ic -tran
Checking
b r
Passed
Fa Checking
non swap space: must be greater than 150 MB. Actual 4095
MB Passed
/tmp/OraInstall2008-08-14_01-05-23PM. Please wait ...Oracle
Universal Installer, Version 11.1.0.6.0 Production

23PM.log
.............................................. 100% Done.
...
End of install phases.(Thu Aug 14 14:15:29 EDT 2008)

WARNING:
The following configuration scripts need to be executed as the
"root" user in each cluster node.
#!/bin/sh

#Root script to run
/u01/app/oracle/product/11.1.0/db_1/root.sh #On nodes
eg6608,eg6610
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The installation of Oracle Database 11g was successful.

Please check '/u01/app/oraInventory/logs/silentInstall2008-08-
14_01-05-23PM.log' for more details.
running root.sh
8. make db
This silently creates the RAC database and installs the sample schemas. If you did not
install the clusterware by using make crs, you will be prompted for the database name
here.
$ make db s a
./create_db )h a
creating the RDBA database.
m ฺbr
Copying database files
i ฺ c o deฺ
...
r e d Gui
Creating and starting Oracle instance
... @ sic dent
Creating cluster database views
e r de Stu
...
v i lav e this
Completing Database Creation
... ic i o_ o us
f abr nse t
Look at the log file "Look at the log file
(
rde le lice
"/u01/app/oracle/cfgtoollogs/dbca/RDBA/RDBA.log" for further
details. e
V ab
... i l
V sfera
i i o
c tran
b r
HA Resource
-
Target State
Fa -----------
non
ora.RDBA.RDBA1.inst
------
ONLINE
-----
ONLINE on eg6608
ora.RDBA.RDBA2.inst ONLINE ONLINE on eg6610
ora.RDBA.db ONLINE ONLINE on eg6610
9. make rman
This configures the cluster database for archivelog mode and updates the necessary
RMAN persistent configuration parameters.
$ make rman
./configure_rman
Stopping the RDBA database instances.
ORACLE instance started.
Database mounted.
...
Database altered.
System altered.

Database closed.
Database dismounted.
ORACLE instance shut down.
Starting the RDBA database instances.
Recovery Manager: Release 11.1.0.6.0 - Production on Thu Aug 14

13:26:46 2008
connected to target database: RDBA (DBID=871507933)

RMAN>
using target database control file instead of recovery catalog
new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored
RMAN>
Recovery Manager complete. s a

HA Resource Target State
)h a
----------- ------ -----
m ฺbr
ora.RDBA.RDBA1.inst ONLINE
c
ONLINE on eg6608
i ฺ o deฺ
ora.RDBA.RDBA2.inst ONLINE d Gui
ONLINE on eg6610
r e
ora.RDBA.db ONLINE
@ sic dent
ONLINE on eg6610
e r de Stu
10. make service
v i lav e this
ic i o_ oactivates
This creates a cluster service called SERV1, u s the service, and configures the
tnsnames.ora files on bothb r nodes
a cluster e t to support the service.
( f
e licen s
$ make service r d
e ble
a V
Creating V ilthe SERV1
./configure_service
f e ra Service.
i o
ci trathe s
n SERV1 Service.
b r
Starting
on-
Fa Configuring
HAnResource
the SERV1 Service tnsnames.
Target State
----------- ------ -----
ora.RDBA.SERV1.RDBA1.srv ONLINE ONLINE on eg6608
ora.RDBA.SERV1.cs ONLINE ONLINE on eg6608
11. make ocr

This mirrors the OCR file.
$ make ocr
./configure_ocr_mirror
Adding /dev/sdb2 as the OCR mirror.
Version : 2
ID : 1473866508

Device/File integrity check succeeded
Device/File integrity check succeeded
12. make vdisk

This mirrors the voting disk file.
$ make vdisk
./configure_vdisk_mirror
0. 0 /dev/sdb5
1. 0 /dev/sdb6
s a
2. 0 /dev/sdb7
)h a
Located 3 voting disk(s).
m ฺbr
i ฺ c o deฺ
r e d Gui
13. make bashrc s ic ent
This configures the .bashrc file on both nodes for the
d @ t udThese
eoracleSaccount to set shell
variables and to create useful functions to switchv e r
homes as required.
s functions
l a h i
include:
o _vi uhome.
• crs: Set variables for theiclusterware s et
b ic ASM thome.
rthe o
• asm: Set variables a for e
(f for theecluster
ns database home.
d e
• db: Set variables
c
V er ble li
ila fera
$ make bashrc
V
r i c io rans.bashrc
./configure_bashrc
t
Fab non-
Configuring
14. make all

This executes steps 3 through 13 and 16.
$ make all
./configure_ssh
Setting up SSH user equivalency.
oracle@eg6610's password: oracle@eg6610's password:
Checking SSH user equivalency.
eg6608
eg6610
./install_crs
Enter the cluster name: clustorl01
Enter the database name: RDBA

Checking installer requirements...
...
located 3 votedisk(s).
./configure_rlwrap
Compiling and installing rlwrap.
./configure_bashrc
Configuring .bashrc
15. make progress

This checks and reports on which components have already been installed or configured
on this cluster.
$ make progress
./check_progress
SSH user equivalency has been configured. s a
Oracle Clusterware is installed.
)h a
The ASM instances are created.
m ฺbr
The database software is installed.
i ฺ c o deฺ
The RDBA database is created.
r e d Gui
The SERV1 Service has been created.
The OCR is mirrored. @ sic dent
The voting disk is mirrored.
e r de Stu
RMAN has been configured.
v i lav e this
o_ o us
rlwrap is installed.
.bashrc is configured.
ic i
( f abr nse t
e rde le lice
l a V ab
16. make i er rlwrap to give SQL*Plus and RMAN history and command-
rlwrap
V andsfinstalls
Thisio compiles n
b r ic editing
line - a
trfunctionality by typing:
F a o n
n
• rlwrap sqlplus
• rlwrap rman
$ make rlwrap
./configure_rlwrap
Compiling and installing rlwrap.
17. make clean

This disables the clusterware on both nodes and reboots both nodes. Wait for the reboot
to complete. The next time this suite is used to install something, it will automatically
finish the cleanup. To force the cleanup to finish immediately, run make clean again.
After this is finished, the environment is refreshed.

$ cd /home/oracle/PCPS
$ make clean
./util_cleanup
About to remove EVERYTHING from this cluster!
Continue? (yes/no): yes
Setting up SSH user equivalency.
oracle@eg6610's password:
Checking SSH user equivalency.
eg6608
eg6610
Pseudo-terminal will not be allocated because stdin is not a
terminal.
Rebooting eg6610
Rebooting eg6608
...
s a
$ cd PCPS
)h a
ฺbr
$ make clean <After node 1 reboots, run make clean a second time>
./util_cleanup
c m
o deฺ
continuing cleanup procedure.
i ฺ
d Gui
Done.
r e
@ sic dent
e r de Stu
v i lav e this
ic i o_ o us
( f abr nse t
e rde le lice
l a V ab
i
V sfer
i o
ic -tran
b r
Fa non


D50311GC11 sg2

Uploaded by

Copyright:

Available Formats

D50311GC11 sg2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D50311GC11 sg2

Uploaded by

Copyright:

Available Formats

s a

James Womack Disclaimer

This document contains proprietary information and is protected by copyright and

1 Oracle Clusterware Installation and Configuration

2 RAC Software Installation

Summary 2-31 ( f abr nse t

8 High Availability of Connections

10 Diagnosing Oracle Clusterware and RAC Components

11 Node Addition and Removal

12 Design for High Availability

Appendix A: Practices and Solutions

Appendix B: Miscellaneous Topics

Appendix C: Practice Environment Scripts

After completing this lesson, you should be able to:

Three main methods:

1. Install and configure OS and hardware for new node.

Execute <Oracle Clusterware home>/oui/bin/addNode.sh.

Use the racgons add_config command to add new node

1. Delete the instance on the node to be deleted.

• The SYSAUX tablespace combines the storage needs for

In this lesson, you should have learned how to:

This practice covers the following topics:

After completing this lesson, you should be able to:

Unplanned Down time

Software failures Hardware failures Human errors Disasters

Operating system CPU Operator error Fire

Database Memory User error Flood

Middleware Power supply DBA Earthquake

Planned down time

Routine operations Periodic maintenance New deployments

Backups Storage maintenance HW upgrade

Performance mgmt Initialization parameters OS upgrades

Security mgmt Software patches DB upgrades

Resource Cause Protection

• Symmetric configuration with RAC at all sites:

Primary instance A Standby receiving instance C

• OC manages intrasite HA operations.

• Fast-Start Failover implements automatic failover to a

Client Client Client Client

Blocks validated and

1. Upgrade Oracle Clusterware

ex0043 ex0044 ex0045

• The OUI employs a timed lock on the inventory list

• OPatch supports four different methods:

Clients 1 Clients Logs 2

Version n Version n Version n Version n+1 Major

Use SPFILE. Create two or Set Multiplex

Enable block Use Automatic Use locally managed Use Automatic s a

• Two disk groups are recommended.

– Backups can be stored on one ERP DB

FRA disk group. CRM DB

– Lower performance may be HR DB

used for FRA (or inner tracks).

• Shared storage across several databases:

Depends on business requirement and budget s a

RAID 1 (Mirroring) RAID 5 (Parity)

• Best choice for low-cost storage

A. ASM only striping (no RAID 0)

• Use external RAID protection when possible.