HA Admin Tasks

PowerHA SystemMirror Common Tasks
for HA Administrators
Session ID: 41CO
Michael Herrera
PowerHA SystemMirror (HACMP) for AIX
ATS Certified IT Specialist
[email protected]
© 2010 IBM Corporation

IBM Power Systems
Agenda
Management: Configuration Optimization

– Start & Stop of cluster services – Hostname Changes
– Moving Resources – Naming requirements in V7.x
– Saving off Configuration – Auto start or not of cluster
services
Maintenance – Dynamic Node Priority
– Upgrading AIX & Cluster Software – Application Monitoring
– CSPOC - LVM Changes – DLPAR Integration
– Adding / Removing Physical volumes – Resource Group Dependencies
– Network Changes (dynamic) Tunables
– Setting up Pager Notification – Cluster Security
– Deploying File Collections – Failure Detection Rate (FDR)
– Custom Cluster Verification Methods – Adding Users
– Practical use of UDE events – Password Changes
– Online Planning Worksheets Common Commands (CLI)
– clmgr, lscluster
2 © 2010 IBM Corporation

IBM Power Systems
How do you check what version of code you are running ?
Historically we have run:

# lslpp –l cluster.es.server.rte
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
cluster.es.server.rte 7.1.1.1 COMMITTED Base Server Runtime
Path: /etc/objrepos
cluster.es.server.rte 7.1.1.1 COMMITTED Base Server Runtime
Now you can also run:

# halevel –s
7.1.1 SP1 even though machine may be running SP2
Also useful:
# lssrc –ls clstrmgrES | grep fix
cluster fix level is "3"
Attention:
Be aware that HA 7.1.1 SP2 or SP3 does not get reported back properly. The halevel command
probes with the wrong option and since the “server.rte” fileset is not updated it will not catch the
updates to the cluster.cspoc.rte filesets.

IBM Power Systems
Upgrade Considerations
There are two main areas that you need to consider – OS & HA Software
Change Controls: what is your ability to apply and test the updates ?
Consider things like Interim Fixes locking down the system
– Will they need to be reapplied?
– Will they need to be rebuilt?
Operating System: Cluster Software Code:

Should you do AIX first or HA code? What type of Migration
– Should you combine the upgrade – Snapshot Migration
– New OS requirements for HA – Rolling Migration
– What is your back-out plan? – Non-Disruptive Update
• Alternate disk install
Evaluate source to target level
• Mksysb
– Can you perform a NDU update?
BOS updates will typically require a – New minimum OS requirements
reboot (hence a disruption) – New required settings
• IP Multicasting, Hostname restrictions
• Required topology changes

IBM Power Systems
AIX Upgrade Flow in Clustered Environment

Hypothetical Example – 2 Node Cluster running AIX 7.1
Active Production Environment – Starting Point – Standby System

- Operating System @ AIX 7.1.0.0 Operating System @ AIX 7.1.0.0
You can start the upgrade on either node - Stop Cluster Services
but obviously an update to the node - OS update TL1 & SPs
hosting the application would cause a - Reboot
disruption to operations
Reintegrate into cluster with AIX 7.1.1.5
- Stop with Takeover
- Acquire Resource Group / Application

- OS update TL1 & SPs
- Reboot
- Reintegrate into cluster with AIX 7.1.1.5
- Issue rg_move back or continue to run on

Standby System running New Level the standby System
Common Question: Can the cluster run with the nodes running different levels?

IBM Power Systems
Flow of PowerHA Software Upgrade

Hypothetical Example – 2 Node Cluster HA version 5.5 to 6.1
Active Production Environment – Starting Point – Standby System

- HA Version 5.5 HA Version @ HA Version 5.5
- UNMANAGE resources
- Application is still running
- smit update_all We advise against stopping the
- HA Level & Patches cluster with the UNMANAGE option
- Be mindful of new base filesets on more than one node at a time.
- smit clstart Note that it can be done but there
- Start scripts will get reinvoked are various factors to consider
Node Running at New 6.1 version

- Application still active
- UNMANAGE resources
- smit update_all
- smit clstart
Node Running Version 6.1
Common Question: How long can the cluster run in a mixed mode ? What operations are supported ?
IBM Power Systems
Client Scenario – Database Binary Upgrade
Scenario:
- Client had an environment running independent Oracle databases in a mutual takeover cluster
configuration. They wanted to update the Oracle binaries one node at a time and wanted to avoid
an unexpected fallover during the process. They wished to UNMANAGE cluster resources on all
nodes at the same time.
Lessons Learned:
Do not do an upgrade of the cluster filesets while unmanaged on all nodes
– This would recycle the clstrmgrES daemon and the cluster would lose its internal state
Application monitors are not suspended when you UNMANAGE the resources
– If you manually stop the application and forget about the monitors existing application
monitors could auto-restart it or initiate a takeover depending on your configuration
Application Start scripts will get invoked again on restart of cluster services
– Be aware of what happens when you invoke your start script while already running, or
comment out the scripts prior to restarting cluster services
Leave the Manage Resources attribute set to “Automatic”

– Otherwise it will continue to show the RG as UNMANAGED until you do an RG move
ONLINE

IBM Power Systems
PowerHA SystemMirror: Cluster Startup Behavior

All currently supported
releases perform a
cluster verification on
start up and will validate
What is the “Best Practice” ? whether the node can
enter the cluster
Cluster Services are

set to automatically
start up on boot up

IBM Power Systems
PowerHA SystemMirror - Cluster Start up Behavior
The cluster manager daemon is now running all of the time

# clshowsrv -v
Status of the RSCT subsystems used by HACMP:
Subsystem Group PID Status
cthags cthags 4980948 active # lssrc -ls clstrmgrES | grep state
ctrmc rsct 4063376 active Current state: ST_STABLE
Status of the HACMP subsystems:

Subsystem Group PID Status
clstrmgrES cluster 4915234 active
clcomd caa 6422738 active
Status of the optional HACMP subsystems:

Subsystem Group PID Status - Default Start up behavior is false
clinfoES cluster 8847544 active
- Verify Cluster should be left to true
Settings can be altered within the cluster panels:

IBM Power Systems
So how do you start up Cluster Services ?

smitty sysmirror System Management PowerHA SystemMirror Services Start / Stop
smitty clstart (FastPath)
clmgr start cluster

– clmgr online node nodeA
– clmgr start node nodeA
IBM Systems Director Plug-In

IBM Power Systems
PowerHA SystemMirror: Cluster Stop Options

What is the purpose of each option ?
For non-disruptive updates

stop services on only one
node at a time to allow for
one node to retain the status
of the cluster resources
• You cannot Non-Disruptively upgrade from pre-version 7.X to newer releases
• The upgrade from 7.1.0 to 7.1.1 is also disruptive

IBM Power Systems
UNMANAGE Resource Group Feature in PowerHA

Function used for Non-Disruptive Updates (one node at a time)
– Previously known as the Forced Stop
HA Daemons will continue to run but resources will not be monitored
Application
Monitors will
continue to run.
Depending on the
implementation it
might be wise to
suspend monitors
prior to this
operation

IBM Power Systems
Moving Resources between Nodes

clRGmove –g <RGname> –n <nodename> -m If multiple RGs are selected
the operation and resources
clmgr move rg <RGname> node=<nodename> will be processed
sequentially
IBM Systems Director Plug-In
smitty cl_admin

IBM Power Systems
Types of Available RG Dependencies
Parent Child Dependencies Made Available in V5.2
Location Dependencies Made Available in V5.3

– Online on Same Node
– Online on Different Nodes
– Online on Same Site
Start After & Stop After Made Available in V7.1.1
Most of this is old news, but the use of dependencies can affect where and how
the resources get acquired. More importantly it can affect the steps required to
move resource groups and more familiarity with the configuration is required

IBM Power Systems
Moving Resource Groups with Dependencies

Invoked –
– clRGmove –g <RGname> –n <nodename> -m

IBM Power Systems
Automatic Corrections on Verify & Sync
There are Verify & Sync options in the first two

paths, however, note that they do not include
the Auto-Corrective option. You need to follow
the Custom Cluster Configuration Path for that.
The custom path will allow to make corrective

actions only if ALL cluster nodes are not running
cluster services. By default it will not perform
any corrective actions.

IBM Power Systems
Automatic Nightly Cluster Verification

By Default the cluster will run a nightly Verification check at midnight
Be aware of the
clcomd changes for
version 7 clusters
The clutils.log file should show the results of the nightly check

IBM Power Systems
Cluster Custom Verification Methods

Cluster Verification is made up of a bunch of data collectors
Checks will return PASSED or FAILED
– Will often provide more details than what is reported in the smit.log output
Custom Verification Methods may be defined to run during the Verify / Sync operations
Note: Automatic verify & sync on node start up does not include any custom verification methods

IBM Power Systems
Adding Custom Verification Methods

Problem Determination Tools > PowerHA SystemMirror Verification > Configure Custom Verification Method
Add a Custom Verification Method and press Enter
Output in smit.log and clverify.log files:

Currently Loaded Interim Fixes:
NODE mutiny.dfw.ibm.com
PACKAGE INSTALLER LABEL
======================================================== =========== ==========
bos.rte.security installp passwdLock
NODE munited.dfw.ibm.com
PACKAGE INSTALLER LABEL
======================================================== =========== ==========
bos.rte.security installp passwdLock
Please Ensure that they are consistent between the nodes!

IBM Power Systems
Custom Verification Methods

Custom methods should be in a common path between the cluster members
– ie. /usr/local/hascripts/custom_ver_check.sh
The Methods are stored in the cluster ODM stanzas
Script Logic & Return Codes #!/bin/ksh

– How fancy do you want to get echo "Currently Loaded Interim Fixes:"
clcmd emgr -P
echo "Please Ensure that they are consistent between the nodes!"

IBM Power Systems
PowerHA SystemMirror: Cluster Snapshots

/usr/es/sbin/cluster/snapshots/ <snapshotname>.info
<snapshotname>.odm
Snapshot files: Snapshots are saved
off automatically any
Snapshot C .odm Snapshot C .info
time a Verify / Sync
Snapshot
cluster B .odm Snapshot
cluster reportB .info operation is invoked
ODM stanzas
cluster
Snapshot A .odm Snapshot
cluster reportA .info
ODM stanzas
cluster cluster report
The .info file is not
ODM stanzas necessary in order to
able to restore the
configuration
Cluster Configuration Cluster Report & CLI output
HACMPcluster <html tags> The snapshot menu will

cllsnode ask for a <name> and a
...infoT T.. <description> as the
HACMPnode cllscf only required fields
TinfoT T..
HACMPadapter cllsif
The snapshot upgrade
TinfoT. T.. migration path requires
the entire cluster to be
down

IBM Power Systems
PowerHA SystemMirror: Changing the Hostname

CAA does not currently support changing a system’s hostname
– Basically means do not attempt to do this in a Version 7.X cluster
Inet0 - hostname Inet0 - hostname
Only the service IP should

be swapping between nodes
# lscluster output
Service IP The same is true for
TT.
UUID as well the cluster repository
Volume Group disk. The UUID is
/filesystems stored hence you
start.sh #!/bin/ksh should not attempt to
set new Hostname replicate the volume
Application
Controller or create an mirrors
stop.sh #!/bin/ksh to the volume
unset hostname caa_private volume
group.
* This is restriction currently under evaluation by the CAA development team and may
be lifted in a future update

IBM Power Systems
Naming requirements in V7 clusters
The COMMUNICATION_PATH has to resolve to the hostname IP

– In prior releases the CP could be any path to the node
Node name can be different than the hostname
The use of a “-” is not supported in the node name

– We had clients further highlight this limitation by using clmgr to create the
cluster. If a node name is not specified and the hostname has a “-” the default
node name assigned will also try to use a “-”
– ksh restrictions were removed to allow the use of a “-” in service IP labels so
both V6.1 and V7.X support their use in the name

IBM Power Systems
Changes to Node outbound traffic

There were changes made to AIX & PowerHA alias processing
Cluster running HA V6.1 SP7

with AIX 6.1 TL2
• Service IP Alias is listed after

persistent & base address
Cluster running HA V7.1 SP3

with AIX 7.1 TL1 SP4
• Service IP Alias is automatically

listed before the base address.
Note that no persistent IP is
configured in this environment

IBM Power Systems
Number of Resources & Fallover Times
Common Questions:
– Will the number of disks or volume groups affect my fallover time?
– Should I configure less larger luns or more smaller luns?
Versions 6.1 and earlier allowed Standard VGs or Enhanced Concurrent VGs
– Version 7.X require the use of ECM volume groups
Your Answers:
Standard VGs would require an openx call against each physical volume
– Processing could take several seconds to minutes depending on the number of LUNs
ECM VGs are varied on all nodes (ACTIVE / PASSIVE)
– It takes seconds per VG
Parallel processing will attempt to varyon on all VGs in parallel

IBM Power Systems
Number of Resource Groups
RG Decisions beyond: Startup Fallover & Fallback behavior

NODE A NODE B
RG1 (NodeA, NodeB) RG2 (NodeB, NodeA) Further Options

Service IP
1 RG vs. Multiple RGs
Service IP
VG2 – Selective Fallover behavior (VG / IP)
VG1
APP Server 1 APP Server 2 RG Processing
– Parallel vs. Sequential
RG3 (NodeA, NodeB) RG4 (NodeB, NodeA) Delayed Fallback Timer
– When do you want to fail back
Service IP Service IP
VG3 VG4 RG Dependencies
APP Server 3 APP Server 4 – Parent / Child, Location
– Start After / Stop After
Best Practice:
Always try to keep it simple, but stay current with new features and take advantage
of existing functionality to avoid added manual customization.

IBM Power Systems
Filesystem Definitions in a Resource Group

Should you explicitly define the filesystems in a Resource Group?
PowerHA default behavior is to mount ALL
Reasons to explicitly define:

– Nested Filesystems
– Only mount Filesystems specified
What are the implications going
forward if you add new Filesystems
Scenario: via CSPOC and forget to append
– 10 Filesystems in volume group & only 1 defined in RG them to the resource group
definition?
• HA processing will only mount the one FS

IBM Power Systems
Event Processing of resources

Resource Groups are processed in Parallel unless you implement RG dependencies or
set a customized serial processing order (HA 4.5 +)
The new process_resources event script is organized around job types: ACQUIRE,
RELEASE, ONLINE, OFFLINE, DISKS, TAKEOVER_LABELS, APPLICATIONS and more
i.e. JOB_TYPE = VGS
Invoked during Parallel Processing: Not invoked:

acquire_svc_addr get_disk_vg_fs
acquire_takeover_addr node_down_local
node_down node_down_remote
node_up
node_down_local_complete
release_svc_addr
node_down_remote_complete
release_takeover_addr
start_server node_up_ local
stop_server node_up_remote
node_up_local_complete
node_up_remote_complete
release_vg_fs
28
* Be mindful of this with the implementation of Pre/Post Events © 2010 IBM Corporation
IBM Power Systems
Defining Pre / Post Events

Pre/Post-Event Commands are NOT the same thing as User Defined Events
A custom Event will never

get invoked unless you
explicitly define it as a Pre or
Post event command to an
existing Cluster Event

IBM Power Systems
User Defined Events - UDE

This option allows you to exploit RMC resource monitors to trigger EVENTs
Familiarize yourself with the “lsrsrc” command
– A Practical Guide for Resource Monitoring and Control - SG24-6615
Notes: # odmget HACMPude

Recycle cluster services after updating UDE events
Scripts must exist on all cluster nodes: (Path, permissions) HACMPude:
Logic in recovery program can be configured to send name = "Herrera_UDE_event"
notification, append more space, etcT state = 0
Can specify multiple values in Selection String field recovery_prog_path =
Actions logged in clstrmgr.debug and hacmp.out files "/usr/local/hascripts/Herrera_UDE“
recovery_type = 2
recovery_level = 0
res_var_name = "IBM.FileSystem"
instance_vector = "Name = \"/\""
predicate = "PercentTotUsed > 95"
rearm_predicate = "PercentTotUsed < 70"

IBM Power Systems
PowerHA SystemMirror: File Collections

Introduced in HA 5.2
– Ability to automatically push files every 10 min from source node specified
– Default collections created but not enabled by default
Configuration_Files SystemMirror_Files
– /etc/hosts – Pre, Post & Notification
– /etc/services – Start & Stop scripts
– /etc/snmpd.conf – Scripts specified in monitors
– /etc/snmpdv3.conf – Custom pager text messages
– /etc/rc.net – SNA scripts
– /etc/inetd.conf – Scripts for tape support
– /usr/es/sbin/cluster/netmon.cf – Custom snapshot methods
– /usr/es/sbin/cluster/etc/clhosts – User defined events
– /usr/es/sbin/cluster/etc/rhosts
– /usr/es/sbin/cluster/etc/clinfo.rc
Not intended to maintain users & passwords between cluster nodes

IBM Power Systems
File Collections Application script Scenario

# smitty sysmirror System Management File Collections
If set to yes files

will be propagated
every 10 minutes
Node A Node B
/usr/local/hascripts/app* /usr/local/hascripts/app*
#!/bin/ksh #!/bin/ksh
Application Start Logic Application Start Logic
RED Updates BLUE Logic

#!/bin/ksh #!/bin/ksh
Application Stop Logic Application Stop Logic
RED Updates Blue Logic

IBM Power Systems
PowerHA SystemMirror - User & Group Administration

# smitty sysmirror System Management Security and Users
Can select
– Local (files)
– LDAP
Select Nodes by
Resource Group
– No selection
means all nodes
Users will be
propagated to all of
the cluster nodes
applicable
Password command
can be altered to
ensure consistency
across al nodes

IBM Power Systems
PowerHA SystemMirror - User Passwords (clpasswd)

# smitty sysmirror System Management Security and Users Passwords in a PowerHA SystemMirror cluster
Optional List of
Users whose
passwords will be
propagated to all
cluster nodes
– passwd
command is
aliased to
clpasswd
Functionality
available since
HACMP 5.2
(Fall 2004)

IBM Power Systems
Repository Disk Failure

IBM Power Systems
Pager Notification Events

As long as sendmail is enabled you can easily receive EVENT notification
smitty sysmirror Custom Cluster Configuration Events Cluster Events

Remote Notification Methods Add a Custom Remote Notification Method
Sample Email:
From: root 10/23/2012 Subject: HACMP

Node mhoracle1: Event acquire_takeover_addr occurred at Tue Oct 23 16:29:36 2012, object =

IBM Power Systems
Pager Notification Methods

HACMPpager:
methodname = "Herrera_notify"
desc = “Lab Systems Pager Event"
nodename = "connor kaitlyn"
dialnum = "[email protected]"
filename = "/usr/es/sbin/cluster/samples/pager/sample.txt"
eventname = "acquire_takeover_addr config_too_long
event_error node_down_complete node_up_complete"
retrycnt = 3
timeout = 45
# cat /usr/es/sbin/cluster/samples/pager/sample.txt
Node %n: Event %e occurred at %d, object = %o
Action Taken: Halted Node Connor
Sample Email:
From: root 09/01/2009 Subject: HACMP

Node kaitlyn: Event acquire_takeover_addr occurred at Tue Sep 1 16:29:36 2009, object =
Attention:
Sendmail must be working and accessible via the firewall to receive notifications
IBM Power Systems
Online Planning Worksheets Discontinued in Version 7

The fileset is still there, but the content is no longer there
There is a push to
leverage IBM Systems
Director which will guide
you through the step by
step configuration of the
cluster

IBM Power Systems
PowerHA SystemMirror – Deadman Switch (CAA)

Version 7 cluster software changes the old behavior
Recent Client Failure Scenario
- Repository Disk LUN had been

locked and had not been
responsive for days. Client was
unaware and standby node had a
problem. Primary system was
brought down when it was unable
to write to repository disk
TT..............
CAA DMS tunable (deadman_mode) allows two different actions

– Assert (crash) the system (default behavior)
– Generate AHAFS event

IBM Power Systems
LVM Dynamic Updates
The cluster is easy to set up, but what about changes going forward
ECM Volume Groups (required at HA V7)

– New lvs will get pushed across, filesystems will not
• LV updates get pushed across but do not update the /etc/filesystems.
• Lazy Update would resolve this issue
– ECM Limitations lifted for:

• reorgvg & chvg -g size changes
Cluster Import Option

– Correcting out of sync timestamps auto-corrections or import
Built-In Lazy Update

IBM Power Systems
CSPOC allows for a multitude of DARE operations

The Cluster Single Point of Control options facilitate dynamic operations
# smitty cl_admin
Follow these panels to

dynamically add or remove
resources from the cluster or
perform resource group
movements between nodes
There are CSPOC specific

logs in the HA cluster that will
provide details in the event of
a problem

IBM Power Systems
CSPOC: Storage & LVM Menus

IBM Power Systems
Tunable Failure Detection Rate in 7.1.1

Note that the SMIT menu to alter values was missing prior to HA 7.1.1 SP1
Attributes stored
in HACMPcluster
object class
Checking current settings:

root@mhoracle1 /> clctrl -tune -o node_down_delay
sapdemo71_cluster(07552a84-057b-11e1-b7cb-46a6ba546402).node_down_delay = 10000
root@mhoracle1 /> clctrl -tune -o node_timeout
sapdemo71_cluster(07552a84-057b-11e1-b7cb-46a6ba546402).node_timeout = 20000
Modifying via command line:

clmgr modify cluster HEARTBEAT_FREQUENCY= 10000 GRACE_PERIOD=5000
*** The settings will take effect only after the next sync

IBM Power Systems
FDR Comparison to Version 6.1 & earlier versions
RSCT (topsvcs) CAA
Heartbeat settings are same for all networks

Heartbeat settings can be defined for each in the cluster.
network type (nim). One perspective is that we only support
Ethernet networks.
The settings for heartbeat are The settings for heartbeat are
Grace period Grace period – (5 - 30 Seconds)
Failure Cycle Failure cycle – (1 - 20 seconds)
Interval between Heartbeats
The combination of heartbeat rate and failure Failure cycle is the time that another node
cycle determines how quickly a failure can be may consider the adapter to be DOWN if it
detected and may be calculated using this receives no incoming heartbeats.
formula: Actual heartbeat rate is calculated
(heartbeat rate) * (failure cycle) * 2 seconds depending on the Failure cycle.
Grace period is the waiting time period after Grace period is the waiting time period after
detecting the Failure before it is reported. detecting the Failure before it is reported.
*** Note that HA 7.1.0 had self-tuning failure detection rate

© 2010 IBM Corporation
IBM Power Systems
Application Monitoring within PowerHA SystemMirror
Some are provided in Smart Assistants

– ie. cluster.es.assist.oracle /usr/es/sbin/cluster/sa/oracle/sbin/DBInstanceMonitor
A Monitor is bound to the Application Controller

– Example OracleDB
Startup Process Custom

Monitor Monitor Monitor
Only
invoked on 60 sec 60 sec
application interval interval
startup
Confirm the
startup of the Long Running Monitors will
application continue run locally with the
running application
New
Application
Startup Mode Checks the Invokes the
in HA 7.1.1 process table custom logic

IBM Power Systems
PowerHA SystemMirror: Application Startup 7.1.1

The cluster invokes the start script but doesn’t confirm its success
Consider at least an application start up monitor
Resource Group A
Service IP
Enhancement was introduced in HA Version 7.1.1
Volume Group - Application start may be set to run in the foreground
/filesystems
start.sh
Application
Controller
stop.sh
Start up Monitor
Long-Running Monitor

IBM Power Systems
PowerHA SystemMirror – HMC Definition
There was no
SDMC support.
No longer much
of an issue
Information
stored in HA
ODM object
classes
Multiple HMC
IPs may be
defined
separated by a
space
Food for Thought: How many DLPAR operations can be handled at once?

IBM Power Systems
PowerHA SystemMirror – Integrated DLPAR Menu

Add Dynamic LPAR and CoD Resources for Applications
HMC
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[TOP] [Entry Fields]

* Application Controller Name Application_svr1 HMC IPs are
defined and stored
* Minimum number of processing units [ 0.00] in a different HA
* Desired number of processing units [ 0.00] panel
* Minimum number of CPUs [0] #
* Desired number of CPUs [0] #
* Minimum amount of memory (in megabytes) [0] #

* Desired amount of memory (in megabytes) [0] #
* Use CoD if resources are insufficient? [no] +

* I agree to use CoD resources [no] +
(Using CoD may result in extra costs)
You must ensure that

* CoD enablement keys are activated
* CoD resources are not used for any other purpose

IBM Power Systems
The many uses of the clmgr utility

V7 Clustering introduces many applications for this command
# clmgr add cluster clmgr_cluster REPOSITORY=hdisk2 CLUSTER_IP=228.1.1.36 Add a new cluster

# clmgr add node clmgr2 Add a new node
# clmgr add network net_ether_01 TYPE=ether
# clmgr add interface clmgr2b2 NETWORK=net_ether_02 NODE=clmgr2 INTERFACE=en1
# clmgr add persistent clmgr1p1 NETWORK=net_ether_01 NODE=clmgr1

# clmgr add service_ip clmgrsvc1 NETWORK=net_ether_01
# clmgr add application_controller test_app1 STARTSCRIPT="/home/apps/start1.sh" Add an Application

STOPSCRIPT="/home/apps/stop1.sh" STARTUP_MODE=background Controller
# clmgr add volume_group test_vg1 NODES="clmgr1,clmgr2" PHYSICAL_VOLUMES=hdisk3
TYPE=original MAJOR_NUMBER=35 ACTIVATE_ON_RESTART=false
# clmgr add resource_group clmgr_RG1 NODES="clmgr1,clmgr2" STARTUP=OHN

FALLOVER=FNPN FALLBACK=NFB VOLUME_GROUP=test_vg Add a new
SERVICE_LABEL=clmgrsvc1 APPLICATIONS=test_app1 Resource Group
# clmgr verify cluster CHANGES_ONLY=no FIX=yes LOGGING=standard Verify / Sync cluster

# clmgr sync cluster CHANGES_ONLY=no FIX=yes LOGGING=standard
# clmgr online cluster WHEN=now MANAGE=auto BROADCAST=true CLINFO=true Start Cluster Services
# clmgr modify clusterNAME=my_new_cls_label Change cluster name

# clmgr manage application_controller suspend test_app1 RESOURCE_GROUP="clmgr_RG1" Suspend / Resume
# clmgr manage application_controller resume test_app1 RESOURCE_GROUP="clmgr_RG2"
Application Monitors

IBM Power Systems
Summary
There are some notable differences between V7 and HA 6.1 and earlier
– Pay careful attention to where some of the options are available
– Appended Summary Chart of new features to the presentation
Version 7.1.2 Scheduled GA on Nov 9th

– Brings Enterprise Edition to V7 clusters
This session is an attempt to make you aware of available options in

PowerHA
– Take my recommendations with a grain of salt!
Take advantage of integrated features & interfaces like:

– Application monitoring infrastructure
– File Collections
– Pre/Post Events and User Defined Events
– Pager Notification Methods
– New clmgr CLI
SG24-8030
IBM Power Systems
Summary Chart
New Functionality & Changes
– New CAA Infrastructure 7.1.X Smart Assistants (Application Integration)
• IP Multicast based Heartbeat Protocol – SAP Live Cache with DS or SVC 7.1.1
• HBA Based SAN Heartbeating – MQ Series 7.1.1
• Private Network Support
• Tunable Failure Detection Rate
• New Service IP Distribution Policies DR Capabilities
• Full IPV6 Support 7.1.2 – Stretch & Linked Clusters 7.1.2
– DS8000 Hyperswap 7.1.2
– Disk Fencing Enhancements 7.1.0
– Rootvg System Event 7.1.0
– Disk rename Function 7.1.0 Management
– Repository Disk Resilience 7.1.1 – New Command Line Interface 7.1.0
• Backup Repository Disks 7.1.2 • clcmd
• clmgr utility
– New Application Startup Mode 7.1.1 • lscluster
– Exploitation of JFS2 Mount Guard 7.1.1 – IBM Systems Director Management 7.1.0
– Adaptive Fallover 7.1.0
– New RG Dependencies 7.1.0
• Start After, Stop After
Extended Distance Clusters
– Federated Security 7.1.1 – XIV Replication Integration (12/16/2011)
• RBAC, EFS & Security System Administration – XP12000, XP24000 (11/18/2011)
– HP9500 (8/19/2011)
– Storwize v7000 (9/30/2011)
– SVC 6.2 (9/30/2011)

IBM Power Systems
Questions?
Thank you for your time!

IBM Power Systems
Additional Resources
PowerHA SystemMirror 7.1.1 Update – SG24-8030

http://www.redbooks.ibm.com/redpieces/abstracts/sg248030.html?Open
PowerHA SystemMirror 7.1 Redbook – SG24-7845 Removed from Download site

http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg247845.html?Open
Disaster Recovery Redbook

SG24-7841 - Exploiting PowerHA SystemMirror Enterprise Edition for AIX
http://www.redbooks.ibm.com/abstracts/sg247841.html?Open
RedGuide: High Availability and Disaster Recovery Planning: Next-Generation Solutions for Multi
server IBM Power Systems Environments
http://www.redbooks.ibm.com/abstracts/redp4669.html?Open
PowerHA SystemMirror Marketing Page

http://www-03.ibm.com/systems/power/software/availability/aix/index.html
PowerHA SystemMirror Wiki Page

http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/High+Availability

HA Admin Tasks

Uploaded by

Copyright:

Available Formats

HA Admin Tasks

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HA Admin Tasks

Uploaded by

Copyright:

Available Formats

PowerHA SystemMirror Common Tasks

© 2010 IBM Corporation

Management: Configuration Optimization

2 © 2010 IBM Corporation

How do you check what version of code you are running ?

Historically we have run:

Now you can also run:

3 © 2010 IBM Corporation

Operating System: Cluster Software Code:

4 © 2010 IBM Corporation

AIX Upgrade Flow in Clustered Environment

Active Production Environment – Starting Point – Standby System

- Stop with Takeover

- Acquire Resource Group / Application

- Issue rg_move back or continue to run on

5 © 2010 IBM Corporation

Flow of PowerHA Software Upgrade

Active Production Environment – Starting Point – Standby System

Node Running at New 6.1 version

Node Running Version 6.1

Client Scenario – Database Binary Upgrade

Leave the Manage Resources attribute set to “Automatic”

7 © 2010 IBM Corporation

PowerHA SystemMirror: Cluster Startup Behavior

Cluster Services are

8 © 2010 IBM Corporation

PowerHA SystemMirror - Cluster Start up Behavior

The cluster manager daemon is now running all of the time

Status of the HACMP subsystems:

Status of the optional HACMP subsystems:

Settings can be altered within the cluster panels:

9 © 2010 IBM Corporation

So how do you start up Cluster Services ?

smitty clstart (FastPath)

clmgr start cluster

IBM Systems Director Plug-In

10 © 2010 IBM Corporation

PowerHA SystemMirror: Cluster Stop Options

For non-disruptive updates

• You cannot Non-Disruptively upgrade from pre-version 7.X to newer releases

• The upgrade from 7.1.0 to 7.1.1 is also disruptive

11 © 2010 IBM Corporation

UNMANAGE Resource Group Feature in PowerHA

HA Daemons will continue to run but resources will not be monitored

12 © 2010 IBM Corporation

Moving Resources between Nodes

IBM Systems Director Plug-In

13 © 2010 IBM Corporation

Types of Available RG Dependencies

Parent Child Dependencies Made Available in V5.2

Location Dependencies Made Available in V5.3

Start After & Stop After Made Available in V7.1.1

14 © 2010 IBM Corporation

Moving Resource Groups with Dependencies

15 © 2010 IBM Corporation

Automatic Corrections on Verify & Sync

There are Verify & Sync options in the first two

The custom path will allow to make corrective

16 © 2010 IBM Corporation

Automatic Nightly Cluster Verification