Chapter

Adding and removing cluster nodes

This chapter contains the following topics:

About adding and removing nodes Adding a node to a cluster Removing a node from a cluster

About adding and removing nodes

After installing VCS and creating a cluster, you can add and remove nodes from the cluster. You can create clusters of up to 32 nodes.

2 Adding and removing cluster nodes Adding a node to a cluster

Adding a node to a cluster

The system you add to the cluster must meet the hardware and software requirements. See Preparing to install and configure VCS on page 23. Table 7-1 specifies the tasks involved in adding a cluster. The example demonstrates how to add a node east to already existing nodes, north and south. Table 7-1 Task
Set up the hardware. Install the software manually.

Tasks involved in adding a node to a cluster Reference

Setting up the hardware on page 3 Preparing for a manual installation Installing VCS RPMs for a manual installation

Add a license key. Configure LLT and GAB. Add the node to the existing cluster.

Adding a license key on page 7 Configuring LLT and GAB on page 7 Adding the node to the existing cluster on page 9 Starting VCS and verifying the cluster on page 9

Start VCS and verify the cluster.

Adding and removing cluster nodes Adding a node to a cluster

Setting up the hardware

Before configuring a new system to an existing cluster, you must physically add the system to the cluster. Figure 7-1 Adding a node to a three-node cluster using two independent hubs

Public Network

Private Network

To set up the hardware 1 Connect the VCS private Ethernet controllers.

If you are expanding from a two-node cluster, you need to use independent hubs for the private network connections, replacing crossover cables if they are used.

If you already use independent hubs, connect the two Ethernet controllers on the new node to the independent hubs. Figure 7-1 illustrates a new node being added to an existing three-node cluster using two independent hubs.

Connect the system to the shared storage, if required.

Preparing for a manual installation

Before you install, log in as the superuser. You then mount the disc and put the files in a temporary folder for installation. See Mounting the product disc on page 79. To prepare for installation

Depending on the OS distribution, replace the dist in the command with rhel4 or sles9. Depending on the architecture, replace the arch in the command with i686, i586, or x86_64. Type the command:
# cd /mnt/cdrom/dist_arch/cluster_server/rpms

4 Adding and removing cluster nodes Adding a node to a cluster

Installing VCS RPMs for a manual installation

VCS has both required and optional RPMs. Install the required RPMs first. All RPMs are installed in the /opt directory. When selecting the optional RPMs, note:

Symantec recommends that you install the RPMs for VCS manual pages (VRTSvcsmn) and VCS documentation (VRTSvcsdc). Install the documentation RPM on nodes where you want access to the documentation. The I/O fencing RPM (VCSvxfen) can be used only with shared disks that support SCSI-3 Persistent Reservations (PR). See the Veritas Cluster Server Users Guide for a conceptual description of I/O fencing. You need to test shared storage for SCSI-3 PR and to implement I/O fencing. See Setting up I/O fencing on page 139. The VCS configuration wizard (VRTScscw) RPM includes wizards for the installation and configuration of Veritas products that require VCS configuration. To use the Java Console with VCS Simulator, you must install the VRTScssim and VRTScscm RPMs.

Perform the steps to install VCS RPMs on each node in the cluster. To install VCS RPMs on a node 1 Install the required VCS RPMs in the order shown. Do not install any RPMs already installed on the system. Pay special attention to operating system distribution and architecture.

RHEL4/x86_64, required RPMs:

# # # # # # # # # # # # # # # # # # # rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i SYMClma-4.1.3.5-5.i386.rpm VRTSatClient-4.3.23.0-0.i386.rpm VRTSatServer-4.3.23.0-0.i386.rpm VRTSicsco-1.3.18.4-4.i386.rpm VRTSpbx-1.3.17.4-4.i386.rpm VRTSperl-5.0.2.1-linux.i386.rpm VRTSsmf-1.3.13.4-4.i386.rpm VRTSspt-5.0.00.0-GA.noarch.rpm VRTSvlic-3.02.16.8-0.x86_64.rpm VRTSllt-5.0.00.0-GA_RHEL4.x86_64.rpm VRTSgab-5.0.00.0-GA_RHEL4.x86_64.rpm VRTSvxfen-5.0.00.0-GA_RHEL4.x86_64.rpm VRTSvcs-5.0.00.0-GA_RHEL4.i686.rpm VRTSvcsmg-5.0.00.0-GA_GENERIC.noarch.rpm VRTSacclib-5.0.00.0-0.i386.rpm VRTSvcsag-5.0.00.0-GA_RHEL4.i686.rpm VRTSvcsdr-5.0.00.0-GA_RHEL4.x86_64.rpm VRTSjre-1.4-GA1.i386.rpm VRTSjre15-1.5-GA3.i386.rpm

Adding and removing cluster nodes Adding a node to a cluster

# rpm -i VRTScscw-5.0.00.0-GA_GENERIC.noarch.rpm # rpm -i VRTSweb-5.0.1-GA4_GENERIC.noarch.rpm # rpm -i VRTScutil-5.0-GA_GENERIC.noarch.rpm

RHEL4/i686, required RPMs:

# # # # # # # # # # # # # # # # # # # # # # rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i SYMClma-4.1.3.5-5.i386.rpm VRTSatClient-4.3.23.0-0.i386.rpm VRTSatServer-4.3.23.0-0.i386.rpm VRTSicsco-1.3.18.4-4.i386.rpm VRTSpbx-1.3.17.4-4.i386.rpm VRTSperl-5.0.2.1-linux.i386.rpm VRTSsmf-1.3.13.4-4.i386.rpm VRTSspt-5.0.00.0-GA.noarch.rpm VRTSvlic-3.02.16.8-0.i686.rpm VRTSllt-5.0.00.0-GA_RHEL4.i686.rpm VRTSgab-5.0.00.0-GA_RHEL4.i686.rpm VRTSvxfen-5.0.00.0-GA_RHEL4.i686.rpm VRTSvcs-5.0.00.0-GA_RHEL4.i686.rpm VRTSvcsmg-5.0.00.0-GA_GENERIC.noarch.rpm VRTSacclib-5.0.00.0-0.i386.rpm VRTSvcsag-5.0.00.0-GA_RHEL4.i686.rpm VRTSvcsdr-5.0.00.0-GA_RHEL4.i686.rpm VRTSjre-1.4-GA1.i386.rpm VRTSjre15-1.5-GA3.i386.rpm VRTScscw-5.0.00.0-GA_GENERIC.noarch.rpm VRTSweb-5.0.1-GA4_GENERIC.noarch.rpm VRTScutil-5.0-GA_GENERIC.noarch.rpm

SLES9/x86_64, required RPMs:

# # # # # # # # # # # # # # # # # # # # # # rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i SYMClma-4.1.3.5-5.i386.rpm VRTSatClient-4.3.23.0-0.i386.rpm VRTSatServer-4.3.23.0-0.i386.rpm VRTSicsco-1.3.18.4-4.i386.rpm VRTSpbx-1.3.17.4-4.i386.rpm VRTSperl-5.0.2.1-linux.i386.rpm VRTSsmf-1.3.13.4-4.i386.rpm VRTSspt-5.0.00.0-GA.noarch.rpm VRTSvlic-3.02.16.8-0.x86_64.rpm VRTSllt-5.0.00.0-GA_SLES9.x86_64.rpm VRTSgab-5.0.00.0-GA_SLES9.x86_64.rpm VRTSvxfen-5.0.00.0-GA_SLES9.x86_64.rpm VRTSvcs-5.0.00.0-GA_SLES9.i686.rpm VRTSvcsmg-5.0.00.0-GA_GENERIC.noarch.rpm VRTSacclib-5.0.00.0-0.i386.rpm VRTSvcsag-5.0.00.0-GA_SLES9.i686.rpm VRTSvcsdr-5.0.00.0-GA_SLES9.x86_64.rpm VRTSjre-1.4-GA1.i386.rpm VRTSjre15-1.5-GA3.i386.rpm VRTScscw-5.0.00.0-GA_GENERIC.noarch.rpm VRTSweb-5.0.1-GA4_GENERIC.noarch.rpm VRTScutil-5.0-GA_GENERIC.noarch.rpm

SLES9/i586, required RPMs:

6 Adding and removing cluster nodes Adding a node to a cluster

# # # # # # # # # # # # # # # # # # # # # #

rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm

-i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i -i

SYMClma-4.1.3.5-5.i386.rpm VRTSatClient-4.3.23.0-0.i386.rpm VRTSatServer-4.3.23.0-0.i386.rpm VRTSicsco-1.3.18.4-4.i386.rpm VRTSpbx-1.3.17.4-4.i386.rpm VRTSperl-5.0.2.1-linux.i386.rpm VRTSsmf-1.3.13.4-4.i386.rpm VRTSspt-5.0.00.0-GA.noarch.rpm VRTSvlic-3.02.16.8-0.i586.rpm VRTSllt-5.0.00.0-GA_SLES9.i586.rpm VRTSgab-5.0.00.0-GA_SLES9.i586.rpm VRTSvxfen-5.0.00.0-GA_SLES9.i586.rpm VRTSvcs-5.0.00.0-GA_SLES9.i686.rpm VRTSvcsmg-5.0.00.0-GA_GENERIC.noarch.rpm VRTSacclib-5.0.00.0-0.i386.rpm VRTSvcsag-5.0.00.0-GA_SLES9.i686.rpm VRTSvcsdr-5.0.00.0-GA_SLES9.i586.rpm VRTSjre-1.4-GA1.i386.rpm VRTSjre15-1.5-GA3.i386.rpm VRTScscw-5.0.00.0-GA_GENERIC.noarch.rpm VRTSweb-5.0.1-GA4_GENERIC.noarch.rpm VRTScutil-5.0-GA_GENERIC.noarch.rpm

Install the optional RPMs, in the order shown. Omit those that you do not want to install.

RHEL4/x86_64, optional RPMs:

# # # # # # rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i VRTSvcsmn-5.0.00.0-GA_GENERIC.noarch.rpm VRTSvcsdc-5.0.00.0-GA_GENERIC.noarch.rpm VRTScscm-5.0.00.0-GA_GENERIC.noarch.rpm VRTScssim-5.0.00.0-GA_RHEL4.i686.rpm VRTScmcs-5.0.00.00-GA_RHEL4.i686.rpm VRTScmccc-5.0.00.00-GA_RHEL4.i686.rpm

RHEL4/i686, optional RPMs:

# # # # # # rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i VRTSvcsmn-5.0.00.0-GA_GENERIC.noarch.rpm VRTSvcsdc-5.0.00.0-GA_GENERIC.noarch.rpm VRTScscm-5.0.00.0-GA_GENERIC.noarch.rpm VRTScssim-5.0.00.0-GA_RHEL4.i686.rpm VRTScmcs-5.0.00.00-GA_RHEL4.i686.rpm VRTScmccc-5.0.00.00-GA_RHEL4.i686.rpm

SLES9/x86_64, optional RPMs:

# # # # # # rpm rpm rpm rpm rpm rpm -i -i -i -i -i -i VRTSvcsmn-5.0.00.0-GA_GENERIC.noarch.rpm VRTSvcsdc-5.0.00.0-GA_GENERIC.noarch.rpm VRTScscm-5.0.00.0-GA_GENERIC.noarch.rpm VRTScssim-5.0.00.0-GA_SLES9.i686.rpm VRTScmcs-5.0.00.00-GA_SLES9.i686.rpm VRTScmccc-5.0.00.00-GA_SLES9.i686.rpm

SLES9/i586, optional RPMs:

# rpm -i VRTSvcsmn-5.0.00.0-GA_GENERIC.noarch.rpm # rpm -i VRTSvcsdc-5.0.00.0-GA_GENERIC.noarch.rpm # rpm -i VRTScscm-5.0.00.0-GA_GENERIC.noarch.rpm

Adding and removing cluster nodes Adding a node to a cluster

# rpm -i VRTScssim-5.0.00.0-GA_SLES9.i686.rpm # rpm -i VRTScmcs-5.0.00.00-GA_SLES9.i686.rpm # rpm -i VRTScmccc-5.0.00.00-GA_SLES9.i686.rpm

Adding a license key

After you have installed all RPMs on each cluster node, use the vxlicinst command to add the VCS license key on each system:
# cd /opt/VRTS/bin # ./vxlicinst -k XXXX-XXXX-XXXX-XXXX-XXXX-XXX

Checking licensing information on the system

Use the vxlicrep utility to display information about all Veritas licenses on a system. For example, enter:
# cd /opt/VRTS/bin # ./vxlicrep

From the output, you can determine the license key, the type of license, the product for which it applies, and its expiration date, if any. Demo keys have expiration dates, while permanent keys and site keys do not.

Configuring LLT and GAB

Create the LLT and GAB configuration files on the new node and update the files on the existing nodes. To configure LLT 1 Create the file /etc/llthosts on the new node. You must also update it on each of the current nodes in the cluster. For example, suppose you are adding east to a cluster consisting of north and south:

If the file on one of the existing nodes resembles:

0 north 1 south

Update the file for all nodes, including the new one, resembling:
0 north 1 south 2 east

Create the file /etc/llttab on the new node, making sure that line beginning set-node specifies the new node. The file /etc/llttab on an existing node can serve as a guide. See /etc/llttab on page 186. The following example describes a system where node east is the new node on cluster number 2:

8 Adding and removing cluster nodes Adding a node to a cluster

set-node east set-cluster 2 link eth1 eth1 - ether - link eth2 eth2 - ether - -

On the new system, run the command:

# /sbin/lltconfig -c

To configure GAB 1 Create the file /etc/gabtab on the new system.

If the /etc/gabtab file on the existing nodes resembles:

/sbin/gabconfig -c

then the file on the new node should be the same, although it is recommended to use the -c -nN option, where N is the number of cluster nodes.

If the /etc/gabtab file on the existing nodes resembles:

/sbin/gabconfig -c -n2

then, the file on all nodes, including the new node, should change to reflect the change in the number of cluster nodes. For example, the new file on each node should resemble:
/sbin/gabconfig -c -n3

See /etc/gabtab on page 186. The -n flag indicates to VCS the number of nodes required to be ready to form a cluster before VCS starts. 2 On the new node, run the command, to configure GAB:
# /sbin/gabconfig -c

To verify GAB 1 On the new node, run the command:

# /sbin/gabconfig -a

The output should indicate that Port a membership shows all nodes including the new node. The output should resemble:
GAB Port Memberships ==================================== Port a gen a3640003 membership 012

See Verifying GAB on page 195. 2 Run the same command on the other nodes (north and south) to verify that the Port a membership includes the new node:
# /sbin/gabconfig -a GAB Port Memberships ==================================== Port a gen a3640003 membership 012 Port h gen fd570002 membership 01 Port h gen fd570002 visible ; 2

Adding and removing cluster nodes Adding a node to a cluster

Adding the node to the existing cluster

Perform the tasks on one of the existing nodes in the cluster. To add the new node to the existing cluster 1 2 3 4 Enter the command:
# haconf -makerw

Add the new system to the cluster:

# hasys -add east

Enter the following command:

# haconf -dump

Copy the main.cf file from an existing node to your new node:
# rcp /etc/VRTSvcs/conf/config/main.cf east:/etc/VRTSvcs/conf/ config/

5 6 7

Start VCS on the new node:

# hastart

If necessary, modify any new system attributes. Enter the command:

# haconf -dump -makero

Starting VCS and verifying the cluster

Start VCS after adding the new node to the cluster and verify the cluster. To start VCS and verify the cluster 1 2 From the new system, start VCS with the new system added to the cluster:
# hastart

Run the GAB configuration command on each node to verify that Port a and Port h include the new node in the membership:
# /sbin/gabconfig -a GAB Port Memberships =================================== Port a gen a3640003 membership 012 Port h gen fd570002 membership 012

10 Adding and removing cluster nodes Removing a node from a cluster

Removing a node from a cluster

Table 7-2 specifies the tasks involved in removing a node from a cluster. In the example procedure, the cluster consists of nodes A, B, and C; node C is to leave the cluster. Table 7-2 Task

Tasks involved in removing a node Reference

Verify the status of nodes and service groups on page 10

Back up the configuration file. Check the status of the nodes and the service groups.

Switch or remove any VCS service groups Deleting the leaving node from VCS configuration on page 11 on the node leaving the cluster. Delete the node from VCS configuration. Modifying configuration files on each remaining node on page 13 Unloading LLT and GAB and removing VCS on the leaving node on page 13

Modify the llthosts and gabtab files to reflect the change. On the node leaving the cluster:

Modify startup scripts for LLT, GAB, and VCS to allow reboot of the node without affecting the cluster. Unconfigure and unload the LLT and GAB utilities. Remove the VCS RPMs.

Verify the status of nodes and service groups

Start by issuing the following commands from one of the nodes to remain, node A or node B. To verify the status of the nodes and the service groups 1 Make a backup copy of the current configuration file, main.cf.
# cp -p /etc/VRTSvcs/conf/config/main.cf \ /etc/VRTSvcs/conf/config/main.cf.goodcopy

Check the status of the systems and the service groups.

# hastatus -summary --A A A SYSTEM STATE System State A RUNNING B RUNNING C RUNNING

Frozen 0 0 0

Adding and removing cluster nodes Removing a node from a cluster

11

Probed AutoDisabled State Y N ONLINE Y N OFFLINE Y N ONLINE Y N OFFLINE Y N ONLINE Y N ONLINE The example output from the hastatus command shows that nodes A, B,

--B B B B B B

GROUP STATE Group System grp1 A grp1 B grp2 A grp3 B grp3 C grp4 C

and C are the nodes in the cluster. Also, service group grp3 is configured to run on node B and node C, the leaving node. Service group grp4 runs only on node C. Service groups grp1 and grp2 do not run on node C.

Deleting the leaving node from VCS configuration

Before removing a node from the cluster, you must remove or switch from the leaving node the service groups on which other service groups depend. To remove or switch service groups from the leaving node 1 Switch failover service groups from the leaving node. You can switch grp3 from node C to node B.
# hagrp -switch grp3 -to B

Check for any dependencies involving any service groups that run on the leaving node; for example, grp4 runs only on the leaving node.
# hagrp -dep

If the service group on the leaving node requires other service groups, that is, if it is a parent to service groups on other nodes, then unlink the service groups.
# haconf -makerw # hagrp -unlink grp4 grp1

These commands enable you to edit the configuration and to remove the requirement grp4 has for grp1. 4 5 Stop VCS on the leaving node:
# hastop -sys C

Check the status again. The state of the leaving node should be EXITED. Also, any service groups set up for failover should be online on other nodes:
# hastatus -summary --A A A SYSTEM STATE System State A RUNNING B RUNNING C EXITED

Frozen 0 0 0

12 Adding and removing cluster nodes Removing a node from a cluster

--B B B B B B

GROUP STATE Group System grp1 A grp1 B grp2 A grp3 B grp3 C grp4 C

Probed Y Y Y Y Y Y

AutoDisabled N N N N Y N

State ONLINE OFFLINE ONLINE ONLINE OFFLINE OFFLINE

Delete the leaving node from the SystemList of service groups grp3 and grp4.
# hagrp -modify grp3 SystemList -delete C # hagrp -modify grp4 SystemList -delete C

For service groups that run only on the leaving node, delete the resources from the group before deleting the group.
# hagrp -resources grp4 processx_grp4 processy_grp4 # hares -delete processx_grp4 # hares -delete processy_grp4

8 9

Delete the service group configured to run on the leaving node.

# hagrp -delete grp4

Check the status.

# hastatus -summary -- SYSTEM STATE -- System State A A RUNNING A B RUNNING A C EXITED --B B B B GROUP STATE Group System grp1 A grp1 B grp2 A grp3 B

Frozen 0 0 0

Probed Y Y Y Y

AutoDisabled N N N N

State ONLINE OFFLINE ONLINE ONLINE

10 Delete the node from the cluster.

# hasys -delete C

11 Save the configuration, making it read only.

# haconf -dump -makero

Adding and removing cluster nodes Removing a node from a cluster

13

Modifying configuration files on each remaining node

Perform the following tasks on each of the remaining nodes of the cluster. To modify the configuration files on a remaining node 1 If necessary, modify the /etc/gabtab file. No change is required to this file if the /sbin/gabconfig command has only the argument -c, although Symantec recommends using the -nN option, where N is the number of cluster systems. If the command has the form /sbin/gabconfig -c -nN, where N is the number of cluster systems, then make sure that N is not greater than the actual number of nodes in the cluster, or GAB does not automatically seed. Note: Symantec does not recommend the use of the -c -x option for /sbin/ gabconfig. The Gigabit Ethernet controller does not support the use of -c -x. 2 Modify /etc/llthosts file on each remaining nodes to remove the entry of the leaving node. For example, change:
0 A 1 B 2 C

to:
0 A 1 B

Unloading LLT and GAB and removing VCS on the leaving node
Perform the tasks on the node leaving the cluster. To stop LLT and GAB and remove VCS 1 Stop GAB and LLT:
# /etc/init.d/gab stop # /etc/init.d/llt stop

2 3

To determine the RPMs to remove, enter:

# rpm -qa | grep VRTS

To permanently remove the VCS RPMs from the system, use the rpm -e command.
# # # # # rpm rpm rpm rpm rpm -e -e -e -e -e VRTScmccc VRTScmcs VRTScssim VRTScscm VRTSvcsdc

14 Adding and removing cluster nodes Removing a node from a cluster

# # # # # # # # # # # # # # # # # # # # # # #

rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm rpm

-e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e -e

VRTSvcsmn VRTScutil VRTSweb VRTScscw VRTSjre15 VRTSjre VRTSvcsdr VRTSvcsag VRTSacclib VRTSvcsmg VRTSvcs VRTSvxfen VRTSgab VRTSllt VRTSvlic VRTSspt VRTSsmf VRTSperl VRTSpbx VRTSicsco VRTSatServer VRTSatClient SYMClma

Remove the LLT and GAB configuration files.

# rm /etc/llttab # rm /etc/gabtab # rm /etc/llthosts