Veritas Doctumentation & Troubleshooting techniques

Guides Cluster and Cluster File System Server Provisioning 4.2 (Formerly OpForce) OpForce 4.0 4.1 Veritas vxconfigd fix Install Veritas Veritas Disk Commands Veritas cluster commands and files Processes OpForce/Server Provisioning Restore files Sending Crash info to Veritas CFS Commands Troubleshooting

Guides
Veritas File System

Need to add pdfs for vxfs

Cluster and Cluster File System

Veritas Storage Foundation Cluster File System 4.1 Install and Admin Guide Veritas Storage Foundation Cluster File System 4.0 Install and Admin Guide Veritas Cluster Server 4.0 Bundled Agents Reference Guide Veritas Cluster Server 4.0 User's Guide

Server Provisioning 4.2 (Formerly OpForce)

Release Notes Getting Started Guide Install Guide Provisioning Guide System Administration Server Management Software Management Troubleshooting Guide

OpForce 4.0 4.1

Opforce Install guide Opforce release notes Opforce User guide Opforce Troubleshooting

Veritas vxconfigd fix

This should only be installed after Veritas 4.1 and MP1. 1.Download the following files and copy to /tmp, also chmod 700 libvxscsi (a) Veritas Hitachi driver (b) Veritas scsi lib file (c) Script to install above files 2.Run the /tmp/libvxscsi file. 3.Reboot server.

Install Veritas
To install Veritas software on your servers follow these steps

1. Setup ssh on your servers, refer to ssh setup 2. Setup the network so install doesn't fail.

You will have to do this on all of your servers. This only applies to SLES, not Redhat. If you have any typos or miss any ' you will have to redo your configure Run yast and configure your heartbeat cards to at least do dhcp. Once that is done go into a shell and ifconfig eth#, note the mac address of the cards. vi the /etc/sysconfig/network/ifcfg-eth-file example:

vi /etc/sysconfig/network/ifcfg-eth-id-00:11:0a:5b:f1:4c BOOTPROTO='dhcp' MTU='' REMOTE_IPADDR='' STARTMODE='onboot' UNIQUE='LRaa.NhO9mtEJcG8' _nm_name='bus-pci-0000:07:0a.0'

Make the above file look like this

BOOTPROTO='static' MTU='' REMOTE_IPADDR='' STARTMODE='onboot' UNIQUE='LRaa.NhO9mtEJcG8' _nm_name='bus-pci-0000:07:0a.0' PERSISTENT_NAME='eth#'

Change the eth# to match your port. Later in the install when it asks for your heartbeat ports and you enter in eth3.You will see this

Enter the NIC for the first private heartbeat link on server: [b,?] eth3 eth3 is probably active as a public NIC on server Are you sure you want to use eth3 for the first private heartbeat link? [y,n,q,b,?] (n) y 3. If you have copied the iso file from this server do a mount -o loop /locationofiso /mountpoint 4. cd in to the /mountpoint/sles9_x86_64 5. run the ./installer - it must be run from this location 6. Select the Veritas software you are installing, normally 1 or 4 7 Enter the names of the servers with spaces 8 Enter license key Once these steps are all done your Veritas configure will probably crash with a /dev/llt error. At which point then it is time to install Veritas Maintence Pack 1. 1 Copy the veritas mp1 file from our Downloads to your server 2 On the server do a tar -xvf veritasmp1.tar 3 Cd into /mp1 4 Type ./installmp 5 Insert the server names 6 After install reboot the server
If you are still having any kind of issue after all of this try to cd to /opt/VRTS/install and do a ./installsf or ./installsfcfs -configure. The program you run will be determined by your software license. This will rerun the configuration daemon for the Veritas and should fix any errors after everything above this is completed.

Veritas Disk Commands

One minor note from the issues we have encountered over the last 24 hours. For our NGPR box we needed to recover the disk, we need to find out how long a 100GB+ disk with at least that much data takes to recover.

The enabled state is the normal operating state. Most configuration operations are allowed in this state. Entering the enabled state imports all disk groups, and begins the management of device nodes stored in the /dev/vx/dsk and /dev/vx/rdsk directories.

vxdctl enable

Displays the cvm state (master/slave)

vxdctl -c mode

Displays what disks Veritas sees and there current state

vxdisk list

This lists all disk groups on current server

vxdisk -o alldgs list

Add or initialize disks. Under selection 1 you will be asked for a diskgroup

vxdiskadm

Similar to vxdisk list

vxdg list

Shows available bytes

vxdg free

Creates in the diskgroup a volume at whatever size vxassist is used for converting and other useful stuff try man vxassist

vxassist -g diskgroup make volume volumesize

Removed the DCO log partition so the volume the can be resized. Try this if the vxresize command fails.

vxsnap -g (disk group) unprepare (volume name)

Only works for cfs/ha/cvm /

/usr/lib/vxvm/bin/vxclustadm nidmap

Clear the failing flag from the Disk Group so that it can be used. Only do this if you are sure there is no hardware issue.

vxedit -g (diskgroup) set failing=off (disk name)

Veritas cluster commands and files

This is one of the config files needed for llt

/etc/llttab

This is the cmnodelist

/etc/llthosts

This file specifies how many nodes are needed to form a cluster and the timeout value for the cluster

/etc/gabtab

These are the cluster config files

/etc/VRTSvcs/conf/config

This command shows you what servers are in the cluster and there heartbeat connections

lltstat -nvv |more

which servers are in the cluster

hasys -list

The current state of the cluster

hasys -state

Looks at status of all servers in the cluster (pipe this to a file, very long)

hasys -display

Provides summary of cluster and package status (like cmviewcl)

hastatus -sum

which server are in the group

hagrp -list

current state of the group in the cluster

hagrp -state

looks at status of groups in cluster (can be very long)

hagrp -display

Processes
For Clustering

/opt/VRTSvcs/bin/had /opt/VRTSvcs/bin/hashadow /opt/VRTSvcs/bin/CmdServer /opt/VRTSvcs/bin/CVMCluster/CVMClusterAgent -type CVMCluster /opt/VRTSvcs/bin/CFSfsckd/CFSfsckdAgent -type CFSfsckd /opt/VRTSvcs/bin/CVMVxconfigd/CVMVxconfigdAgent -type CVMVxconfigd lltd lltdlv lsmod |grep gab (should return a line) lsmod |grep llt (should return a line)
For volume

vxiod's vxconfigd -x syslog vxfs_thread /usr/lib/vxvm/bin/vxrelocd root /usr/lib/vxvm/bin/vxcached root /usr/lib/vxvm/bin/vxconfigbackupd vxnotify's

OpForce/Server Provisioning
Login to the Server Provisioning server at http://wwhq230m.whq.ual.com.

Restore files
/usr/openv/netbackup/bin/bp

Sending Crash info to Veritas

Use the following Binary to gather the info that Vertias needs to analyse a problem. The VRTSexploer Install

Install the binary with the commands

zcat VRTSspt-linux.tar.Z | tar xf rpm -i ./VRTSspt-linux/VRTSspt-*.i386.rpm

and run with the following.

/opt/VRTSspt/VRTSexplorer/VRTSexplorer

when asked to start vxconfigd leave default (n) Send the information to the following ftp site. ftp ftp.veritas.com login: anonymous passwd: (your email address) ftp> cd /incoming ftp> bin ftp> put (VRTSexplorer file)

CFS Commands
To add new Nodes to a CFS Cluster (CVM Service Group)

Add the nodes to the cluster via editing the llttab, llthosts, gabtab files. Verify the Volume Manager has been configured correctly via the ''vxinstall'' command. Add the new Node to the cvm SystemList and AutoStartList. Add the new Node to the cvm_clus (CVMNodeID) Resource The nodes list must match that in the llthosts file On each of the existing nodes in the cluster: Run the following command to rescan for the new nodes
/etc/vx/bin/vxclustadm -m vcs -t gab reinit

Run the following command to verify that the new nodes were found. This should be in the ''Out of the Cluster'' State
/etc/vx/bin/vxclustadm nidmap

On the new node in the cluster: Run the following command to join it into the CVM communication
/etc/vx/bin/vxclustadm -m vcs -t gab startnode /etc/vx/bin/vxclustadm nidmap

After run the following command to verify that the node has changed the ''Joined'' state. To Add Mount Points to new Nodes in the cluster To modify the Diskgroup info cfsdgadm modify (disk group) (node name)=sw To modify the mount point info cfsmntadm modify (mount point) add (node name)=suid,rw

To bring the mount point online cfsmount (mount point) (node name)

Troubleshooting
VEA won't run? /etc/init.d/isisd start