AIX Network Install Manager

When we replaced our old hardware platform with IBM AIX RS/6000 systems, a main concern was controlling how our new systems would be installed. With the old platform, we ran into several instances where the software (moving from our development group to QA and into production) was not the same, which caused numerous build breaks. We chose the AIX Network Installation Management (NIM) setup, because we needed to ensure consistent, standard OS installations between systems within our four separate data centers. Our company also has a security rule that doesn't allow any one machine access to all networks within our organization. Our four separate data centers have four separate networks with no connection between them, and each network is segmented into either inside or outside the firewall.

NIM Resources
NIM allows you to customize installations and maintain clients on the network from a centralized location (the NIM master) or the NIM client itself. The master contains the NIM database and can serve resources. Resources in NIM are files or directories containing data that NIM will use to install, customize, and maintain NIM clients. A NIM client is any machine configured and defined in the NIM database. Some key NIM resources used in our setup are: Licensed Program Product Source Directory (lpp_source): This directory contains backup file format (BFF) images, which AIX installp uses to load software. One way to understand the role of the lpp_source directory in a BOS installation is to compare it to all the installation images needed to support any configuration (specifically different device configurations) along with a base core set of software (called simages) that are on the BASE installation CDs. We created a base 433 lpp_source, multiple lpp_sources containing different maintenance levels, and separate lpp_sources for our 32-bit and 64-bit third-party application software. Shared Product Object Tree (SPOT): This directory is created from an lpp_source and is equivalent in content to a /usr file-system on AIX. The purpose of a SPOT in a NIM installation is similar to the boot images and BOS installation scripts (bi_main, rc.boot, and rc.bosinst) on volume 1 of the BASE install CD. The SPOT must contain support for all boot environments (platform, network type, kernel type). We created several different SPOTs for the different data centers and maintenance levels we use to support our systems. bosinst_data: This data file contains information that drives the BOS install (e.g., prompt vs. no-prompt, which disk to install the OS on, and the type of installation (Overwrite, Preservation, or Migration) to name a few). First, we created separate bosinst_data resources for each machine type (S80, H70, B50, M80, P680, and 43P). Then, by specifying two disks to target in our bosinst_data resource and specifying copies in the image_data resource, we could set up mirroring during the initial load. image_data: This data file contains information about the characteristics of the OS being installed. For example, it includes the size of file systems, whether or not to mirror, and whether or not to disk stripe. We created

separate image_data resources for each machine type (S80, H70, B50, M80, P680 and 43P). Script: This is a customization file the systems administrator can create, which executes near the end of the BOS installation. You can execute multiple NIM script resources during a BOS installation. If you need to control the order of execution, I recommend putting it all in one large script. We created one NIM script to set up the OS environment and hardware parameters, configure certain system files, set up security, install third-party software, create and set up dump devices, customize paging space to correspond to the system RAM, and help create a complete inventory by machine type and serial number. I've included a copy of this script at the end of this article (Listing 1). All listings for this article are available from the Sys Admin Web site: http://www.sysadminmag.com/code/ Installp_bundles: This data file contains a customized list of additional software to install after the base AIX software is loaded. If you have different configurations that you need to duplicate on a repeatable basis, this resource is very useful. In our environment, we have different OS software requirements for development, QA, and production above the minimal AIX software needed to support different hardware systems. The easiest way to facilitate and maintain these different requirements, which need to be consistent, is to use installp_bundles. mksysb: This is a backup archive file that contains a system image of rootvg. Because of our network security restrictions (no one machine could be connected to all the networks within our organization), we used mksysb and savevg tapes to replicate the NIM master to the other data centers. If we had one machine connected to the different data centers, we could have used NIM to replicate and update the NIM masters in the different data centers by BOSinstalling a NIM mksysb resource and using a NIM script to restore the other volume group data. mac_group: This is a logical grouping of machine types (standalone, diskless, or dataless) that enables the systems administrator to target one or more machines with a single command or NIM operation. We did not use this feature, but we could have taken advantage of this by grouping all like systems and like configurations to install to more than one machine at a time.

Working with NIM

We used the 43P systems as our NIM masters for each data center because they could complete remote installations of machines or be moved and directly connected to a server for OS installations. These NIM masters were also designated as the resource servers in our environment. To ensure consistency and standardization of each NIM master (for the different data centers), we created a standard NIM master machine, which we cloned. We made a stacked tape containing a mksysb image and a savevg image of the standard NIM master to sync up and update the other NIM masters. Here are the commands we ran on the standard NIM master to create this stacked single tape:

1. 2. 3. 4.

mksysb -i /dev/rmt0 tctl -f/dev/rmt0.1 fsf4 savevg -i -m {volume_group_name} -f/dev/rmt0.1 mt -f/dev/rmt0 rewind

To restore the tape to the other NIM masters, we did the following:

1. Booted and restored the mksysb image from the stacked tape 2. tctl -f/dev/rmt0.1 fsf4 3. restvg volume_group_name
These NIM servers were placed throughout our environment and are used by multiple operators with different training levels. Thus, we developed a strict directory tree for storage of resources on the NIM server, but this was our own restriction, not a restriction created by NIM. I recommend creating either one large separate filesystem or a few file systems that will contain your NIM resource files. Also, use naming conventions that make sense to you in terms of the resource name and directory structure. The following is a breakdown of our directory structure and a description of what is contained in the directory: Master directory of bundle lists: /export/installp_bundles Master directory of all NIM resources: /export/nim lpp sources for use by the NIM server during SPOT generation and system installation: Installations/export/nim/aix433 32-bit and 64-bit compiled third-party applications: /export/nim/aix433/32bit-lpp and /export/nim/aix433/64bit-lpp Master bosinst.data files for each server type: /export/nim/bosinst Master image.data files for each server type: /export/nim/imagedata In-house customization scripts: /export/nim/scripts Master OS system files: /export/nim/system

Flat file inventories by serial number of each system installed by the NIM server: /export/nim/system/{system type}/{serial number}

Master directories of additional special filesets: /export/nim/packages

Any mksysb file images of servers: /export/nim/mksysb

Master installable SPOT images: /export/nim/spot

NIM custom scripts: /export/nim/custom

Manual backups of the NIM configurations: /export/nim/nim-backup

We used SPOT copies (customized to contain only the minimal OS software) to complete the initial loads of our NIM client systems to support our six different hardware platforms. To build a SPOT, we first loaded all AIX 4.3.3.0 filesets into a directory called raw-lpp_source. This directory essentially contained all volumes of the BASE install CDs, bonus pack CDs, and base and extended documentation CDs for 4.3.3. We also had an equivalent raw-lpp_source for each maintenance level we needed to support. We made sure that every fix for that maintenance level was present. Once completed, we generated a listing of lpp's into a bundle format (an installp_bundle resource that contained our customized minimal OS software) to create the AIX433-base lpp_source. We then created an MLXX lpp_source containing only the maintenance level we approved for our standard. This was customized similar to the AIX433-base lpp_source except it contained only the ML fixes. After our customized SPOTs were created, we were ready to install a system from this base lpp_source with any ML level. During the installation process, we used the bosinst.data and image.data files to control the layout of the rootvg onto the hard disks of the server. Using NIM has been facilitated by establishing strict hardware configuration standards for each each IBM server type at our site. This provides the ability to move machines around without conducting a hardware upgrade. It also makes installation easy, because we always know what kind of hardware we are using.

Customizing

After the installation of the OS was complete, we had the NIM server automatically run a customization script to enforce more unique standards within our computing environment. This customization script (Listing 1) was written, in-house, to facilitate turning on and off selected functions. It uses these functions to control the flow of each operational section. Each command used was echoed to the NIM console log along with a description to aid in diagnostics in case something went wrong during the NIM installation. By defining this script as a NIM resource, NIM is able to perform the administrative tasks discussed below. Note that this script is designed to meet the objective of a system being 95% pre-configured for our operating environment after NIM has completed its operations. The other 5% is what our operations staff performs on the server to turn it into a Web server, ftp server, gateway, or even a database server. Also note that this script is good for teaching new AIX systems administrators about essential system commands in AIX. Each command executed is echoed to the screen with a description of what it's doing, it's command syntax, and it's output.

Set Up OS Environment Parameters

During this phase of the script, we adjust AIX to meet our standards for items such as: Timezone Remote boot facility Network Options Starting certain demons on restart

Setup Hardware Parameters AIX will set the 10/100 NIC cards to "autonegotiate". We found this caused a network bottleneck in our infrastructure. We utilize these commands to force the adapter to 100/full duplex on startup of the server. Configure System Files We have created a master directory of certain system files that are copied onto each system when a NIM installation takes place. Utilizing this repository allows control of what the initial system files will conform to. This is also the first time in the script that you see special programming for a particular platform. Note at the end of this function that we do not want the "httplite docsearch" utility running on anything other than a 43P workstation. Setup Security Here we utilize the NIM server to create another script that will only run once on bootup and then remove itself, the /etc/firstboot. By placing commands in the /etc/firstboot script, the NIM-installed client will execute these commands and then remove the script after it gets an OS installed. This allows us to lock a system down to our security standards immediately. We do this after NIM is finished with the server, because you will lose connectivity to the NIM server and your install will hang if you run these commands before NIM has completed.

Install Third-Party Applications Here we have created our own "BFF"-formatted install packages for 32-bit and 64-bit compiled software. We keep these in separate directories and use this script to determine whether we are installing a 32-bit or 64-bit machine, and then to install the correct versions. Create and Set Dump Devices Depending on the configuration of an AIX server, the initial dump space defined may not be big enough. The two main parts of this function are to calculate whether the dump space is big enough and to determine whether a mirror drive exists. If the function determines the dump space is too small, it will increase it automatically and set up a secondary dump file on the mirror drive if available. Adjust Paging Space to Complement RAM Because AIX has changed its swap algorithms, it is no longer necessary to have swap space equal to size of RAM. (This really helps on a 64-Gig RAM system.) We use this function to calculate the size of RAM and then adjust the swap space to the values noted in the script. Create a Hardware Inventory -- By Machine Type or by Serial Number This function calls another script that will create an exhaustive inventory of a NIM installed server (Listing 2). We use this to verify a server install or for disaster recovery of a down server. We could literally rebuild a like server from this listing. Create a Complete Inventory -- By Machine Type or by Serial Number The purpose of this function is to call a smaller system inventory script (Listing 3) that will only give the amount of information about each NIM-installed server for our inventory-tracking database.

Conclusion
In summary, we needed to ensure consistent standard OS installations between four separate data centers. By using AIX 43P systems and NIM system software, we succeeded in our efforts. NIM allowed for remote installation and management of our networked systems. Specifically, the NIM script resource handled tedious and timeconsuming tasks that cause systems administrators to dread post-install configuration. From setting up hardware parameters to creating inventory lists, NIM allowed complete customization while leaving us free to handle other vexing tasks...like resetting passwords.