WPAR and Multibos

How to
By Ahmed Mashhour
EMEA Technical Lead
[email protected]

Contents:
Page

01. What Is WPAR .........................................

02. Methods of WPAR creation.....................
03. Checking and modifying WPAR ...............
04. WPAR Cloning and backup/restore..........
05. Wpar troubleshooting.............................
06. What Is Multibos.....................................
07. Multibos Creation and Configuration ......
08. Mulltibos Troubleshooting and DEBUG...

03
04
06
09
12
13
13
17

01. What Is WPAR:

Workload Partitions (WPARs) are virtualized operating system environments that are
Created within a single AIX image. While they may be self-contained in the sense that
each
WPAR has its own private execution environment with its own filesystems and network
Addresses, they still run inside the global environment.
The global environment -- the actual LPAR -- owns all the physical resources of the
Logical partition.
It is important to also note that the global environment can see all the processes running
Inside the specific WPARs.
WPAR has few types:
S => System workload partition
L => Versioned system workload partition (VWPAR)
A => Application workload partition
The re-reuisites to create WPARs are:
I)
AIX 6.1 as a Global Environment.
II)
The filesets:
bos.wpars => for WPAR binaries.
wio.common, wio.fcp, wio.vscsi => For WPAR I/O

02. Methods of WPAR creation:

Creating a WPAR can be done through multiple methods:

02.1. SMIT menu: # smitty wpar || # smitty mkwpar:

# smitty mkwpar
Create a System Workload Partition
Create a System Workload Partition (Advanced)
Create a Specification File from a System Workload Partition
Create a System Workload Partition from a Specification File
Each WPAR should have a unique name. The name GLOBAL is reserved surely
For the global environment.
* Workload Partition Name
HOSTNAME
Base Directory
Default Volume group name
Copy Global Name Resolution settings
Network INTERFACE
Internet ADDRESS (dotted decimal)
Network MASK
-ORIPv6 ADDRESS (colon delimited)
Prefix Length
WPAR-Specific Routing?
Create the Workload Partition?
Specification Output File
START Workload Partition?
Start at system boot?
Automatically RESOLVE conflicts? No
Architecture Compatibility

[]
[]
[]
[]
Yes

+
+
+

[]
[]
[]
[]
No
Yes
[]
No
No

+
+
/
+
+
+
+

02.2. Actual mkwpar command: # mkwpar:

For reference, The mkwpar can use the following options/flags:
-g Will create all file systems owned by wpar using the capacity of the indicated
Volume group .
-h Specifies the host name assigned to this wpar.
-l Create private and writable /opt and /usr .
-n Workload partition name .
-N Network info follows .
-P Set the root password interactively.
-r Duplicate network name resolution services (/etc/resolv.conf and so forth).
-s Start the wpar after creation .
-o Create wpar.config file that can be used to re-create the wpar.
-D [devname=device name] To allocate device when creating the WPAR.
Example:
# mkwpar -g wparvg -h wpar1 -l -n wpar1 -P -r -s -o /var/wpar1.config
The example will create a WPAR with a specific hostname and wpar name, Also with
A dedicated ethernet adapter configured with network details.
We also can create rootvg wpars , which is a system WPAR configured with its own
rootvg.
On one or more dedicated storage devices. Configuring a rootvg WPAR gives the WPAR
Admin complete control over managing:
The storage devices exported to the WPAR.
The VGs on those devices.
The LVs and file systems within those volume groups.
A system WPAR- which is not a rootvg WPAR - does not have its own root volume
Group, but it has file systems created in logical volumes created out of the rootvg
Of the global system.
For a rootvg WPAR, storage devices must be exported (or allocated) to the WPAR when
It is created. After it has been created, the chwpar command can be used to allocate
Additional disks to the WPAR's rootvg or, if it contains multiple disks, to deallocate
A disk from the WPARs root volume group.
It is not possible to change a WPAR without its own root volume group into a rootvg
WPAR after it has been created.

A configuration with a WPAR owned root volume group (a RootVG WPAR) helps to
Isolate the filesystems and volume groups of a WPAR from the global system.
Rootvg WPAR can be created using the below command:
# mkwpar -l -n wpar2 -o /etc/wpar2.config -D devname=hdisk3 rootvg=yes

03. Checking and modifying WPAR:

# lsvg -l wparvg
# lswpar
# startwpar [-v] wpar1
# stopwapar [-v] wpar1
# lswpar wpar1
# lswpar -L wpar1
# lswpar -t S testwpar
-t {[A][S][L][C]}: Filters the output based on workload partition types.
Name
State
Type
Hostname Directory
RootVG WPAR
------------------------------------------------------------------------------------------------------wpar1
D
S
wpar1
/wpars/wpar1
no

WPAR States:
-----------------D => Defined
L => Loaded
A => Active
F => Frozen
P => Paused
N => Maintenance
M => Moving
T => Transitional
B => Broken
E => Error

Logging into WPAR:

# clogin wpar1
# uname -W --> To know the wpar ID (Global lpar will have ID 0).
# exit (Log out from the wpar environment into global Lpar)

Stopping the WPAR:

# stopwpar testwpar (From the global Lpar).
# shutdown (From the wpar itself.)

Removing the WPAR:

# rmwpar wpar1 (It should be stopped firstly.)
# rmwpar -p wpar1
This will do a preservation removal.
(i.e: Will remove the WPAR from the database without removing its local filesystems).
Hence, it can be used when creating a new wpar using existing
Filesystems.
-p
Performs a preservation removal of this workload partitionworkload
Partition. Configured local file systems (logical volumes or
Subdirectories within pre-existing logical volumes) are not
Emptied or removed. This flag is for System wpars only.
File systems preserved using this flag can be used with "mkpwar -p"
To create a new workload partitionworkload partition attached to them.

Example:
# mkwpar -n NEW_WPAR_NAME -p OLD_WPAR_NAME -d
/wpars/NEW_WPAR_NAME
The entries in /etc/filesystems will get updated with the new mount points.

Changing wpar configuration:

- Add new IP address attribute:
# chwpar -N address=10.70.90.12 testwpar
This will add an alias IP to the default wpar ethernet.
- Delete the IP address attribute:
# chwpar -K -N address=10.70.90.12 testwpar
This will delete the IP address.
- Modify hostname attribute:

# chwpar -h NEW_HOSTNAME WPAR_NAME

Wpar information database(s):
1- /etc/corrals/index file
Contains name and ID information of wpars
Example :
1 : S : wpar1 : 1
Definition: WparID:Type:Kernel_CID_value
- Kernel CID value is the index of entry in the kernel
WPAR table used for this wpar. 0 means not not running,
Any other value means it's running.
2- /etc/corrals/<wpar_name>.cf
Contains wpar configuration
3- /etc/filesystems -> Of the global Lpar
Contains all information about all filesystems of the global Lpar and wpars inside.

Specification file usage:

# mkwpar -e existing_wpar_name -o /tmp/wpar1.spec -w
-w: Writes the specification file only and do not create the wpar.
- Specification file can then be used to create a workload partition at a later time using
The -f flag.
- To create a workload partition based on an existing specification file, enter the
Following command:
# mkwpar -f /tmp/wpar1.spec
Or,
# mkwpar -f /tmp/wpar1.spec -n NEW_WPAR_NAME

04. WPAR Cloning and backup/restore:

To clone a wpar:
# mkwpar -e EXISTING_WPAR -n NEW_WPAR
To create a rootvgwpar and clone it (like alt_disk_copy):
1) Create rootvg wpar on hdisk18, check the status and start it up:
# mkwpar -D devname=hdisk18 rootvg=yes -l -n rootvgwpar
# lswpar rootvgwpar
Name
State Type Hostname Directory
RootVG WPAR
-------------------------------------------------------------------rootvgwpar D S rootvgwpar /wpars/rootvgwpar yes
# lswpar -Br rootvgwpar
Name
Device Name Virtual Device RootVG Bootset
------------------------------------------------------------rootvgwpar hdisk18
hdisk0
yes 0
# startwpar rootvgwpar

2) Allocate a new disk hdisk17 to the rootvgwpar, and set the bootset on it
In the same step:
# chwpar -B devname=hdisk17 rootvgwpar
Creating a bootset for WPAR rootvgwpar. Please wait...
3) Check the bootset:
# lswpar -Br rootvgwpar
Name
Device Name Virtual Device RootVG Bootset
------------------------------------------------------------rootvgwpar hdisk17
hdisk1
no
1
rootvgwpar hdisk18
hdisk0
yes 0
# lswpar -b rootvgwpar
Name
Bootlist
------------------------rootvgwpar
4) Make sure the wpar will boot from hdisk17 which is the bootset "1":
# chwpar -b bootlist=1 rootvgwpar
# lswpar -b rootvgwpar
Name
Bootlist
--------------------rootvgwpar 1
# clogin rootvgwpar
# lspv
hdisk0
00050a8562a598e5
hdisk1
00050a85616632c3

rootvg
altvg.1

active

5) Reboot the rootvgwpar from global environment, and check the results:
# stopwpar rootvgwpar
# startwpar rootvgwpar
# lswpar -Br rootvgwpar
Name
Device Name Virtual Device RootVG Bootset
------------------------------------------------------------rootvgwpar hdisk17
hdisk0
yes 1
rootvgwpar hdisk18
hdisk1
no
0

# lswpar -D rootvgwpar | grep hdisk

rootvgwpar hdisk17
disk hdisk0
rootvgwpar hdisk18
disk hdisk1

yes
no

6) Login into the wpar and list the disks:

# clogin rootvgwpar
# lspv
hdisk0
00050a85616632c3
hdisk1
00050a8562a598e5

rootvg
altvg.0

EXPORTED
EXPORTED

active

7) From global environment, the old disk (hdisk18) can be removed,

But the bootset should be removed firstly:
# chwpar -K -B bootset=0 rootvgwpar
# chwpar -K -D devname=hdisk18 rootvgwpar
# cfgmgr
Backup & Restore:
# savewpar -f BACKUP_FILE/DEVICE_PATH wpar_name
- It invokes the command # mkwpardata
Which creates the default directory: /tmp/wpardata/wpar_name
- The directory has the following files:
filesystems image.data image.info wpar.spec
# restwpar -f BACKUP_FILE/DEVICE_PATH
- You can add the flags:
-n new_wpar -d /wpars/new_wpar

05. Wpar troubleshooting:

- Turn on WPAR debugging:
# export WPAR_SCREEN_LVL=D
# export WPAR_LOGFILE_LVL=D
- You can use the shell debug:
# sh -x /usr/sbin/wpar_command wpar_name >/var/adm/wpars/wpar.debug 2>&1
- Check for '-v' flag in a few wpar commands.
- Ask ask for the below files :
/var/adm/wpars/wpar.debug
/var/adm/wpars/event.log
/var/adm/ras/wpar.<wpar_name>.log
/etc/corrals/<wpar_name>.cf
/etc/corrals/index
/etc/filesystems

06. What Is Multibos:

Multibos is an AIX utility (Started in AIX 5.3 TL and higher) in which it allows as root
To create two separate instances of AIX in the same rootvg. Since it creates a standby
BOS that boots from another BLV.
The benefit is, you can install, update and maintain the standby BOS. Plus you can
Copy/Share the filesystems data.
Before attempting to run multibos, ensure there's a free space in the rootvg, and your
AIX version is AIX 5.3 TL or higher.
One more thing to bear in mind, the total number of copied Lvs cannot exceed 128.

07. Multibos Creation and Configuration:

To create a multibos instance, use the command:
# multibos -s
However, It'd be better if you put the above command in preview mode beforehand by
Adding -p option and also you may include -X to allow for automatic file system
Expansion If space is needed:
# multibos Xsp
Once you got Return Status = SUCCESS, then you are good to go without -p .
The log file for every multibos operation is:
/etc/multibos/logs/op.alog
The most important options you may use are:
-a = Update_all
-N = Block bosboot
-B = Bosboot operation
-R = Remove operation
-b = Install bundle file
-p = Preview only
-e = Exclude file
-S = Shell operation
-f = Fix List file
-s = Setup operation
-i = image.data file
-t = Block bootlist change
-u = Unmount operation
-X = Expand file systems as needed
-m = Mount operation
-M = Mksysb setup operation
-l = Install device or directory

Check the bootlist after creating the multibos:

# bootlist -om normal
hdisk0 blv=bos_hd5 pathid=0
hdisk0 blv=hd5 pathid=0
In verbose output, use:
# bootlist -om normal -v
'ibm,max-boot-devices' = 0x5
NVRAM variable: (bootdevice=/pci@80000002000000a/pci1014,0339@0/sas/disk@30000:4
/pci@80000002000000a/pci1014,0339@0/sas/disk@30000:2)
Path name: (/pci@80000002000000a/pci1014,0339@0/sas/disk@30000:4)
match_specific_info: ut=disk/sas/scsd
hdisk0 blv=bos_hd5 pathid=0
Path name: (/pci@80000002000000a/pci1014,0339@0/sas/disk@30000:2)
match_specific_info: ut=disk/sas/scsd
hdisk0 blv=hd5 pathid=0
Changing the boot instance order can be done using:
# bootlist m normal o hdisk0 blv=bos_hd5 hdisk0 blv=hd5
Use bootinfo -v To determine which instance you have booted from,
It will either return hd5 or bos_hd5.
Rebuild the standby BOS boot image (Similar function of bosboot for normal hd5
BLV):
# multibos XB
Mount/umount standby instance:
To mount a standby BOS
# multibos Xm
To unmount a standby BOS
# multibos Xu

Wake up ( get interactive shell with chroot access) the standby instance:
# multibos S
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt
+-----------------------------------------------------------------------------+
Multibos Root Shell
+-----------------------------------------------------------------------------+
Starting multibos root shell ...
Active boot logical volume is hd5.
Script command is started. The file is /etc/multibos/logs/scriptlog.151110132604.txt.
MULTIBOS>
So, it will give you MULTIBOS> prompt.
Installing a fileset and an efix:
# multibos -Xc -b /tmp/file.list -l <Filesets_Location>
file.list contains the filsets lists which you want to install, The format
Should be as below:
I: abc.xyz.rte
I:xxx.yyy.lib
The same command applies for efixes but with slightly different file.list format:
E:IZZZYYY.epkg.Z
E:IYXXXZZ.epkg.Z

Removing a fileset or efix:

You should only go into the interactive shell using 'multibos -S' then apply
The normal operations by installp, smitty remove and emgr -rL.

If you want to migrate your multibos, or create a multibos from another

mksysb image, use the below:
multibos M <mksysb_location> -sX
- Note: The mksysb image used to create multibos should be created using -M option
Added to mksysb command.
From mksysb manual page:
-M:
Creates a backup file that is intended for use with the multibos command. The -M flag
backs up the /, /usr, /var, and /opt file systems. Do not use the backup to reinstall
A system.
To remove a multibos instance, use:
# multibos RX
Sometimes, the normal removal of multibos fails with lots of errors, You may remove it
Manually following the below procedure:
# putlvcb -f '' hd5
# putlvcb -f '' hd4
# putlvcb -f '' hd2
# putlvcb -f '' hd9var
# putlvcb -f '' hd10opt
# chfs -a mb= /opt
# chfs -a mb= /var
# chfs -a mb= /usr
# chfs -a mb= /
# rmlv -f bos_hd5
This may return information message 0516-1246. That is expected.
# rmfs -r /bos_inst/opt
# rmfs -r /bos_inst/var
# rmfs -r /bos_inst/usr
# rmfs -r /bos_inst
# rmitab mbverify
# bosboot -ad /dev/ipldevice
#bootlist -m normal -o hdisk0 blv=hd5

08. Mulltibos Troubleshooting and DEBUG:

1- On an affected system, run the 'script' command to
Capture all output displayed to screen:
# script /tmp/multibos.out
2- Run the failing 'multibos' command with adding '-XD' flags to
The actual flags you use in the command. (-D option is not documented):
# multibos -...XD
3- After the 'multibos' finishes running, type "exit" to close the
Script session:
# exit
4- Bundle up the multibos directory:
# tar -cvf /tmp/multibos.dir.tar /etc/multibos/*
- Ask the customer to send you the files:
/etc/filesystems
/tmp/multibos.out
/tmp/multibos.dir.tar

Thank you.
Ahmed Mashhour
EMEA Technical Lead