IBM VIOS Maintenance

7/22/2014
FORSYTHE TALKS
VIOS MAINTENANCE
Jaqui Lynch
[email protected]
This document and the material contained herein is confidential and

proprietary and intended solely for the use and information of the client or
customer to whom it is addressed. It should not be copied, disseminated,
or used in any other manner without the prior written consent of Forsythe
AGENDA
Best Practices Setup
Network
Installation
Maintenance and Upgrades
Backup and recovery
Monitoring
FBO
Other
Wrap-up/Questions
7/22/2014
BEST PRACTICES SETUP
STAY CURRENT
VIOS Lifecycle
Version
GA
1.5
11/07
2.1
11/08
2.2.00
9/10
2.2.1
10/11
2.2.2
10/12
2.2.3
4Q13
2.2.4
2Q15
Latest release:
2.2.3.3 FP27 SP03
EOM
2008
2010
2011
10/12
10/13
EOS/EOL
09/11
09/12
09/13
04/15
09/16
Basically AIX 6100-09-03 Requires NIM at 6100-09-03 or 7100-03-03
Can be applied to 2.2.3.0, 2.2.3.1 or 2.2.3.2
Download updates from Fix Central:

http://www-933.ibm.com/support/fixcentral/
Download base from entitled software:

https://www-05.ibm.com/servers/eserver/ess/ProtectedServlet.wss
4
7/22/2014
USE FLRT AND CHECK PREREQS

FLRT Home Page:
http://www14.software.ibm.com/webapp/set2/flrt/home
VIOS to NIM Master Mapping:

http://www14.software.ibm.com/webapp/set2/sas/f/flrt/viostable.html
System Software Maps for VIOS:

http://www-01.ibm.com/support/docview.wss?uid=ssm1platformvios
AIX/VIOS Security Tables:

http://www14.software.ibm.com/webapp/set2/sas/f/flrt3/Sec_APARs.html
VIOS Hiper Tables:

http://www14.software.ibm.com/webapp/set2/flrt/doc?page=hiper#vios_hiper
Also check MPIO driver versions as there are specific requirements for each VIO release
5
GENERAL
Keep it simple
Ensure LMB is the same on all servers if you want to use LPM
Use hot pluggable adapters rather than built in ones
Use dual VIO to allow for concurrent updates

All adapters should be desired, not required
Dont mix multipath drivers / HBAs
Backup regularly use NIM or scripts
Run HMC Scanner and/or Sysplan before and after all changes
Plan for at least one update per year (IBM normally puts out 2)
Separate VIOs for production and non prod on large systems
Test failover (SEA failover and disk if vio goes down)
Use VIO commands wherever possible rather than going into oem_setup_env
mirror vio rootvg
Check errpt regularly
NEVER run at 100% entitlement ensure it is high enough and there are
plenty of VPs and memory
Run VIOS Advisor regularly
NOTE v2 requires at LEAST 30GB in rootvg
Easier maintenance
7/22/2014
PAGING
Clean up paging
By default VIO has a 512MB hd6 and a 1.5GB paging00 on the same LUN
Swapoff paging00 and delete it
Increase hd6 to 4GB
Reboot
Change from:
lsps -a
Page Space
paging00
hd6
Physical Volume Volume Group Size

%Used Active Auto Type Chksum
hdisk0
rootvg
1024MB 18
yes
yes lv 0
hdisk0
rootvg
512MB 35
yes
yes lv 0
TO:
# lsps -a
Page Space
hd6
Physical Volume Volume Group Size

%Used Active Auto Type Chksum
hdisk3
rootvg
4096MB 73
yes
yes lv 0
SETUP DUMP DEVICES AND LOGGING

Set up dump devices
sysdumpdev e
to get correct size
mklv y lv_dumplv1 rootvg 32 hdisk3
mklv y lv_dumplv2 rootvg 32 hdisk2
sysdumpdev P p /dev/lv_dumplv1
sysdumpdev P s /dev/lv_dumplv2
rmlv lg_dumplv
Logging
vi /etc/syslog.conf
user.info
user.debug
mail.debug
*.emerg
*.alert
*.crit
*.err
auth.notice
*.info
/var/adm/lpm/lpm.log rotate size 10m files 2 compress

/var/adm/lpm/lpm.dbg rotate size 20m files 8 compress
/usr/local/logs/mailog
/usr/local/logs/syslog
/usr/local/logs/messages
cd /usr/local/logs
touch syslog mailog infolog messages
stopsrc -s syslogd
startsrc -s syslogd
Try not to log to /var as bad things happen if it fills up
Make /usr/local/logs a new filesystem
7/22/2014
TUNING
Make the same tuning changes you would make on AIX
Set num_cmd_elems and max_xfer_size on the fiber adapters on VIO

chdev -l fcs0 -a max_xfer_size=0x200000 -a num_cmd_elems=1024 -P
If NPIV also set on clients

Client setting cannot be higher than the VIOs
Pay attention to adapter layout and priorities
ADAPTER TUNING
fcs0
bus_intr_lvl
bus_io_addr
115
0xdfc00
Bus interrupt level

Bus I/O address
bus_mem_addr 0xe8040000
Bus memory address
False
False
False
init_link
al
intr_priority
lg_term_dma
3
Interrupt priority
Long term DMA
True
False
max_xfer_size
0x100000
Maximum Transfer Size
True
(16MB DMA)
num_cmd_elems
200
Maximum number of COMMANDS to queue to the adapter True
pref_alpa
sw_fc_class
0x1
2
Preferred AL_PA
FC Class for Fabric
True
True
True
0x800000
INIT Link flags
True
Changes I often make (test first)

max_xfer_size
0x200000
128MB DMA area for data I/O
num_cmd_elems 1024
Maximum number of COMMANDS to queue to the adapter True
Often I raise this to 2048 check with your disk vendor

lg_term_dma is the DMA area for control I/O
Check these are ok with your disk vendor!!!
At AIX 6.1 TL2 VFCs will always use a 128MB DMA memory area even with default max_xfer_size
Remember make changes too both VIO servers and client LPARs if using NPIV
VIO server setting must be at least as large as the client setting
See Dan Braden Techdoc for more on tuning these:
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105745
10
7/22/2014
MY VIO SERVER AND NPIV CLIENT ADAPTER SETTINGS
VIO SERVER
#lsattr -El fcs0
lg_term_dma
max_xfer_size
num_cmd_elems
the adapter True
0x800000
0x200000
2048
Long term DMA

True
True
Maximum number of COMMANDS to queue to
NPIV Client (running at defaults before changes)

#lsattr -El fcs0
lg_term_dma
0x800000
Long term DMA
True
max_xfer_size
0x200000
True
num_cmd_elems
2048
Maximum Number of COMMAND Elements True
NOTE NPIV client must be <= to settings on VIO
11
ADAPTER PRIORITIES AFFECT PERFORMANCE
Check the various Technical Overview Redbooks at http://www.redbooks.ibm.com/
12
7/22/2014
MEMORY PLANNING
13
VIRTUAL ETHERNET
Link aggregation
Put vio1 aggregate on a different switch to vio2 aggregate
Provides redundancy without having to use NIB
Allows full bandwidth and less network traffic (NIB is pingy)
Basically SEA failover with full redundancy and bandwidth
Pay attention to entitlement

VE performance scales by entitlement not VPs
If VIOS only handling network then disable network threading on the
virtual Ethernet
chdev dev ent? thread=0
Turn on large send on VE adapters

chdev dev ent? attr large_send=yes
Turn on large send on the SEA

chdev dev entx attr largesend=1
14
7/22/2014
NETWORK
15
STARTER
SET OF TUNABLES
NETWORK
Typically we set the following:

NETWORK
no -p -o rfc1323=1
no -p -o tcp_sendspace=262144
no -p -o tcp_recvspace=262144
no -p -o udp_sendspace=65536
no -p -o udp_recvspace=655360
Also check the actual NIC interfaces and make sure they are set to at least
these values
You cant set udp_sendspace > 65536 as IP has an upper limit of 65536 bytes
per packet
Check sb_max is at least 1040000 increase as needed
16
7/22/2014
MY VIO SERVER SEA
# ifconfig -a
en6:
flags=1e080863,580<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTIC
AST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>
inet 192.168.2.5 netmask 0xffffff00 broadcast 192.168.2.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
lo0:
flags=e08084b,1c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST
,GROUPRT,64BIT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/0
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
17
VALID ADAPTERS
FOR
P7 AND P7+
770
Multifunction Cards up to one per CEC

1768 Integrated Multifunction Card with Copper SFP+ - Dual 10Gb copper and dual 10/100/1000MB copper ethernet
1769 Integrated Multifunction Card with SR Optical - Dual 10Gb optical and dual 10/100/1000MB copper ethernet
PCIE Adapters
5284/5287 PCIE2 2 port 10GbE SR (5284 is low profile)

5286/5288 PCIE2 2 port 10GbE SFP+ Copper (5286 is low profile)
5769 PCIE1.1 1 port 10GbE SR
5772 PCIE1.1 1 port 10GbE LR
EC27/EC28 PCIE2 2 port 10GbE RoCE SFP+ (EC27 is low profile)
EC29/EC30 PCIE2 2 port 10Gbe RoCE SR (EC29 is low profile)
5708 PCIE 2 port 10Gb FCoE converged network adapter
Basically SR is fibre and SFP+ is copper twinax

If using SFP+ IBM only supports their own cables they come in 1m, 3m and 5m and are
10GbE SFP+ active twinax cables
Use the PCIE2 cards wherever possible
RoCE Supports the InfiniBand trade association (IBTA) standard for remote direct memory
access (RDMA) over converged Ethernet (RoCE)
More information on adapters at:
http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/topic/p7hcd/pcibyfeature_77x_78x.htm
18
7/22/2014
POWER8 - S814 & S824 LAN ADAPTERS

eConfig: Adapters => LAN
NIM not supported

NIM not supported
NIM not supported
NIM not supported
eConfig: Adapters => HiPerf/RIO
You can ignore the FCoE support on this

adapter a simply use the 10Gb ports for
Ethernet. This adapter supports NIM.
2 ports of 10GbE + (2) ports 1GbE
NIM = Network Install Manager. Used for installing AIX or VIOS over the network instead of DVD drive.
19
NETWORK PERFORMANCE AND THROUGHPUT
Depends on:
Available CPU power

MTU size
Distance between receiver and sender
Offloading features
Coalescing and aggregation features
TCP configuration
Firmware on adapters and server
Ensuring all known efixes are on for 10GbE issues
20
10
7/22/2014
NOTES ON 10GBE
Using jumbo frames better allows you to use the full bandwidth coordinate with
network team first
Jumbo frames means an MTU size of 9000
Reduces CPU time needed to forward packets larger than 1500 bytes
Has no impact on packets smaller than 1500 bytes
Must be implemented end to end including virtual Ethernet, SEAs, etherchannels, physical adapters,
switches, core switches and routers and even firewalls or you will find they fragment your packets
Throughput can improve by as much as 3X on a virtual ethernet
Manage expectations
Going from 1GbE to 10GbE does not mean 10x performance

You will need new cables
You will use more CPU and memory
Network traffic gets buffered

This applies to the SEA in the VIOS
Check that the switch can handle all the ports running at 10Gb
Make sure the server actually has enough gas to deliver the data to the network
at 10Gb
21
10GBE TIPS
Use flow control everywhere this stops the need for retransmissions
Need to turn it on at the network switch,

Turn it on for the adapter in the server
chdev l ent? a flow_cntrl=yes
If you need significant bandwidth then dedicate the adapter to the LPAR
There are ways to still make LPM work using scripts to temporarily remove the adapter
TCP Offload settings largesend and large_receive
These improve throughput through the TCP stack
Set largesend on (TCP segmentation offload) should be enabled by default on a 10GbE SR

adapter
AIX - chdev l en? a largesend=on

On vio chdev dev ent? attr largesend=1
With AIX v7 tl1 or v6 tl7 chdev l en? l mtu_bypass=on
Try setting large_receive on as well (TCP segment aggregation)
AIX - chdev l en? a large_receive=on

VIO chdev dev ent? attr large_receive=1
If you set large_receive on the SEA the AIX LPARs will inherit the setting
Consider increasing the MTU size (talk to the network team first) this increases the size of the
actual packets
chdev l en? mtu=65535 (9000 is what we refer to as jumbo frames)

This reduces traffic and CPU overhead
If you use ifconfig to make the changes it does not update ODM so the change does not survive
a reboot
22
11
7/22/2014
10GBE TIPS
Low CPU entitlement or too few VPs will impact network performance
It takes CPU to build those packets
Consider using netperf to test

Network speed between two LPARs on the same box is limited to the virtual Ethernet Speed
which is about 0.5 to 1.5 Gb/s
The speed between two LPARs where on is on the SEA and the other is external is the lower of
the virtual Ethernet speed above or the speed of the physical network
But all VMs on a server can be sending and receiving at the virtual ethernet speed concurrently
If 10Gb network check out Gareths Webinar
https://www.ibm.com/developerworks/wikis/download/attachments/153124943/7_PowerVM_10Gbit_Ethernet.pdf?ver
sion=1
23
OTHER NETWORK
SEA failover performance can be improved by setting:
portfast trunk on the switch ports
And by activating:
dead gateway detection with a ping time of 2 seconds
Also look at:
enabling bridge protocol data unit (BPDU) guard on switch
ports to protect against spanning tree loops
24
12
7/22/2014
VIO 2.2.3 SEA CHANGES

Traditional SEA setup
Ent4 is the virtual adapter defined at the HMC with external access
(SEA goes here)
VIO1 is priority 1 and VIO2 is priority 2
Ent5 is the virtual adapter on Vlan 1 with no external
(IP will go here)
Ent6 is the control channel on vlan 255
mkvdev sea ent0 vadapter ent4 default ent4 defaultid 1 attr ha_mode=auto ctl_chan=ent6
Creates ent7 as the SEA
Do not mess up priorities or ctl_chan or you will cause a spanning tree loop
Update with 2.2.3
See chapter 4 of SG248198- Redbook on 2.2.3 Enhancements
SEA setup has been simplified
Requirement removed for dedicated control channel and VLAN ID for each SEA failover configuration
Multiple SEA pairs can now share VLAN 4095 within the same virtual switch and no ctl_chan is needed
HMC (>= 7.8) reserves 4095 for internal management traffic
Requires VIOS 2.2.3, HMC 7.7.8 and firmware 780 or higher
25
INSTALLATION
26
13
7/22/2014
VIOS AND NIM

Use of NIM to back up, install, and update the VIOS is supported.
Note: For install, always create the SPOT resource directly from the VIOS mksysb
image. Do NOT update the SPOT from an LPP_SOURCE.
Use of NIM to update the VIOS is supported as follows:
Ensure that the NIM Master is at the appropriate level to support the VIOS image.
http://www14.software.ibm.com/webapp/set2/sas/f/flrt/viostable.html
On the NIM Master, use the operation updateios to update the VIOS Server.
"nim o updateios a lpp_source=lpp_source1 ... ... ..."
On the NIM Master, use the operation alt_disk_install to update an alternate disk
copy of the VIOS Server.
"nim o alt_disk_install a source=rootvg a disk=target_disk
a fix_bundle=(Value) ... ... ..."
If NIM is not used to update the VIOS, only the updateios or the alt_root_vg
command from the padmin shell can be used to update the VIOS.
27
VIOS AND NIM

Add VIOS partition as a NIM client
Copy the VIOS mksysb image from the CD to your NIM master
On VIOS 2.2 media there are 3 images now the 3rd is on DVD 2
Copy all 3 images individually to a directory and then use cat to combine them
cat /export/mksysb/vios2.2/mksysb_image /export/mksysb/vios2.2/mksysb_image2
/export/mksysb/vios2.2/mksysb_image3 >/export/mksysb/nim_vios2.2mksysb
Define mksysb resource to NIM master

Define spot on NIM master
The source for the SPOT will be the combined mksysb
The SPOT CANNOT be created from an LPP_Source
Copy the bosinst.data from the DVD and create a viosbosinst resource
You can now use bos_inst to do a mksysb install once the partition profile is defined
http://www-01.ibm.com/support/docview.wss?uid=isg3T1011386
28
14
7/22/2014
MAINTENANCE AND UPGRADES
29
UPDATING VIOS
1. Normally upgrade HMC first then firmware then VIOS and then AIX
2. BUT check the readme for all of the above first to make sure there is
not a different required order
3. Download the updates and cross-check compatibility using FLRT
4. Read the readme again
5. Run errpt to check for problems,check there are no stale partitions,
missing disks or paths, etc
lsvg rootvg checks for stale PPs and physical volumes.

lsvg -pv rootvg looks for missing disks.
lspath - checks for missing paths.
errpt checks for errors.
6. Ensure all paths on clients are redundant so LPARs will stay up when
this VIOS is rebooted
7. Run HMC Scanner or sysplan to document prior to changes
8. Backup the VIOS
9. Mount the NFS filesystem or DVD or FBO image to be used for update
30
15
7/22/2014
MIGRATION
Back the VIO up before doing anything and again when done!
1. In order to migrate to v2.* your HMC must be at v7 or later at least 7.7.4
If VIOS is lower than v2.1 then you must migrate to 2.0.0 using the migration DVD
2. Migrating from prior to v1.3
Basically this is a reinstall
3. Migrating from v1.3 or v1.4

Need the migration DVD for VIOS 1.5 or the updates
Need to update to VIOS 1.5.2.6-FP-11.1 SP-02 prior to upgrade to v2
4. Migrating from v1.5.2.6-FP-11.1 SP-02 or higher

Need the migration DVD for VIOS v2
Boot from the DVD in SMS mode and tell it to do a migration upgrade
Note once at v2.1 you need to update to 2.2.3.1 prior to applying 2.2.3.3
2.2.3.3 requires a minimum release of 2.2.3.0 in order to be applied
Single step update requires VIO between 2.2.1.1 and 2.2.2.x
NIM allows you to create a single merged lpp_source to get around this but cannot be used with SDDPCM
5. See Power VM Managing and Monitoring Redbook Chapter 11

http://www.redbooks.ibm.com/redbooks/pdfs/sg247590.pdf
Once you are on v2.1 then upgrades are all done using updateios
There are specific concerns around updates if you are running SSPs (Shared storage pools)
31
UPDATING VIOS WITH FIXPACKS OR SPS

From 2.2.3.2 to 2.2.3.3
As padmin run updateios commit to ensure any uncommitted updates
are committed
Check to ensure there are no missing filesets prior to updates
$ ioslevel
2.2.3.2
$ updateios -commit
All updates have been committed.
$ oem_setup_env
# /usr/sbin/emgr P
There is no efix data on this system.
Now run checks
32
16
7/22/2014
PRE INSTALL CHECKS FOR VIOS 2.2.3.2 TO 2.2.3.3 UPDATE

Did VIO2 (secondary VIO first):
$ ioslevel
2.2.3.2
$ oem_setup_env
#df g - make sure no filesystems are full
#oslevel -s
6100-09-02-1412
# instfix -i | grep ML
All filesets for 6.1.0.0_AIX_ML were found.
All filesets for 6100-00_AIX_ML were found.
# lppchk -v
# lppchk -vm3
# oslevel -s -l 6100-09-02-1412
#errpt | more check there are no errors
33
CONTINUE 2.2.3.3 UPDATE BACKUP 1/2

Back it up:
# ./save-viostuff.sh
mkdir: 0653-358 Cannot create /home/padmin/saveit.
/home/padmin/saveit: Do not specify an existing file.
# ls -l /home/padmin/saveit
total 824
-rw-r--r-- 1 root staff
118 Jul 22 12:33 b740vio2.disktmp.txt
24 Jul 22 12:33 b740vio2.ioslevel.txt
16 Jul 22 12:33 b740vio2.oslevel.txt
8038 Jul 22 12:33 b740vio2.vioadapter.txt
4528 Jul 22 12:33 b740vio2.viodisk.txt
59593 Jul 22 12:33 b740vio2.viodisks.txt
8800 Jul 22 12:33 b740vio2.violsdevv.txt
11967 Jul 22 12:33 b740vio2.violsmapall.npiv.txt
19363 Jul 22 12:33 b740vio2.violsmapall.txt
4595 Jul 22 12:33 b740vio2.vioslots.txt
227944 Jul 22 12:33 b740vio2.viovpd.txt
37 Jul 22 12:33 cfgname.txt
0 Jul 22 12:33 entstat.txt
240 Jul 22 12:33 firewall.txt
652 Jul 22 12:33 hostmap.txt
5970 Jul 22 12:33 optimize.txt
713 Jul 22 12:33 routinfo.txt
240 Jul 22 12:33 user.txt
15071 Jul 22 12:33 view.txt
34
17
7/22/2014
CONTINUE 2.2.3.3 UPDATE BACKUP 2/2

$ viosbr -backup -file /home/padmin/saveit/b740vio2-backup
Backup of this node (b740vio2) successful
$ savevgstruct rootclients_vg
Creating information file for volume group rootclients_vg.
Creating list of files to back up.
Backing up 6 files
6 of 6 files (100%)
0512-038 savevg: Backup Completed Successfully.
oem_setup_env
# mount /usr/local/backups
# su - padmin -c "ioscli backupios -file /usr/local/backups/b740vio2-jul2214.mksysb -mksysb"
/usr/local/backups/b740vio2-jul2214.mksysb doesn't exist.
Creating /usr/local/backups/b740vio2-jul2214.mksysb
Creating information file for volume group rootclients_vg.
Backing up 6 files
6 of 6 files (100%)
Backup in progress. This command can take a considerable amount of time
to complete, please be patient...
Creating information file (/image.data) for rootvg.
Backing up 160374 files..............................
39229 of 160374 files (24%)............................
160374 of 160374 files (100%)
35
CONTINUE 2.2.3.3 UPDATE INSTALL

Download from Fix Central the iso image for 2.2.3.3 I do this to my NIM server
It came down as H52175995.iso
mkdir /cdrom
loopmount -i H52175995.iso -o "-V cdrfs -o ro" -m /cdrom
smitty bffcreate I do this on my NIM server and create a directory to put the files in that the VIO has access to
In this case /usr/local/soft/vios2233
Now on the VIO:
$ updateios -accept -install -dev /usr/local/soft/vios2233
*******************************************************************************
installp PREVIEW: installation will not actually occur.
*******************************************************************************
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
SUCCESSES
--------Filesets listed in this section passed pre-installation verification
and will be installed.
Mandatory Fileset Updates
------------------------(being installed automatically due to their importance)
bos.rte.install 6.1.9.16
# LPP Install Commands
<< End of Success Section >>
Prompts you to reply Y which you do and it installs them
36
18
7/22/2014
CONTINUE 2.2.3.3 UPDATE INSTALL

After bos.rte.install is installed it then prompts you re installing the other 272 fixes
Check estimated space needed and free space and if all is good then:
Reply Y and they begin installing takes about 2 hours depending
$ioslevel
Shows as 2.2.3.3
$oem_setup_env
# oslevel -s
6100-09-03-1415
lspv | grep rootvg
hdisk0
00f6934cc34a30f3
hdisk1
00f6934c30e34699
rootvg
rootvg
active
active
bosboot a d hdisk0
bosboot a d hdisk1
bootlist m normal hdisk0 hdisk1
Now reboot and then run post install tests
37
POST INSTALL CHECKS

$ ioslevel
2.2.3.3
$ oem_setup_env
# oslevel -s
Should show: 6100-09-03-1415
6100-09-03-1415
# instfix -i | grep ML
# lppchk -v
# lppchk -vm3
# oslevel -s -l 6100-09-03-1415
#errpt | more check there are no errors
Once all checks are passed and VIO2 is back up then go do the same upgrade to VIO1
38
19
7/22/2014
UPDATING VIOS PROBLEMS

oem_setup_env
oslevel s
6100-00-00-0000
instfix -i | grep ML
Not all filesets for 6100-08_AIX_ML were found.
This means there are missing filesets
# oslevel -sq
Known Service Packs
------------------6100-08-02-1316
6100-08-01-1245
# oslevel -s -l 6100-08-02-1316
Fileset
Actual Level
Service Pack Level
----------------------------------------------------------------------------bos.alt_disk_install.boot_images
6.1.8.0
6.1.8.15
bos.loc.utf.ES_ES
6.1.7.15
6.1.8.15
These filesets should be corrected prior to updating
Either use updateios to update them or to remove them
39
REMOVE OR UPDATE PROBLEM FILESETS

DO NOT USE SMITTY use updateios
Issues with bos.suma
updateios remove bos.suma
# oslevel -s -l 6100-08-02-1316
Fileset
Actual Level
Service Pack Level
----------------------------------------------------------------------------bos.alt_disk_install.boot_images
6.1.8.0
6.1.8.15
bos.loc.utf.ES_ES
6.1.7.15
6.1.8.15
updateios remove bos.loc.utf.ES_ES
Upgrade alt disk
Copy images to be updated into a directory (/usr/local/soft/missing)
Run inutoc .
updateios commit
Updateios -accept -install -dev /usr/local/soft/missing
Also remove efixes prior to updates:
/usr/sbin/emgr P lists them
To remove:
# /usr/sbin/emgr -r -L <EFIX label>
emgr -r -L IV46869m3a
40
20
7/22/2014
BACKUP AND RECOVERY
41
BACKING
UP
VIOS
Use viosbr to backup user defined virtual resources on the VIO

Make sure to save that backup in rootvg
viosbr backup file /tmp/viosabkupbr
You can also use viosbr to view or restore
http://publib.boulder.ibm.com/infocenter/systems/scope/hw/topic/p7hcg/viosbr.htm
Mount NFS filesystem to backup to (in my case /backups)

mkdir /backups/viosa
Then as padmin:
backupios file /backups/viosa
The above creates a nim_resources.tar package in that directory and it can be used to
clone VIO servers
You can also back it up as a mksysb file that is easy to restore from the NIM server
backupios -file /backups/viosa.mksysb -mksysb
42
21
7/22/2014
BACKING
UP
VIOS FROM ROOT
As root run viosave.sh (see next slide)

lsvg if you have other VGs i.e. datavg then for each ..
su - padmin -c "ioscli savevgstruct datavg"
su - padmin -c "ioscli viosbr backup file /tmp/viosabr.backup
Mount the NFS repository for the backups (/nfsmnt)
su padmin c ioscli backupios file /nfsmnt/vio2-jul2114.mksysb -mksysb
This backs it up to a bootable mksysb file
To use the mksysb with NIM replace the backupios command above with:
backupios -file /nfsmnt/nimbkups
This creates a nim_resources.tar file that can be used for restores described at:
http://public.dhe.ibm.com/software/server/vios/docs/backupios_mod.pdf
43
DOCUMENT VIO INFORMATION VIOSAVE.SH

#! /bin/sh
#
day="`/bin/date +'%d'`"
month="`/bin/date +'%m'`"
year="`/bin/date +'%y'`"
set -- Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
shift $month
lmonth="$1"
machine=`uname -n`
directory="`/bin/date +'%m%d%Y_%H%M'`"
machine_directory=`printf "%s_%s" $machine $directory`
mkdir /home/padmin/saveit
cd /home/padmin/saveit
logit="/home/padmin/saveit/$machine"
logit1="/home/padmin/saveit/$machine"
su - padmin -c "ioscli ioslevel" >>$logit1.ioslevel.txt
su - padmin -c "ioscli lsdev -type disk" >>$logit1.viodisk.txt
su - padmin -c "ioscli lsdev -type adapter" >>$logit1.vioadapter.txt
su - padmin -c "ioscli lsdev -vpd" >>$logit1.viovpd.txt
su - padmin -c "ioscli lsdev -slots" >>$logit1.vioslots.txt
su - padmin -c "ioscli lsmap -all" >>$logit1.violsmapall.txt
su - padmin -c "ioscli lsmap -all -npiv" >>$logit1.violsmapall.npiv.txt
su - padmin -c "ioscli lsdev -virtual" >>$logit1.violsdevv.txt
su - padmin -c "ioscli cfgnamesrv -ls " >cfgname.txt
su - padmin -c "ioscli entstat -all ent9 " >entstat.txt
su - padmin -c "ioscli hostmap -ls" >hostmap.txt
su - padmin -c "ioscli lsuser" >user.txt
su - padmin -c "ioscli netstat -routinfo" >routinfo.txt
su - padmin -c "ioscli optimizenet -list" >optimize.txt
su - padmin -c "ioscli viosecure -firewall view" >firewall.txt
su - padmin -c "ioscli viosecure -view -nonint" >view.txt
oslevel -s >$logit1.oslevel.txt
getlvodm -C > $logit1.disktmp.txt
while read label line
do
echo "\n" >>$logit1.viodisks.txt
echo "Hdisk is $label" >>$logit1.viodisks.txt
echo " " >>$logit1.viodisks.txt
su - padmin -c "ioscli lsdev -dev $label -attr" >>$logit1.viodisks.txt
done <"$logit1.disktmp.txt"
#
exit 0
44
22
7/22/2014
MONITORING
45
NETSTAT -V
ETHERNET STATISTICS (ent18) :

Device Type: Shared Ethernet Adapter
Elapsed Time: 44 days 4 hours 21 minutes 3 seconds
Transmit Statistics:
Receive Statistics:
-------------------------------------Packets: 94747296468
Packets: 94747124969
Bytes: 99551035538979
Bytes: 99550991883196
Interrupts: 0
Interrupts: 22738616174
Transmit Errors: 0
Receive Errors: 0
Packets Dropped: 286155
Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 712
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 50
Broadcast Packets: 3227715
Multicast Packets: 3394222
No Carrier Sense: 0
CRC Errors: 0
DMA Underrun: 0
DMA Overrun: 0
Lost CTS Errors: 0
Alignment Errors: 0
Max Collision Errors: 0
No Resource Errors: 286155 check those tiny, etc Buffers
Late Collision Errors: 0
Receive Collision Errors: 0
Deferred: 0
Packet Too Short Errors: 0
SQE Test: 0
Packet Too Long Errors: 0
Timeout Errors: 0
Packets Discarded by Adapter: 0
Single Collision Count: 0
Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 50
46
23
7/22/2014
NETSTAT
VIO
SEA
Transmit Statistics:
-------------------Packets: 83329901816
Bytes: 87482716994025
Interrupts: 0
Transmit Errors: 0
Packets Dropped: 0
Receive Statistics:
------------------Packets: 83491933633
Bytes: 87620268594031
Interrupts: 18848013287
Receive Errors: 0
Packets Dropped: 67836309
Bad Packets: 0
Max Packets on S/W Transmit Queue: 374
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 0
No Carrier Sense: 0
DMA Underrun: 0
Lost CTS Errors: 0
Max Collision Errors: 0

CRC Errors: 0
DMA Overrun: 0
Alignment Errors: 0
No Resource Errors: 67836309
Virtual I/O Ethernet Adapter (l-lan) Specific Statistics:

--------------------------------------------------------Hypervisor Send Failures: 4043136
Receiver Failures: 4043136
Send Errors: 0
Hypervisor Receive Failures: 67836309
No Resource Errors can occur when the appropriate amount

of memory can not be added quickly to vent buffer space for a
workload situation.
47
BUFFERS
Virtual Trunk Statistics
Receive Information
Receive Buffers
Buffer Type
Min Buffers
Max Buffers
Allocated
Registered
History
Max Allocated
Lowest Registered
532
502
Tiny
512
2048
513
511
Small
512
2048
2042
506
Medium
128
256
128
128
Large
24
64
24
24
2048
354
128
128
24
24
24
24
Huge
24
64
24
24
Max Allocated represents the maximum number of buffers ever allocated

Min Buffers is number of pre-allocated buffers
Max Buffers is an absolute threshhold for how many buffers can be allocated
chdev l <veth> -a max_buf_small=4096 P
chdev l <veth> -a min_buf_small=2048 P
Above increases min and max small buffers for the virtual ethernet adapter configured for the SEA above
Needs a reboot
Max buffers is an absolute threshold for how many buffers can be allocated
Use entstat d (-all on vio) or netstat v to get this information
48
24
7/22/2014
NMON
MONITORING
nmon -ft AOPV^dMLW -s 15 -c 120
Grabs a 30 minute nmon snapshot

A is async IO
M is mempages
t is top processes
L is large pages
O is SEA on the VIO
P is paging space
V is disk volume group
d is disk service times
^ is fibre adapter stats
W is workload manager statistics if you have WLM enabled
If you want a 24 hour nmon use:

nmon -ft AOPV^dMLW -s 150 -c 576
May need to enable accounting on the SEA first this is done on the VIO
chdev dev ent* -attr accounting=enabled
Can use entstat/seastat or topas/nmon to monitor this is done on the vios
topas E
nmon -O
VIOS performance advisor also reports on the SEAs
49
DPO (DYNAMIC PLATFORM OPTIMIZER)

PowerVM feature requiring firmware 760 and HMC 7.6.0 or greater
DPO aware O/S
AIX v6.1 TL08, VIOS 2.2.2.0, AIX v7.1 TL02, IBM I 7.1 PTF MF56058, RHEL 7, SLES
12
Earlier systems will work but will need a reboot after running DPO
No value on single-socket servers

Used to improve system wide partition memory and processor placement
(affinity)
Tries to assign local memory to CPUs, reducing memory access time
Launched via HMC using the optmem command
lsoptmem command shows current and predicted memory affinity
See chapter 15 of the PowerVM Managing and Monitoring Redbook SG247590
50
25
7/22/2014
DSO (DYNAMIC SYSTEM OPTIMIZER)
PowerVM and AIX feature
P7 or P7+
AIX v6.1 TL08 SP1 or AIX v7.1 TL02 SP1
Cannot be using AMS (Active memory sharing)
Chargeable feature via an enablement fileset
Dynamically tunes the allocation of system resources within an LPAR
Identifies and optimizes workloads

Tries to optimize cache affinity, memory affinity, use of large pages, hardware prefetch
See chapter 16 of the PowerVM Managing and Monitoring Redbook

SG24-7590
Whitepaper at:
http://www-01.ibm.com/common/ssi/cgibin/ssialias?infotype=SA&subtype=WH&htmlfid=POW03093USEN
51
SHARED PROCESSOR POOL MONITORING

Turn on Allow performance information collection on the LPAR properties
This is a dynamic change
topas C
Most important value is app available pool processors
This represents the current number of free physical cores in the pool
nmon option p for pool monitoring
To the right of PoolCPUs there is an unused column which is the number of free
pool cores
nmon analyser LPAR Tab
lparstat
Shows the app column and poolsize
52
26
04:29
04:29
04:28
04:27
04:26
04:26
04:25
04:24
04:23
04:23
04:22
04:21
04:20
04:20
04:19
04:18
OtherLPARs
04:17
04:17
04:16
04:15
PhysicalCPU
04:14
04:14
04:13
04:12
04:11
04:11
04:10
04:09
04:08
04:08
04:07
04:06
04:05
04:05
04:04
04:03
NMON
04:02
04:02
04:01
04:00
7/22/2014
ANALYSER LPAR TAB
Shared Pool Utilisation - b750nl1 10/4/2013
35
PoolIdle
30
25
20
15
10
53
FBO FILE BACKED OPTICAL
54
27
7/22/2014
FBO
Provides ISO image access to LPARs connected to the VIO
Uses vSCSI
Add third disk to one of the VIO servers and put in its own VG
Use that VG as the repository
Rip DVDs to ISO images (or download the ISOs)
Can also put mksysb images in the repository
55
USING FBO
So using hdisk2 as my third disk I create a VG:
mkvg y datavg hdisk2
mkrep sp datavg size 10G
This can be increased later and creates:
/dev/VMLibrary 10.00
5.85 42%
7
1% /var/vio/VMLibrary
We can now load images into the repository from an ISO we have on NFS:
mkvopt -name rhelboot64 -file /usr/local/nfsmnt/rhel-64bit-basedvd.iso
This takes the ISO and copies it as follows:
$ ls -l /var/vio/VMLibrary
-rw------- 1 root staff 237981696 Jul 25 13:14 rhelboot64
To list the repository:
$ lsrep
Size(mb) Free(mb) Parent Pool
10199
Name
rhelboot64
5989
datavg
Parent Size
279552
Parent Free
157696
File Size Optical

227 None
Access
rw
56
28
7/22/2014
USING FBO
In order to use the repository, the virtual host has to be assigned a virtual SCSI FBO adapter; do so
with:
mkvdev -fbo -vadapter vhost0
vtopt0 available
Multiple vtopt devices for a client can be created, if desired.
Now make the image available to vtopt0:
loadopt -vtd vtopt0 -disk rhelboot64
loadopt -disk p8aix71base1 -vtd vtopt0
This example would mount rhelboot64 or p8aix71base1 as if it were a local CD. To unmount it, use:
unloadopt -vtd vtopt0
Also, rmvopt removes an image; rmrep removes a repository; and lsvopt lists the state of the vtopts:
$ lsvopt
VTD
vtopt0
Media
No Media
Size(mb)
n/a
57
MY POWER8 VIO FBO

$ lsrep
Size(mb) Free(mb) Parent Pool
511414 490594 fbovg
Name
p8aix61base1
p8aix61base2
p8aix71base1
p8aix71base2
p8rhel7
p8ubuntu
p8viosbase1
p8viosbase2
$ lssp
Pool
rootvg
fbovg
$ lsvopt
VTD
vtopt0
vtopt1
vtopt2
vtopt3
Parent Size
1089024
Parent Free
577024
File Size Optical

4423 None
2265 None
4089 None
3035 None
1334 None
655 None
3913 None
1106 None
Access
rw
rw
rw
rw
rw
rw
rw
rw
Size(mb) Free(mb) Alloc Size(mb) BDs Type

540672 407552
512
2 LVPOOL
1089024 577024
512
1 LVPOOL
Media
No Media
No Media
No Media
No Media
Size(mb)
n/a
n/a
n/a
n/a
58
29
7/22/2014
OTHER FBO COMMANDS

padmin
chrep
chvopt
loadopt
lsrep
lsvopt
mkrep
mkvopt
rmrep
rmvopt
unloadopt
lssp
mkvdev
You can then run help <command>
As root
lsvg
mkvg
59
OTHER
60
30
7/22/2014
USEFUL COMMANDS
Command History
$ fc -l
725 lsrep
726 backupios -file /usr/local/backups/b750viobkp
727 exit
728 lsmap -vadapter vhost0
729 fc l
Global command log
$ lsgcl | grep "Aug 9 2013"
Aug 9 2013, 08:25:35 root ioslevel
Aug 9 2013, 08:59:22 padmin license
Aug 9 2013, 09:00:29 padmin lsmap -vadapter vhost0
Aug 9 2013, 09:01:29 padmin lsgcl
Redirecting output when running as padmin
lsmap all npiv | tee npivdata.txt
61
USEFUL COMMANDS
vSCSI Commands
mkvdev -vdev hdisk2 -vadapter vhost0
mkvdev fbo vadapter vhost0
NPIV
Setup NPIV mappings
vfcmap vadapter vfchost0 fcp fcs0
lsmap npiv all
lsmap vadapter vfchost0 npiv
lsdev virtual
lsnports
lsdev slots
lscfg vpl vfchost0
62
31
7/22/2014
USEFUL COMMANDS
$ lsdev -virtual
name
status
description
ent5
Available Virtual I/O Ethernet Adapter (l-lan)
ent6
ent7
vasi0
Available Virtual Asynchronous Services Interface (VASI)
vbsd0
Available Virtual Block Storage Device (VBSD)
vfchost0
Available Virtual FC Server Adapter
vfchost1
vfchost2
vfchost3
vhost0
Available Virtual SCSI Server Adapter
vhost1
vhost2
vhost3
vsa0
Available LPAR Virtual Serial Adapter
b740ios1_rv1 Available Virtual Target Device - Logical Volume
b740l1_rv1
Available Virtual Target Device - Logical Volume
vtopt0
Available Virtual Target Device - File-backed Optical
vtopt1
vtopt2
vtopt3
vtscsi0
Available Virtual Target Device - Disk
vtscsi1
vtscsi2
vtscsi3
ent8
Available Shared Ethernet Adapter
63
USEFUL COMMANDS
$ lsmap -vadapter vhost0

SVSA
Physloc
Client Partition ID
--------------- -------------------------------------------- -----------------vhost0
U8205.E6B.10934CP-V1-C21
0x00000003
VTD
Status
LUN
Backing device
Physloc
Mirrored
b740l1_rv1
Available
0x8300000000000000
lv_b740l1
VTD
Status
LUN
Backing device
Physloc
Mirrored
vtopt0
Available
0x8200000000000000
VTD
Status
LUN
Backing device
Physloc
Mirrored
vtopt1
Available
0x8100000000000000
N/A
N/A
N/A
64
32
7/22/2014
USEFUL COMMANDS
$ lsmap -vadapter vfchost0 -npiv

Name
Physloc
ClntID ClntName
ClntOS
------------- ---------------------------------- ------ -------------- ------vfchost0
U8205.E6B.10934CP-V1-C31
3
Status:NOT_LOGGED_IN
FC name:fcs0
FC loc code:U78AA.001.WZSG8PD-P1-C5-T1
Ports logged in:0
Flags:4<NOT_LOGGED>
VFC client name:
VFC client DRC:
$ lsmap -vadapter vfchost4 -npiv
Name
Physloc
ClntID ClntName
ClntOS
------------- ---------------------------------- ------ -------------- ------vfchost4
U8205.E6B.10934CP-V1-C36
8 b740nl1
AIX
Status:LOGGED_IN
FC name:fcs0
FC loc code:U78AA.001.WZSG8PD-P1-C5-T1
Ports logged in:3
Flags:a<LOGGED_IN,STRIP_MERGE>
VFC client name:fcs0
VFC client DRC:U8205.E6B.10934CP-V8-C36
65
USEFUL COMMANDS
$ lsnports
name
physloc
fcs0
U78AA.001.WZSG8PD-P1-C5-T1
fabric tports aports swwpns awwpns

1 64 63
2048
2041
$ lsdev -slots
# Slot
Description
Device(s)
HEA 1
Logical I/O Slot lhea0 ent0
U8205.E6B.10934CP-V1-C0
Virtual I/O Slot vsa0
U8205.E6B.10934CP-V1-C11 Virtual I/O Slot ent5
U8205.E6B.10934CP-V1-C21 Virtual I/O Slot vhost0
U8205.E6B.10934CP-V1-C31 Virtual I/O Slot vfchost0
U8205.E6B.10934CP-V1-C32769 Virtual I/O Slot vasi0
66
33
7/22/2014
USEFUL HMC COMMANDS

ssh to HMC as hscroot or your userid
Useful Commands:
lshmc
vtmenu
lshwres
- way better than ascii console
monhmc r mem n 0
monhmc r proc n 0
monhmc r swap n 0
monhmc r disk n 0
chhmcfs r disk n 0
lshmcfs
how much memory do I have?

CPU usage
Page space
What is my disk utilization?
Clear out all temp files
hmcshutdown r t now
Reboot HMC
67
USEFUL HMC COMMANDS 7042-CR6

hscroot@bpichmc:~>monhmc -r mem -n 0
Mem: 4043216k total, 3885308k used, 157908k free, 484132k buffers (has 4GB)
hscroot@bpichmc:~>monhmc -r proc -n 0
Cpu0 : 0.0%us, 0.7%sy, 0.0%ni, 98.3%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
hscroot@bpichmc:~>monhmc -r swap -n 0
Swap: 2040244k total, 137456k used, 1902788k free, 1036824k cached
hscroot@bpichmc:~>monhmc -r disk -n 0
Filesystem
1K-blocks
/dev/sda2
16121184
/dev/sda3
6040320
/dev/mapper/HMCDataVG-HomeLV
10321208
/dev/mapper/HMCDataVG-LogLV
8256952
/dev/mapper/HMCDataVG-DumpLV
123854820
/dev/mapper/HMCDataVG-ExtraLV
20642428
/dev/mapper/HMCDataVG-DataLV
227067260
Used Available Use% Mounted on

7100064 8202208 47% /
297672 5435808 6% /var
245052 9551868 3% /home
1292372 6545152 17% /var/hsc/log
319672 117243692 1% /dump
198692 19395160 2% /extra
455376 215077548 1% /data
hscroot@bpichmc:~>lshmcfs
filesystem=/var,filesystem_size=8063,filesystem_avail=6390,temp_files_start_time=07/14/2014 13:11:00,temp_files_size=783
filesystem=/dump,filesystem_size=120951,filesystem_avail=114495,temp_files_start_time=07/14/2014 21:09:00,temp_files_size=0
filesystem=/extra,filesystem_size=20158,filesystem_avail=18940,temp_files_start_time=none,temp_files_size=0
filesystem=/,filesystem_size=15743,filesystem_avail=8009,temp_files_start_time=07/22/2014 23:18:00,temp_files_size=0
68
34
7/22/2014
USEFUL HMC COMMANDS 7042-CR7

hscroot@bpic8hmc:~>monhmc -r mem -n 0
Mem: 41263576k total, 3608896k used, 37654680k free, 551600k buffers
Either it has 41GB memory or there is a bug
hscroot@bpic8hmc:~>monhmc -r proc -n 0
Cpu1 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
hscroot@bpic8hmc:~>monhmc -r swap -n 0
Swap: 2040244k total,
0k used, 2040244k free, 934024k cached
hscroot@bpic8hmc:~>monhmc -r disk -n 0
Filesystem
1K-blocks Used Available Use% Mounted on
/dev/sda2
16121184 6715032 8587240 44% /
/dev/sda3
6040320 270112 5463368
5% /var
/dev/mapper/HMCDataVG-HomeLV
10321208 244856 9552064
3% /home
/dev/mapper/HMCDataVG-LogLV
8256952 479796 7357728
7% /var/hsc/log
/dev/mapper/HMCDataVG-DumpLV
61927420 187024 58594668
1% /dump
/dev/mapper/HMCDataVG-ExtraLV
20642428 198692 19395160
2% /extra
/dev/mapper/HMCDataVG-DataLV
144497320 195428 136961860 1% /data
hscroot@bpic8hmc:~>lshmcfs
filesystem=/var,filesystem_size=8063,filesystem_avail=7185,temp_files_start_time=07/14/2014 16:33:00,temp_files_size=318
filesystem=/dump,filesystem_size=60475,filesystem_avail=57221,temp_files_start_time=07/14/2014 20:15:00,temp_files_size=0
filesystem=/extra,filesystem_size=20158,filesystem_avail=18940,temp_files_start_time=none,temp_files_size=0
filesystem=/,filesystem_size=15743,filesystem_avail=8385,temp_files_start_time=07/22/2014 22:43:00,temp_files_size=0
69
USEFUL HMC COMMANDS

lshmc
-V
-v
-r
-n
-b
-l
-L
-h
-i
- Displays HMC version information.

- Displays HMC VPD information.
- Displays HMC remote access settings.
- Displays HMC network settings.
- Displays the BIOS level of the HMC.
- Displays the current locale for the HMC.
- Displays all supported locales for the HMC.
- Displays HMC hardware information.
- Displays HMC Integrated Management Module (IMM)
settings.
-e
- Displays HMC settings for Events Manager for Call
Home.
-F [<attribute names>] - delimiter-separated list of the names of the
attributes to be listed for the specified HMC
setting. If no attribute names are specified,
then all attributes will be listed.
--header
- prints a header of attribute names when -F is
also specified
--help
- prints this help
70
35
7/22/2014
USEFUL HMC COMMANDS

hscroot@bpichmc:~>lshmc -V
"version= Version: 8
Release: 8.1.0
Service Pack: 0
HMC Build level 20140602.3
MH01421: Required fix for HMC V8R8.1.0 (06-03-2014)
MH01436: Fix for OpenSSL,GnuTLS (06-11-2014)
MH01441: Fix for HMC V8R8.1.0 (06-23-2014)
","base_version=V8R8.1.0
hscroot@bpichmc:~>lshmc -v
"vpd=*FC ????????
*VC 20.0
*N2 Wed Jul 23 04:45:57 UTC 2014
*FC ????????
*DS Hardware Management Console
*TM 7042-CR6
*SE 102EEEC
*MN IBM
*PN 0B20PT
*SZ 4140253184
*OS Embedded Operating Systems
*NA 10.250.134.20
*FC ????????
*DS Platform Firmware
*RM V8R8.1.0.0
"
71
USEFUL HMC COMMANDS

hscroot@bpichmc:~>lshmc -b
"bios=D6E149AUS-1.09
"
hscroot@bpichmc:~>lshmc -r
ssh=enable,sshprotocol=,remotewebui=enable,xntp=disable,xntpserver=127.127.1.0,syslogserver=,syslog
tcpserver=,syslogtlsserver=,altdiskboot=disable,ldap=disable,kerberos=disable,kerberos_default_realm=,k
erberos_realm_kdc=,kerberos_clockskew=,kerberos_ticket_lifetime=,kpasswd_admin=,trace=,kerberos_k
eyfile_present=,kerberos_allow_weak_crypto=,legacyhmccomm=disable,security=legacy,sol=disabled
hscroot@bpichmc:~>lshmc -e
emch=disabled,callhome=enabled,registered_hmcs=
On HMC check LMB sizes
hscroot@bpichmc:~>lshwres -r mem -m p740-Server-8205-E6B-SN10934CP --level sys -F
mem_region_size
256
72
36
7/22/2014
USEFUL HMC COMMANDS HMC UPDATES

Use with great care
saveupgdata r disk
getupgfiles -h public.dhe.ibm.com -u anonymous --passwd anonymous -d
/software/server/hmc/network/v8810
ls la /hmcdump
chhmc c altdiskboot s enable mode upgrade
tail -f /tmp/HmcInstall.log
during upgrade
Directories on FTP Server (ftp.software.ibm.com)

Upgrades:
/software/server/hmc/network/v8810
Fixes:
/software/server/hmc/fixes
Service Packs:
/software/server/hmc/updates
73
HMC SCANNER
Latest HMC Scanner is available at http://tinyurl.com/HMCscanner

Java program that uses SSH to connect to HMC, FSM or IVM to gather information about the
system configuration
I run it on one of the AIX Systems as follows:
./hmcScanner.ksh servername hscroot -p password -stats sanitize

Sanitize causes it to produce two spreadsheets one that has been cleansed of identifying data
Information is organized in tabs in an excel spreadsheet:
System summary: name, serial number, cores, memory, service processor IP for each server
LPAR Summary: list of all LPAR by serve with status, environment, version, processor mode
LPAR CPU: processor configuration of each LPAR

LPAR MEM: memory configuration of each LPAR
Physical Slots: list of all slots of each system with LPAR assignment, description, physical location and drc_index
Virtual Ethernet: network configuration of each virtual switch and each LPAR
Virtual SCSI: configuration of all virtual SCSI adapters, both client and server
VSCSI Map: devices mapped by each VIOS to partitions
Virtual Fibre: virtual fibre channel configuration of client and server with identification of physical adapter assigned
SEA: SEA configuration and statistics for all VIOS
SW Cores: LPAR and virtual processor pool configuration matrix to compute the number of software licenses. Simulation of
alternative scenarios is possible.
CPU Pool Usage: monthly average history of CPU usage of each system. Based on last 12 months of lslparutil data.
Sys RAM Usage: monthly average history of physical memory assignement to each LPAR. Based on last 12 months of lslparutil
data.
LPAR CPU Usage:monthly average history of CPU usage of each LPAR. Based on last 12 months of lslparutil data.
CPU Pool Daily Usage: 1 year of CPU usage of every pool and subpools of each system. Based on daily averages.
LPAR Daily Usage: 1 year of CPU usage of every LPAR of each system. Based on daily averages.
CPU Pool HourlyUsage: 2 months of CPU usage of every pool and subpools of each system. Based on hourly averages.
LPAR Hourly Usage: 2 months of CPU usage of every LPAR of each system. Based on hourly averages.
74
37
7/22/2014
RUNNING HMC SCANNER

I run it from AIX as Windows and Java issues have caused problems
Right now I have HMCScanner11
./hmcScanner.ksh hmcname hscroot -p password stats
hmcScanner version 0.11.0
Detecting manager type: HMC
Detecting managed systems: 3 systems present.
Starting managed system configuration collection:
Scanning p720-Server-8202-E4B-SN10934BP: ............... DONE
Scanning p740-Server-8205-E6B-SN10934CP: ............... DONE
Scanning p750-Server-8233-E8B-SN069348P: ............... DONE
Collection successfully finished. Data is in /software/hmcscanner-11/bpichmc/
Performance data collection:
Loading p720-Server-8202-E4B-SN10934BP: . .
Loading p740-Server-8205-E6B-SN10934CP: . .
Loading p750-Server-8233-E8B-SN069348P: . .
......... DONE
bpichmc_20140722_111805_scan.xls
bpic8hmc_20140722_104308_scan.xls
75
VIOS ADVISOR
https://www.ibm.com/developerworks/community/wikis/home/wiki/Power%20Systems/page/VIOS%20Advisor
Application that collects performance metrics and does a health check

Productized in VIOS 2.2.2
Current downloadable version is 030813A
Examples follow
These were run on a production VIO during a regular production day
Duration is between 10 and 60 minutes. Samples collected every 15 seconds

To run for 10 minutes:
$ part -i 10
Creates a tar file
The report to read is the vios_advisor_report.xml report
76
38
7/22/2014
VIOS ADVISOR
$ part -i 10
part: Reports are successfully generated in b740vio2_140722_19_46_12.tar
$oem_setup_env
#tar tvf b740vio2_ .tar
# pwd
/home/padmin/jaqui/b740vio2_140722_19_46_12
# ls -l
total 704
249619 Jul 22 19:56 b740vio2_140722_1946.nmon
drwxr-xr-x 2 padmin staff
4096 Jul 22 19:56 images
-r--r--r-- 1 padmin staff
8122 Jul 22 19:56 popup.js
6971 Jul 22 19:56 style.css
16869 Jul 22 19:56 vios_advisor.xsl
-rw-r--r-- 1 padmin staff
32677 Jul 22 19:56 vios_advisor_report.xml
29156 Jul 22 19:56 vios_advisorv2.xsl
Can also process nmon files:
$ part -f b750vio1_140721_2359.nmon
part: Reports are successfully generated in b750vio1_140721_2359.tar
# tar -tvf b750vio1_140721_2359.tar
-r--r--r-- 10 1 16869 Jul 22 19:48:10 2014 ./b750vio1_140721_2359/vios_advisor.xsl
-rw-r--r-- 10 1 36446 Jul 22 19:48:09 2014 ./b750vio1_140721_2359/vios_advisor_report.xml
-r--r--r-- 10 1 29156 Jul 22 19:48:10 2014 ./b750vio1_140721_2359/vios_advisorv2.xsl
b750vio1_140721_2359\vios_advisor_report.xml
77
VIOS ADVISOR
78
39
7/22/2014
VIOS ADVISOR
79
VIOS ADVISOR
80
40
7/22/2014
VIOS ADVISOR
81
VIOS ADVISOR
82
41
7/22/2014
VIOS ADVISOR
83
VIOS ADVISOR
84
42
7/22/2014
SSP LIMITATIONS
http://www14.software.ibm.com/webapp/set2/sas/f/vios/vios2233.readme.html
Limitations for Shared Storage Pool
Software Installation
All VIOS nodes must be at version 2.2.1.3 or later.
When installing updates for VIOS Update Release 2.2.3.3 participating in a Shared Storage Pool, the Shared Storage Pool
Services must be stopped on the node being upgraded.
SSP Configuration
Feature
Min
Max
Number of VIOS Nodes in Cluster
16
Number of Physical Disks in Pool
1024
Number of Virtual Disks (LUs) Mappings in Pool
8192
Number of Client LPARs per VIOS node
200
Capacity of Physical Disks in Pool
10GB
16TB
Storage Capacity of Storage Pool
10GB
512TB
Capacity of a Virtual Disk (LU) in Pool
1GB
4TB
Number of Repository Disks
Capacity of Repository Disk
512MB
1016GB
85
REFERENCES
86
43
7/22/2014
USEFUL LINKS
Charlie Cler Articles
http://www.ibmsystemsmag.com/authors/Charlie-Cler/
Andrew Goade Articles

http://www.ibmsystemsmag.com/authors/Andrew-Goade/
Jaqui Lynch Articles

http://www.ibmsystemsmag.com/authors/Jaqui-Lynch/
https://enterprisesystemsmedia.com/author/jaqui-lynch
Jay Kruemke Twitter chromeaix

https://twitter.com/chromeaix
Nigel Griffiths Twitter mr_nmon

https://twitter.com/mr_nmon
Gareth Coates Twitter power_gaz

https://twitter.com/power_gaz
Jaquis Upcoming Talks and Movies

Upcoming Talks
http://www.circle4.com/forsythetalks.html
Movie replays
http://www.circle4.com/movies
87
USEFUL LINKS
Nigel Griffiths
AIXpert Blog
https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/?lang=en
10 Golden rules for rPerf Sizing
Youtube channel
https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/size_with_rperf_if_you_must_but_don_t_fo
rget_the_assumptions98?lang=en
http://www.youtube.com/user/nigelargriffiths
AIX Wiki
https://www.ibm.com/developerworks/wikis/display/WikiPtype/AIX
HMC Scanner
http://www.ibm.com/developerworks/wikis/display/WikiPtype/HMC+Scanner
Workload Estimator
http://ibm.com/systems/support/tools/estimator
Performance Tools Wiki
http://www.ibm.com/developerworks/wikis/display/WikiPtype/Performance+Monitoring+Tools
Performance Monitoring
https://www.ibm.com/developerworks/wikis/display/WikiPtype/Performance+Monitoring+Documentation
Other Performance Tools
https://www.ibm.com/developerworks/wikis/display/WikiPtype/Other+Performance+Tools
Includes new advisors for Java, VIOS, Virtualization
VIOS Advisor
https://www.ibm.com/developerworks/wikis/display/WikiPtype/Other+Performance+Tools#OtherPerformanceTools-VIOSPA
Capturing Debug output for padmin
http://www-01.ibm.com/support/docview.wss?uid=isg3T1012362
88
44
7/22/2014
REFERENCES
Gareth Coates Tricks of the POWER Masters
http://public.dhe.ibm.com/systems/power/community/aix/PowerVM_webinars/30_Tricks
_of_the_Power_Masters.pdf
Nigel PowerVM User Group

https://www.ibm.com/developerworks/mydeveloperworks/wikis/home?lang=en#/wiki/Po
wer%20Systems/page/PowerVM%20technical%20webinar%20series%20on%20Powe
r%20Systems%20Virtualization%20from%20IBM%20web
SG24-7940 - PowerVM Virtualization - Introduction and Configuration

SG24-7590 PowerVM Virtualization Managing and Monitoring

SG24-8080 Power Systems Performance Guide Implementing and

Optimizing
SG24-8079 Power 7 and 7+ Optimization and Tuning Guide

Redbook Tip on Maximizing the Value of P7 and P7+ through Tuning and
Optimization
http://www.redbooks.ibm.com/technotes/tips0956.pdf
89
VIOS SPECIFIC REFERENCES

SDD and SDDPCM Specific procedures for VIOS
http://www-01.ibm.com/support/docview.wss?uid=ssg1S7002686&aid=1
VOS 2.2.3.3 README

http://www14.software.ibm.com/webapp/set2/sas/f/vios/vios2233.readme.html
SG24-7940 - PowerVM Virtualization - Introduction and Configuration

SG24-7590 PowerVM Virtualization Managing and Monitoring

SG24-8080 Power Systems Performance Guide Implementing and

Optimizing
SG24-8079 Power 7 and 7+ Optimization and Tuning Guide

Redbook Tip on Maximizing the Value of P7 and P7+ through Tuning and
Optimization
http://www.redbooks.ibm.com/technotes/tips0956.pdf
90
45
7/22/2014
THANK YOU FOR YOUR TIME
If you have questions please email me at:

[email protected]
Handouts will be at:
http://www.circle4.com/papers/viosmaint.pdf
Replay will be at:
http://www.circle4.com/movies/
91
46

IBM VIOS Maintenance

Uploaded by

Copyright:

Available Formats

IBM VIOS Maintenance

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IBM VIOS Maintenance

Uploaded by

Copyright:

Available Formats

7/22/2014

This document and the material contained herein is confidential and

BEST PRACTICES SETUP

Basically AIX 6100-09-03 Requires NIM at 6100-09-03 or 7100-03-03

Can be applied to 2.2.3.0, 2.2.3.1 or 2.2.3.2

Download updates from Fix Central:

Download base from entitled software:

USE FLRT AND CHECK PREREQS

VIOS to NIM Master Mapping:

System Software Maps for VIOS:

AIX/VIOS Security Tables:

VIOS Hiper Tables:

Use dual VIO to allow for concurrent updates

Physical Volume Volume Group Size

Physical Volume Volume Group Size

SETUP DUMP DEVICES AND LOGGING

/var/adm/lpm/lpm.log rotate size 10m files 2 compress

Make the same tuning changes you would make on AIX

Set num_cmd_elems and max_xfer_size on the fiber adapters on VIO

If NPIV also set on clients

Bus interrupt level

Bus memory address

Maximum Transfer Size

Maximum number of COMMANDS to queue to the adapter True

Maximum Transfer Size

INIT Link flags

Changes I often make (test first)

Maximum number of COMMANDS to queue to the adapter True

Often I raise this to 2048 check with your disk vendor

MY VIO SERVER AND NPIV CLIENT ADAPTER SETTINGS

Long term DMA

NPIV Client (running at defaults before changes)

ADAPTER PRIORITIES AFFECT PERFORMANCE

Check the various Technical Overview Redbooks at http://www.redbooks.ibm.com/

Pay attention to entitlement

Turn on large send on VE adapters

Turn on large send on the SEA

Typically we set the following:

MY VIO SERVER SEA

Multifunction Cards up to one per CEC

5284/5287 PCIE2 2 port 10GbE SR (5284 is low profile)

Basically SR is fibre and SFP+ is copper twinax

POWER8 - S814 & S824 LAN ADAPTERS

NIM not supported

eConfig: Adapters => HiPerf/RIO

You can ignore the FCoE support on this

NETWORK PERFORMANCE AND THROUGHPUT

Available CPU power

Going from 1GbE to 10GbE does not mean 10x performance

Network traffic gets buffered

Need to turn it on at the network switch,

chdev l ent? a flow_cntrl=yes

TCP Offload settings largesend and large_receive

These improve throughput through the TCP stack

Set largesend on (TCP segmentation offload) should be enabled by default on a 10GbE SR

AIX - chdev l en? a largesend=on

Try setting large_receive on as well (TCP segment aggregation)

AIX - chdev l en? a large_receive=on

chdev l en? mtu=65535 (9000 is what we refer to as jumbo frames)