#CLUS

Troubleshooting
Cisco UCS Ethernet
Networking:
Like a TAC Engineer
John Hibbs, TAC Engineer
BRKINI-2764

#CLUS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKINI-2764

by the speaker until June 18, 2018.

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• What, Why, and How of Adapter Virtualization
• Demystifying UCS-B Networking
• Life of a Packet from OS to Fabric Interconnect
• TAC Troubleshooting Tips and Tricks

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
What, Why, and How of
Adapter Virtualization
Adapter Virtualization
Fewer Required Adapters, Same Functionality

Traditional Server UCS Server with VIC

NIC NIC NIC Converged

Network
Adapter (CNA)
NIC NIC NIC
NIC HBA
HBA HBA

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Adapter Virtualization
Adapter Virtualization Benefits
• Multiple paths
• Redundancy
• Just in time provisioning
• Add more as you need them
• Multiple vNICs
• Essentially multiple NICs
• VLAN isolation/consolidation
• Granularly allow all or only a few VLANs on each vNIC

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Adapter Virtualization
Adapter Virtualization Benefits
• Easy to configure pools of MACs/WWPNs
• Control over MACs/WWPN naming scheme
• A side MACs vs B side MACs naming convention
• MAC Addresses from pool can all say AABB:BEEF:001A

• Less hardware, more functionality

• Energy conservation

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Virtual Interface Card 1340

• 2 10Gbps links per

fabric

• 4 10Gbps links per

fabric with port
expander

• Modular LOM
Form Factor

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Virtual Interface Card 1380

• 4 10Gbps links per

fabric

• Mezzanine Form Factor

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Displaying Adapter
Information
Adapter Virtualization
Watching adapter logging in real time

CiscoLive-2018-A# connect adapter 1/1/1

adapter 1/1/1 # connect
No entry for terminal type "dumb";
using dumb terminal settings.
adapter 1/1/1 (top):1# show-log
180111-11:31:36.672197 mcp.vif vnic14/lif3/vif3.1: vif_delete_rx
180111-11:31:36.672275 mcp.vif vnic14/lif3/vif3.1: DOWN
180111-11:31:38.595682 mcp.vif vnic13/lif2/vif2.0: vif_delete_rx
180111-11:31:38.596125 mcp.vif vnic13/lif2/vif2.0: DOWN

When connecting to adapter the syntax is Chassis/Blade/Adapter

Ex: Chassis 2, Blade 3, Adapter 2 would be

CiscoLive-2018-A# connect adapter 2/3/2

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Adapter Virtualization
Displaying configured vNICs on Adapter

• 2 Ethernet vNICs and 1 used for FC

• FC interfaces also show as a vNIC

CiscoLive-2018-A# connect adapter 1/1/1

adapter 1/1/1 # connect
adapter 1/1/1 (top):2# attach-mcp
adapter 1/1/1 (mcp):2# vnic
-------------------------------------- --------- --------------------------
v n i c l i f v i f
id name type host state lif state uif ucsm idx vlan state
---- -------------- ------- ---- ----- --- ----- --- ----- ----- ---- -----
12 vnic_1 enet 0 UP 2 UP - 0 1196 66 1 UP
=>1 1195 77 1 UP
13 vnic_2 enet 0 UP 3 UP =>0 1197 67 1 UP
- 1 1198 78 1 UP
14 vnic_3 fc 0 UP 4 UP =>1 1170 70 600 UP

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Adapter Virtualization
vNIC Programming
CiscoLive-2018-A# connect adapter 1/1/1
adapter 1/1/1 # connect
adapter 1/1/1 (top):2# attach-mcp This will show us information
adapter 1/1/1 (mcp):3# vnic 12
vnicid : 12 about the vNIC’s programming
which is more detailed than the
name : vnic12
type : enet
state : UP
adminst : UP
flags : OPEN, INIT, LINKUP, NOTIFY_INIT, ENABLE
“vnic” command will show
ucsm name : vnic_1
spec_flags : FAILOVER, TRUNK
...
mac : 00:25:b5:aa:aa:1e We can also see the MAC
address to help with vNIC
vifid : 1195
...
vnic : [12] (n=1)
lif : [2] (n=1)
...
mapping

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Adapter Virtualization
vNIC Programming
CiscoLive-2018-A# connect adapter 1/1/1
adapter 1/1/1 # connect
adapter 1/1/1 (top):2# attach-mcp
adapter 1/1/1 (mcp):3# vnic 12 notify
hostpa: 0x1e60000
intr: 3
sz: 40
nsends: 2
nsenderrs: 0
csum: 29006

Adding the “notify” switch lets

link_state: UP
port_speed: 20000
mtu: 9000
msglvl: 0
uif: 1
us see the programmed MTU
status: PFC_PAUSE
error: 0
link_down_cnt: 0
perbi_rebuild_cnt: 0

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Adapter Virtualization
vNIC Programming
CiscoLive-2018-A# connect adapter 1/1/1
adapter 1/1/1 # connect
adapter 1/1/1 (top):2# attach-mcp
adapter 1/1/1 (mcp):13# lifstats -a 2 • Use the LIF that corresponds to
your vNIC
DELTA TOTAL DESCRIPTION
0 6090 Tx unicast frames without error
0 6 Tx multicast frames without error
0
0
13
414120
Tx broadcast frames without error
Tx unicast bytes without error
• -a includes 0 and non-0 fields
0
0
516
884
Tx multicast bytes without error
Tx broadcast bytes without error
• Real time traffic counters
0
0
0
0
Tx frames dropped
Tx frames with error • Only includes delta output since
last time command was run
0 0 Tx TSO frames
0 6095 Rx unicast frames without error
1 271684 Rx multicast frames without error
21 29765633 Rx broadcast frames without error
0 390146 Rx unicast bytes without error
64 39110920 Rx multicast bytes without error
5208 7291812805 Rx broadcast bytes without error
0 0 Rx frames dropped
0 0 Rx rq drop pkts (no bufs or rq disabled)
0 0 Rx rq drop bytes (no bufs or rq disabled)
0 0 Rx frames with error
0 0 Rx good frames with RSS
0 0 Rx frames with Ethernet FCS error
….

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Questions?
Demystifying UCS-B
Networking
 Ethernet Switching Modes
• UCS has two types of switching mode
• The mode affects how Layer 2
forwarding concepts are applied
1. End-Host Mode
• Appears like a hypervisor host to
upstream network
• Default and recommended best practice
2. Switch Mode
• FI works like a normal layer 2 switch with
spanning-tree

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Physical and Logical Ports in UCS

Uplink Ports
Fabric
Interconnect Server Ports
Network Interface
(NIF) Ports
IOM / FEX Host Interface (HIF)
Ports
Adapter Port (DCE)
Adapter vNIC
vNIC / vEthernet /
Virtual Interface (VIF)
#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
UCS Networking Tips
General Tips and Reminders
• Port Channels must use LACP
• No switching occurs between Fabric Interconnect’s L1 and L2 ports
• To traverse between both FIs, traffic must go north
• Same VLAN and same fabric will be switched locally on the FI

• When in end-host mode, the upstream switch ports should not

participate in spanning-tree
• spanning-tree port type edge trunk
• Templates = less misconfiguration risk

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
UCS Networking Tips
General Tips and Reminders
• When vNIC failover occurs, we send a GARP
• DO NOT use both fabric failover and OS failover
• This will cause failover problems
• FIs are primary/secondary for management services only
• Both FIs actively forward data
• FCoE runs between the FI and IOM
• Pause frames are expected to be seen and are expected behavior for
FCoE traffic

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Questions?
Life of a Packet

Tracing the MAC

 Start From The Most Southbound Endpoint
Gather information from the hypervisor about the VM…
• Hypervisor Host (blade)

• VM MAC Address
• vSwitch Port Group
• IP Address

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Understand the Virtual Network Path
Use “esxtop” command with the ‘n’ option
[root@localhost:~] esxtop

4:50:33pm up 2 days 23:29, 696 worlds, 3 VMs, 6 vCPUs; CPU load average: 0.00, 0.00, 0.00

PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/s PSZRX %DRPTX %DRPRX
33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
33554434 vmnic0 - vSwitch0 0.00 0.00 0.00 12.25 0.01 82.00 0.00 0.00
33554435 Shadow of vmnic0 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
33554439 vmk0 vmnic1 vSwitch0 10.11 0.01 181.00 8.16 0.00 73.00 0.00 0.00
33554440 vmnic1 - vSwitch0 10.11 0.01 181.00 20.02 0.01 79.00 0.00 0.00
33554441 Shadow of vmnic1 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
33554442 512108:rhel7-1 vmnic1 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
33554443 512185:ucs-central vmnic1 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
33554444 510584:Win7-Tool vmnic0 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331649 Management n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331650 vmnic2 - vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331651 Shadow of vmnic2 n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331652 vmnic4 - vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331653 Shadow of vmnic4 n/a vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331654 vmk1 vmnic2 vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
50331655 vmk2 vmnic4 vSwitch1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Determining Fabric Path in UCS
vmnic to vNIC mapping
We use the vmnic MAC Address and match it with the vNIC in UCS
[root@localhost:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:06:00.0 enic Up 10000Mbps Full 00:25:b5:a1:a1:a0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1 0000:07:00.0 enic Up 10000Mbps Full 00:25:b5:b1:b1:b1 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2 0000:08:00.0 enic Up 10000Mbps Full 00:25:b5:a1:a1:b0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:85:00.0 enic Up 10000Mbps Full 00:25:b5:b1:b1:b0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4 0000:86:00.0 enic Up 10000Mbps Full 00:25:b5:b1:b1:c0 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic5 0000:87:00.0 enic Up 10000Mbps Full 00:25:b5:a1:a1:a1 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

Common mistake – assuming vmnic# in ESXi and vNIC# in UCSM are the
same without verifying

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Determining Fabric Path in UCS
Match the MAC of the vmnic to the vNIC on the service profile

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Are we learning the MAC Address on the FI?
Send traffic from the VM and see what is working
• Based on what we have found, we expect the following:
• VM traffic should be traversing Fabric Interconnect A
• VLAN ID should be 211
• VM MAC Address – 00:50:56:8d:29:15
CiscoLive-2018-A# connect nxos a
CiscoLive-2018-A(nxos)# show mac address-table vlan 211
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 211 0050.568d.2915 dynamic 10 F F Veth4173

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Viewing all virtual interfaces for a server
Verifying that traffic is flowing as expected
CiscoLive-2018-A# show service-profile circuit server 1/1
Service Profile: CiscoLive/CiscoLive-ESX-01
Server: 1/1
Fabric ID: A
Path ID: 1
VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport
---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
12371 Up Active No Protection Unprotected 0/0/0 0/0/0 Ether
4173 vNIC2 Up Active No Protection Unprotected 0/0/0 0/0/1 Ether
4175 vNIC4 Up Active No Protection Unprotected 0/0/0 0/0/1 Ether
4179 fc0 Up Active No Protection Unprotected 0/0/0 2/0/15 Fc
Fabric ID: B
Path ID: 2
VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport
---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
12372 Up Active No Protection Unprotected 0/0/0 0/0/0 Ether
4176 vNIC3 Up Active No Protection Unprotected 0/0/0 0/0/2 Ether
4177 vNIC5 Up Active No Protection Unprotected 0/0/0 0/0/2 Ether
4180 fc1 Up Active No Protection Unprotected 0/0/0 0/0/44 Fc

Shows all programmed virtual interfaces for referenced server

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Which uplink is being used?
Understanding how pinning works
Basic rules to define which interface to pin to:
1. Which uplink interface carries ALL of the vNIC’s configured
VLANs?
2. Which uplink interfaces are active?
3. Which uplink has the least amount of vifs pinned to it currently?
Severity: Major
Code: F0283
Last Transition Time: 2014-02-18T23:08:51.270
ID: 1157440
Status: None
Description: ether VIF 1369 on server 6 / 4 of switch B down, reason: ENM source pinning failed
Affected Object: sys/chassis-6/blade-4/fabric-B/path-1/vc-1369
Name: Dcx Vc Down
Cause: Link Down
Type: Network
Acknowledged: No
Occurrences: 7
Creation Time: 2014-02-11T12:57:11.768
Original Severity: Major
Previous Severity: Cleared
Highest Severity: Major

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Which uplink is being used?
Uplink pinning commands CiscoLive-2018-A(nxos)# show pinning border-interfaces active

--------------------+---------+----------------------------------------
Border Interface Status SIFs
--------------------+---------+----------------------------------------
Po1 Active sup-eth2 Veth4137 Veth4145 Veth4173
Veth4175 Veth4178 Veth4183 Veth4195
Veth4197 Veth4200 Veth4208 Veth4210
Veth4212 Veth4214 Veth4216
Eth1/18 Active
Two common ways to view
CiscoLive-2018-A(nxos)# show pinning server-interfaces

---------------+-----------------+------------------------+----------------
-
SIF Interface Sticky Pinned Border Interface Pinned Duration
---------------+-----------------+------------------------+----------------
-
Eth1/1 No - -
Eth1/2 No - -
Eth1/3 No - -
Eth1/4 No - -
Eth1/11 No - -
Eth1/12 No - -
Veth4137 No Po1 1d 58:3:23
Veth4145 No Po1 1d 57:47:47
Veth4173 No Po1 1d 57:54:31

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Forwarding seems to be working as expected
What’s upstream?
CiscoLive-2018-A(nxos)# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/31(P) Eth1/32(P)

Port-channel is Switched and Up, with interfaces Up in Po

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Which link am I using?
Port-Channel Load Balancing
CiscoLive-2018-A(nxos)# show port-channel load-balance forwarding-path interface port-channel 106 vlan 177 src-ip
14.17.177.9 dst-ip 14.17.177.254

Missing params will be substituted by 0's.

Load-balance Algorithm on switch: source-dest-ip
crc8_hash: 21 Outgoing port id: Ethernet1/18
Param(s) used to calculate load-balance:
dst-ip:
src-ip:
14.17.177.254
14.17.177.9
dst-mac: 0000.0000.0000
Tip!
src-mac: 0000.0000.0000

Here we can see which interface in the port-channel the specific

source/destination traffic is using

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
What To Do Next?
Time to go North!
• If the UCS appears to be working as expected, move north
• If source and destination are:
• Same subnet or vlan = same broadcast domain
• Different subnet or vlan = different broadcast domain

• Use show cdp neighbors to determine upstream device and ports

• If not available, trace cables

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Reviewing the upstream switches
Are we learning the MAC Address?

MAC Address not learned for CiscoLive-2018-5k-A# show mac address-table vlan 211
Legend:
our VM on either upstream * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
switch VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 211 002a.6a35.4a41 static 0 F F sup-eth2
* 211 002a.6a39.2a41 static 0 F F Po3
* 211 547f.ee2f.3381 dynamic 60 F F Po33

CiscoLive-2018-5k-B# show mac address-table vlan 211

Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 211 002a.6a35.4a41 static 0 F F Po3
* 211 002a.6a39.2a41 static 0 F F sup-eth2
* 211 547f.ee2f.3381 dynamic 300 F F Po33

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Reviewing the upstream switches
Configuration correct?
CiscoLive-5k-A# show run interface ethernet 1/8

Upstream switchport is Ethernet 1/8 interface Ethernet1/8

on Nexus 5K switchport mode trunk

switchport trunk allowed vlan 1-210,212-4094
channel-group 11 mode active

CiscoLive-5k-A# show run interface port-channel 11

interface port-channel11
switchport mode trunk
Looking at the port-channel switchport trunk allowed vlan 1-210,212-4094
spanning-tree port type edge trunk
configuration, we can see that speed 10000
vpc 11
VLAN 211 is not allowed

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Reviewing the upstream switches
vPC Inconsistency Parameters
CiscoLive-2018-A# show vpc consistency-parameters global

Name
Legend: Type 1 : vPC will be suspended in case of mismatch
Type Local Value Peer Value
Tip!
------------- ---- ---------------------- -----------------------
STP Mode 1 Rapid-PVST Rapid-PVST
STP Disabled 1 None None
STP MST Region Name 1 "" ""
STP MST Region Revision 1 0 0
STP MST Region Instance to 1
VLAN Mapping
Type 1 inconsistency will
cause the vPC to be suspended
STP Loopguard 1 Disabled Disabled
STP Bridge Assurance 1 Enabled Enabled
STP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,
BPDUFilter, Edge BPDUGuard Disabled Disabled
STP MST Simulate PVST 1 Enabled Enabled

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
We learn the MAC
address, but on the
wrong VLAN?
 Is it being tagged in the OS?
To tag or not to tag

If you’re tagging in the OS, make sure you aren’t removing the VLAN tag on the NIC

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Is the Hypervisor tagging?
The vSwitch could also be tagging

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Check which VLAN is native
Management VLAN shouldn’t be configured as native

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
What if we don’t learn
the MAC address on the
FI?
Did we have issues traversing IOM and VIC?
• Three components to investigate:
• OS/Driver issues – Did the OS actually send the frame northbound?
• VIC Adapter
• IOM (NIF and HIF ports)

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Cisco VIC Adapter
Connecting and identifying logical interfaces…
CiscoLive-2018-A# connect adapter 1/1/1
adapter 1/1/1 # connect

adapter 1/1/1 (top):2# attach-mcp

adapter 1/1/1 (mcp):36# vnic
vnic id : internal id of vnic, use for other vnic cmds
vnic name/mac : ucsm provisioned name (-n) or mac address (-m)
vnic type : enet=ethernet, enet_pt=dynamic ethernet, fc=fcoe
vnic host : host
vnic state : state of vnic
lif : internal logical if id, use for other lif/vif cmds
lif state : state of lif
vif uif : bound uplink 0 or 1, =:primary, -:secondary, >:current
vif ucsm : ucsm id for this vif
vif idx : switch id for this vif
vif vlan : default vlan for traffic
vif state : state of vif
-------------------------------------- --------- --------------------------
v n i c l i f v i f
id mac type host state lif state uif ucsm idx vlan state
---- -------------- ------- ---- ----- --- ----- --- ----- ----- ---- -----
14 0025:b5a1:a1a0 enet 0 UP 4 UP =>1 4173 30 1 UP
15 0025:b5b1:b1b1 enet 0 UP 5 UP =>0 4174 29 1 UP
16 0025:b5a1:a1b0 enet 0 UP 6 UP =>1 4175 31 1 UP
17 aa25:b5a1:a1a0 fc 0 UP 7 UP =>1 4179 21 1000 UP

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Cisco VIC Adapter
Logical Interface Counters
adapter 1/1/1 (mcp):28# lifstats -a 4
DELTA TOTAL DESCRIPTION
0 0 Tx unicast frames without error
0 0 Tx multicast frames without error
…
0 0 Tx frames dropped
0
0
0
0
Tx
Tx
frames with error
TSO frames
• Tx would mean we
0
216
0
479103
Rx
Rx
unicast frames without error
multicast frames without error
sent frames with
5321
0
10558692
0
Rx
Rx
broadcast frames without error
unicast bytes without error
errors to the IOM
19077 44142182 Rx multicast bytes without error
386336 778358713 Rx broadcast bytes without error
0 0 Rx frames dropped
0 0 Rx rq drop pkts (no bufs or rq disabled)
0 0 Rx rq drop bytes (no bufs or rq disabled)
0
0
0
0
Rx
Rx
frames with error
good frames with RSS • Rx would mean OS
0
24
0
42055
Rx
Rx
frames with Ethernet FCS error
frames len == 64 sent bad frame to
5403
27
10715806
81854
Rx
Rx
frames 64 < len <= 127
frames 128 <= len <= 255 the adapter
83 198061 Rx frames 256 <= len <= 511
…
0.632bps Tx rate
6.499kbps Rx rate Tip!

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
We don’t see any issues
on the adapter
 IOM Troubleshooting – 2nd Gen
CiscoLive-2018-A# connect iom 1
fex-1# show platform software woodside sts
Board Status Overview: Uplink #: 1 2 3 4 5 6 7 8
legend: Link status: | | | |
' '= no-connect +-+--+--+--+--+--+--+--+-+
X = Failed SFP: [$][$][$][$][ ][ ][ ][ ]
- = Disabled +-+--+--+--+--+--+--+--+-+
: = Dn | N N N N N N N N |
| = Up | I I I I I I I I |
[$] = SFP present | 0 1 2 3 4 5 6 7 |
[ ] = SFP not present | |
[X] = SFP validation failed | NI (0-7) |
------------------------------ +------------+-----------+
|
+-------------------------+-------------+-------------+---------------------------+
| | | |
+------------+-----------+ +-----------+------------+ +------------+-----------+ +-------------+----------+
| HI (0-7) | | HI (8-15) | | HI (16-23) | | HI (24-31) |
| | | | | | | |
| H H H H H H H H | | H H H H H H H H | | H H H H H H H H | | H H H H H H H H |
| I I I I I I I I | | I I I I I I I I | | I I I I I I I I | | I I I I I I I I |
| 0 1 2 3 4 5 6 7 | | 8 9 1 1 1 1 1 1 | | 1 1 1 1 2 2 2 2 | | 2 2 2 2 2 2 3 3 |
| | | 0 1 2 3 4 5 | | 6 7 8 9 0 1 2 3 | | 4 5 6 7 8 9 0 1 |
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+
[ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ]
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+
- - | | - | - | | | | | - | | |
1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1
6 5 4 3 2 1 0
\__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/
blade8 blade7 blade6 blade5 blade4 blade3 blade2 blade1

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 IOM Troubleshooting – 3rd Gen
fex-1# show platform software tiburon sts
Board Status Overview:
legend:
' '= no-connect
X = Failed
- = Disabled
: = Dn
| = Up
. = SFP waiting for t_start_up timer expiration
i = SFP waiting for prom info read
c = SFP waiting for t_start_up_cooled timer expiration
* = SFP present
[X] = SFP validation failed
------------------------------
* * * * <- QSFPs
+-----------------------------------------------------------------------------------------------+
- - - - - - - - : : : : : : : : : : : : | : : :
+-----------------------------------------------------------------------------------------------+
|0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23|
|I |
|N |
| Tiburon |
| Asic: 0 |
|H |
|I |
|0 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4|
| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7|
+-----------------------------------------------------------------------------------------------+
- - - - - - - - - - - - - - - - - - - - : : : : - - - - - | - | - - - - - - - - - - - - - - - -
3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1
2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
\_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ \_\_/_/ (r = retimer lane configured)
blade8 blade7 blade6 blade5 blade4 blade3 blade2 blade1

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 IOM Troubleshooting
fex-1# show platform software {tiburon/woodside} rmon 0 hi31
+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+
| TX | Current | Diff | RX | Current | Diff |
+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+
| TX_PKT_LT64 | 0| 0| RX_PKT_LT64 | 0| 0|
| TX_PKT_64 | 0| 0| RX_PKT_64 | 386| 15|
| TX_PKT_65 | 379| 15| RX_PKT_65 | 13| 0|
| TX_PKT_128 | 8| 0| RX_PKT_128 | 754| 75|
| TX_PKT_256 | 717| 51| RX_PKT_256 | 0| 0|
| TX_PKT_512 | 12| 0| RX_PKT_512 | 22| 4|
| TX_PKT_1024 | 0| 0| RX_PKT_1024 | 0| 0|
| TX_PKT_1519 | 24| 0| RX_PKT_1519 | 0| 0|
| TX_PKT_2048 | 0| 0| RX_PKT_2048 | 0| 0|
| TX_PKT_4096 | 0| 0| RX_PKT_4096 | 0| 0|
| TX_PKT_8192 | 0| 0| RX_PKT_8192 | 0| 0|
| TX_PKT_GT9216 | 0| 0| RX_PKT_GT9216 | 0| 0|
| TX_PKTTOTAL | 1140| 66| RX_PKTTOTAL | 1175| 94|
| TX_OCTETS | 341435| 20207| RX_OCTETS | 163687| 15984|
| TX_PKTOK | 1140| 66| RX_PKTOK | 1175| 94|
| TX_UCAST | 384| 15| RX_UCAST | 588| 55|
| TX_MCAST | 756| 51| RX_MCAST | 543| 38|
| TX_BCAST | 0| 0| RX_BCAST | 44| 1|
| TX_VLAN | 0| 0| RX_VLAN | 0| 0|
| TX_PAUSE | 0| 0| RX_PAUSE | 0| 0|
| TX_USER_PAUSE | 0| 0| RX_USER_PAUSE | 0| 0|
| TX_FRM_ERROR | 0| 0| | | |
| | | | RX_OVERSIZE | 0| 0|
| | | | RX_TOOLONG | 0| 0|
| | | | RX_DISCARD | 0| 0|
| | | | RX_UNDERSIZE | 0| 0|
| | | | RX_FRAGMENT | 0| 0|
| | | | RX_CRC_NOT_STOMPED | 0| 0|
| | | | RX_CRC_STOMPED | 0| 0|
| TX_OCTETSOK | 341435| 20207| RX_OCTETSOK | 163687| 15984|
+----------------------+----------------------+-----------------+----------------------+----------------------+-----------------+

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 IOM Troubleshooting
Do we have errors on the NIF or HIF ports?
fex-1# show platform software woodside loss
+-------+-------------------------------------+------------+-+-----------------------------------+---------------------------------------+
| | | | | | |
| | | | | | frm_to |
| | |Port Extra | | +---------------------------------------|
| | RMON | Drop |S| SS Loss Counters | COS | XOFF |
| +------------+-----------+------------+------------|S|-----------+-----------+-----------+---------------------------------------|
| Port | Tx Pause | Rx Pause | Errors | Counters |x| RX SS | Tx SS | SS Total |0 |1 |2 |3 |4 |5 |6 |7 |0 |1 |
+-------+------------+-----------+------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+
| 0- NI3| 0| 59896| 0| 7|0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |5| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+
| 0-HI31| 770| 0| 0| 0|0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |1| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |3| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |4| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |5| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
| | | | | |7| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
+-------+-------------------------------------+------------+-+-----------+-----------+-----------+---+---+---+---+---+---+---+---+---+---+

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 IOM Troubleshooting
What is the traffic rate for the HIF port?
fex-1# show platform software woodside rate
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| Port || Tx Packets | Tx Rate | Tx Bit || Rx Packets | Rx Rate | Rx Bit |Avg Pkt|Avg Pkt| |
| || | (pkts/s) | Rate || | (pkts/s) | Rate | (Tx) | (Rx) |Err|
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| 0-BI || 28 | 5 | 5.26Kbps || 22 | 4 | 8.66Kbps | 97 | 226 | |
| 0-CI || 65 | 13 | 26.09Kbps || 52 | 10 | 36.23Kbps | 230 | 415 | |
| 0-NI7 || 14 | 2 | 12.61Kbps || 12 | 2 | 6.69Kbps | 543 | 329 | |

Tip!
| 0-NI6 || 18 | 3 | 7.66Kbps || 34 | 6 | 19.64Kbps | 246 | 341 | |
| 0-NI5 || 27 | 5 | 15.96Kbps || 24 | 4 | 7.81Kbps | 349 | 183 | |
| 0-NI4 || 16 | 3 | 10.22Kbps || 20 | 4 | 11.60Kbps | 379 | 342 | |
| 0-NI3 || 65 | 13 | 37.61Kbps || 62 | 12 | 35.64Kbps | 341 | 339 | |
| 0-NI2 || 7 | 1 | 6.88Kbps || 13 | 2 | 4.09Kbps | 594 | 177 | |
| 0-NI0 || 59 | 11 | 27.28Kbps || 57 | 11 | 13.22Kbps | 269 | 125 | |
| 0-HI31 || 11 | 2 | 18.93Kbps || 88 | 9 | 18.88 bps | 317 | 382 | |
…
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+

Is your HIF or NIF transmitting traffic at an expected rate?

Is it comparable to the other interfaces?

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 IOM Troubleshooting
When did the port come up or down?
fex-1# show platform software woodside oper
ASIC 0:
+---+-----+-+----+-----------+-----------------+
| | | | | MAC | | PHY | | |
| | | | | |S|---+---| | |
| | |A| |-+-+-+-+-+-|e|XFI|SFI| | |
| P | |d| | |L|T|R|R|T|r|---+---| | |
| o | |m| |P|o|R|R|F|F|d|p|p|p|p|S| |
| r | |i|Oper|C|c|d|d|l|l|e|m|c|c|m|F|ucode|----------------------------+----------------------------+-----+
| t |Name |n| St |S|k|y|y|t|t|s|d|s|s|d|P| ver | Time last came Up | Time last went Down |Flaps|
+---+-----+-+----+-+-+-+-+-+-+-+-+-+-+-+-+-----+----------------------------+----------------------------+-----+
| 0 |HI0 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 06/07/2018 13:58:50.071434 | 06/07/2018 13:58:19.281217 | 49|
| 1 |HI1 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 06/07/2018 13:58:48.775615 | 06/07/2018 13:58:19.282638 | 49|
| 2 |HI2 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 06/07/2018 13:58:48.978491 | 06/07/2018 13:58:19.283948 | 50|

Tip!
| 3 |HI3 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 06/07/2018 13:58:51.116372 | 06/07/2018 13:58:19.285257 | 49|
| 4 |HI4 |-| Dn |0|0|1|1|0|0|0|0|0|0|0| | 0.00| 12/31/1969 19:00:00.000000 | 12/31/1969 19:00:00.000000 | 0|
| 5 |HI5 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 04/18/2018 12:24:02.434054 | 04/18/2018 12:23:49.405967 | 84|
| 6 |HI6 |-| Dn |0|0|1|1|0|0|0|0|0|0|0| | 0.00| 12/31/1969 19:00:00.000000 | 12/31/1969 19:00:00.000000 | 0|
| 7 |HI7 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 04/18/2018 12:24:01.186227 | 04/18/2018 12:23:49.407197 | 84|
| 8 |HI8 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 02/13/2018 11:07:16.356256 | 02/13/2018 11:07:04.206411 | 12|
| 9 |HI9 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 02/13/2018 11:07:17.449138 | 02/13/2018 11:07:04.207779 | 12|
…
|31 |HI31 |E| Up |1|1|1|1|1|1|1|0|0|0|0| | 0.00| 02/13/2018 11:03:17.262892 | 02/13/2018 11:02:56.836439 | 9|
…
+---+-----+-+----+-+-+-+-+-+-+-+-+-+-+-+-+-----+----------------------------+----------------------------+-----+

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
IOM Troubleshooting
Tip: Check the IOM ports from NX-OS…
• Ethernet ports x/y/z correlate
to HIFs on the IOM CiscoLive-2018-A(nxos)# show interface counters errors

• x = IOM --------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

• y = module on IOM
--------------------------------------------------------------------------------

Eth1/1/1 0 0 0 0 0 0

• z = port Eth1/1/2
Eth1/1/3
0
0
0
1170
0
0
0
1170
0
0
0
0
Eth1/1/4 0 0 0 0 0 0
Eth1/1/5 0 0 0 0 0 0
Eth1/1/6 0 0 0 0 0 0
Eth1/1/7 0 0 0 0 0 0
Eth1/1/8 0 0 0 0 0 0
Eth1/1/9 0 0 0 0 0 0
Eth1/1/10 0 0 0 0 0 0
Eth1/1/11 0 0 0 0 0 0
Eth1/1/12 0 0 0 0 0 0
Eth1/1/13 0 0 0 0 0 0

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 IOM Troubleshooting
Example where errors on IOM are indicating issues downstream…
CiscoLive-2018-A(nxos)# show interface counters errors

--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Uplink interfaces rcvd
Eth1/6
Eth1/21
0
0
103
103
0
0
103
103
0
0
0
0
bad frames
Po100 0 206 0 206 0 0
Uplink port-channel
counters
Po1351 0 207 0 207 0 0
Adapter-IOM Po
Eth3/1/1 0 0 0 0 0 0
Eth3/1/2 0 0 0 0 0 0
Eth3/1/3
Eth3/1/4
0
0
0
207
0
0
0
207
0
0
0
0 HIF ports on IOM

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
My disjointed network
has connectivity issues?
What is Disjoint L2?
Two different L2 Domains…
Prod DMZ
• When UCS needs to connect to two
different L2 Domains
• Non-adjacent network

• Typical Use Cases

• DMZ
• Backup Solution
• Legacy

Prod vNIC DMZ vNIC

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Disjoint Layer 2 Fault
What’s this fault?
Severity: Major
Code: F0283
Last Transition Time: 2014-02-18T23:08:51.270
ID: 1157440
Status: None
Description: ether VIF 1369 on server 6 / 4 of switch B down, reason: ENM source pinning failed
Affected Object: sys/chassis-6/blade-4/fabric-B/path-1/vc-1369
Name: Dcx Vc Down

• This fault indicates a pinning

Cause: Link Down
Type: Network
Acknowledged: No
Occurrences: 7
Creation Time: 2014-02-11T12:57:11.768
failure
Original Severity: Major
Previous Severity: Cleared
Highest Severity: Major
• It tells us which blade and VIF has
a config issue

• Now we need to look at that

blade’s config and find the issue

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Configuration is half way done

What about these

VLANs?

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Uplink Port Configuration In This Scenario
Uplink Configuration
CiscoLive-2018-B(nxos)# show running-config interface ethernet 1/17

interface Ethernet1/17
description U: Uplink
pinning border
pinning server nf-exporter
switchport mode trunk
switchport trunk allowed vlan 1,104,111,204,211,304,311,900
udld disable
All the VLANs are still allowed!
no shutdown

CiscoLive-2018-B(nxos)# show running-config interface port-channel 2

interface port-channel2
description U: Uplink
switchport mode trunk
switchport trunk allowed vlan 1,104,111,204,211,304,311
pinning border
speed 10000

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Understanding the Designated Receiver
• Absence of STP means we rely on other mechanisms to avoid
loops
• An uplink will be selected as the broadcast and multicast receiver
• This is referred to as the Designated Receiver
• This is on a per VLAN basis

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Who is the Designated Receiver?
Viewing Designated Receiver
CiscoLive-2018-B(nxos)# show platform software enm internal info vlandb all

vlan_id 1
-------------
Designated receiver: Po2

Prod
Membership:
Eth1/17 Po2 DMZ
vlan_id 104 VLAN 900 Only
-------------
Designated receiver: Eth1/17
Po2
Membership:
Eth 1/17
Eth1/17 Po2

vlan_id 111
-------------
Designated receiver: Po2
Membership:
Eth1/17 Po2

vlan_id 900
-------------
Designated receiver: Eth1/17
Membership: Prod vNIC DMZ vNIC
Eth1/17

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Disjoint Layer 2 Configured in full…

You must explicitly define

which interfaces the VLAN
should traverse for ALL
VLANs in DJL2

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Correct configuration from CLI
CiscoLive-2018-B(nxos)# show running-config interface ethernet 1/17

interface Ethernet1/17
description U: Uplink
pinning border
pinning server nf-exporter
switchport mode trunk
switchport trunk allowed vlan 1,900
udld disable
The extra VLANs are removed
no shutdown

CiscoLive-2018-B(nxos)# show running-config interface port-channel 2

interface port-channel2
description U: Uplink
switchport mode trunk
switchport trunk allowed vlan 1,104,111,204,211,304,311 VLAN 900 is not present
pinning border
speed 10000

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Questions?
TAC Troubleshooting
Tips
Troubleshooting for UCS
General Tips
• You know your systems better than anybody else!
• Generate tech supports files sooner than later!
• Have a detailed Problem Description
• Recent changes, timestamps, faults and error messages
• Have a network topology at a port level, not just system

??????????????????
#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Network Troubleshooting for UCS
Methodology
• Simplify the issue, not just “It’s slow”
• Question to the void!
• UCS only deals with Layer 2
• Are we learning the MAC address on the FI?
• Is the issue fabric specific?
• If multiple servers are affected, pick one to work with
• Check each step systematically
• Have you checked the drivers?

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Problem Isolation

The Network UCS Domain UCS Server DB-VM-01

Is Down Is Offline Not Working Unreachable

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Problems related to
CRCs
Layer 1—FCS Errors
Frame Check Sequence Errors
• FCS errors indicate the frame has failed the Frame Check Sequence and
has been flagged.
• Since the FI is a cut-through switch, we don’t check the FCS field prior to
forwarding the frame
• Instead we forward, STOMP it, and increment the CRC counter
• Could be a Layer 1 issue or an MTU mismatch

CiscoLive-2018-A(nxos)# show hardware internal carmel port ethernet 1/6 | grep STOMP

RX_PKT_CRC_NOT_STOMPED | 5123 First port to see bad FCS check

RX_PKT_CRC_STOMPED | 0 Not the first port to see bad FCS check

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Layer 1–CRC Errors
Looking at Interface Errors
CiscoLive-2018-A(nxos)# show interface counters errors

--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
First, look at all
Eth1/6
Eth1/21
0
0
103
103
0
0
103
103
0
0
0
0 interfaces to see if any
interfaces have errors
Po100 0 206 0 206 0 0

Po1351 0 207 0 207 0 0

Eth3/1/1 0 0 0 0 0 0
Eth3/1/2 0 0 0 0 0 0
Eth3/1/3 0 0 0 0 0 0
Eth3/1/4 0 207 0 207 0 0

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Layer 1–CRC Errors
Looking at ASIC level errors
Tip!
CiscoLive-2018-A(nxos)# show hardware internal carmel crc
+----------+------------+------------+------------+------------+------------+------------+------------+
| Port | MM rx CRC | MM Rx Stomp| FI rx CRC | FI Rx Stomp| FI tx CRC | FI tx Stomp| MM tx CRC |
+----------+------------+------------+------------+------------+------------+------------+------------+
| Eth 1/1 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/2 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/3 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/4 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/5 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/6 | --- | --- | 103 | --- | --- | --- | --- |
| Eth 1/7 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/8 | --- | --- | --- | --- | --- | --- | --- |
| Eth 1/9 | --- | --- | --- | --- | --- | --- | --- |

• MM indicates “off the wire”

• If MM counter increments, FI counter should increment too
• If FI counter increments, the MM counter won’t necessarily
increment

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Layer 1 – Eye Height
Optics Information
• If eye height values are too low, this usually indicates a bad optic
• Can cause CRC errors and link failures
• If flapping interface doesn‘t increase reading, then likely a bad SFP

CiscoLive-2018-A(nxos)# show hardware internal carmel port ethernet 1/17 eye_info

Serdes Eye Info:
NI Serdes:
height: 153 mv width: 703 mUI (raw=31/2d) 04/29/2018 16:49:01.102640:
Min height: 137 mv Min width: 656 mUI (raw=2c/2a)
Max height: 165 mv Max width: 734 mUI (raw=35/2f)

FI lane 0 Serdes:
height: 200 mv width: 843 mUI (raw=40/36) 04/29/2018 16:49:01.107761:
Min height: 200 mv Min width: 750 mUI (raw=40/30)
Max height: 200 mv Max width: 859 mUI (raw=40/37)

FI lane 1 Serdes:
height: 200 mv width: 828 mUI (raw=40/35) 04/29/2018 16:49:01.112893:
Min height: 200 mv Min width: 781 mUI (raw=40/32)
Max height: 200 mv Max width: 859 mUI (raw=40/37)

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Layer 1–Interface Errors?
Key interface output
CiscoLive-2018-A(nxos)# show interface ethernet 1/17
Ethernet1/17 is up
Dedicated Interface
Hardware: 1000/10000 Ethernet, address: 547f.eec5.4e58 (bia 547f.eec5.4e58)
Description: U: Uplink
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk What mode are we in?
full-duplex, 10 Gb/s, media type is 10G Duplex and media speed
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 13week(s) 6day(s) Last link flap
Last clearing of "show interface" counters never Last clearing of counters
30 seconds input rate 13480 bits/sec, 1685 bytes/sec, 5 packets/sec
30 seconds output rate 19488 bits/sec, 2436 bytes/sec, 5 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 134.72 Kbps, 191 pps; output rate 18.16 Kbps, 5 pps
…

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Layer 1–Interface Errors?
Important interface output
…
RX
36469395 unicast packets 30768771 multicast packets 16827231 broadcast packets
84065397 input packets 27257083301 bytes
6770660 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
35483045 unicast packets 1184003 multicast packets 30394067 broadcast packets
67061115 output packets 21523455983 bytes 554268 jumbo packets
0 output errors 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
1 interface resets

• In this output we see RX and TX counters

• The last line shows how many interface resets have occurred

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Jumbo frames don’t
work?
QoS
QoS Configuration
• Fabric Interconnect only matches classes, defaults to “Best Effort”
• 2 no drop classes, 1 is for Fibre-Channel
• 9216 MTU for jumbo frames, overhead!

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
QoS
• After configuring QoS, you still have to apply the policy to the
adapter
• Don’t forget about the MTU!

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Tips!
• Don’t run twinax cables through same conduit as power cables
• Can cause CRC errors. True story.

• CoS remarking
• If upstream switch changes the frames’ CoS value, it may be improperly switched
or dropped when it returns to the FI
• Configure Best Effort class to have at least as high an MTU as any other class
• On a Nexus 5000 switch, the MTU setting is global

• 6332 FI requires reload when modifying global QoS policies

Tip!

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
MAC is moving?
MAC Moves
Why would a MAC move?
• Typically reported by VM/Host owner for intermittent connectivity
• Could be caused by:
• Upstream STP loop
• Duplicate MAC Addresses could cause MAC moves Tip!
• Cloned OS boot LUNs
• Migrating Service Profiles
• vPC Peer-Link is down
• Server NIC teaming is using unsupported Active/Active

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
MAC Flapping
CiscoLive-2018-A (nxos)# show platform fwm info mac 0026.5108.2f48 128 MAC and VLAN

mac vlan 1.4044 mac 0026.5108.2f48: vlan 1.128

mac vlan 1.4044 mac 0026.5108.2f48: learned-on Eth1/1/33 age 0 ref_map = 'vlan if’
mac vlan 1.4044 mac 0026.5108.2f48: nohit_count 0 hw_programmed 1 mac_clone 0
mac vlan 1.4044 mac 0026.5108.2f48: old_if_index 'null'mac vlan 1.128 mac 0026.5108.2f48: pss_flags 0
mac vlan 1.4044 mac 0026.5108.2f48 cfg attrs - not-cli-cfg not-static movable no-drop no-regmac non-netstack-learnt not-secure not-src-drop
mac vlan 1.4044 mac 0026.5108.2f48: mcec_flags 0x1, mac_info_flags 0, rem_if 0, sync_count 0 rcv_count 0
mac vlan 1.4044 mac 0026.5108.2f48: CDCE Address 15:0:0:bc:a:6

Mac history (Last 35 operations):

Total operations: 3:
Operation: Mac create (9)
(flags: Loc (0x1) mac_info_flags (0x0) if: 0x1f000800 hint: 0) <--- This will tell us what ports it was learned on from the 0x numbers
at Thu May 1 12:59:25 2014
Operation: Mac learned from hw (40)
(flags: Loc (0x1) mac_info_flags (0x0) if: 0x1f000800 hint: 0) <--- This will tell us what ports it was learned on from the 0x numbers
at Thu May 1 12:59:25 2014
Operation: Mac hardware update successful (24)
(flags: Loc (0x1) mac_info_flags (0x0) if: 0x1f000800 hint: 0) <--- This will tell us what ports it was learned on from the 0x numbers
at Thu May 1 12:59:25 2014

CiscoLive-2018-A(nxos)# show interface snmp-ifindex | inc 0x1f000800

...
Eth1/1 520095744 (0x1f000800) <----- This is the interface where we learned the MAC

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
MAC Flapping Investigation
Other Helpful Commands
• show platform fwm event-history vlan X
• show platform fwm info stm-stats
• show platform fwm errors
• show platform fwm info error history
Tip!

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Questions?
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKINI-2764

by the speaker until June 18, 2018.

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Complete your online session evaluation

Give us your feedback to be entered

into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing
on demand after the event at www.CiscoLive.com/Online.

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings

#CLUS BRKINI-2764 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Thank you

#CLUS
#CLUS