BRKDCT 3101 PDF

Nexus9000(Standalone)
Architecture And
Troubleshooting
Shridhar V. Dhodapkar Technical Leader (Services)
CCIE 6367 (Routing & Switching)
BRKDCT-3101
Session Abstract
This session presents briefly the architecture of the latest generation
of Nexus 9000 Series Modular switches. Topics include supervisors,
fabrics, I/O modules, forwarding engines, and physical design elements, as
well as the Top of the Rack Nexus9300 Switches.
The session will also cover how to monitor the health of the system.
We will walk you through in depth troubleshooting Tools and Techniques.
Session Goal
To provide an overall understanding of the Nexus 9000 switching

architecture, supervisor, fabric, and I/O module design, packet flows, and
key forwarding engine functions
This session will introduce System Telemetry, Troubleshooting tool Kits

and troubleshooting case scenarios
This session will not examine NX-OS software architecture or other Nexus
platform architectures
Related Sessions
BRKARC-2222 - Cisco Nexus 9000 architecture
BRKARC-3471 - Cisco NX-OS Software Architecture
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches
Agenda
Introduction
Architecture
System Health check Telemetry
Troubleshooting Toolkit
Nexus 9000 Troubleshooting
Common Link Layer Issues-L1

Fabric Connectivity and
In band
L2/L3 Packet Forwarding
vPC
Nexus9000 Specific Limitation and Goodies
Introduction
Introduction-What is Nexus9000 Family ?
Nexus 9500 Series Switches

Nexus9504/Nexus9508/Nexus9516
Nexus 9300 Series Switches

N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396
Architecture
9500 Field Upgradeable Units (FRU)

9500 has the following modular components which can upgraded or
replaced in the field
Nexus 9508 Front View
Nexus 9508 Rear View
Supervisor
Fabric Module
Line Card
System Controller
Fan Tray
Power Supply
The Supervisor, System controller ,Fabric Module and LC have OBFL

(On-Board Failure Logging) for failure analysis
Nexus 9500 Platform FRU

Supervisor Module-What it is Role
Redundant Half-width supervisor engine

Common for 4-, 8-, and 16- slot chassis
External Clock Input (PTP)
Responsible for control-plane functions
System Controller-What it is Role
Offload supervisor from internal device management tasks

Central Point of Chassis Control
EOBC Switch (Ethernet Out of Band Channel)
EPC Switch (Ethernet Protocol Channel)
Power Supplies via SMB (System Management Bus)
Fan Trays
Nexus 9500 Platform Line Card

Line Cards
FM6
I/O module with Merchant and

Merchant+ ASIC
FM4
FM5
HG
MUX1
HG
MUX2
HG
MUX4
Have Various Forwarding Tables

L2 Mac Table And L3 Host Table
ACL and Buffers for Queuing
45
01
23
FM3
FM2
HG
MUX3
HG
MUX5
89
10
11
7
MN6Port
01
23
Northstar 1
ASIC Name
MF Port
0
2
NFE=Network Forwarding Engine-Trident 2(T2)

ALE=Application Leaf Engine-North Star(NS)
7
5
-Donner
3
5
68
91
1
2
62
4
1Warpcor
0
2
e 9
T2
45
7
MN6 Port
HG
MUX6
89
10
11
Northstar 2
MF Port
0
2
HG
7
5
N9K-X9564PQ
91
1
2
62
4
T2
Note: Internal ports are called as Hi-Gig/HG ports

F
F F F F F F F F F
P
PPPPPPPPP
1
1 2 3 4 5 6 7 8 9
0
FM1
40G
QSFP
F
P
1
1
F
P
1
2
10G SFP+ Ports

F
P
1
3
F
P
1
4
F
P
1
5
F
P
1
6
F
P
1
7
F
P
1
8
F
P
1
9
F
P
2
0
F
P
2
1
F
P
2
2
F
P
2
3
F
P
2
4
F
P
2
5
F
P
2
6
F
P
2
7
F
P
2
8
F
P
2
9
F
P
3
0
F
P
3
1
F
P
3
2
F
P
3
3
F
P
3
4
F
P
3
5
F
P
3
6
F
P
3
7
F
P
3
8
F
P
3
9
F
P
4
0
F
P
4
1
F
P
4
2
F
P
4
3
F
P
4
4
F
P
4
5
F
P
4
6
F
P
4
7
F
P
4
8
FP FP FP FP
49 50 51 52
Nexus 9500 Fabric Module

Fabrics Modules
Interconnect Line Card slots

Installed at the rear of the chassis
Leverages Broadcom Trident II ASICs
Max 1.92 Tbps per line card slot (6 Fabric Cards)
960 Gbps per line card slot (3 Fabric Cards)
All Fabric Cards are active and carry traffic
Fan Tray requires Fabric Card to be present in even slot
Trident II
ASIC-NFE
32 x 40G
Hi-Gig2
Trident II
ASIC-NFE
32 x 40G
Hi-Gig2
Nexus 9500 Fabric Module

Data Plane Scaling for 8-Slot Chassis
An 8-Slot chassis fabric module can provide up to 320Gbps to each Line Card slot
With 6 fabric modules, each Line Card slot can have up to 1.92Tbps duplex
forwarding bandwidth
T2
Fabric 1
T2
320 Gbps
(8x 40Gbps)
Fabric 2
T2
T2
Fabric 3
T2
T2
Fabric 4
T2
T2
Fabric 5
T2
T2
Fabric 6
T2
T2
320 Gbps
(8x 40Gbps)
320 Gbps
(8x 40Gbps)
320 Gbps
(8x 40Gbps)
320 Gbps
(8x 40Gbps)
320 Gbps
(8x 40Gbps)
320 Gbps
640 Gbps
960 Gbps
Line Card Slot

1.28 Tbps
1.60 Tbps
1.92 Tbps
Distributed Data Plane of Nexus 9500 Series Switches

Fabric 1
Fabric 2
Fabric 3
Fabric 4
Fabric 5
Fabric 6
Nx NFE
Nx NFE
Nx NFE
Nx NFE
Nx NFE
Nx NFE
N = 1 for N9504
N = 2 for N9508
N = 4 for N9516
2 x 42 Gbps
2 x 42 Gbps
ALE
ALE
12 x 42 Gbps
12 x 42 Gbps
NFE
NFE
ALE
ALE
12 x 42 Gbps
12 x 42 Gbps
NFE
NFE
Note: Internal ports are called as Hi-Gig/HG ports
Nexus9500 Series Line Card Summary

Information
X9600 Series Line

Cards
X9500 Series Line

Cards
X9400 Series
ASIC Technology
Merchant only
N9K-X9636PQ
Merchant+
N9K-X9564PX
N9K-X9564TX
N9K-X9536PQ
Merchant only
N9K-X9432PQ
N9K-X9464PX
N9K-X9464TX
Number of ASIC
3 T2
2 T2 + 2 NS
2 T2 + 2 NS
2 T2 + 2 NS
2 T2 40 gig 32 Ports
1 T2 48 1/10 gig , 4
QSFP
Non Blocking
Non Blocking
Line rate > 200 byte

packet
36 MB
104 MB
12 MB with one T2
24 MB with two T2
Buffer Size
High Level Block Diagram-N9500
All PSU, SC, SUP, FM, and

LC plug into the same
Power Supply Interface
N9K-C9300 Series
Fixed Chassis
Port QSFP+ Uplink Module
1 RU or 2RU or 3RU
AC/DC Power Supply
Front-to-Back & Back-to-Front Airflow
Latency: 1-2 usec
Wire-Speed L2/L3 Forwarding
Switch will not boot up without GEM
Expansion Module
Nexus 9300 Series Switch Summary

N9396TX/PX
N93128TX
N9372TX
N9372TX
N9372PX
NFE (BCM T2)
ALE ( NorthStar)/GEM
GEM-1 NS
GEM-1 NS
No GEM-1
Donner
No GEM -1
Donner
No GEM- 1
Donner
Oversubscribed
No
1.5:1
No
No
No
Line Rate
Yes
Yes (packets >
Yes
Yes
194-Bytes)
QoS Classes
Buffer (MB)
36 (12*3)
104 (12*2+40*2)
24 (12*2)
104 (12*2+40*2)
104 (12*2+40*2)
High Level Block Diagram-N9300

GEM 4x 40GE QSFP+ Uplinks
12
x 40G
(12+12)x12
Hi-Gig2
CPU
2C 1.5GHz
(12+12)x12
Trident II
ASIC
BRCM Trident2
12 x 40G
48 10G
Ethernet
PCIe
DDR3
DIMM2
16GB Total
NorthStar
Northstar ASIC 1
Ingress
Egress
x 12 40G
Network Interfaces
Front Panel 48x 1GE/10GE Ports
The last 2/3 numbers stand for

total bandwidth in Gigabits
1000BaseT
Mgmt Port
2 USB
Ports
eUSB
Boot Flash
12C
93128 128G (96 x 10G + 8 x 40G)
9396 96G (48 x 10G + 12 x 40G)
9372 72G ( 48 x 10G + 8 x 40 G)
Nexus9500 Unicast Packet Flow

Fabric Module
Performs L3 LPM
lookup and resolves
Egress port and
next-hop
Fabric Module
L3 LPM Lookup & Forwarding
Ingress Line Card
Classify traffic
based on 802.1q
COS, IP Pres,
DSCP &ACL
Remark if needed
L2/L3 Lookup in
MAC Table and IP
Host Table
Parse the first 128

Byte and extract
header info
Egress Line Card
EoQ
ALE-NS
Ingress
Accounting &
Policing
Traffic
Classification
& Remarking
I-ACL
L2/L3
Lookup &
forwarding
T2-NFE
Output Q
& Shaping
E-ACL
Packet
Modification
Parser
Network Interface
EoQ
ALE-NS
OOBFC
Signaling
Ingress
Accounting &
Policing
Traffic
Classification
& Remarking
I-ACL
L2/L3
Lookup &
forwarding
T2-NFE
OOBFC
Signaling
Output Q
& Shaping
E-ACL
Packet
Modification
Parser
Network Interface
Additional buffer is
available for
extended out put
Ques EoQ
Class-based output
queues. Support 6
classes including
control traffic class
Egress Line card

sends packet to
egress port based
on DMOD/DPORT
N9K-C9300 High Level Block Diagram
(16 x 10G) x 3 =
480G FP Bandwidth
(12 x 40G) = 480G

Bandwidth to GEM
Module
HiGiG2 Interface on T2
MACF ports on the GEM and to MACN ports
(12 x 40G) = 480G FP

Bandwidth Uplink Ports
MACN ports.
Main Features of Trident2 1280Gbps Switch ASIC

Features
DCB Engine
Content aware Engine
L2
MAC
L2/L3
Processing
Dynamic Memory
Manager
Packet
Buffer
128 Integrated SerDes

128 SERDES@10Gbps
OR
32 SERDES@40Gbps
L2/L3
Multicast
L3
Route
Information
Maximum IO and Core bandwidth
1280G
MAC(L2) Entries
32K min -288K max
L3 Hosts
IPv4:16K min-112Kmax
IPv6:8K min-56 max
L3 Multicast Group
8K
Virtual Ports
16K
Maximum number of Physical ports
104
Counters
Host IF
North Star
Features
Information
Support Mixed Speed but in Fixed

configuration.
Network Interface:12 Ports Fabric

Interface: 12 40 Gig
Forwarding
720Mpps lookup rate on Ingress

Datapath
720Mpps lookup rate at Egress
Datapath
Shared Memory Subsystem

Ingress Path Buffer
Egress Path Buffer
10 Mbytes
30 Mbytes
Maximum number of Physical ports
24
Broadcom Unified Forwarding Table

T2 has the following Unified Forwarding Table:
SUPPORTED COMBINATIONS
Mode
L2
L3 Host
LPM
288K
16K
16K
224K
56K
16K
160K
90K
16K
98K
122K
16K
32K
16K
128K
Routing Mode for Nexus9300

LPM Routing Mode
Broadcom T2
Mode
Default system routing mode
ALPM Routing mode
CLI Command
System routing max-mode l3
N93K#show system routing mode

Configured System Routing Mode: Hierarchical
Applied System Routing Mode: Hierarchical (Default)
N93K#show hardware internal forwarding table utilization module 1
Max Host Route Entries (shared v4/v6): 124928
Max LPM Table Entries : 16384
Routing Mode for Nexus9500

LPM Routing Mode
Broadcom T2 Mode
Default System routing mode
3 (For Line card)

4 (For Fabric Module)
2--Line Card- V6 in LPM
3--For Fabric Module
3--For Line Card
4--With max-l3-mode option
For Line card
No Routes on Fabric Module
Max-host routing mode

Nonhierarchical routing mode
64-bit ALPM routing mode
Sub mode of mod 4 for

Fabric modules
show hardware internal forwarding table utilization mod 1

Max Host Route Entries (shared v4/v6):16384
Max LPM Table Entries : 131072
show hardware internal forwarding table utilization mod 21
Max Host Route Entries (shared v4/v6): 0
Max LPM Table Entries :0
Cli Command
System routing max-mode host

System routing non-hierarchical
Option [max-l3-mode]
System routing mode hierarchical
64b-alpm
Non hierarchical
routing mod
ACL TCAM TABLE

Characteristic
Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks
Egress ACL: 1K TCAM entries - 4x 256 banks
Each ACL type needs its own dedicated bank/banks
IPv4, IPv6 or MAC each needs dedicated bank/banks

MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks
VACL is programmed symmetrically in both egress and ingress ACL
Interface
Type
Ingress ACL
Egress ACL
SVI
TCAM Shared
TCAM Not shared
L3
TCAM Shared
TCAM Shared
ACL Characteristics
Atomic/hitless update of existing applied ACL while modified
Temporary label swap (no use of default-result)
Two acl copies in tcam, if there is no enough space, process fails
ACL TCAM banks chaining not supported
L4OPs/LOUs only used for expansion beyond 5 lines, configurable
10 L4op per acl limit
Specific applications (dhcp, bfd) may install their own ACLs which must merge
with user configured racl, vacl, pacl
TCAM Carving for Nexus 9000

TCAM Region-N9500
Size Per Region
IPV4 RACL
1536
IPv4 L3 QOS
256
Ingress System
256
SPAN
256
Ingress CoPP
256
Redirect
256
vPC Convergence 512

Egress IPv4
RACL
768
Egress System
256
Ingress
Ingress
TCAM Region-N9300
Size Per Region
IPv4 PACL
512
512
IPV4 VACL
512
512
IPV4 RACL
512
512
256
IPv4 Port QOS
256
256
256
Ingress System
256
256
256
SPAN
256
256
Ingress CoPP
256
256
Redirect
256
256
vPC Convergence
512
512
Egress IPv4 RACL
256
Egress IPv4 VACL
512
256
512
Egress System
256
256
3X512
256
256
512
Egress
3X256
256
Egress
ACL TCAM Default Region and Carving
TCAM Banks will first get assigned to Feature which has largest region.
Next TCAM Bank will get assigned to Feature which need double Width.
TCAM Carving requires Line Card/TOR reload to take effect
To read current TCAM allocation

N9K#Show system internal access-lists global
To reconfigure TCAM Region

N9K(config)hardware access-list tcam <feature name> <size>
Buffer And Queuing-T2
T2 has 12 Mbytes of
Buffer shared by all
ports for all Traffic
Shared buffer divided Into

Control and default service
pool if module is T2 only
Shared buffer divided

into Control, default and
OOBFC service Pool if
Module is T2 and NS
based
Control
Shared
Buffer
12 MB
Control
Default
Module with T2 only
OOBFC: Out of band flow control unicast service pool
Shared
Buffer
12 MB
OOBFC
Default
Module with T2 And NS
Buffer And Queuing-North Star

10 MB
Buffer
NorthStar
ASIC 1
20 MB
Buffer
10 MB
Buffer
North Star has 40 Mbytes of Buffer

Divided in to Three Pool
Control , SPAN , Default
12 x 40G
Hi-Gig2
Control
Trident II
ASIC
12 x 40G
Ethernet
Shared
Buffer
SPAN
Default
Buffer Boost Function with T2 and NS

Fabric Module
ALE-NS
10 MB
Buffer
NFE
T2
10 MB
Buffer
20 MB
Buffer
12 MB Buffer Shared by all

ports
Network Interface
1/10GE
1/10GE
1/10GE
1/10GE
Buffer boost is function which allow T2 to use extra

buffer of NS
When Buffer boost is enabled on a port , T2 Local switch
traffic is Sent to NS for extra buffer space When Buffer boost is disabled on a port, T2 local traffic
to this port remains local on this NFE
Buffer Boost is enabled by default and can be disabled
on a per port basis
System Health check Telemetry
Most Common System Health Check
What is the Best Recommended NX-OS Release

CPU & Memory usage
Inter Process Messaging usage-MTS
Traffic Stats/Drop To CPU
CoPP/Hardware Rate Limiter Drops
Interface Errors for STP/Error disable
Ethernet Out of Band Drops/Error
Inter ASIC Utilization
Instant Buffer usage Stats
Hardware Capacity Check
FATAL System Errors
Consistency Checkers Various Tables
GOLD Diagnostic Checks
Sev1/2 Syslog
General Recommendation for New and Existing

Deployments
Software Recommendation
Platform Series
Minimum Release
Recommended Release
Cisco Nexus 9500
6.1(2)I2(2b)
6.1(2)I3(4a)
Cisco Nexus 9300
6.1(2)I2(2b)
6.1(2)I3(4a)
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommend
ed_releaseb_Minimum_and_Recommended_Cisco_NXOS_Releases_for_Cisco_Nex
us_9000_Series_Switches.html
Verified Scale limits for different features and protocol for each release
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_61
2I34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34_chapter_01.
html
CPU & Memory Usage

N9K#show system resources
Load average: 1 minute: 0.00 5 minutes: 0.03 15 minutes: 0.05
Processes
:
432 total, 1 running
CPU states :
2.76% user, 0.75% kernel, 96.48% idle
CPU0 states : 0.00% user, 0.00% kernel, 100.00% idle
Memory usage:
16402328K total,3443588K used, 12958740K free
Current memory status: OK
N9K#show system internal memory-usage-per-module in-KB

Slot 01:Used:1647420 Kbytes,Free:425680 Kbytes,Total:2073100 Kbytes
N9K#show system internal memory-alerts-log
Make sure log is clean
CPU
D
R
A
M
D
R
A
M
CPU & Memory Usage

Provides top process using CPU cycle
show processes cpu sort | head lines
PID
Runtime(ms) Invoked
uSecs
----- ----------- -------- ----3357
220
3100
7099
5853
31655
10181
3109
5859
9489
52308
181
3477
672
3107
216
3478
268
175
1535
12
1Sec
-----45.50%
0.50%
2.00%
0.50%
0.50%
Process
----------adjmgr
ipqosmgr
diag_port_lb
netstack
ospf
Possibly ARP Table Churn
Top Command-display top CPU processes

top provides an ongoing look at processor activity in real time
N9K#run bash
Auto update
bash-4.2$ top
top - 11:13:32 up 9 days, 3:34, 4 users, load average: 0.11, 0.11, 0.08
Tasks: 226 total, 1 running, 220 sleeping, 0 stopped, 5 zombie
Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 98.5%id, 0.0%wa, 0.1%hi, 0.3%si, 0.0%st
Mem: 16402328k total, 3445044k used, 12957284k free, 72676k buffers
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root
20 0 2152 620 556 S
0 0.0 0:08.05 init
2 root
20
0S
0 0.0
0:00.00 kthreadd
3 root
20
0S
0 0.0
0:00.58 ksoftirqd/0
Inter Process Messaging Usage

N9K#sh sockets client detail | inc pim|drops|Errors
select drops:
Errors:
select drops:
Errors:
select drops:
Errors:
10
Make Sure Drops/Errors not incrementing
0
Message and transaction service-MTS
N9K#sh system internal mts buffers sum | diff

node
sapno
recv_q pers_q npers_q log_q
sup
320
0
0
4592
0
sup
284
0
19
0
0
sup
250
2
0
0
0
For SAP 320 own

by OSPF npers_q
increasing
Inband Driver Statistics-CPU Drops

N9K#show hardware internal cpu-mac inband stats
eth2 stats:
RMON counters
Rx
Tx
total packets
601163425
318962431
Per Queue Stats
Queue Idx
Packet Count Bytes Drops Csum Errors Allocation Failure
Queue 0
17677525
111822449180 0
0
0
- - - - - - - - - - - - - - -SNIP- - - - - - - - - - - - - - - - - - - - - - - Queue 7
17677525
111822449180 0
0
CRC 0errors/Collisions/late
Collisions
Alignment errors
Interrupt Counters
Symbol errors
Rx overrun
0
Sequence errors/Rx jabbers
Error counters
RX errors/Rx length errors
Rate statistics
Rx packet rate (current/peak) 717 / 80695 pps
Tx packet rate (current/peak) 360 / 1338 pps
Traffic Stats/Drops to CPU (Contd)

N9K# show system internal frame traffic | in drops
Global input drops:
bad-interface 0, bad-encap 0, failed-decap 0,
Global output drops:
Drops From PKTmgr

eth_output_err 0, gre_err 0 otv_err 0 span_drop_en: 0 span_drops: 0
Crossbar down drops : 0
Flood_to_core LTL: Hits: 0 Misses: 0
N9K# show system inband queuing statistics | in drop

bpdu: recv 68, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop 0
(q0): recv 1249377,
drop 0, congested 0
rcvbuf 2097152, sndbuf 262142 no drop
(q1): recv 4138154,
drop 0, congested 0
rcvbuf 2097152, sndbuf 262142 no drop
Instant Buffer Utilization For CPU Port

show hardware internal buffer info pkt-stats cpu
[Q00-07]
0
0
0
0
0
0
0
0
[Q08-15]
0
0
0
0
0
0
0
0
[Q16-23]
0
0
0
0
0
0
0
0
[Q24-31]
0
0
0
0
0
0
0
0
[Q32-39]
0
0
0
0
0
0
0
0
[Q40-47]
0
0
0
0
0
0
0
0
Total 48 Queues
Each Line Display Cell utilized

for 8 queues
One Cell represent approximately

208 Bytes
Congestion encountered if Counters

keep incrementing
Ethernet Out Of Band Drops/Errors

N9K#show hardware internal eobc stats | inc dropped
RX packets:248308217 errors:0 dropped:0 overruns:0 frame:0
TX packets:71554006 errors:0 dropped:0 overruns:0 carrier:0
N9K# show system internal emon stats
EMON MOD ONLINE BMP: 37f00067
FSM ID: 0 EOBCMON
=======================================
HB tx_req
186396
Provides Stats for all Modules
module 1:
including Fabric module
rx_req
176410
rx_resp
176426
Heart bit miss
rx_miss
7
tx_resp
176410
Instant Buffer Usage Stats

Buffer polling interval for 7.0
N9K#show hardware internal buffer info pkt-stats mod 1
Release is 500msecs
INSTANCE: 0
---------------------------------------------------------Output Shared Service Pool Buffer Utilization (in cells)
SP-0
SP-1
SP-2
SP-3
----------------------------------------------------------Total Instant Usage
4474
89
2939
Remaining Instant
Usage
25466
14255
3405
Peak/Max Cells Used
4821
327
3060
29940
14344
6344
Switch Cell Count
---------------------------------------------------------show hardware internal ns buffer info pkt-stats
Instant Buffer utilization per queue

per port
One cell represents 208 bytes
Show hardware internal buffer info pktstats input mod 1
SP-3-Dedicted resource for Control
Traffic
SP-0-Resource for Locally Switched
Unicast ,Multicast and SPAN
SP-2 Extended Output queue for

Unicast using buffers from North
Star
Instant Buffer Usage Stats - With Buffer Usage

INSTANCE: 0
Output Shared Service Pool Buffer Utilization (in cells)
SP-0
SP-1
SP-2
SP-3
------------------------------------------------------------------------Total Instant Usage

Remaining Instant Usage
4474
89
2939
25466
14255
3405
SP-3 Started filling

the Queue
-------------------------------------------------------------------------
ASIC Port
[13]
Q3
Q2
Q1
Q0
Port 13 onwards are Front Panel Port
UC(OOBFC)->0
CPU
SPAN
Only printed if there is congestion
UC->
1249
332
MC->
3247
1996
CPU buffer filling

up
CoPP Drops
N9K# show policy-map interface control-plane mod 1 | in dropped
dropped 0 packets;
dropped 0 packets;
dropped 0 packets;
dropped 0 packets;
dropped 7800 packets;
Drops Seen for Default-Class at minimal rate is normal
We recommend that you use the strict default CoPP policy initially and then later modify the CoPP
policies based on the data center and application requirements.
Parameters
Default
Default policy
Strict
Default Policy
9 policy entries
CoPP Drops-Exception drops

class-map copp-system-p-class-l3uc-data (match-any)
match exception glean
class-map copp-system-p-class-redirect (match-any)
match access-group name copp-system-p-acl-ptp
class-map copp-system-p-class-exception (match-any)
Goal is to Classify all Traffic Using CoPP
match exception ip option
match exception ip icmp unreachable
match exception ipv6 option
match exception ipv6 icmp unreachable
class-map copp-system-p-class-exception-diag (match-any)
match exception ttl-failure
match exception mtu-failure
Hardware Rate Limiter

N9K# show hardware rate-limiter mod 1
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
Module: 1
R-L Class
Config
Allowed
Dropped
Total
+----------+-----+------------+------------+-------------+
L3 glean
100
L3 mcast loc-grp
3000
access-list-log
100
bfd
10000
1352890
fex
3000
span
50
1352890
FATAL System Errors

N9K#show logging onboard mod 1 exception-log | incl FATAL prev 15
-----------------------------------------------------------------------Date (mm/dd/yy)=01/15/15
Time (hs:mn:sec): 00:16:58
OBFL Exception log data for THIS SUP Module:0
********* Exception info for module 0 ********

exception information --- exception instance 1 ---Device Name
: System Manager
Device Errorcode
: 0x0000023a
ErrNum (devInfo)
: 58 (0x3a)
System Errorcode
Error Type
: 0x401e0089 Service in VDC has had a hap-reset

: FATAL error
Common Interface Error counters and Status

N9K# show interface counters errors mod 4
Port
Align-Err
FCS-Err
Xmit-Err
Rcv-Err
UnderSize OutDiscards
-------------------------------------------------------------------------Eth4/1
100
581
N9K# show interface status err-disabled

Port
Name
Status
Reason
--------------------------------------------------------------------------
Eth4/1
err-disable
link-flap
Interface Queuing Stats

N9K#show queuing interface 4/18
Egress Queuing for Ethernet4/18 [System]

QoS-Group# Bandwidth% PrioLevel
3
Shape
Min
Max
Qlimit
Units
-
6(D)
-------------------------SNIP--------------------------
100
6(D)
---------------------------------------------------QOS GROUP 0
Unicast
Dropped Pkts |
0|
| OOBFC Unicast
0|
Multicast
0|
-----------------------------------------------------------QOS GROUP 7
Unicast
Dropped Pkts |
0|
| OOBFC Unicast
0|
Multicast
0|
Inter ASIC Utilization-HG Ports

T2 #0
T2 #1
HG00
HG00
Fabric Module
T2 #0
T2 #1
T2 #2
N9K#show system internal interface counters mod 1
Line Card
Internal Port Counters (150 secs rate) for Slot: 1

====================================================
Interface
ASIC
ASIC
BCM
Port
Inst
Port
TxBitRate(BwUtil) TxPktRate
(bps)
(pps)
RxBitRate(BwUtil) RxPktRate
(bps)
(pps)
----------------------------------------------------------------------------------------ii1/1/1
HG0
170512 (0.00)
0(0.00)
-------------------------------------------Snip-----------------------------------------ii1/1/14
HG1
ii1/1/25
HG0
0( 0.00)
1790648
(0.00)
1129882872(2.51)
960753
1043
22864(0.00)
20
Verify Consistency Between Software and Hardware

Table
Table
CLI
Physical Interface
show consistency-checker link-state
Port-Channel
Membership
Mac Address Table
show consistency-checker membership port-channels
Vlan Membership
show consistency-checker membership vlan
L3 interface-LIF
programming
For RIB and FIB
L3 interface-LIF programming Logical Interface for Routing
show consistency-checker l2
show consistency-checker forwarding ipv4 unicast
Consistency Checkers-Link and STP state

N9K#show consistency-checker link-state mod 1
Link State Checks: Link state only
Consistency Check: PASSED

No inconsistencies found for:
Ethernet1/1
2015 Mar 24 03:23:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_LINK_STATE: Consistency
Check: PASSED
N9K# show consistency-checker stp-state

Checks: Spanning tree state
vlan 18
2015 Mar 24 03:25:21 N9508a-SJ %$ VDC-1 %$ vshd: CC_VLAN_STP_STATE:

Consistency Checkers-Port Channel-Vlan Membership

N9K#show consistency-checker membership port-channels
Checks: Trunk group and trunk membership table.
Consistency Check: Failed
Inconsistency found for port-channel1:
Module:1, Unit:
['Ethernet3/49', 'Ethernet2/49']
Module:26, Unit: ['Ethernet3/49', 'Ethernet2/49]

N9K# show consistency-checker membership vlan
18
Checks: Port membership of Vlan in vlan and egr_vlan table

Ports configured as "switchport monitor
will be skipped

Vlan:18, Hardware state consistent for:
Ethernet2/49
2015 Mar 24 03:28:31 N95a%$ VDC-1 %$ vshd: CC_VLAN_MEMBERSHIP: Consistency
Check: PASSED
Consistency Checkers-Mac address Table

N9K# show consistency-checker l2 module 1
Consistency check: PASSED
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen, + - primary entry using vPC Peer-Link,
(T) - True, (F) - False
Missing entries in the HW MAC Table
VLAN
MAC Address
Type
age
Secure NTFY
Ports
---------+-----------------+--------+---------+------+----+-----------------Extra and Discrepant entries in the HW MAC Table

VLAN
MAC Address
Type
age
Secure NTFY
Ports
---------+-----------------+--------+---------+------+----+------------------
Consistency Checkers-L3 Interface

N9K# show consistency-checker l3 mod 1
L3 LIF Checks: L3 Vlan, CML Flags, IPv4 Enable
No inconsistencies found for:

Ethernet1/1
Ethernet1/2
Ethernet1/3
2015 Mar 24 04:07:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_L3_LIF: Consistency Check:

PASSED
Consistency Checker Unicast Forwarding

N9K#test consistency-checker forwarding
Consistency check started.

N9K# show consistency-checker
forwarding
ipv4 unicast module 1
IPV4 Consistency check (in progress): table_id(0x1) slot(1)

Elapsed time : 8257 ms
N9K# show consistency-checker
forwarding
ipv4 unicast
IPV4 Consistency check : table_id(0x1) slot(1)

Execution time : 13244 ms ()
No inconsistent adjacencies.
No inconsistent routes.
Consistency-Checker: PASS for 1
module 1
Gold Diagnostic Checks

N9K# show diagnostic result mod 2
On Demand Diagnostic can be executed
Module 2: 48x1/10G-T 4x40G Ethernet Module
Test results:(.=Pass, F=Fail,I=Incomplete,U=Untested,A=Abort,E=Error disabled)

1) ASICRegisterCheck------------>
2) PrimaryBootROM--------------->
3) SecondaryBootROM------------->
4) OBFL------------------------->
6) BootFlash-------------------->
7) AsicMemory------------------->
8) FpgaRegTest---------------- ->
9) PortLoopback:--------------- >
Port
RewriteEngineLoopback
9 10 11 12 13 14 15 16
----------------------------------------------------U
Sev1/2 Syslog
show logging logfile | incl -1-|-22015 Feb 25 10:30:17 N9508a-SJ %PLATFORM-2-MOD_PWRUP: Module 26 powered up
(Serial number SAL1738D37W)
2015 Feb 25 10:32:37 N9508a-SJ %XBAR-2-XBAR_HGLINK_NOT_UP: fabric link 1 on
module 2 unit 0 connected to fabric module 26 unit:0 is not up during module
bring up
2015 Feb 25 10:32:39 N9508a-SJ %MODULE-2-MOD_FAIL: Initialization of module 26
(Serial number: SAL1738D37W) failed
2015 Feb 25 10:32:39 N9508a-SJ %PLATFORM-2-MOD_PWRDN: Module 26 powered down
(Serial number SAL1738D37W)
Ethanalyzer
TCP Dump
ELAM
Packet Tracer
Flex Counter
ERSPAN
Consistency Checkers
Ethanalyzer-When To Use it
To Analyze the traffic sent and received by CPU
It uses wiresharks code (an open source software)
Netstack
Pseudo Inband
Troubleshooting High CPU
Troubleshoot Control Plane issues Ex. OSPF , PIM , STP
SUP
NIC-ETH2
Flap.
Note: Ethanalyzer does not allow capturing of hardware switched traffic between data
ports of the switch
Ethanalyzer-CLI
N9K# ethanalyzer local interface inband capture-filter "pim
detail
Capturing on inband
Frame 1 (60 bytes on wire, 60 bytes captured)
Arrival Time: Mar 24, 2015 10:01:10.018889000
-------Snip-----------------[Protocols in frame: eth:ip:pim]
N9K#ethanalyzer local interface inband display-filter "ospf detail
Capturing on inband
Frame 1 (82 bytes on wire, 82 bytes captured)
Arrival Time: Mar 24, 2015 10:04:11.425523000
-------------------Snip-------------------[Frame is marked: False]
[Protocols in frame: eth:ip:ospf]
Some Available Options

autostop
:Autostop
decode-internal
header decoding
:Internal
limit-captured-frames :Maximum
number of
TCP Dump
Tcpdump command works on most flavors of Linux operating system
Helps to prints out a description of the contents of packets on a network interface
Tcpdump will, if not run with the -c flag, continue capturing packets until it is
interrupted by a SIGINT signal CTRL-C
Tcpdump output can be saved to file for further reference
More info at http://www.tcpdump.org/
Tcpdump -syntax
N9K# show feature | in bash
Feature Name Instance

bash-shell
State
tcpdump version 4.1.1
enabled
libpcap version 1.2.1
N9K# run bash

bash-4.2#
Syntax: tcpdump -h
sudo su
Usage: tcpdump [-aAbdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count]

[ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds]
Password:******
[ -i interface ] [ -M secret ] [ -r file ]
bash-4.2# whoami
[ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ]
root
[ -y datalinktype ] [ -z command ] [ -Z user ]
bash-4.2# tcpdump c 10 I ps-inb
[ expression ]
bash-4.2#
Tcpdump-Examplesbash-4.2# tcpdump -c 100 -w tcpdump.pcap -vvvv -i ps-inb
Capturing 100 packets And

tcpdump: WARNING: ps-inb: no IPv4 address assigned
writing to file
tcpdump: listening on ps-inb, link-type EN10MB (Ethernet), capture size 65535
bytes
100 packets captured
102 packets received by filter
bash-4.2#cd /bootflash
bash-4.2# tcpdump -tttt -r tcpdump.pcap | more

reading from file tcpdump.pcap, link-type EN10MB (Ethernet)
Reading captured file
2015-04-26 03:21:31.309350 00:0e:ee:01:1b:01 (oui Unknown) > 00:00:00:ff:ff:01

(oui Ethernet), ethertype Unknown (0x8833), length 160:
0x0000:
0000 fc08 0b00 0000 0000 0800 0000 0ffd
...............
-------------------------------------more---------------------------------
tshark
bash-4.2$ tshark -i ps-inb
Capturing on inband
0.000000 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
12.328377 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
^C2 packets captured
bash-4.2$
Elam-Embedded Logic Analyzer Module-NS
Elam Allows to capture single packet based on Trigger

Triggers are configured using Packet information
Only Supported on North Star Based(ALE) Line Cards and GEMs
Use with TAC Supervision
Help to Answer following Questions
Was the Packet indeed Received by device on given Line card?
How did the Packet Look like?
How was the packet rewritten based on forwarding Decision made by T2?
Was the Packet correctly forwarded or Dropped?
ELAM Configuration
Init Initialize the ELAM select the Asic instance, pipeline and
select lines
module-1# debug platform internal ns elam asic
module-1(NS-elam)# trigger init ingress in-select 3 out-select 5
2. Config
Config Configure the trigger based on different fields in the packet

module-1(NS-elam-insel3)# set outer ipv4 src_ip 13.13.13.10
3. Arm
Trigger
Arm Arm the trigger by setting the fields to match in hardware

module-1(NS-elam-insel3)# start
Read Once the trigger is triggered, read the report

module-1(NS-elam-insel3)# report
Reset Once the process is complete, reset the trigger to restart

the process
module-1(NS-elam-insel3)# reset
1. Init
4. Read
5. Reset
Elam Ingress & Egress Direction-TOR

Traffic entering GEM ports which has NS and
exiting T2 is Egress Pipeline
Egress
Ex. trigger init egress in-select 3 out-select 5

set outer ipv4 dst_ip 13.13.13.10
Ingress
NorthStar
ASIC 1
12 x 40G
Hi-Gig2
Traffic Entering T2 and Exiting GEM ports is

Ingress Pipeline
Trident II
ASIC
12 x 40G
Ethernet
Network Interfaces
Ex. trigger init ingress in-select 3 out-select 5

set outer ipv4 src_ip 13.13.13.10
IP.Add=13.13.13.10
Elam Ingress & Egress Direction-EOR

Traffic entering from Fabric Module in to NS of
Line Card is Egress Pipeline
Fabric 3
Fabric 1
Ex. trigger init egress in-select 3 out-select 5

set outer ipv4 dst_ip 13.13.13.10
Egress
N
FE
N
FE
Ingress
Line Card
North Star ASIC
Traffic Entering NS and exiting towards Fabric

Module is Ingress Pipeline
12 x 40G
Hi-Gig2
12 x 40G
Ethernet
Ex. trigger init ingress in-select 3 out-select 5

set outer ipv4 src_ip 13.13.13.10
Trident II
ASIC
Network Interfaces

13.13.13.10
ELAM Sample Configuration & Key Info

Nexus9508 with N9K-X9564TX
13.13.13.1/30
Eth6/52
Eth5/1
N9K-X9564TX
13.13.13.10/30
4 40Gig Port On NS
40 1/10 Gig On T2
N9K# attach mod 6

module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
module-6(NS-elam-insel3)#
start
status
report
If Packet Captured
Status: Triggered
Important ELAM Fields

GBL_C++ [INFO] hg2_srcmod: 0E
GBL_C++ [INFO] hg2_srcpid: 0D
GBL_C++ [INFO] hg2_dstmod: 11
Information
is in Hex
Convert to
Dec.
GBL_C++ [INFO] hg2_dstpid: 0A

GBL_C++ [INFO] ip_da: 000000000000D0D0D0A
GBL_C++ [INFO] ip_sa: 000000000000D0D0D01
GBL_C++:
GBL_C++:
[MSG] - sideband is complete

[INFO]
ovector: 000FFF
N9K# show interface hardware-mappings

---------------------------------------------------------------------Name Ifindex Smod Unit HPort FPort NPort VPort
-----------------------------------------Eth5/2 1a280000 14 0 13 255 0

-1
Eth6/52 1a286600 17 1 10 255 51
-1
Sideband is the result where

packet will be sprayed.
Should never be 0
Packet Tracer-T2
FM Mod
Helps to Trace the packet inside Switch.
Only packets in the direction of the flow are traced
Trident II
ASIC
Two Acls are installed for each filter on each Line card
One ACL for Front Panel Port Group
Second ACL for traffic exiting Fabric Module and ingressing Line
card
Network Interfaces
Packet Tracer Configuration

test packet-tracer dst-ip 13.13.13.10 detail-fp
Configure Filter
test packet-tracer dst-ip 13.13.13.10 detail-hg

13.13.13.10/30
Start Tracer
test packet-tracer start
rt
Stop Tracer
Check Counter
Filter
Clear/Remove-all
test packet-tracer stop
test packet-tracer show
test packet-tracer clear remove
Sample Configuration & Identify Front Port-LC

N9K#test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
N9K#test packet-tracer show
Module 6:
Filter 1 installed: src-ip
Module 21:
Module 26:
filter 1 non-zero Packet-tracer stats

13.13.13.10/30
13.13.13.1 dst-ip 13.13.13.10 detail-fp

13.13.13.1 dst-ip 13.13.13.10 detail-fp
13.13.13.1 dst-ip 13.13.13.10 detail-fp

Eth6/1
13.13.13.1/30
Eth6/52
13.13.13.10/30
Packet Tracer Sample Configuration & Key Info

N9K# test packet-tracer start filter 1
N9K# test packet-tracer show filter 1
13.13.13.10/30
mod 6 non-zero
Packet-tracer stats
Eth6/1
Module 6:
Filter1 installed: src-ip 13.13.13.1 dst-ip
13.13.13.10 detail-fp
Eth6/52
13.13.13.1/30
13.13.13.10/30
ASIC instance 0:
Entry 1: id = 7426, count = 5, active, fp, port 13
N9K# show interface hardware-mappings | grep 6/1

Name
Ifindex
Smod
Unit
Eth6/1
1a280000
16
Hport
13
FPort
Nport
255
VPort
-1
Sample Configuration Identify Fabric Port LC From FM

N9K# test packet-tracer dst-ip 13.13.13.10
src-ip 13.13.13.1 detail-hg
N9K# test packet-tracer start filter 1

N9K# test packet-tracer show
mod 6 non-zero
Module 6:
Filter 1 installed:
src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-hg
ASIC instance 0:
Entry 0: id = 7425, count = 68, stopped, fp,
ASIC instance 1:
Eth8/1
Eth6/52
13.13.13.1/30
Entry 1: id = 7426, count = 13, stopped, hg, port 1

Entry 2: id = 7427, count = 11, stopped, hg, port 2
13.13.13.10/30
13.13.13.10/30
Flex Counters Adjacency Statistics

Flex counters used to count Next hop Adjacency stats
One can attach Stats to multiple Adjacency at same time
One Stat Counter per adjacency
Total Flex Counters are 16K per Switch
How To Configure Flex Counters

Eth6/1
Eth6/52
13.13.13.1/30
13.13.13.10/30
13.13.13.10/30
N9K# sh ip route 13.13.13.10

IP Route Table for VRF "default"
'%<string>' in via output denotes VRF <string>
13.13.13.8/30, ubest/mbest: 1/0
*via 13.13.13.6, Eth6/52, [110/41], 00:33:14, ospf-10, intra
N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6
interface ethernet 6/52 (enable |disable | show)
Sample Configuration
interface ethernet 6/52 show
13.13.13.10/30
Module:21 Unit:0
-----------------Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 738 Bytes 90036
Mcast: Packets 0 Bytes 0
Eth6/1
Module:22 Unit:1
13.13.13.1/30
-----------------Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 946 Bytes 115412
Mcast: Packets 0 Bytes 0
Eth6/52
13.13.13.10/30
SPAN & ERSPAN
Switch Port Analyzer

Provides efficient, high-performance traffic monitoring service
Duplicates network traffic to one or more monitor interfaces
Types Of SPAN
Local SPAN
Encapsulated Remote SPAN(ERSPAN)
Applications:
Troubleshooting connectivity issues
Base lining network utilization/performance
Detecting anomalous traffic flows
On Nexus9000 Span Traffic uses dedicated queue
Queue carrying SPAN traffic has low Priority over other queues
during congestion
SPAN QOS Queue

N9K# show queuing interface ethernet 4/18 | begin SPAN
|
SPAN QOS GROUP
+-----------------------------------------------------------------+
|
Unicast
| OOBFC Unicast
Multicast |
+------------------------------------------------------------------+
|
Tx Pkts |
0|
0|
0|
Tx Byts |
0|
0|
0|
Dropped Pkts |
0|
0|
0|
Dropped Byts |
0|
0|
0|
Q Depth Byts |
0|
0|
0|
SPAN Configuration
N9K(config)# monitor
N9K(config-monitor)#
Sup-eth
session 1
source interface sup-eth 0 both
source interface ethernet 6/1
e6/1
destination interface ethernet 6/2
Local SPAN
No Shut
N9K(config)#int et 6/2
show monitor
Session State
Reason
--- ---------------1
up The session is up
Local
e6/2
N9K(config-if)# switchport monitor
Description
-------------------Local SPAN Session
ERSPAN Configuration
Only Supports Source ERSPAN
N9K(config)# monitor erspan origin ip-address 13.13.13.2
global
Type-3 Header 32-bit Timestamp
N9K(config)# monitor session 1 type erspan-source
Supports on Nexus9300 only
N9K(config-erspan-src)# header-type 3
Layer 3
N9K(config-erspan-src)# source interface ethernet 6/1
N9K(config-erspan-src)# erspan-id 1
L3
N9K(config-erspan-src)# ip ttl 16
N9K(config-erspan-src)# vrf default
e6/2
e6/1
N9K(config-erspan-src)# destination ip 9.1.1.2
ERSPAN
N9K(config-erspan-src)# marker-packet-2
Marker packet carry original UTC time
N9K(config-erspan-src)# no shut
stamp to over come 32-bit wrapper
issue
Consistency Checkers-Summary
Show consistency-checker
stp-state vlan
link-state
membership vlan
membership port-channels
membership port-channels
l2
l3
forwarding ipv4 unicast
Nexus 9000
Troubleshooting
Understanding T2 interfaces-Xe0/hg
N9K# bcm-shell mod 1 "show unit"
Unit 0 chip BCM56852_A2 (current)
Unit 1 chip BCM56852_A2
hg0
N9K#bcm-shell mod 1 0:ps

ena/ speed/ link auto
port link duplex scan neg?
STP
state
hg11
hg0
lrn
inter max
T2
pause discrd Instance
ops
face frame
0
hg11
loop
T2
back
Instance 1
hg0 up
42G FD HW No
Forward
None
FAF F FXGMII
16360
F F F F F F F
F F F F FQSPF
F F F F
F F F F FQSPF
hg2 up
42G FD HW No
Forward
None
16360
P P P P P P P
P P P P PPorts
PPorts
P P PXGMII
P P P P
P P P P FA
13 14 15 16 17 18 19 20 21 22 23 24
01 02 03 04 05 06 07 08 09 10 11 12
--------------------------------Snip---------------------------------Hg11 up
42G FD HW No
Forward
None FA XGMII 16360
Xe0
Xe11
Xe0
Xe11
Xe0 !ena
40G FD HW No
Disable
None
FA XGMII
1582
Eth1/13
Eth1/24
Eth1/1
xe1 up
40G FD HW No
Disable
None
FA Eth1/12
XGMII 1582
--------------------------------Snip---------------------------------Xe11 !ena
40G FD HW No
Disable
None FA XGMII 1582
Hg=Internal Ports
Xe=Front Panel Port
Layer -1 Issues- Transceiver Not Recognized

N9K# show interface ethernet 4/18 transceiver details
Ethernet4/18
transceiver is not present
module-4# show hardware internal bcm-usd event-history xcvr 18
1) Event:E_STRING, length:135, at 220346 usecs after Thu Apr 16 20:50:17 2015
bcm_usd_xcvr_fcot_notify_default(941): [unit=0 nxosport=18 bcmport=30]
fcot_state:0x2 fcot_type:0 sent MTS_OPC_FCOT_EVENT_INFO, rc 0x0
2) Event:E_STRING, length:93, at 647132 usecs after Thu Apr 16 20:50:14 2015
bcm_usd_xcvr_fcot_scan_sfp(3003): [unit=0 nxosport=18 bcmport=30]
FCOT not supported err=-1
Interface MTU/Speed/Flow Control Verification

N9K# show interface Ethernet 4/18
Ethernet4/18 is up
admin state is up, Dedicated Interface Belongs to Po10
Hardware: 10000/40000 Ethernet, address: 7c69.f66e.d860 (bia 7c69.f66e.d860)
MTU 9216 bytes, BW 40000000 Kbit, DLY 10 usec

N9K# bcm-shell module 4 1: ps Xe17"
ena/
speed/
port
link
duplex
xe17
up
40G
FD
link
auto
scan
neg?
HW
No
STP
state
Disable
lrn inter
pause
discrd
ops
None
FA
max
face frame
SR4
9298
loop
back
Interface Flow Control Check

N9K#Show interface ethernet 1/1 flowcontrol
Port
Send
FlowControl
admin
Receive
oper
FlowControl
admin
RxPause
TxPause
oper
-----------------------------------------------------------------------------
Eth1/1
0
off
off
off
off
N9K#bcm-shell module 1 "ps"
port
xe0
Wrong programming
ena/
speed/
link
auto
link
duplex
scan
neg? state
up
10G
FD
HW
No
TP
pause
Disable
discrd
TX RX
None
lrn
inter
ops
face
FA
SFI
max
loop
frame back
9298
Interface Input Drops

Ethernet1/30 is up
Hardware: 1000/10000 Ethernet, address: 7426.acea.ceb9 (bia 7426.acea.ceb9)
EtherType is 0x8100
0 input with dribble 1316 input discard
N9K#bcm-shell mod1 cstat xe29
+------------------Programmable Statistics Counters[Port xe29]------+
| Type | No. |
Value
|
Enabled For
|
+----------------------------------------------------------------- -+
| RX | 0(R)|
19163028| RIPD4 RIPD6 RDISC RPORTD
|
|
|
|
| PDISC VLANDR
|
|
| 1(R)|
28744286| IMBP
|
|
| 4
|
993820| RPORTD FcmPortClass3RxDiscards
|
|
| 6
|
19163407| RFILDR FcmPortClass2RxDiscards
|
|
| 7
|
19163048| RDROP
|
|
| 8
|
18169208| VLANDR
| | gre VLANDR
bcm-shell mod 6 "cstat info"
+-------------------------------------------------------------------+
VLANDR
Rx VLAN drops
|
| 3(R)|
14704| TPKTD
|
|
| 4(R)|
968303| TGIP4 TGIP6 FcmPortClass3TxFrames|
|
| 6
|
968303| TGIP4 FcmPortClass3TxFrames
|
+-------------------------------------------------------------------+
Fabric Connectivity and Troubleshooting

T2
T2
T2
T2
T2
T2
T2
N9K-C9508-FM-4
N9K-C9508-FM-8
N9K-C9516-FM-16
In an 4-slot chassis N9K-C9504-FM has 1 T2 per module

FMs provides redundancy for internal data flow, the loss of FMs just increases
the oversubscription factor.
Full-Rate Mode(FRM) V/S Oversubscribed Mode(OSM)

Each T2 have 32 40Gigport with total capacity of 1.2Tbps with 2 switching
mode
OSM(Default) - Uses all 32 40 Gig ports Line Rate achieved for packets > 200 Bytes
FRM - Uses only 24 40 Gig ports Line rate achieved for > 64 Bytes
Configuration Knob to Change the mode.

N9K(config)# system fabric-mode full-rate
Configuration effective after Reboot
N9K#show system fabric-mode
Applied System Fabric Mode:Full rate mode
Use FRM mode to achieve line rate for 64 byte packets on 9636PQ , 9564PQ ,
9564TX cards
All other 94xx line cards will not be powered up in this mode
RTAG7 and DLB

Two Packet Hashing algorithm available from LC to FM
RTAG7-To Select HG Port use Packet Header.
For a flow same HG Link is used
FM1
FM6
FM-2
DLB-Dynamic Load Balancing- Default algorithm

Initial Hash same as RTAG7
HGPorts
Based on Link Quality pick up optimum HG Port
HGPorts
LC1
Better utilization of all HG links
N9K(config)# port-channel load-balance internal [dlb/rtg7]
N9K# show port-channel load-balance internal algorithm
HighGig port-channel load balance algorithm: dlb
LC2
Higig Link Failures Fabric Module Policy

For any single Higig link failure between FM and LC
Bring down the FM, if there is more than one FM
Else bring down LC
Multiple Higig links failures for a Single LC going to Multiple FM - Bring down
the LC module.
Multiple Higig links failures on LC to one of the FM - Bring down the LC module
4/8 slot Chassis Fabric Connectivity
9500/9600 Series Line Cards T2

have connectivity to all 6 Fabric
Modules T2
9400 series Line cards connects to

all T2 but use only 4 Fabric Modules
-No Connection to Slot 21 & 25
Traffic between 9500/9600 Series

Line Card and 9400 Line card will
use subset Hi Gig links .
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
40 Gig Link
T2
T2
N9K-X9536PQ
16 slot Chassis Fabric Connectivity
9500 Series Line Cards T2 will have

connection to all 6 Fabric Module but to
only 2 T2s from each Fabric Module
9500 series Line Cards T2 will have

connection to all 4 T2s of Fabric module if
there are only 3 Fabric module present
9400 series cards connects to all T2 but

use only 4 FM-No Connection to Slot 21 &
25
N9K-C9516-FM
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
40 Gig Link
Traffic between 9500 Series Line Card and

9400 will use subset Hi gig links.
N9K-X9636PQ line card module is not
supported in 16 slot chassis
T2
T2
N9K-X9536PQ
16 slot Chassis Fabric Connectivity
With 3 FM configuration All 4 T2 units in

each FM are connected to 9500 series LC
modules' T2 units
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
Each blue line represents one 40 Gig link
T2
T2
N9K-X9536PQ
T
2
T
2
T
2
T
2
Line Cards With Mux to FM

FM25
FM26
HG
MUX1
FM24
HG
MUX2
HG
MUX4
012
3
456
MN7Port
75
35
6-8
911
2-
31-
2624
0
29
Warpcore
T2
012
3
02
Standby Mux Link
456
MN7Port
HG
MUX6
MF Port
911
75
2624
T2
Fabric Modules
Line Card
Line cards N9K-X9464PX/TX ,

N9K-X9564PQ/TX have Mux
Mux used for connecting HiG Link

from Line Cards to multiple Fabric
Module
Mux available only for Half of the

HiG interface of LC
By Default Mux Link Active to Odd

number of Fabric Module
8 9 10
11
Northstar 2
Active Mux Link
MF Port
FM21
HG
MUX3
HG
MUX5
8 9 10
11
Northstar 1
02
FM22
FM23
FM Connectivity For N9K-X9564PX With MUX

show system internal fabric connectivity mod 5 | in B
HiGIG Link-info Linecard slot:5
LC-Slot LC-Unit
5
5
LC-HGLink
MUX
FM-Slot
FM-Unit FM-HGLink
HG02
1B
25
HG12
HG03
1B
25
HG12
Line cards N9K-X9464PX/TX ,

N9K-X9564PQ/TX have Mux
Mux used for connecting HiG Link
from Line Cards to multiple Fabric
Module
Mux available only for Half of the
HiG interface of LC
By Default Mux Link Active to Odd
number of Fabric Module
show system internal fabric connectivity mod 5
HiGIG Link-info Fabriccard slot:5

LC-Slot LC-Unit LC-HGLink
MUX
With FM from Slot 25 Down
FM-Slot FM-Unit FM-HGLink
HG02
1A
26
HG14
HG03
1A
26
HG14
FM-25
T2-0 T2-1
FM-26
T2-0 T2-1
HG012
HG012
B
HG02
HG014
HG03
MUX
LC
T2-0 T2-1
HG014
FM Connectivity For N9K-X9564PX With MUX

show system internal fabric connectivity mod 5 | in B
LC-Slot LC-Unit
5
5
LC-HGLink
MUX
FM-Slot
FM-Unit FM-HGLink
HG02
1B
25
HG12
HG03
1B
25
HG12
FM-26
T2-0 T2-1
FM-25
T2-0 T2-1
HG012
HG012
HG014
HG014
HG03
HG02
MUX
LC
T2-0 T2-1

MUX
With FM from Slot 25 Down
HG012
HG012
HG02
1A
26
HG14
HG03
1A
26
HG14
FM-26
T2-0 T2-1
FM-25
T2-0 T2-1
B
HG02
HG014
A
HG03
MUX
LC
T2-0 T2-1
HG014
Fabric Troubleshooting commands

Fabric Module Slot-21

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink
HG00
21
HG00
HG01
21
HG00
T2
#1
T2
#0
HG00
HG00

21
HG00
HG00
21
HG00
HG01
MUX
T2
#0
T2
#1
Line Card Slot-1
T2
#2
Fabric Port Drops and Link Status

N9K# show hardware internal fabric interface asic counters mod 21
Counters for Fabric Ports:
FabricInterface Forward
Forward Error
Pkt Error
Pkt
QOS Rx
RxDrops
TxDrops
RxDrops
TxDrops
Drops
0 / 1 / HG0
0
0
0
0
0
1 / 1 / HG0
0
0
1 0
0
0
QOS Tx
Drops
0
0
N9K# bcm-shell mod 21 "ps | inc hg0

ena/
port
hg0
speed/
link
up
link
duplex
scan
auto
neg?
42 FD
HW
No
STP
state
pause discrd
Forward None
lrn
ops
inter max loop

face frame back
FA
XGMII 16360
Fabric Port STP State HW point of View

N9K# sh vlan id 100
VLAN Name
Status
Ports
---- ------------------ --------- 100
VLAN0100 active
Po1, Eth1/1
N9K# bcm mod 21 " stg show

STG 5: contains 1 VLAN (100)
Forward: hg
show sys internal xbar event-history {trace|errors|msgs|sw}

show sys internal xbar-client event-history {trace|errors|msgs|sw}
show tech-support xbar
Path of the Packet -Inband

CPU
Netstack
System Controller-SC1
Mod29
Fabric Module
Mod23
Fabric Module
Fabric Module
Mod21
NIC-Eth3
NIC-Eth2
Traffic from all ingress Line Card

to Supervisor will hash to one
Fabric module
Traffic from Supervisor Card to
Egress Line cad will hash on one
FM. May not be same
CoPP is operational on all LC.
However aggregate CoPP is on
FM
Mod26
Eth6/1
Line Card
OSPF Hello
Check for Drops/Errors-Line Card
Line Card
North Star ASIC
Trident II
ASIC
Network Interfaces
N9K#show hardware internal interface ethernet 6/1 asic counters
Important Counters/Drops
--------------- --------- --------- --------- --------- --------- --------Interface Name
Forward
Forward Error Pkt Error Pkt
QOS Rx
QOS Tx
RxDrops
TxDrops
RxDrops
TxDrops
Drops
Drops
--------------- --------- --------- --------- --------- --------- --------Ethernet6/1
870
0
100
0
0
0
--------------- --------- --------- --------- --------- --------- --------Forward Rx Drops = [ RDBGC0 RDBGC4 RDBGC6 RDBGC7 RDBGC8 ]
Forward Tx Drops = [ TDBGC1 TDBGC3 TDBGC5 (excludes expected Multicast drops)]
ErrorPkt Rx Drops= [ IUNHGI IUNKOPC RFCS RALN RFLR RERPKT RJBR RSCHCRC RUND RMTUE]
ErrorPkt Tx Drops= [ TJBR TFCS TRPKT RMTUE TUFL TPCE ]
QOS Rx Drops
= [ RDISC DROP_PKT_ING DROP_PKT_IMTR DROP_PKT_YEL DROP_PKT_RED ]
QOS Tx Drops
= [ MCQ_DROP_PKT(0) MCQ_DROP_PKT(1) MCQ_DROP_PKT(2)
RDBGC0
Use slot <#> show hardware internal interface indiscard-stats instance <#>
N9K#bcm-shell mod 6 "listreg RALN"| grep Description
Description: Receive Alignment Error Frame Counter
Instant Buffer Usage Stats-With Buffer Usage

INSTANCE: 0
Output Shared Service Pool Buffer Utilization (in cells)

SP-0
SP-1
SP-2
SP-3
------------------------------------------------------------------------ SP-3 Started filling

Total Instant Usage
4474
0
89
2939
the Queue
Remaining Instant Usage 25466
0
14255
3405
-----------------------------------------------------------------------ASIC Port
Q3
Q2
Q1
Q0
CPU
SPAN
[13]
UC(OOBFC)->
Only printed if there is congestion
UC->
1249
332
MC->
3247
1996
CPU buffer filling

up
CoPP Drops on Line Card

Line Card
North Star ASIC
Trident II
ASIC
Network Interfaces
N9K# show policy-map interface control-plane mod 6 class copp-system-p-classcritical | in ospf|trans|dropped

match access-group name copp-system-p-acl-ospf
transmitted 21898 packets;
dropped 0 packets;
Identify FM -Check CoPP Drops

N9K# show hardware internal cpu-mac inband active-fm traffic-to-sup
Active FM Module for traffic to sup:
0x00000015
Fabric Module in Slot 21 carry all traffic to Sup
N9K# show policy-map interface control-plane mod 21 class copp-system-p-classcritical | in ospf|trans|dropped

match access-group name copp-system-p-acl-ospf
match access-group name copp-system-p-acl-ospf6
transmitted 21898 packets;
dropped 0 packets;
Check for Drops/Errors-Fabric Module

Identify HG Port on LC and FM
N9K# show system internal fabric connectivity mod 6 | grep 21
LC-Slot
6
LC-Unit
LC-HGLink
HG10
MUX
FM-Slot
3B
FM-Unit
21
FM-HGLink
HG15
N9K# sh hardware internal fabric interface asic counters module 6 instance 0 asic-port 11
Important Counters/Drops
FabricInterface
0 / 11 / HG10
Verify Drops/Error on HG port on LC
Forward
Forward
Error Pkt
Error Pkt
QOS Rx
QOS Tx
RxDrops
TxDrops
RxDrops
TxDrops
Drops
Drops
N9K# sh hardware internal fabric interface asic counters mod 21 in 0
RxDrops
0 / 11 / HG15
TxDrops RxDrops
0
TxDrops
0
Drops
0
asic-port 16
Drops
0
Verify Drops Between FM and SC
System Controller
MVDXN-SW
module-21# show mvdxn internal port-status

Switch type: Marvell 98DXN11 - 10 port switch
Port
3
Descr
Enable Status ANeg
SC1EPCswitch Yes
UP
Fabric Module in Slot 21
Speed
Mode
No
MVDXN-SW
FABRIC CARD
InByte
OutByte
109548011
117051401
InPkts
OutPkts
274144
587285
10 port switch on System

controller and Fabric
module connect SC to FM
module-29# show mvdxn internal port-status

Switch type: Marvell 98DXN11 - 10 port switch System Controller in Slot 29
Port
7
Descr
FM1EPCswitch
Enable Status ANeg

Yes
UP
No
Speed
2
Mode
6
InByte
OutByte
InPkts OutPkts
746159513
60543666
620863
269592
Drops/Errors On Supervisor
N9K#show hardware internal cpu-mac inband counters
inb|dro
eth2
Link encap:Ethernet
in eth|ps-
HWaddr 00:00:00:01:1b:01

eth3
Link encap:Ethernet
HWaddr 00:00:00:01:1b:01

ps-inb
Link encap:Ethernet
Netstack
HWaddr 00:00:00:01:1b:01

Pseudo Inband
NIC-Eth2
NIC-Eth3
Supervisor Card
Drops/Errors On Supervisor-Cont.
N9K#show hardware internal cpu-mac inband stats | in
Queue Idx
Packet Count
Bytes
Drops
errors|rate|Queue
Csum Errors
Allocation Failure
Queue 0
65429
580195964
Queue 7
65429
580195964
CRC errors ...................... 0

Alignment errors ................ 0
Symbol errors ................... 0
Carrier extension errors .........0
Rx packet rate (current/peak)
812 / 1097 pps
Tx packet rate (current/peak)
454 / 741 pps
Related show tech(s)

Nexus9500# sh tech-support inband
counters
Nexus9500# show tech-support pktmgr
Nexus9500# show tech-support <service>
L2 Mac And Vlan Table Verification

N9K# sh mac address-table dynamic vlan 100
Legend:
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
True, (F) - False
VLAN
*
100
Eth6/1
MAC Address
Type
547f.ee1c.06fc
age
dynamic
Mac=547f:ee1c.06fc
interface Ethernet6/1
Secure NTFY Ports
(T) -
Eth6/1
switchport
switchport access vlan 100
N9K# bcm-shell mod 6 " l2 show"
| in Hit
mac=54:7f:ee:1c:06:fc vlan=100 GPORT=0x800800d modid=16 port=13/xe0 Hit
N9K# bcm-shell mod 6 "vlan show 100

vlan 100
ports xe0,hg ....... untagged xe0
no shutdown
Spanning Tree Verification

N9K# sh spanning-tree interface ethernet 6/1
Vlan
VLAN0100
Role Sts
Desg FWD
Cost
128.1537
Prio.Nbr
Type
P2p
VLAN.ipipe0[100]: <VP_GROUP_BITMAP=0x00000STG=0X67
FID_ID=0x64
FID_ID=Vlan ID.
Block: xe1-xe47
Forward: xe0,hg
no shutdown
N9K# Dec 0x67=103

STG= STP Group ID
N9K# bcm-shell mod 6 "stg stp 103

STG 103:
switchport
switchport access vlan 100
N9K# bcm-shell mod 6 "dump vlan 100
N9K# Dec 0x64=100
interface Ethernet6/1
Eth6/1
Mac=547f:ee1c.06fc
Unicast L3 Forwarding
T2 has combination of dedicated TCAM table space and shared hash table
memory known as Unified Forwarding Table (UFT)
The UFT is partitioned into three forwarding tables
MAC Address Table
IP Host Table
Longest Prefix Match-LPM Table
To maximize the system-wide forwarding scalability UFT tables on line
cards and fabric modules for different forwarding lookup functions
Feature
Scale
L3 LPM
Table
128K
FM
Feature
Scale
120K
LC
L3 Host Table
And L2/L3
Multicast
L2 Mac Table
96K
Unicast L3 Forwarding- Component Information

Theory of Operation
Software/Hardware Programming
OSPF communicates with uRIB to build the
routing table
AM builds the next-hop adjacency entry
uFDM distributes the information to the line
cards
IP FIB (running on the line cards) programs the
ASIC components with the forwarding and
adjacency information.
Remember: Software forwarding by the SUP is only

used for control and exception packets
OSPF
AM
ARP
uRIB
uFDM
Supervisor
Hardware-T2
FIB Manager
Forwarding Hardware
L3 Unicast Troubleshooting Flow

Next-Hop
Check the routing table
ARP/MAC
Check the ARP Table
Checking Route on
RIB And FIB.
Check Forwarding Route
Show ip route [ipv4] [<prefix>]

Show ip arp [ipv4]
show ip adjacency (Ipv4]
show forwarding adjacency platform [ipv4]
module <mod>
show forwarding [ipv4] route module <mod>
HW Programming
On LC/FM
Use BCM commands
bcm-shell mod 22 "l3 defip show"
Unicast L3 Forwarding- Two Possible Scenarios

Case 1: If incoming packet hit /32 host route on LC, forwarding decision is made on LC
Case 2: If incoming packet miss /32 host route on LC. Now for Longest Prefix
match (LPM) the packet get forwarded to FM
Install a default route 0.0.0.0/0 on Line Cards using the virtual MOD ID for Fabric Module
as the DMOD to force Line Cards to forward LPM packets to Fabric Modules
Fabric Modules perform LPM lookup and forward packets to the resolved Destination
MOD/Destination PORT
Also will verify How to Check ECMP Route
Network Diagram-Problem Definition

13.13.13.12/30
.13
.14
13.13.13.0/30
.1
.2
13.13.13.8/30
.9
.18
.17
13.13.13.16/30
Nexus3064Q-ESC#
N9K#
.10
N9508c-SJ#
Nexus3064Q-ESC# ping 13.13.13.10

PING 13.13.13.10 (13.13.13.10): 56 data bytes
Request 0 timed out
Nexus3064Q-ESC# traceroute 13.13.13.10
traceroute to 13.13.13.10 (13.13.13.10), 30 hops max, 40 byte
packets
1
2
13.13.13.2 (13.13.13.2)
* * *
1.124 ms
0.911 ms
0.752 ms
N9508d-SJ#
Router MAC Programming Check

Router Mac address must be programmed in Hardware
N9K1#show interface ethernet 6/1 | grep address
Hardware: 100/1000/10000 Ethernet, address: 003a.99fc.dd7f
N9K1# bcm-shell mod 6 "0:d chg my_station_tcam" | grep dd7f
MY_STATION_TCAM.ipipe0[0]: <VALID=1,------snip----MAC_ADDR=0x003a99fcdd7f,
Verify /32 Host Route on Line card-Case 1

N9K1#show ip route 13.13.13.14
/32 Host Entry
13.13.13.14/32, ubest/mbest: 1/0, attached
*via 13.13.13.14, Eth6/33, [250/0], 00:37:24, am
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 13.13.13.14
Entry VRF IP address
Mac Address
INTF
MOD PORT CLASS HIT
10
1
13.13.13.14 00:00:00:00:00:00 100010
0
0
0
y
N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100010

Entry Mac
Vlan INTF
PORT MOD MPLS_LABEL
100010 88:f0:31:bf:ad:17 4095 4432
45
16
-1
ToCpu
no
Drop
no
N9K1#show system internal ethpm info interface ethernet 6/33 | grep -i STATIC
IF_STATIC_INFO: port_name=Ethernet6/33,if_index:0x1a284000,ltl=40875,slot=5,
nxos_port=32,dmod=16,dpid=45,
Next Hop Reached via L3-Port Channel

N9K1#show ip route 10.164.112.22
/32 Host Entry
10.164.112.22/32, ubest/mbest: 1/0
*via 13.13.13.14, Po200, [110/3], 00:09:33, ospf-10, intra
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22
Mac Address
INTF
MOD PORT CLASS HIT
175660 1 10.164.112.22 00:00:00:00:00:00 100012
0
0
0
y
N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100012

Entry Mac
Vlan INTF
PORT MOD MPLS_LABEL
100010 88:f0:31:bf:ad:17 665
4761
3t
0
-1
ToCpu
no
Drop
no
N9K1#show system internal ethpm info interface port-channel 200 |grep I STATIC
IF_STATIC_INFO: port_name=port-channel200,if_index:0x160000c7,ltl=2597,slot=0,
nxos_port=02,dmod=0,dpid=3,
Verify HW-Programming on LC or FM ? Case 2

N9K# show ip route 13.13.13.10
This is not /32 host Route.
IP Route Table for VRF "default
Packet forwarding decision

responsibility is of the Fabric
Module
13.13.13.8/30, ubest/mbest: 1/0

*via 13.13.13.6, Eth6/52, [110/41],
00:22:29, ospf-10, intra
ALL FM will be programmed
N9K# show forwarding route 13.13.13.10 module 21
with this Route
IPv4 routes for table default/base

Prefix
| Next-hop
13.13.13.8/30
Interface
13.13.13.6
| Labels
Ethernet6/52
Line Card Punting Packets to Fabric For LPM ?

N9K# show hardware internal forwarding adjacency statistics default-route mod 6
Module:6 Unit:0
Traffic matched adjacency for default route (destined to FM):
Unicast: Packets 148 Bytes 13382
N9K# bcm-shell mod 6 "0:l3 defip show"
Unit 0, Total Number of DEFIP entries: 12288
#
VRF
Net addr Next Hop Mac
INTF
VLAN
3072Override 0.0.0.0/0 00:00:00:00:00:00
149149 0
MODID PORT PRIO CLASS HIT

0
Mod 100 is assign to Fabric Module

N9K# bcm-shell mod 6 "l3 egress show" | inc 149149
Entry
Mac
Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop
149149 00:12:12:12:12:12 4095 8189 1
100
-1
no
no
Longest Prefix Match on Fabric Module

N9K# bcm-shell mod 22 "l3 defip show" | grep 13.13.13.8
#
VRF Net addr
196620 1 13.13.13.8/30
Next Hop Mac
INTF
00:00:00:00:00:00 100008
MODID PORT PRIO CLASS HIT VLAN

0
N9K# bcm-shell mod 22 "l3 egress show" | grep 100008

Entry
100008
Mac
88:f0:31:bf:ad:17
Vlan
INTF
4095
4520
PORT MOD MPLS_LABEL ToCpu Drop

10
17
-1
no
no
Mac add used for rewrite

N9K# show system internal ethpm
info interface eth 6/52 | grep dmod
IF_STATIC_INFO:
port_name=Ethernet6/52,if_index0x1a286600,ltl=40856,slot=5,nxos_port=51,
dmod=17,dpid=10,unit=1,
ECMP Route Validation

N9K#show ip route 10.164.112.22
10.164.112.22/32, ubest/mbest: 2/0
Multi-Path

N9K#sh routing hash
13.13.13.2 10.164.112.22 mod 6
Hashing to path *13.13.13.18
N9K#bcm-shell mod 6 "l3

multipath show"
Multipath Egress Object 200256
Out Interface: Eth6/34
Interfaces: 100008 100010
N9K#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22

Mac Address
INTF
17
00:00:00:00:00:00
200256
10.164.112.22
MOD PORT
0
CLASS HIT
0
n (ECMP)
Follow same steps demonstrated for /32 Host entry to learn about Interface in multipath show cli
Use Tools From Toolkit

ELAM- IF Line Card has North Star
module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
Packet Tracer- For All FM and LC having T2

N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
Flex Counter- Check Adjacency hit counter

interface ethernet 6/52 enable
show tech-support forwarding l3 unicast
Consistency Checker
show consistency-checker
show tech-support adjmgr

forwarding
ipv4 unicast show tech routing unicast
Virtual Port-Channel-vPC
Allow a single device to use a port channel across

two upstream switches
Eliminate STP blocked ports
Dual-homed server operate in active-active mode
HSRP-Both active and standby peers forward

packets-ARP response by Active
Configuration steps Same as other Nexus

Products
Logical Topology with vPC
Case:1 All vPC Leg UP

Scenario: Traffic of a Host in Vlan 10 connected to Switch-A hash to N9K1 to reach Host in Vlan 20
connected to Switch-B
N9k1
vPC Peer Link =Eth1/1,4/1
N9k2
Keep Alive
PC1-PeerLink
SVI10
SVI-Mac 78da.6e71.9a3f
Standby 10.10.10.3
SVI10
MCT-1/1, 4/1
Eth6/20
Eth4/18
10.10.10.1/24
10.10.10.2/24
Eth6/20
Eth4/18
HSRP-Mac 0000.0c07.ac0a
Standby 10.10.10.3
HSRP-Mac 0000.0c07.ac0a
SVI20
vPC10
SVI-mac 78da.6e71.9a3f
vPC20
SVI20
SVI-mac 003a.99fc.dd7f
10.10.20.1/24
Standby 10.10.20.3
SVI-mac 003a.99fc.dd7f
Switch-A
Switch-B
10.10.20.2/24
Standby 10.10.20.3
HSRP-Mac 0000.0c07.ac14
HOST-A
Vlan-10
10.10.10.x/24
HOST-B
Vlan-20 HSRP-Mac 0000.0c07.ac14
20.20.20.x/24
vPC-Router MAC Programming Check

Both Active and Standby Peer responsible for L3 switching
Virtual Mac address must be programmed in Hardware on Both peers
Interface
Vlan10
Grp Prio P
10
100
State Active addr

Active 10.10.10.2
Standby addr
local
Group addr
10.10.10.3
N9K1# bcm-shell mod 4 "0:d chg my_station_tcam" | grep

VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,
Interface
Vlan10
Grp Prio P State Active addr

10
100
Standby 10.10.10.2
Standby addr
local
Group addr
10.10.10.3
N9K2# bcm-shell mod 4 "0:d chg my_station_tcam" | grep

VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,
vPC Peer Gateway Programming Check

Are N9Ks Configured with Peer-Gateway
N9K1-SJ# show mac address-table vlan 10 | in G
G
10 0000.0c07.ac0a static
F
F
vPC Peer-Link(R)
G 10 003a.99fc.dd7f static
F F
sup-eth1(R)
N9K2 SVI MAC
G
10 78da.6e71.9a3f static
F
F
vPC Peer-Link
N9K# bcm-shell mod 4 "0:d chg my_station_tcam" | egrep 0x003a99fcdd7f

MY_STATION_TCAM.ipipe0[0]:
<VALID=1,MAC_ADDR_MASK=0xffffffffffff,MAC_ADDR=0x003a99fcdd7f,KEY=0x00000000003
a99fcdd7f,IPV6_TERMINATION_ALLOWED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_R
ARP_TERMINATION_ALLOWED=1>
vPC Check For Traffic Ingressing Peer Link

Egress Block Mask
vPC Check-Traffic from Peer Link should Not L2/L3 Switch with local and remote
Legs up
N9K1# show vpc brief | grep Po
id Port
Status
Active vlans
1
Po1
up
10-20
id Port Status Consistency Reason Activevlans
10 Po10
up
success
success
10-20
20 Po20
up
success
success
10-20
N9K2# show vpc brief | grep Po
id Port
Status
Active vlans
1
Po1
up
10-20
id Port Status Consistency Reason Activevlans
10 Po10
up
success
success
10-20
20 Po20
up
success
success
10-20
N9k1
N9k2
Keep Alive
PC1-PeerLink
MCT-1/1, 4/1
Eth6/20
Eth4/18
Eth4/18
Eth6/20
vPC10
Switch-A
vPC20
Switch-B
vPC Check for Traffic Ingressing Peer Link (Contd)

N9K1#show port-ch summary |
Group Port- Type Protocol
1
Po1(SU) Eth
LACP
10
Po10(SU)Eth
LACP
20
Po20(SU)Eth
LACP
in Po
Member Ports
Eth1/1(P)Eth4/1(P)
Eth4/18(P)
Eth6/20(P)
N9K1# show system internal vpcm info mask

module 6
Masked ports for Module 6, Unit 0:
[Src Port None]: Eth6/20
[Src Port Eth1/1]: Eth6/20
[Src Port Eth4/1]: Eth6/20
Masked ports for Module 6, Unit 1:
N9k1
N9k2
Keep Alive
PC1-PeerLink
MCT-1/1, 4/1
Eth6/20
Eth4/18
Eth4/18
Eth6/20
vPC10
Switch-A
vPC20
Switch-B
Traffic Ingressing on Eth1/1 and

Eth4/1 will not exit Eth 6/20
ACL redirect logic for routed packets-vPC Leg Down
Redirect ACL installed to redirect routed packets for the

vPC for which local interface goes down
N9k1
Mac address learned from vPC points virtual port
PC1-PeerLink
MCT-1/1, 4/1
N9K1# show hardware access-list tcam region | grep vpc

VPC Convergence [vpc-convergence] size =
512
N9K1# sh mac address-table address30f7.0d9b.d401

VLAN MAC Address
20
Type
30f7.0d9b.d401 dynamic
age Secure NTFY Ports

0
vPC Peer-Link
N9k2
Keep Alive
Eth6/20
Eth4/18
Link Down
Eth4/18
Eth6/20
vPC10
Switch-A
vPC20
Switch-B
ACL redirect logic for routed packets-vPC Leg Down
On N9K1 traffic entering Eth6/20 after L3 switch

should egress Peer Link
N9k1
PC1-PeerLink
N9K2 Should not drop traffic entering Peer link and

forward traffic out to Eth 4/8
MCT-1/1, 4/1
Eth6/20
N9K# bcm-shell module 6 "fp show group 57

InPorts->L3Routable
N9k2
Keep Alive
Eth4/18
Eth4/18
Ln Down
Eth6/20
vPC10
vPC20
DstTrunk
Switch-A
Offset: 213 Width: 16

DATA=0x00008003
Trunk-id of 3 Down vPC
action={act=RedirectTrunk, param0=1(0x1)
Trunk-id of vPC Peerlink
Switch-B
ACL redirect logic for routed packets-Verify TrunkID

N9Ka# show system internal ethpm info int port-channel1 | grep dpid
IF_STATIC_INFO: port_name=port-channel1,if_index:0x16000000,ltl=2595,slot=95
dpid=1,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
N9508a-SJ# show system internal ethpm info int port-channel10 | grep dpid
IF_STATIC_INFO: port_name=port-channel10,if_index:0x16000000,ltl=2595,slot=95
dpid=3,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
show tech-support vPC

show tech-support cfs
show tech-support port-channel
Some important info to capture
ACL redirect logic for routed packets-Verify TrunkID

NX-OS -7.0(3)I1(2)
N9508a-SJ# show system internal access-list vpc-convergence mod 6
-----------------------------------------------------------VPC Convergence Entries
-----------------------------------------------------------Instance: 0
Trunk-id of 3 Down vPC
==========
Ingress:
Trunk-id of vPC Peerlink
---------Entry-ID DstTrunk-GID RedirectTrunk-GID Packet-Count
-----------------------------------------------------------------------1539
3
1
6082015
Nexus9000 Specific
Limitation and Goodies
Email from Nexus9000 To Cisco SR
Commands output directly sent to email address
Information from Nexus9000 Can be directly attached to Service Request.
Information is sent as body to email- not as attachment

N9K(config)# email
N9K(config-email)# smtp
N9K(config-email)# smtp-host 173.37.37.37
N9K(config-email)# from [email protected]
N9K(config-email)# smtp-port 25
show run | email subject <SR-number> [email protected]
Bash Support !!!!

Goes beyond what standard CLI can provide
Customers demand more capabilities/freedom Creativity
Feature: bash-shell
User Role: dev-ops or network-admin or vdc-admin*
Strongly recommended: Some experience with shell/Linux-Use with
extreme care
Broadcom ASIC shell access on the Nexus 9000 !!!
The Nexus 9000 is based largely on the Broadcom Trident II ASIC-Known as T2
The modular unit Fabric Modules (FM) and Line Cards (LC) each contain multiple
instances of the T2 ASIC, as well as the TOR (top of rack) units
Access is provided to each and every instance of the T2 ASIC
No additional license is required to access the bcm-shell
Permitted by default role network-admin
Role based access control (RBAC) can be used to limit user access
Accounting log available for BCM activity
BCM Access some Examples
hg0
T2
Instance
N9K# bcm-shell mod 6 "show unit"
Unit 1 chip BCM56852_A2
Xe0
N9K# bcm-shell mod 6 "ps" | in 19

xe19 up
1G FD
SW Yes Disable None
N9K# show accounting log
Eth1/1
FA
XGMII
hg11
T2
Instance 1
F F F F F F F
F F F F FQSPF
P P P P P P P
P P P P PPorts
01 02 03 04 05 06 07 08 09 10 11 12
Unit 0 chip BCM56852_A2 (current)
hg0
hg11
Xe11
Eth1/12
F F F F F F F
F F F F FQSPF
P P P P P P P
P P P P PPorts
13 14 15 16 17 18 19 20 21 22 23 24
Xe0
Eth1/13
1582
| last 2
Mon Apr 20 08:31:52 2015:type=update:id=console0:user=admin:cmd=bcm-shell

module 6 "show unit" (SUCCESS)
Mon Apr 20 08:32:14 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "ps" | in 19 (SUCCESS)
Xe11
Eth1/24
BCM Access some Examples (Contd)

N9K# bcm-shell mod 21 "config show l3"
l3_alpm_enable=2
l3_max_ecmp_mode=1
l3_mem_entries=16384
N9K# bcm-shell mod 4 "config show l2
l2xmsg_hostbuf_size=16384
l2_mem_entries=98304
Python !!!!
Python is - Established, Modern and Powerful, Clean, lots of libraries, liberal

license
Perl is available in gdb images only not available in final images
Tcl is there but no one uses it in NX-OS
The license that Python has (GPL-Like with very few restrictions on modification,
distribution and commercial use) make it very attractive to embed and distribute
On the box applications that can currently use Python scripts
Embedded Event Manager

Power On Auto Provisioning (POAP)
Create your own scripts that are like Super commands
Create your own command modifiers the things that act on commands applied with a
pipe |
Python-Continued
There are two Python environments on the N9000

One executed from VSH
One executed from Bash
Both run in their own forked process
The main differences comes from the environment that they get initialized into
These differences between them should be minimal
There is a sandbox that should primarily contain lower privileged users
Network-admin users get basically a pure 2.7.5 python environment
That sandbox mostly applies to lower privileged users, they may be prevented from doing certain things
in python
Also prevents file operations on files outside of bootflash
Python-Example
N9K# python
Python 2.7.5 (default, Oct
8 2013, 23:59:43)
switching between VSH and Python
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or

"license" for more information.
>>>
switch between VSH and the
N9K# run bash python
Interpreter (Bash 1)
Python 2.7.5 (default, Oct
8 2013, 23:59:43)
[GCC 4.6.3] on linux2

Type "help", "copyright", "credits" or
"license" for more information.
>>>
Python Script Example
Why Patching?
Many customers spend extensive time and effort to test and qualify software prior to deployment. In todays
environments, if a defect is found, effectively root-caused, and integrated, since it is rolled out through a
maintenance release, customers would need to restart their qualification cycle, wasting time, and pushing out
deployment dates
Bug Found, Diagnose, Root
Cause
Begin Code Test &

Qualification Cycle
Maint. Released
Restart Qual Cycle
Defect Resolved, integrated

into Maint.
Actual Deployment
Target Deployment
6 Months
10 Months
NX-OS Image Patching

The Nexus9000 Standalone platforms introduces new patching capabilities that allows specific defects to be
rolled out in an independent package that can be applied to existing base software binaries. This will help
reduce customer code certification times, leading to greater customer satisfaction.
Bug Found, Diagnose, Root
Cause
Begin Code Test &

Qualification Cycle
Continue Qual
With additional tests
Defect Resolved, Patch

Released
6 Months
7 Months
Actual Deployment
Target Deployment
Patching Overview
NXOS platforms release major versions when introducing new features and engineering
special builds to provide bug fixes.
The new goal will be to allow customers to deploy patches for specific fixes only without
affecting the data plane of the device.
The patching architecture comes from IOS XR (SMU Software Maintenance Upgrade)
used to deliver Quick, Effective and Focused patches for specific sections of code.
Both binaries and libraries can be patched.
Supervisors and Line Card services can be patched.
Software patching will leverage process restart/reload or ISSU
Patch Uninstall Workflow - Detailed
User invokes install deactivate <patch_name>
System manager gracefully shuts down each impacted process
Softlinks are changed from active SMU to one in backup folder (if present).
Relevant SMU is removed from the /var/installer/activated/SMU directory.
System Manager triggers restart of impacted processes
(Optional) install remove deletes the patch from the local repository
CLI Commands Patch Install

Command
Syntax
Function
Notes
Install add
install add <uri> [activate]
Download patch from URI and add

patch to repository.
Only one patch can be added at

a time. Optionally can activate
patch in this step.
Install remove
install remove [<package> |

inactive]
User can remove only nonactivated patches
Confirmation y/n will be prompted
Install activate
install activate <package> [test]
Installs a patch from the local

repository. If not present, an error
will be returned.
Only one patch can be activated

at a time. No show commands
permitted during operation.
Install deactivate
install deactivate <package>
Uninstall patch and move it to nonactivated repository
Only one patch can be

deactivated at a time. **Patches
must have no other patch
dependencies
Install commit
install commit
Preserves all activated patches

across reloads.
Activated patches are committed

to a list kept in the patch
repository
CLI Commands Show Commands

Command
Function
Sample
show install request
Shows current install operation along

with time stamp, package name,
initiating user and % complete.
Fri May 10 09:06:55.921 UTC

Install operation 13 '(admin) install activate n9000-dk.6.0.2.U1.1.CSCuf08219.bin
Started by user 'cisco' via CLI at 09:06:48 UTC Fri May 10 2013
The operation is 10% complete
show install log [id | detail

| from | last | reverse]
Shows user information on previous

installation operations. Optional [detail]
command for verbose information.
Install operation 1 by user admin at Tue Sep 28 01:37:02 2004:

install commit
Operation completed successfully
Install operation 2 by user admin at Mon Oct 18 17:26:36 2004:
install add tftp://10.52.241.252/bcarter/n3000-uk9.6.0.2.U1.1.CSCuf08219.bin
Operation completed successfully
Install operation 7 by user lab at Mon Oct 18 17:31:13 2004:
install activate n3000-uk9.6.0.2.U1.1.CSCuf08219
Operation failed because service failed to come up.
show install active [onreload]
Displays boot images and active or

committed patches
switch# show install active

Boot Images:
Kickstart Image: bootflash:/n9000-dk.6.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Active Packages:
n9000-dk.6.1.1.CSCui56298.bin
CLI Commands Show Commands (Contd)

Command
Function
Sample
show install inactive [onreload]
Shows patches in the repository not

yet activated
switch# show install inactive

Boot Images:
Image: bootflash:/inseor.6.1.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Inactive Packages:
switch#
show install pkg-info

<package>
Shows details of a specific patch.

Requires that patch has been added
using install add first.
switch# show install pkg-info n9000-dk.6.1.1.CSCui56298.bin

Contents of Package file 'n9000-dk.6.1.1.CSCui56298.bin':
Expiry date : Jan 19, 2015 02:55:56 UTC
Uncompressed size : 17892613
Vendor : Cisco Systems
Desc : Bug Fix for CDET: CSCui56298
Build : Built on Wed May 10 08:04:58 UTC 2013
Source : By n9k-infra-bld
Platform: Nexus-9000.
Supersedes: n9000-uk9.6.1.1.U1.1.CSCuf09119, n9000-uk9.6.1.1.U1.1.CSCuf02229
Pre-requisite: n9000-uk9.6.1.1.U1.1.CSCuf09219
Restart information: BGP process restart.
Sample Patch Install Copy Patch to Switch

N9K# copy
scp://[email protected]/home/sdn/n9k/inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
bootflash:
Enter vrf (If no input, current vrf 'default' is considered): management
[email protected]'s password:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
100% 233KB
232.7KB/s
00:01
Copy complete, now saving to disk (please wait)...
N9508#
N9508# dir | grep .gbin

238230
Jan 15 10:52:31 2014inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9508#
Sample Patch Install Add patch to repository & verify

N9K# install add bootflash:inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
Install operation 19 completed successfully at Wed Jan 15 10:55:14 2014
N9508#
N9K# show install packages

----------------------------------------------------------inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin inactive-commit
Modules
Module #27: inactive-commit
Module #28: inactive-commit
----------------------------------------------------------N9K# show install inactive
Inactive Packages:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9K#
Important Limitations
For every Feature please review Guidelines and Limitations
Cisco Nexus 9000 Series NX-OS Verified Scalability Guide
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_612I34.html
Only one software image (called nx-os) is required to load the Cisco NX-OS operating system.
EPLD Upgrade are recommended but are not mandatory
User Configured MAC address for SVI- Packets will not be flooded if Layer 2 Adjacency is missing
Diagnostic-The Port Loop back and Boot up Port Loop back tests are not supported
ASIC Memory-NS test is applicable only for the N9K-X9564PX and N9K-X9564TX line cards.
Priority flow control (PFC) is supported on Cisco Nexus 9500 Series switches with the N9KX9636PQ line card.
FEX is supported only on the Cisco Nexus 9372PX and 9396PX switches.
Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of
running 8-queue mode.
Participate in the My Favorite Speaker Contest

Promote Your Favorite Speaker and You Could Be a Winner
Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
Send a tweet and include

Your favorite speakers Twitter handle <Speakerenter your Twitter handle here>
Two hashtags: #CLUS #MyFavoriteSpeaker
You can submit an entry for more than one of your favorite speakers
Dont forget to follow @CiscoLive and @CiscoPress
View the official rules at http://bit.ly/CLUSwin
Complete Your Online Session Evaluation
Give us your feedback to be

entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
Complete your session surveys

though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Table Topics
Meet the Engineer 1:1 meetings
Related sessions
Thank you
Backup Slides
Backup Slides !!!!
Fabric Module
Fabric Module for Nexus 9504
NFE

NFE
NFE
Chassis Type
NFEs per Fabric Module
Nexus 9504 Nexus 9508 Nexus 9516

1
NFE
NFE
NFE
NFE
Nexus 9500 Platform FRU- Line Card

Connect to Fabric Modules
12 x 42 Gbps
Fabric 2
Fabric 1
N
F
E
N
F
E
N
F
E
N
F
E
Fabric 3
N
F
E
Fabric 4
N
F
E
N
F
E
N
F
E
Fabric 5
N
F
E
Fabric 6
N
F
E
N
F
E
N
F
E
ALE 1
12 x 42
Gbps
1 x 42
Gbps
1 x 42
Gbps
NFE 1
18x 40
Gbps
Ethern
et
NFE
NFE
Network
Interfaces
NFE
18x 40Gbps
12 x 40 Gbps
12 x 40 Gbps
12 x 40 Gbps
Connect to Hosts or
Network
N9K-X9636PQ
FM1
FM2
HG Ports
FM3
FM4
HG Ports
FM5
FM6
HG Ports
T2
T2
T2
Instance 0
Instance 1
Instance 2
QSPF Ports
QSPF Ports
QSPF Ports
FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12
13 14 15 16 17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32 33 34 35 36
N9K-X9464PX
FM2
FM3
FM4
MUX1-2
FM6
MUX3-4
HG Ports
T2
HG Ports
10G SFP+ Ports

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
40G QSFP
FP FP FP FP
49 50 51 52
N9K-X9464TX
FM2
FM3
FM4
MUX1-2
FM6
MUX3-4
HG Ports
HG Ports
T2
100/1000/10000 T Ports
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
40G QSFP
10G
PHY
10G
PHY
10G
PHY
10G
PHY
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
FP FP FP FP
49 50 51 52
N9K-X9432PQ
FM2
FM3
FM4
FM6
HG Ports
FP
01
FP
02
FP
03
FP
04
FP
05
FP
06
HG Ports
T2
T2
Instance 0
Instance 2
QSPF Ports
QSPF Ports
FP
07
FP
08
FP
09
FP
10
FP
11
FP
12
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
N9K-X9564PQ
FM6
FM4
FM5
HG MUX1
HG MUX2
HG MUX4
4567
0123
8 9 10 11
FM3
FM2
HG MUX3
HG MUX5
0123
7-5
4567
HG MUX6
8 9 10 11
MN Port
MN Port
Northstar 1
Northstar 2
MF Port
0-2
FM1
MF Port
3-5
6-8
9-11
0-2
9-11
2-0
31-29
26-24
7-5
26-24
Warpcore
T2
T2
40G QSFP
10G SFP+ Ports

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
FP FP FP FP
49 50 51 52
N9K-X9564TX
FM6
FM4
FM5
HG MUX1
HG MUX2
HG MUX4
4567
0123
FM3
FM2
HG MUX3
HG MUX5
8 9 10 11
4567
0123
MN Port
Northstar 1
Northstar 2
MF Port
0-2
3-5
6-8
9-11
0-2
9-11
7-5
2-0
31-29
26-24
7-5
26-24
T2
T2
40G QSFP
100/1000/10000 T Ports
10G
PHY
10G
PHY
10G
PHY
10G
PHY
HG MUX6
8 9 10 11
MN Port
MF Port
10G
PHY
FM1
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
FP FP FP FP
49 50 51 52
Multicast L3 Forwarding
Before hardware can forward any Multicast packets,

forwarding information has to propagate from Sup to the LC
Several layers are to be verified:
MRIB (control-plane is created here)
MFDM PI /PD (platform independent & forwarding
information)
MFIB-IPFIB
IP FIB process programs hardware:
FIB Table contains (*,G) and (S,G) forwarding entries and RPF
information
GROUP table contains forwarding and pointers replication
information (pointers to MC VLAN)
MC VLAN tables contain replication information (~OIF lists)
Hardware (packets are forwarded here) & SDK
PIM
IGMP
Supervisor
MSDP
MRIB
MF DM
IP FIB
T2
Line Card
FIB Table MC VLAN Table

IPMC_GR
L2/L3 Multicast Packet Walk

Fabric Module
Lookup to resolve egr.

modules;
Sends one copy to each
egr. module;
Trident II
Lkup in Host Table
& L2 Table
L2/L3 mcast lookup;

Replicate pckts to local
receiving ports;
Send 1 copy to fabric
module;
Trident II
Trident II
IACL
Traffic
Classification&
Remarking
Egress Q
EACL
Parser
Network Interfaces
10GE
Egress Q
EACL
L2/L3
Lookup &
Pkt rewrite
Parser
L2/L3
Lookup &
pkt rewrite
Examines ingress
packet. Get packet
headers for
processing.
IACL
Traffic
Classification
& Remarking
Network Interfaces
40GE
10GE
40GE
Lookup for local

receiving ports;
replicate pkts onto
those ports.
Multicast L3 Forwarding-MRIB
N9K# show ip mroute 239.10.10.10 shared-tree
Supervisor
IP Multicast Routing Table for VRF "default
(*, 239.10.10.10/32), uptime: 00:23:32, ip pim

Incoming interface: Ethernet6/1, RPF nbr:
13.13.13.1
Outgoing interface list: (count: 1)
Ethernet6/52, uptime: 00:22:42, pim
PIM
IGMP
MRIB
IP FIB
MF DM
MSDP
Multicast L3 Forwarding-mFDM PI-Supervisor

N9K# show forwarding distribution ip multicast route group 239.10.10.10 source
13.13.13.14 | in 13|Index
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 1
N9K# show forwarding distribution multicast outgoinginterface-list l3 1
Reference Count: 4
Supervisor
PIM
IGMP
MSDP
Platform Index: 0xb00001
MRIB
MF DM
Number of Outgoing Interfaces: 1 t6/52
IP FIB
Multicast L3 Forwarding IPFIB-Line card

N9K# show forwarding ip multicast route group 239.10.10.10 source
13.13.13.14 mod 6 | inc 239|Eth
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 0x1
Ethernet6/52
T2
Line Card

IPMC_GR
Mod 6 is N9K-X9564TX
To reach Ethernet 6/52 which is on NS from front port of T2,Packets need to
cross Fabric module
Multicast L3 Forwarding Entries on LC BCM Shell

N9K# bcm-shell mod 6 "ipmc table show"
SRC IP ADDRESS
MC IP ADDRESS
MC GROUP
VID VRF COS
13.13.13.14
239.10.10.10
0x2000007
75680
no
0.0.0.0
239.10.10.10
0x2000007
86578
no
N9K#bcm-shell module 6 "mc show group=0x2000007"

Group 0x2000007 (L3)
HWIDX CLASS HIT
T2
Line Card
FIB Table MC VLAN

Table IPMC_GR
port hg0, encap id -1

-------snip-----------port hg11, encap id -1
Traffic spared to Hig towards Fabric
Multicast L3 Forwarding Entries on LC BCM Shell

N9K# bcm-shell mod 6 " search l3_entry_ipv4_multicast group_ip_addr=0xef0a0a0a
source_ip_addr=0x0d0d0d0e
L3_ENTRY_IPV4_MULTICAST.ipipe0[75680]:
SOURCE_IP_ADDR=0xd0d0d0e,
GROUP_IP_ADDR=0xef0a0a0a,
L3MC_INDEX=7
T2
Line Card

IPMC_GR
N9K# bcm-shell mod 6 " dum chg l3_entry_ipv4_multicast 75680

show tech-support multicast`
IPV4MC:EXPECTED_L3_IIF=0x112e,
show tech-support forwarding multicast
N9K# show system internal eltm info interface ethernet 6/1 | in LIF
cr_flags = INTF LIF , LIF = 4398 (0x112e), LTL = 40959 (0x9fff) (S 0x0, P 0x0)
IGM Snooping
Forwarding programming in vPC Scenario
IGMP Process Provides both Layer 3 IGMP Processing , and Layer 2 IGMP snooping functionality
Receivers use IGMP (Internet Group Management Protocol) to report their multicast group
Membership to router
Layer 2 IGMP Snooping functions of IGMP process include processing snooped multicast router
Packets Including IGMP reports and leaves sent by receiver
Once the group membership is learned , the Supervisor Engine informs I/O modules , which
program Hardware
This will Constrain data-plane multicast packets to only those ports with multicast routeror interested
receivers in HW
IGMP Snooping continued
BCM on FM are in Mode 4. This will have L2 Table size of 32K & L3 Host Table 16K
L3 Host table will be used to program (*,G) /(S,G) entry. This will will accommodate
maximum of 8K entry.
MFDM sends two OIF List information to MFIB. One for LC (S,G) OIF List and other for
FM ( Mac, Group) OIF List in PIM disable Vlan.
MFIB will use (S,G) OIF list to program LC and Mac Group to Program FM in 32K L2 Table.
If PIM is enable FM can accommodate 8K(VRF, S,G) and will program Hardware.
Address aliasing is possible because on FM we use L2 table to program Mac Group information
IGMP Snooping (Contd)
With vPC IGMP will have knowledge of multi chassis Ether Channel trunk (MCT) interface.
When one of the vPC peer receives IGMP join , it will sync up this with peer over MCT link
using cFS-Cisco Fabric Services over Ethernet .
Duplication of traffic crossing MCT is avoided using Port block Mask
VPC Support PIM-SM Only
For source in VPC domain dual Forwarders are used
For Source in Layer 3 Cloud , Unicast best metric determines active forwarder
VPC Operational Primary in case of tie. CFS used to negotiate active Forwarder role
Configuration-IGMP Snooping enable by default

Nexus9508-13# sh ip igm snooping vlan 103
IGMP Snooping information for vlan 103
IGMP snooping enabled
Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 10.10.103.5, version: 2, i/f Po30
Nexus9508-13# sh ip igm snooping vlan 100
IGMP Snooping information for vlan 100
IGMP snooping enabled

Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 192.168.100.2, version: 2, i/f Vlan100
Querier interval: 125 secs

Querier last member query interval: 1 secs
Reference Topology for Troubleshooting

vPC Keep Alive
N9508-12
vPC Peer Link PO-10

Eth 3/1-2
N9508-13
Eth 3/1-2
Eth 6/9/1-4
Eth1/3/1-4
vPC30
Eth 1/17-18 ,Eth 1/33-34
N93k
Eth 1/48
Ixia 10/2-Source
vPC 35
Eth 1/17,Eth 1/19 , Eth 1/33-34
N35K
Eth1/48
Ixia 10/1-Receiver
IGMP Snooping Troubleshooting

Stream will enter one of the VPC-Peer , Which will get forwarded across Peer link to other VPC Peer
Both boxes will have (S ,G)
Upon Creation of (S,G) , VPC Peers negotiate best metric
Both realize source is VPC-Connected
Install Entry as Win-Force

If either peer gets a PIM/IGMP Join for the given source , they both add Interface to OIF
Nexus9508-12(config)# sh ip pim internal vpc rpf-source
Nexus9508-13# sh ip pim internal vpc rpf-source
PIM vPC RPF-Source Cache for Context "default" - Chassis

Role Primary
PIM vPC RPF-Source Cache for Context "default" - Chassis

Role Secondary
Source: 192.168.100.10
Source: 192.168.100.10
Pref/Metric: 0/0
Pref/Metric: 0/0
Source role: primary
Source role: secondary
Forwarding state: Win-force (forwarding)
Forwarding state: Win-force (forwarding)
MRIB Forwarding state: forwarding
MRIB Forwarding state: forwarding
vPC Peer receiving Join
IGMP Join from one of the receiver enter one of the VPC Pee.
This Peer encapsulates IGMP in CFS , sends to other Peer
Both Peer have identical State
Both Peer install OIF
Data traffic flows down to Receiver, also forwarded to other Peer on Peer Link
Other Peer drop the packet either by PORT BLOCK MASK blocking or no OIF
Nexus9508-ESC-12# sh ip mroute 239.10.10.10 192.168.100.10
Nexus9508-ESC-13# sh ip mroute 239.10.10.10 192.168.100.10
IP Multicast Routing Table for VRF "default"
IP Multicast Routing Table for VRF "default"
(192.168.100.10/32, 239.10.10.10/32), uptime: 01:00:09, ip pim

mrib
(192.168.100.10/32, 239.10.10.10/32), uptime: 04:25:36, ip pim

mrib
Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:

01:00:09, internal
Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:

04:25:36
Vlan101, uptime: 00:59:40, mrib
Vlan101, uptime: 02:04:41, mrib
Nexus9508-ESC-12#
Nexus9508-ESC-13#
Step to verify PI On Supervisor. Verify on Both Peers

Nexus9508-ESC-12# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.10"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address
239.10.10.10
Type Interface
D
Vlan101
Uptime
Expires Last Reporter
00:01:23 00:02:56 192.168.101.13
Nexus9508-ESC-12#
Nexus9508-ESC-13# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.1
0"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address
239.10.10.10
Type Interface
D
Nexus9508-ESC-13#
Vlan101
Uptime
Expires Last Reporter
00:01:18 00:03:01 192.168.101.13
CFS Provide info

Nexus9508-ESC-12# sh ip igmp snooping groups vlan 101 detail
IGMP Snooping group membership for vlan 101
Group addr: 239.10.10.10
Nexus9508-ESC-13# sh ip igm snooping groups vlan

Group ver: v2 [old-host-timer: not running] 101 det
Last reporter: 192.168.101.10
IGMP Snooping group membership for vlan 101
IGMPv2 member ports:
IGMPv1/v2 memb ports:
Po35 [1 GQ missed], cfs:false, native:true
Group addr: 239.10.10.10

Group ver: v2 [old-host-timer: not running]
Last reporter: 192.168.101.10
vPC grp peer-link flag: exclude
IGMPv2 member ports:
M2RIB vPC grp peer-link flag: exclude
IGMPv1/v2 memb ports:
Nexus9508-ESC-12#
Po35 [0 GQ missed], cfs:true, native:false

vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: exclude
Nexus9508-ESC-13#
Verifying Multicast forwarding Distribution Module

Platform Independent On Supervisor
Nexus9508-ESC-12# sh forwarding distribution multicast route group 239.10.10.10
source 192.168.100.10
(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:
Received Packets: 1073 Bytes: 36977
Number of Outgoing Interfaces: 2
Vlan100
( Mem L2 Ports: port-channel10 )
Not showing PC 10 for Vlan 101 because of

exclude flag seen while checking igmp
snooping stats.
Vlan101
( Mem L2 Ports: port-channel35 )
Note: On shutting down local vpc only, igmp does not send update to mfdm/ipfib to update the mroute state.
That is why you did not see mfdm/ipfib removing local vpc. So if local leg of vPC is down we will still PC in the above output.

Platform Independent On Supervisor-(Contd)
Nexus9508-12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1

Number of next hops: 2

Vlan: 101
port-channel35
bridged Vlan
port-channel10
Hardware Outgoing Interface List Index: 33554443

Platform Independent On Supervisor-IGMP-Snooping
Nexus9508-12# sh forwarding distribution ip igmp snooping vlan 101 group 239.10.10.10 det
Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
Reference Count: 1
Nexus9508-13# sh forwarding distribution ip igmp snooping vlan 101 group
Platform Index: 0xa00004
239.10.10.10 det
Vpc peer link exclude flag set
Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
port-channel10
Reference Count: 1
port-channel35

port-channel10
port-channel35
Verifying Multicast Forwarding Distribution Module

Platform Independent On Supervisor-Snooping Group.
Nexus9508-12# sh forwarding distribution l2 multicast mac-based vlan
101
Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
Reference Count: 1
Nexus9508-13# sh forwarding distribution l2 multicast mac-based vlan 101
Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
port-channel10
Reference Count: 1
port-channel35

port-channel10
port-channel35
IPFIB on LC for IGMP Snooping programming.

Nexus9508--12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1
port-channel30 (Vlan: 101)
port-channel10 (bridged)
Nexus9508-13# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 6

port-channel30 (Vlan: 101)
port-channel10 (bridged)
Verifying Hardware Programming

Nexus9508--12# bcm-shell mod 1 "mc show group=33554441"
If we see encap id a positive #
Executing mc show group=33554441 on bcm shell on module 1
then it is LIF
Group 0x2000009 (L3)

port hg0, encap id 400005
If we see encap id = -1 then it is
L2 bridge copy.
port xe10, encap id 21

port xe11, encap id 21
Nexus9508-12# bcm-shell mod 3 "mc show group=33554441"
Executing mc show group=33554441 on bcm shell on module 3
Group 0x2000009 (L3)
port xe0, encap id -1
port xe1, encap id -1
Nexus9508-12# sh system internal eltm info interface vlan 101 | in LIF

cr_flags = INTF VLAN , LIF = 21 (0x15), LTL = -1 (0xffffffff) (S 0x0, P 0x0)
Nexus9508-ESC-12#
From BCM to check what is HW index for given Group

Static entry of Mcast group
Hit Bit indicate flow is present
Mcast Index is where the traffic need to bridge
Nexus9508-12# bcm-shell module 1 "l2 show" | in MCast
mac=01:00:5e:0a:0a:0a vlan=101 GPORT=0x0 modid=0 port=0 Static Hit MCast=33554435
mac=01:00:5e:0a:0a:14 vlan=100 GPORT=0x0 modid=0 port=0 Static MCast=33554435
Nexus9508-12# sh ip igmp gr vlan 100
show tech-support ip igmp snooping

show tech-support ip multicast

BRKDCT 3101 PDF

Uploaded by

Copyright:

Available Formats

BRKDCT 3101 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 3101 PDF

Uploaded by

Copyright:

Available Formats

Nexus9000(Standalone)

To provide an overall understanding of the Nexus 9000 switching

This session will introduce System Telemetry, Troubleshooting tool Kits

BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches

System Health check Telemetry

Nexus 9000 Troubleshooting

Common Link Layer Issues-L1

Nexus9000 Specific Limitation and Goodies

Introduction-What is Nexus9000 Family ?

Nexus 9500 Series Switches

Nexus 9300 Series Switches

9500 Field Upgradeable Units (FRU)

The Supervisor, System controller ,Fabric Module and LC have OBFL

Nexus 9500 Platform FRU

Redundant Half-width supervisor engine

System Controller-What it is Role

Offload supervisor from internal device management tasks

Nexus 9500 Platform Line Card

I/O module with Merchant and

Have Various Forwarding Tables

NFE=Network Forwarding Engine-Trident 2(T2)

Note: Internal ports are called as Hi-Gig/HG ports

10G SFP+ Ports

Nexus 9500 Fabric Module

Interconnect Line Card slots

Nexus 9500 Fabric Module

Line Card Slot

Distributed Data Plane of Nexus 9500 Series Switches

Note: Internal ports are called as Hi-Gig/HG ports

Nexus9500 Series Line Card Summary

X9600 Series Line

X9500 Series Line

Line rate > 200 byte

High Level Block Diagram-N9500

All PSU, SC, SUP, FM, and

Port QSFP+ Uplink Module

AC/DC Power Supply

Front-to-Back & Back-to-Front Airflow

Latency: 1-2 usec

Wire-Speed L2/L3 Forwarding

Switch will not boot up without GEM

Nexus 9300 Series Switch Summary

NFE (BCM T2)

Yes (packets >

High Level Block Diagram-N9300

Front Panel 48x 1GE/10GE Ports

The last 2/3 numbers stand for

93128 128G (96 x 10G + 8 x 40G)

9396 96G (48 x 10G + 12 x 40G)

9372 72G ( 48 x 10G + 8 x 40 G)

Nexus9500 Unicast Packet Flow

Parse the first 128

Egress Line Card

Egress Line card

N9K-C9300 High Level Block Diagram

(12 x 40G) = 480G

(12 x 40G) = 480G FP

Main Features of Trident2 1280Gbps Switch ASIC

128 Integrated SerDes