HPE Mellanox Switch Training, Part 1

HPE Mellanox
Switch Training
Suchart Boonpan
MASE, CCNP, ACE-A, ACMP
May 2020
Network Diagram
Confidential – For Training Purposes Only 2

PPTV NETWORK DIAGRAM
Cisco core L3 #1, #2

MAGP 2:
Interface vlan: 400
State : Master VLAN 400 inter-peer link (IPL) Management Plane (Active/Standby)
Virtual IP : 10.0.102.1
SW1 IP : 10.0.102.10/24 Mpo30 Mgmt0, 1G
P24, 10G MLAG Mgmt IP
Sw2 IP : 10.0.102.10/24
Virtual MAC : 00:00:5E:00:01:02 Mgmt0 SW1: 10.0.99.61
SN2410M SN2410M Mgmt0 SW1: 10.0.99.62
Mgmt0, 1G MLAG VIP: 10.0.99.63
MAGP 1: P49-50
Interface vlan: 300 Mpo1-6 100G
State : Master P1-6 25G Cisco Mgmt/iLO switch
Virtual IP : 10.0.1.254
SW1:IP: 10.0.1.252/23
SW2:IP: 10.0.1.253/23
Virtual MAC : 00:00:5E:00:01:01
Link Aggregation
Data VLAN 300

Qumulo
ILO Servers 1-6
iLO
3
HPE M-SERIES SWITCHES SPECIFICATION
SN2010M SN2100M SN2410bM SN2410M SN2700M/SN2745M

18 Port 8 / 16 Port 24 / 48 Port 24 / 48 Port 16 / 32 Port
Half Width Half Width Full Width Full Width Full Width / Full & Short
Depth
18 x 1/10/25G SFP28 + 16 x 40/100 QSFP28 48 x 10G SFP + 48 x 10/25G SFP28 + 32 x 40/100G QSFP28
4 x 40/100G QSFP28 8 x 40 QSFP 8 x 40/100G QSFP
34 x 10/25G SFP28 64 x 10/25 SFP28 64 x 10GbE SFP 64 x 10/25G SFP28 64 x 10/25G SFP28
Flexible pricing & investment

Cost-optimized unique form factors Optimized for storage performance
protection
4
DOWNLINKS (COMPUTE AND STORAGE CONNECT)
HPE M‐Series SN2010M
(18) 10/25Gb SFP28 Ports
(4) 10/25/40/100Gb QSFP28 Ports
25G
25G
25G SFP+ DAC
25Gb SFP+ Direct Attached Copper (DAC .5m to 3m)

HPE 25Gb SFP28 to SFP28 0.5m Direct Attach Copper Cable (844471-B21)
HPE 25Gb SFP28 to SFP28 1m Direct Attach Copper Cable (844474-B21)
HPE 25Gb SFP28 to SFP28 3m Direct Attach Copper Cable (844477-B21)
5
CROSSLINKS (MLAG 100G PREFERRED BEST PRACTICE)
HPE M‐Series SN2100M 100G 100G

(16) 10/25/40/100Gb QSFP28 Ports
HPE M‐Series SN2010M 100G 100G

(18) 10/25Gb SFP28 Ports
(4) 10/25/40/100Gb QSFP28 Ports
100G QSFP28 DAC
100Gb QSFP28 Direct Attached Copper (DAC 1m)

HPE X240 100G QSFP28 to QSFP28 1m Direct Attach Copper Cable (JL271A)
6
M-SERIES SN2410M ETHERNET SWITCH
7
SN2410M M-SERIES NETWORK CONNECTIONS
Q ports go to the 640SFP28 NIC in
the Qumulo Nodes.
Q Q Comp Comp Edge
1 3 1 3 1 MLAG 1
Q Q Comp Comp Edge MLAG 2

2 4 2 4 2
Comp ports go to the 640SFP28 NIC MGMT
in the Compute Nodes.

Q ports go to the 640SFP28 NIC in
the Qumulo Nodes.
Q Q Comp Comp Edge

1 3 1 3 1 MLAG 1
Q Q Comp Comp Edge MLAG 2

2 4 2 4 2
MGMT
Comp ports go to the 640SFP28 NIC
in the Compute Nodes.
Q1 Q3 Comp Comp Tor1

iLO iLO 1 iLO 3 iLO Mgmt
Q2 Q4 Comp Comp Tor2
iLO iLO 2 iLO 4 iLO Mgmt
25 Gb DAC cables 1Gb, 10Gb, 25 Gb 10 Gb DAC cables

uplink (with
matching transceiver
1 Gb UTP or adapter) 100 Gb DAC cables
8
Q&A
– Question ?
– Break 5 m
9
Mellanox Switch User Interfaces

User Interfaces of the MLNX-OS®
1. Command Line Interface (CLI)

The CLI is accessed through: SSH, Telnet sessions, or directly via the console port on the front pane
2. Web Interface (web GUI)

The Web GUI is accessed through: HTTP or HTTPS
11
Connection with MLNX-OS
1. Access the MLNX-OS CLI via Serial Console.
12
User name and password
3. Log in with default credentials.

– User name: admin
– Password: admin
login as: admin
Mellanox MLNX-OS Switch Management
Using keyboard-interactive authentication.

Password:admin

2. Access the MLNX-OS WebUI via HTTP/HTTPS
Mellanox
Switch
Network
PC
14
3. Access the MLNX-OS CLI via SSH
Mellanox
Switch
Network
PC
15
CLI configuration modes—Standard and enable
Standard
– Identified by the CLI prompt >.
– Most restrictive mode.
– Includes commands that query only a restricted set of state information.
– Users cannot take any actions that directly affect the system—like rebooting the switch, or changing the
configuration.
– Use the enable command to move from standard mode to enable mode.
switchA [standalone: master] > enable
Enable
– Identified by the CLI prompt #.
– Offers commands to view all state information, and take actions like rebooting the system.
– Does not allow you to change any configurations.
– Use the disable command to move from enable mode to standard mode.
switchA [standalone: master] # disable

Global configuration mode—Config
Config
– Identified by CLI prompt (config)#.
– Allowed only for user accounts in the “admin” role.
– Has a full, unrestricted set of commands to view anything, take any action, and change any configuration.
– Use the configure terminal command to move from enable mode to config mode.
g1switchA [standalone: master] # configure terminal

g1switchA [standalone: master] (config) #
– Use ‘exit’ command to move from config mode to enable mode.
g1switchA [standalone: master] (config) # exit

g1switchA [standalone: master] #

Getting help
– Use ‘?’ from any mode to view available commands.

Use the space bar to see more commands, or ‘q’ to quit the display.
switchA [standalone: master] (config) # ?
aaa Configure Authentication, Authorization, and Accounting
access-list Configure access-list action
banner Set system banners
boot Configure system boot parameters
clear Reset certain statistics or clear caches
cli Configure CLI shell options
clock Set the system clock or timezone
configuration Manipulate configuration files
– Use the command followed by ‘?’ to view available command parameters.
switchA [standalone: master] (config) # show ?
aaa Display Authentication, Authorization, and Accounting settings
access-list Display IPV4 information
access-lists List access lists
asic-version Display asic version
banner Display banner settings
bootvar Display installed system images and boot parameters
cli Display CLI options
clock Display system time and date

CLI commands autocomplete
– Use [Tab] to auto-complete commands.

c [Tab] - displays all commands that start with ‘c.’
switchA [standalone: master] # c
clear cli configure crypto
– co [Tab] - autocompletes to ‘configure.’

– Unique prefix of a command can be used, instead of the full command.
Example: ‘co t’ can be used instead of ‘configure terminal.’
switchA [standalone: master] # co t
switchA [standalone: master] (config) #

Saving the configuration
– Save running-config into active-config.
switchA [standalone: master] (config) # configuration write

switchA [standalone: master] (config) # show configuration files
initial (active)
initial.bak
Active configuration: initial

Unsaved changes: no
Or
switchA [standalone: master] (config) # write memory

Mellanox Switch Image (Operating System)

MLNX-OS images
– Two images are stored in the flash memory: Partition 1 and Partition 2.
switchA [standalone: master] (config interface mgmt1) # show images
Installed images:
Partition 1:
version: X86_64 3.8.2204 2019-12-29 16:11:11 x86_64
Partition 2:
version: X86_64 3.7.1134 2019-01-24 13:38:57 x86_64
Last boot partition: 1

Next boot partition: 1
– By default, the image from Partition 1 is loaded at reboot.

– Mellanox Operating system = ONYX = MLNX-OS (same thing)

WebUI Onyx Image Upgrade
1. Use the IP address of each Switch of the

management interface on the address bar of
your browser, example;
(switchA 10.25.19.11)
(switchB 10.25.19.12)
2. Type in user name and password default:

admin, admin
3. Press Login

1. Choose System
2. Choose Onyx Upgrade
3. Select Install from local file:
4. Select Choose file:
5. Select Install Image

1. Choose System
4. Select Choose file: onyx-X86_64-3.8.2204

1. Choose System

1. Choose System

1. Choose System

WebUI Update Status
1. Please note file copy and then Install
2. Please note Image Update Status
3. Please select Reboot

WebUI Update Status

WebUI Update Status

Q&A
– Question ?
– Break 10 m or Lunch
32
MLAG – Multi Chassis LAG

List of Network protocols used in this project.
– MLAG L2
– MLAG L3 (MAGP)
– Interface Port-Channel (Link Aggregation)
– Interface MLAG Port-Channel (Multi Chassis Link Aggregation)
– Spanning tree mode RPVST
– Switch port mode Hybridge
– Static Route
34
MLAG – Multi Chassis LAG:
– Physical ports of two separate switches are aggregated in one

Layer 3Layer 3
logical port. Network
Network
– MLAG switches appear as a single Layer 2 switch.

switchA switchB
– A peering device (host or switch) runs a standard LAG, and is
IPL
not aware of the fact that its LAG is connected to two separate
switches.
– MLAG provides: MLAG
– High bandwidth and load-balancing

– High availability in case of a link failure LAG
– High availability in case of a switch failure or a switch software

upgrade
Qumulo 1
The MLAG protocol components.
• Keepalive
• Unicast and multicast sync
• MLAG port sync
36
MLAG Keepalive and Failover
– Master election in MLAG is based on the highest IPL VLAN interface IPs of the nodes.
– The MLAG pair of switches periodically exchanges a keepalive message (via IPL)
– If the keepalive message fails to arrive for three consecutive intervals the switches break into two
standalone switches.
– If IPL fail, the slave shuts down its interfaces to avoid a split brain scenario and the master becomes a
standalone switch.
37
Unicast and Multicast Sync
– It prevents unicast asymmetric traffic from loading the network with flood traffic
38
MLAG Port Sync
– Under normal circumstances, traffic from the IPL cannot pass through the MLAG ports (the IPL is isolated
from the MLAG ports).
– If one of the MLAG links break, the other MLAG switch opens that isolation and allows traffic from its peer
through the IPL to flow via the MLAG port which accesses the destination of the fallen link.
39
MLAG Global Configurations
1. Enable IP routing
– MLAG may be enabled without IP routing, but without IP routing an IPL VLAN interface cannot be
configured and thus MLAG does not function.
2. Enable IGMP snooping
– MLAG may be enabled without IGMP snooping, but if IGMP snooping is disabled, multicast FDBs do not
synchronize.
3. Enable LACP – if dynamic LAG is used.
4. Enable MLAG protocol commands.
– g1switchA configuration:
g1switchA [standalone: master] (config) # ip routing
g1switchA [standalone: master] (config) # ip igmp snooping
g1switchA [standalone: master] (config) # lacp
g1switchA [standalone: master] (config) # protocol mlag
– g1switchB configuration:
g1switchB [standalone: master] (config) # ip routing
g1switchB [standalone: master] (config) # ip igmp snooping
g1switchB [standalone: master] (config) # lacp
g1switchB [standalone: master] (config) # protocol mlag
IPL Port-Channel
1. Create a port-channel:
– Port-channel indexes on two switches may differ.
2. Set the port-channel as an IPL.

3. Group physical ports to the port-channel.
– LACP or static LAG can be used.
– switchA configuration:
g1switchA [standalone: master] (config) # interface port‐channel 1
g1switchA [standalone: master] (config interface port‐channel 1) # ipl 1
g1switchA [standalone: master] (config interface port‐channel 1) # exit
g1switchA [standalone: master] (config) # interface ethernet 1/19‐1/20
g1switchA [standalone: master] (config interface ethernet 1/19‐1/20) # channel‐group 1 mode active
g1switchA [standalone: master] (config interface ethernet 1/19‐1/20) # exit
– switchB configuration:
g1switchB [standalone: master] (config) # interface port‐channel 1
g1switchB [standalone: master] (config interface port‐channel 34) # ipl 1
g1switchB [standalone: master] (config interface port‐channel 34) # exit
g1switchB [standalone: master] (config) # interface ethernet 1/19‐1/20
g1switchB [standalone: master] (config interface ethernet 1/19‐1/20) # channel‐group 1 mode active
g1switchB [standalone: master] (config interface ethernet 1/19‐1/20) # exit
IPL VLAN Configuration
1. Create a VLAN and a VLAN interface for the IPL.
2. Set an IP address and a netmask for the VLAN interface.
– The switch with highest IP address is elected as the MLAG master.
3. Map the VLAN interface to be used on the IPL and set
the peer’s IP address.
–switchA configuration:
g1switchA [standalone: master] (config) # vlan 4094
g1switchA [standalone: master] (config vlan 4094) # exit
g1switchA [standalone: master] (config) # interface vlan 4094
g1switchA [standalone: master] (config interface vlan 4094) # ip address 172.16.34.253 /30
g1switchA [standalone: master] (config interface vlan 4094) # ipl 1 peer‐address 172.16.34.254
g1switchA [standalone: master] (config interface vlan 4094) # exit
g1switchB [standalone: master] (config) # vlan 4094
g1switchB [standalone: master] (config vlan 4094) # exit
g1switchB [standalone: master] (config) # interface vlan 4094
g1switchB [standalone: master] (config interface vlan 4094) # ip address 172.16.34.254 /30
g1switchB [standalone: master] (config interface vlan 4094) # ipl 1 peer‐address 172.16.34.253
g1switchB [standalone: master] (config interface vlan 4094) # exit
MLAG Cluster Configuration
1. Configure the MLAG cluster:
– Both switches must be configured with an identical
unique group name.
– One of the switches is also configured with the VIP –
it is the cluster master.
– VIP address must be of the management subnet.
switchA [standalone: master] (config) # mlag‐vip MLAG‐G1 ip 10.25.19.13 /16
switchA [MLAG‐G1: master] (config) #
switchB [standalone: master] (config) # mlag‐vip MLAG‐G1
switchB [MLAG‐G1: standby] (config) #
– Wait for a few seconds until prompt is changed to group name and cluster master/standby.
Enable MLAG Protocol
– Enable MLAG protocol:
– MLAG protocol is disabled by default.
switchA [MLAG‐G1: master] (config) # mlag

switchA [MLAG‐G1: master] (config mlag) # no shutdown
– switchB configuration
switchB [MLAG‐G1: standby] (config) # mlag

switchB [MLAG‐G1: standby] (config mlag) # no shutdown
Verify MLAG VIP Configuration
switchA [MLAG‐ACAD: master] (config) # show mlag‐vip

MLAG VIP
========
MLAG group name: MLAG‐G1
MLAG VIP address: 10.25.19.13/16
Active nodes: 2
Hostname VIP‐State IP Address

‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
switchA master 10.25.19.11
switchB standby 10.25.19.12
Cluster master/ standby

Verify MLAG Configuration
switchA [MLAG‐G1: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload‐delay: 30 sec
Keepalive‐interval: 1 sec
Upgrade‐timeout: 60 min
System‐mac: 00:00:5E:00:01:57 MLAG virtual MAC
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:

Inactive: 0
Active‐partial: 0
Active‐full: 1
MLAG IPLs Summary:

ID Group Vlan Operational Local Peer
Port‐Channel Interface State IP address IP address
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
1 Po1 34 Up 172.16.34.253 172.16.34.254 Switch g1switchA ‐ MLAG master
MLAG Members Summary: Switch g1switchB ‐ MLAG Standby

System‐id State Hostname
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
00:02:C9:A8:E2:D8 Up <g1switchA>
00:02:C9:83:84:48 Up g1switchB
MLAG Port-Channel Creation
1. Create a mlag-port-channel: Host Configuration
– ‘mlag-port-channel’ indexes must be identical on both
MLAG switches.
– LACP or static LAG can be used.
2. Qumulo Node physical ports to the mlag-port-channel.
switchA [MLAG‐G1: master] (config) # interface mlag‐port‐channel 16
switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # exit
switchA [MLAG‐G1: master] (config) # interface ethernet 1/16
switchA [MLAG‐G1: master] (config interface ethernet 1/16) # mlag‐channel‐group 16 mode active
switchA [MLAG‐G1: master] (config interface ethernet 1/16) # exit
switchB [MLAG‐G1: standby] (config) # interface mlag‐port‐channel 16
switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # exit
switchB [MLAG‐G1: standby] (config) # interface ethernet 1/16
switchB [MLAG‐G1: standby] (config interface ethernet 1/16) # mlag‐channel‐group 16 mode active
switchB [MLAG‐G1: master] (config interface ethernet 1/16) # exit
MLAG Port-Channel Configuration
1. Disable STP for the mlag-port-channel:
2. Enable mlag-port-channel:
– Default admin state of mlag-port-channel is disabled.
In order to allow administrator to configure both
switches first, and then enable MLAG.
switchA [MLAG‐G1: master] (config) # interface mlag‐port‐channel 16
switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # spanning‐tree port type edge
switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # spanning‐tree bpdufilter enable
switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # no shutdown
switchB [MLAG‐G1: standby] (config) # interface mlag‐port‐channel 16
switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # spanning‐tree port type edge
switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # spanning‐tree bpdufilter enable
switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # no shutdown
Verify mlag-port-channel Configuration
switchA [MLAG‐G1: master] (config) # show interfaces mlag‐port‐channel summary

MLAG Port‐Channel Flags: D‐Down, U‐Up
P‐Partial UP, S ‐ suspended by MLAG
Port Flags: D ‐ Down, P ‐ Up in port‐channel (members)
S ‐ Suspend in port‐channel (members), I ‐ Individual
Group
Port‐Channel Type Local Ports Peer Ports
(D/U/P/S) (D/P/S/I) (D/P/S/I)
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
1 Mpo16(U) LACP Eth1/16(P) Eth1/16(P)
– Physical ports flags:

– Down - port is down IMPORTANT: Please take note, links will be Down until
– Up – ports is up the Qumulo node adapter ports are set to support LACP
– ‘mlag-port-channel’ flags:
– Partial Up – local or remote are down
– Up – both local and remote are up
– Down – ‘admin’ state is disabled
Q&A
– Question ?
– Break 10 m
50
Virtual local area network (VLAN)

Virtual local area network (VLAN)
– A virtual local area network (VLAN) is a logical segment of the Ethernet network that defines a broadcast domain.
– A VLAN is identified by a VLAN ID.
– Each VLAN should be assigned with a unique IP subnet.
– Hosts within the same VLAN can communicate with each other in layer 2.
– Hosts in different VLANs can communicate with each other in layer 3.
VLAN1 VLAN1
VLAN1
VLAN2
VLAN3
VLAN2 VLAN3 VLAN3 VLAN2

Trunk ports
– Trunk ports carry traffic for multiple VLANs across a single link.
– Hosts in the same VLAN, that are connected to different switches, can communicate with each other over
the trunk link.
– When a frame is sent on the trunk port, the sending switch adds a tag that contains the VLAN ID.
The receiving switch reads the VLAN ID and removes the tag.
VLAN1
trunk port VLAN1
VLAN1
VLAN2
VLAN3
VLAN2 VLAN3 VLAN3 VLAN2

IEEE 802.1Q trunking protocol
– The IEEE 802.1Q trunking protocol defines the tag added to Ethernet frames carried over a trunk port.
– The 802.1Q tag is 4 bytes in size, including the 12-bit VLAN ID.
Original Ethernet frame
6 bytes 6 bytes 2 bytes 46-1500 bytes 4 bytes
Destination Source Type/ Data FCS
MAC MAC Length
802.1Q tagged Ethernet frame

6 bytes 6 bytes 4 bytes 2 bytes 46-1500 bytes 4 bytes
Destination Source 802.1Q Tag Type/ Length Data New FCS

MAC MAC
2 bytes 3 bits 1 bit 12 bits

Type Priority CFI VLAN ID

Switch port types
A switch port can be configured in access, trunk, or hybrid mode.

– Access
– The port accepts and sends only untagged frames
– Frames are assigned to the configured port VLAN ID (PVID)
– Usually, an access port is connected to a host
– Trunk
– The port accepts and sends only tagged frames
– Untagged frames are dropped
– Usually, a trunk port is connected to another switch
– Hybrid
– The port accepts and sends both tagged and untagged frames
– Untagged frames are assigned to the configured port VLAN ID (PVID)
– A hybrid port is connected to either a switch or a host

Configure new VLAN’s
– switchA Configuration.
switchA [MLAG-G1: master] (config) # vlan 507
switchA [MLAG-G1: master] (config vlan 507) # exit
switchA [MLAG-G1: master] (config) # vlan 400
switchA [MLAG-G1: master] (config vlan 400) # exit
switchA [MLAG-G1: master] (config) #
* VLANs 1-4094 are supported.
– switchB Configuration
switchB [MLAG-G1: standby] (config) # vlan 507

switchB [MLAG-G1: standby] (config vlan 507) # exit
switchB [MLAG-G1: standby] (config) # vlan 400
switchB [MLAG-G1: standby] (config vlan 400) # exit
switchB [MLAG-G1: standby] (config) #

Configure VLAN Interface and IPV6
switchA [MLAG-G1: master] (config) # interface vlan 507 ip address 192.168.17.3 /24
switchA [MLAG-G1: master] (config) # ipv6 routing
switchA [MLAG-G1: master] (config) # interface vlan 1 ipv6 enable
switchA [MLAG-G1: master] (config) # interface vlan 507 ipv6 enable
switchB [MLAG-G1: standby] (config) # interface vlan 507 ip address 192.168.18.3 /24
switchB [MLAG-G1: standby] (config) # ipv6 routing
switchB [MLAG-G1: standby] (config) # interface vlan 1 ipv6 enable
switchB [MLAG-G1: standby] (config) # interface vlan 507 ipv6 enable

Verify new VLANs
– Verify new VLANs configuration.
switchA [MLAG-G1: master] (config) # show vlan

----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/1, Eth1/2, Eth1/3, Eth1/4, Eth1/5,
Eth1/6, Eth1/7, Eth1/8, Eth1/9, Eth1/10,
Eth1/17, Eth1/18, Eth1/21, Eth1/22, Mpo16
400
507
4094
– VLAN 1 is the default VLAN, and all ports are assigned to it.

Assign VLAN’s to host Interfaces
switchA [MLAG-G1: master] (config) # interface ethernet 1/16 description Qumulo-Node1
switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport mode hybrid
switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport access vlan 1
switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport hybrid allowed-vlan 507
switchB [MLAG-G1: standby] (config) # interface ethernet 1/16 description Qumulo-Node1

switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport mode hybrid
switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport access vlan 1
switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport hybrid allowed-vlan 507

Verify VLANs Assignment
– Verify new VLANs configuration.
switchA [MLAG-G1: master] (config) # show vlan

----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/1, Eth1/2, Eth1/3, Eth1/4, Eth1/5,
Eth1/17, Eth1/18, Eth1/21, Eth1/22, Mpo16
400
507 Mpo16
4094
– VLAN 1 is the default VLAN, and all ports are assigned to it.

Configuring Access Mode and Assigning Port VLAN ID (PVID)
switch > enable

switch # configure terminal
switch (config) # vlan 6
switch (config vlan 6) #
switch (config vlan 6) # exit
switch (config) #
switch (config) # interface ethernet 1/22
switch (config interface ethernet 1/22) #
switch (config interface ethernet 1/22) # switchport mode access
switch (config interface ethernet 1/22) # switchport access vlan 6
switch (config 1/22) # exit
switch (config) #
61
Configuring Hybrid Mode and Assigning Port VLAN ID (PVID)
switch > enableswitch

# configure terminal
switch (config) #
switch (config) # interface ethernet 1/22
switch (config interface ethernet 1/22) # switchport mode hybrid
switch (config interface ethernet 1/22) # switchport access vlan 6
switch (config interface ethernet 1/22) # exit
switch (config) #
62
Configuring Trunk Mode VLAN Membership
switch > enable

switch # configure terminal
switch (config) #
switch [standalone: master] (config) # interface ethernet 1/35
switch [standalone: master] (config interface ethernet 1/35) #
switch [standalone: master] (config interface ethernet 1/35) # switchport mode trunk
switch [standalone: master] (config interface ethernet 1/35) #
63
Q&A
– Question ?
– Break 10 m
64
Spanning Tree Protocol (STP)

Multiple spanning tree (MST)
- MST maps multiple VLANs to an instance, reducing the number of spanning-tree instances.
- MST and PVST+ are compatible
- Backwards compatible with RSTP and STP
- It is the IEEE standard protocol (802.1s)
66
Rapid spanning tree (RSTP)
- The Rapid Spanning Tree Protocol recovers (converges to a new spanning tree) more quickly than STP
- It is backwards-compatible with MST and STP.
- It is the IEEE standard protocol (802.1w)
67
Rapid per-VLAN spanning tree (RPVST)
- Cisco proprietary version of Rapid Spanning Tree Protocol (802.1w)

- It creates a spanning tree for each VLAN, just like PVST.
- Rapid-PVST is backward compatible with standard Per-VLAN Spanning Tree (PVST/802.1d)
68
Ethernet layer 2 loops
– Layer 2 redundant links are required to provide a backup

path in case of link or switch failure.
– Redundant links result in layer 2 loops—There are
multiple paths between a pair of nodes.
– Layer 2 loops cause “broadcast storms.”
– When an Ethernet broadcast frame is sent in the network, it
endlessly circulates in a loop consuming all available
bandwidth.
– Broadcast storms deny bandwidth for normal network traffic.

Spanning Tree Protocol (STP)
– Spanning Tree Protocol (STP) is an IEEE 802.1D

standard.
– STP ensures a loop-free topology for Ethernet networks.
– STP allows a network design to include redundant links
and to provide automatic backup paths, if an active link
fails.
X X
– STP identifies redundant links and puts redundant ports
in blocking state.
– When a topology change occurs, STP reacts and moves
blocked ports to the forwarding state.
– Convergence time is 30 to 50 seconds.

Rapid Per-VLAN Spanning Tree (RPVST)
Configuration

Configure Spanning Tree RPVST
switchA [MLAG-G1: master] (config) # spanning-tree mode rpvst
switchB [MLAG-G1: standby] (config) # spanning-tree mode rpvst

Configure Flowcontrol and Jumbo Frames
switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 flowcontrol receive on force

switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 flowcontrol send on force
switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 mtu 9216 force
switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 flowcontrol receive on force

switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 flowcontrol send on force
switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 mtu 9216 force

Maximum Transmission Unit (MTU) Size
– The largest possible frame size of a communications Protocol Data Unit (PDU) on an OSI Model Layer 2 data network.
– Default frame size is 1518 bytes
– Example of commands to check MTU size;
C:\Users\ScottHogg> ping 192.168.10.1 -l 1500 –f
RedHat# ping -s 1500 -M do 192.168.10.1
Router1# ping 192.168.10.1 size 1500 df-bit
Switch7K# ping 192.168.10.1 packet-size 9216 c 10
RP/0/RP0/CPU0:Router1#ping 192.168.10.1 size 1500 donnotfrag
Junos-root@J4350-1# run ping 192.168.10.1 size 1500 do-not-fragment rapid
74
Q&A
– Question ?
– Break 10 m
75
Link Layer Discovery Protocol (LLDP)

Link Layer Discovery Protocol (LLDP)
– Link Layer Discovery Protocol (LLDP) is a vendor-

neutral protocol defined in IEEE 802.1AB.
– LLDP is used by network devices for advertising their
identity, capabilities, and neighbors on an IEEE 802 switchA switchB
LAN.
– LLDP can be used to discover and verify Ethernet
network topology.
– LLDP is by default globally disabled.
Eth1/1
– LLDP frames are sent every 30 seconds by all LLDP
enabled interfaces.

Configure LLDP
switchA [MLAG-G1: master] (config) # lldp
switchB [MLAG-G1: standby] (config) # lldp

Show local LLDP information
– Show local LLDP information.
switchA [MLAG-G1: master] (config) # show lldp local
LLDP: enabled switchA switchB
Local global configuration
Chassis sub type: Mac Address
Chassis id: b8:59:9f:70:d6:00
System Name: g1switchA
System Description: SN2010M,Onyx,SWv3.8.2204
Supported capabilities: B,R
Supported capabilities enabled: B
Eth1/19
– Show interface LLDP information.
switchA [MLAG-G1: master] (config) # show lldp interfaces ethernet 1/19
TLV flags
PD: port-description
SN: sys-name
SD: sys-description
SC: sys-capabilities
MA: management-address
ETS-C: ETS-Configuration
Interface Receive Transmit TLVs

----------------------------------------------------------------------------------
-
Eth1/19 Enabled Enabled PD, SN, SD, SC, MA, PFC, AP, ETS-C, ETS-R

View Cable/Transceiver
– Show local Transceivers information.
switchA [MLAG-G1: master] (config) # show interfaces ethernet 1/19 transceiver

Port 1/19 state
identifier : QSFP28
cable/module type : Passive copper cable
ethernet speed and type: 100GBASE-CR4
vendor : Mellanox
cable length : 1m
part number : 845404-B21
revision : A1
serial number : 6C2749003C
switchA switchB
Eth1/19
Onyx system features
Feature Description
Software management – Dual software image

– Software and firmware updates
File management – FTP, TFTP, SCP
Logging – Event history log

– Syslog support
Chassis management – Monitoring environmental controls
– Power management
– Auto-temperature control
– High availability
Network management interfaces – SNMP v1,v2c,v3
– Puppet agent
Security – SSH, Telnet
– RADIUS, TACACS+
Date and time – NTP
Cables and transceivers – Transceiver info

Configure Clock and NTP
switchA [MLAG-G1: master] (config) # clock timezone America North United_States Central
switchA [MLAG-G1: master] (config) # clock set 09:00:00 2020/05/12
switchA [MLAG-G1: master] (config) # ntp server 10.187.2.2
switchA [MLAG-G1: master] (config) # ntp enable
switchB [MLAG-G1: master] (config) # clock timezone America North United_States Central
switchB [MLAG-G1: master] (config) # clock set 09:00:00 2020/05/12
switchB [MLAG-G1: master] (config) # ntp server 10.187.2.2
switchB [MLAG-G1: master] (config) # ntp enable

Show NTP and Clock
– Display commands NTP Status.
NTP is administratively : enabled
NTP Authentication administratively: disabled
NTP server role : enabled
Clock is synchronized:
Reference: 10.187.2.2
Offset : -0.620 ms
Active servers and peers:

10.187.2.2:
Conf Type : serv
Status : sys.peer(*)
Stratum : 1
Offset(msec) : -0.620
Ref clock : .GPS.
Poll Interval (sec): 64
Last Response (sec): 53
Auth state : none
– Display Clock settings.

switchA [MLAG-G1: master] (config) # show clock
Time: 14:54:50
Date: 2020/05/12
Time zone: America North United_States Central (US/Central)
UTC offset: -0500 (UTC minus 5 hours)

Reset factory defaults
– Reset the switch to factory defaults.
g1switchA [MLAG-G1: master] (config) # reset factory ?
<cr>
halt Halt system after reset, instead of rebooting
keep-all-config Preserve all configuration files (supercedes keep-basic)
keep-basic Preserve licenses in the active configuration
keep-virt-vols Preserve all virtual disk volumes
only-config Reset only configuration
mseries3 [standalone: master] (config) # reset factory keep-all-config
Warning - confirming will cause system reboot.
Type 'YES' to confirm reset:
Configuration Licenses System profile Management

interfaces
keep-all-config Unchanged Not deleted Unchanged Unchanged
keep-basic Reset Not deleted Reset Reset
only-config Reset Deleted Reset Unchanged

Reset Factory
switchA [MLAG-G1: master] (config) # reset factory
switchB [MLAG-G1: standby] (config) # reset factory

Q&A
– Question ?
– Break 10 m
86
Multi-active gateway protocol (MAGP)

Enable the switch as host’s gateway (Layer 3 MLAG)
There are two protocols of the Layer 3 MLAG;

- VRRP: Virtual Router Redundancy Protocol.
It’s working as Active/Standby.
- MAGP: Multi-active gateway protocol.
It’s working as Active/Active.
Note: MAGP is the recommend protocol for implementing Mellanox’s L3 MLAG.
88
Multi-active gateway protocol (MAGP)
- To solve the default gateway problem when a host is

connected to a set of switch routers via MLAG.
- Each switch routers is an active default gateway router
to the host.
- Directly forwarding IP traffic to the L3 cloud regardless
which SR traffic comes through.
89
Configuring MAGP Example
- switch (config)# ip routing

- switch (config)# vlan 20
- switch (config)# interface vlan 20
- switch (config interface vlan 20)# ip address 11.11.11.11 /8
- switch (config interface vlan 20)# no shutdown
- switch (config)# protocol magp
- switch (config interface vlan 20)# magp 100
- switch (config interface vlan 20 magp 100)# ip virtual-router address 11.11.11.254
- switch (config interface vlan 20 magp 100)# ip virtual-router mac-address AA:BB:CC:DD:EE:FF
90
Verify the MAGP configuration
91
IP Routing
– IP Interfaces (L3)
– MLNX-OS supports the following 3 types of IP interfaces:
•VLAN interface
•Loopback interface
•Router port interface
Note: Router port interfaces are not supported on SX10xx-xxxR and SX60xx-xxxR systems
Note: Routing for this project is using VLAN interface with ip route 0.0.0.0 0.0.0.0 10.0.102.2
92
VLAN interface Attributes
– VLAN interface is a logical IPv4 interface created per subnet over a specific 802.1Q VLAN ID.
– Each interface VLAN has the following attributes:
•Admin state
•Operational state
•MAC address
•IP address and mask
•MTU
•Description
•Set of counters
93
Configure a Router Port Interface
94
Q&A
– Question ?
– Break 10 m
95
Basic Troubleshooting

What happens if the IPL link goes down?
1. Split-brain
2. Only the master switch will pass traffic.
MAGP 2:
Interface vlan: 400
Admin state : Enabled VLAN 400 inter-peer link (IPL) Management Plane (Active/Standby)
State : Master
Virtual IP : 10.0.102.1 mgmt0
Virtual MAC : 00:00:5E:00:01:02 10.0.99.61 ‐ 63 /24
SN2410M SN2410M
mgmt0
MAGP 1:
Interface vlan: 300 Cisco MGMG switch
Admin state : Enabled
State : Master
Virtual IP : 10.0.1.254
Virtual MAC : 00:00:5E:00:01:01
Link Aggregation
Data VLAN 300 Qumulo
ILO Servers 1-6
iLO
97
What happens if no IP communication between the MGMT ports
1. CLI prompt is displayed: [:unknown]#
2. It Split-brain when IPL down
MAGP 2:
Interface vlan: 400
Admin state : Enabled VLAN 400 Management Plane (Active/Standby)
State : Master
Virtual IP : 10.0.102.1 mgmt0
Virtual MAC : 00:00:5E:00:01:02 10.0.99.61 ‐ 63 /24
SN2410M SN2410M
mgmt0
MAGP 1:
Interface vlan: 300 Cisco MGMG switch
Admin state : Enabled
State : Master
Virtual IP : 10.0.1.254
Virtual MAC : 00:00:5E:00:01:01
Link Aggregation
ILO Servers 1-6
iLO
98
Verify interface MLAG port channel
#show interface mlag-port-channel sum
VLAN 400
mgmt0
10.0.99.61 ‐ 63 /24
SN2410M SN2410M
mgmt0
Cisco MGMG switch
mlag-port-channel
Link Aggregation
ILO Servers 1-6
iLO
99
Q&A
– Question ?
100
Thank you
[email protected]

HPE Mellanox Switch Training, Part 1

Uploaded by

Copyright:

Available Formats

HPE Mellanox Switch Training, Part 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HPE Mellanox Switch Training, Part 1

Uploaded by

Copyright:

Available Formats

HPE Mellanox

Confidential – For Training Purposes Only 2

Cisco core L3 #1, #2

Data VLAN 300

SN2010M SN2100M SN2410bM SN2410M SN2700M/SN2745M

Flexible pricing & investment

25G SFP+ DAC

25Gb SFP+ Direct Attached Copper (DAC .5m to 3m)

HPE 25Gb SFP28 to SFP28 1m Direct Attach Copper Cable (844474-B21)

HPE 25Gb SFP28 to SFP28 3m Direct Attach Copper Cable (844477-B21)

HPE M‐Series SN2100M 100G 100G

HPE M‐Series SN2010M 100G 100G

100G QSFP28 DAC

100Gb QSFP28 Direct Attached Copper (DAC 1m)

Q Q Comp Comp Edge MLAG 2

Comp ports go to the 640SFP28 NIC MGMT

in the Compute Nodes.

Q Q Comp Comp Edge

Q Q Comp Comp Edge MLAG 2

Q1 Q3 Comp Comp Tor1

25 Gb DAC cables 1Gb, 10Gb, 25 Gb 10 Gb DAC cables

Confidential – For Training Purposes Only 10

1. Command Line Interface (CLI)

2. Web Interface (web GUI)

1. Access the MLNX-OS CLI via Serial Console.

3. Log in with default credentials.

login as: admin

Mellanox MLNX-OS Switch Management

Using keyboard-interactive authentication.

Confidential – For Training Purposes Only 13

2. Access the MLNX-OS WebUI via HTTP/HTTPS

3. Access the MLNX-OS CLI via SSH

switchA [standalone: master] # disable

Confidential – For Training Purposes Only 16

g1switchA [standalone: master] # configure terminal

– Use ‘exit’ command to move from config mode to enable mode.

g1switchA [standalone: master] (config) # exit

Confidential – For Training Purposes Only 17

– Use ‘?’ from any mode to view available commands.

Confidential – For Training Purposes Only 18

– Use [Tab] to auto-complete commands.

– co [Tab] - autocompletes to ‘configure.’

Confidential – For Training Purposes Only 19

– Save running-config into active-config.

switchA [standalone: master] (config) # configuration write

Active configuration: initial

switchA [standalone: master] (config) # write memory

Confidential – For Training Purposes Only 20

Confidential – For Training Purposes Only 21

Last boot partition: 1

– By default, the image from Partition 1 is loaded at reboot.

Confidential – For Training Purposes Only 22

1. Use the IP address of each Switch of the

2. Type in user name and password default:

Confidential – For Training Purposes Only 23

2. Choose Onyx Upgrade

3. Select Install from local file:

4. Select Choose file:

5. Select Install Image