UCS Networking 201—Deep Dive

BRKCOM-2003
Agenda
• Overview / System Architecture
•Physical Architecture
•Logical Architecture

• Switching Modes of the Fabric Interconnect

• Fabric Failover
• Ethernet Switching Modes Recommendations
• SAN / LAN Northbound Connections
•FC/FCoE Storage direct attach, NAS appliance port
•Port Channel and vPC uplinks, traffic flows and failures

• Adapter Offerings
• UCS Generation-2 Hardware

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
 Overview

© 2011 Cisco and/or its

Presentation_ID © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 Unified Computing System (UCS)

Single Point of Management

Unified Fabric

Stateless Servers with Virtualized Adapters

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
UCS Building Blocks
UCS Manager
Embedded– manages entire system

UCS Fabric Interconnect

20 Port 10Gb FCoE
40 Port 10Gb FCoE

UCS Fabric Extender

Remote line card

UCS Blade Server Chassis

Flexible bay configurations

UCS Blade or Rack Server

Industry-standard architecture

UCS Virtual Adapters

Choice of multiple adapters

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
 Cisco UCS Networking: Physical Architecture
SAN A ETH 1 ETH 2 SAN B

MGMT MGMT

Uplink Ports
OOB Mgmt 6100 6100
Fabric Switch Fabric A Cluster Fabric B
Server Ports

Chassis 1 Chassis 20
Fabric Extenders I I I I
O O O O
M M M M

Virtualized Adapters A CNA B A CNA CNA B

Compute Blades B200 B250
Half / Full width
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
 Cisco UCS Networking: Physical Architecture
SAN A ETH 1 ETH 2 SAN B

MGMT MGMT

Uplink Ports
OOB Mgmt 6100 6100
Fabric Switch Fabric A Cluster Fabric B
Server Ports

FEX FEX
Chassis 1 VIC
Fabric Extenders I I
O O Rack Mount
M M Management
Plane
Virtualized Adapters A CNA B
Compute Blades B200
Half / Full width
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Network Interface Virtualization (NIV)
Fabric
vNIC (LIF) Interconnect

vFC vEth
Host presented PCI device 1 1
managed by UCSM.

VIF
IOM
Policy application point where
a vNIC connects to UCS fabric
Adapter
VNTag
An identifier that is added to
vHBA vNIC Cable
the packet which contains 1 1
source and destination ID Virtual Cable
(VNTag)
which is used for switching Service Profile
(Server)
within the UCS fabric. Blade

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Abstracting the Logical Architecture
What you see What you get
6100-A 6100-A 6100-A

Switch vFC vEth vFC vEth

1 1 1 1

Eth 1/1
 Dynamic, Rapid
Provisioning
IOM A IOM A
 State abstraction
Cable
 Location
10GE 10GE
A Independence
A
Adapter
 Blade or Rack

Adapter
vHBA vNIC vHBA vNIC Physical Cable

1 1 1 1
Virtual Cable
(VN-Tag)
Service Profile
(Server)
Blade (Server)
Blade

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
 Hardware Components

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
UCS 6100 Hardware Architecture

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
 2104-IOM Architecture
Components
Woodside ASIC 1-4
Aggregates traffic to/from 32 host- Fabric Ports
facing 10G Ethernet ports from/to 8 FLASH to
network-facing 10G Ethernet ports Interconnect
DRAM
CPU (also referred to as CMC)
EEPROM
Controls Redwood and perform other
chassis management functionality Control
Chassis
L2 Switch Management IO Redwood ASIC
Controller
Aggregates traffic from BMCs on the
server blades

Switch
Woodside Interfaces
HIF (Backplane ports)
NIF (FabricPorts)
BIF Chassis 8 Backplane Ports to
CIF Signals Blades

No local switching – All traffic from

HIFs goes upstream for Switching
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
UCS IOM — Slot to Uplink Pinning

Number of Active Fabric Links Blades pinned to fabric link

1-Link All the HIF ports pinned to the active link

2-Link 1,3,5,7 to link-1
2,4,6,8 to link-2
4-Link 1,5 to link-1
2,6 to link-2
3,7 to link-3
4,8 to link-4

 HIFs are statically pinned by the system to individual fabric ports.

 Only 1,2,4 links are supported for pinning. 3 is not a valid pinning configuration.
 On a link failure, only blades pinned to the NIF are brought down

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Individual Links
Static Pinning (IOM-FI)
Fabric Interconnect
•Static Pinning done by the

Fabric Ports
system dependent on number
of fabric ports

• 1,2 4 (2^x) are valid links for IOM

initial pinning

Server Ports

Blade 4
Blade 1

Blade 2

Blade 3

Blade 5

Blade 6

Blade 7

Blade 8
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Individual Links
Fabric Port Failure
Fabric Interconnect
•Pinned HIFs are brought

Fabric Ports
down

•Other blades unaffected

IOM

Server Ports

Blade 4
Blade 1

Blade 2

Blade 3

Blade 5

Blade 6

Blade 7

Blade 8
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
Individual Links
Re-ack of chassis
Fabric Interconnect

•Blades re-pinned to valid

Unused Link
Fabric Ports
number of links – 1,2 or 4

•Pinned blade connectivity

affected IOM

•HIF’s brought down/up for re-

Server Ports
pinning

•May result in unused links

Blade 1

Blade 2

Blade 3

Blade 4

Blade 5

Blade 6

Blade 8
Blade7
•Addition of links requires re-
ack of chassis.
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
 Ethernet Switching Modes

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
 Switching Modes: End Host
LAN • Server vNIC pinned to an Uplink port
Spanning
Tree • No Spanning Tree Protocol
• Reduces CPU load on upstream switches
• Reduces Control Plane load on 6100
• Simplified upstream connectivity
• UCS connects to the LAN like a Server, not
6100 A MAC like a Switch
Learning
• Maintains MAC table for Servers only
vEth 3 vEth 1 • Eases MAC Table sizing in the Access Layer
VLAN 10 MAC
Fabric A Learning • Allows Multiple Active Uplinks per VLAN
• Doubles effective bandwidth vs STP
L2 • Prevents Loops by preventing Uplink-to-
Switching
Uplink switching
• Completely transparent to upstream LAN
VNIC 0 VNIC 0
• Traffic on same VLAN switched locally

Server 2 Server 1

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
End Host Mode Unicast Forwarding
• Server to server traffic on the LAN
same VLAN is locally switched
Server 2

• Uplink port to Uplink port traffic Uplink

not switched Ports
Deja-Vu
RPF

• Each server link is pinned to an

uplink port / port-channel 6100

• Network to server unicast traffic

is forwarded to server only if it vEth 1
VLAN 10
vEth 3
arrives on pinned uplink port.
This is termed as the Reverse
Path Forwarding—(RPF) check
• Packet with source MAC
belonging to a server received VNIC 0 VNIC 0
on an uplink port is dropped
(Deja-Vu Check) Server 2 Server 1

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
End Host Mode Multicast Forwarding

 Broadcast traffic is pinned on LAN

B B
exactly one uplink port (or port-
channel) i.e., it is dropped Uplink
Broadcast
Listener
when received on other uplinks Ports All VLANs

 All multicast groups are pinned

to same uplink port (port- 6100
channel)
 Server to server multicast
vEth 1 vEth 3
traffic is locally switched
 RPF and deja-vu check also
applies for multicast traffic B

VNIC 0 VNIC 0

Server 2 Server 1

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
 Switching Modes: Switch
Root
LAN • Fabric Interconnect behaves like a
normal Layer 2 switch
• Server vNIC traffic follows VLAN
forwarding
• Spanning tree protocol is run on the
6100 A MAC uplink ports per VLAN—Rapid PVST+
Learning
• Configuration of STP parameters (bridge
vEth 3 vEth 1 priority, Hello Timers etc) not supported
VLAN 10
Fabric A
• VTP is not supported currently
L2 • MAC learning/aging happens on both the
Switching server and uplink ports like in a typical
Layer 2 switch
VNIC 0 VNIC 0
• Upstream links are blocked per VLAN via
Spanning Tree logic
Server 2 Server 1

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
 Fabric Failover

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
 Fabric Failover
• Fabric provides NIC failover L1 L1
6100-A 6100-B
capabilities chosen when L2 L2

defining a service profile vEth vEth

1 1
• Traditionally done using NIC
bonding driver in the OS
Physical Cable IOM IOM
• Provides failover for both
unicast and multicast traffic Virtual Cable
10GE 10GE
• Works for any OS.
PHY Adapter
Cisco VIC – M81KR
Menlo – M71KR
vNIC VIRT
1 Adapter

OS / Hypervisor / VM

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
 Fabric Failover with Bare Metal OS
Slam Dunk: Cisco UCS simplifies the redundancy
L1 L1
6100-A L2 L2 6100-B

 No OS NIC Teaming MAC A MAC A

configuration required
 Simple single NIC design
 Fabric failures hidden from
OS Server w/
vNIC MAC A

 NIC stays UP Cisco VIC or Menlo

 6100 sends gratuitous ARP

 Everything to gain WINDOWS / LINUX

 Nothing to lose

Implicit MAC Learned MAC

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Fabric Failover with HYPER-V
Slam Dunk: Cisco UCS provides the missing redundancy
 Hyper-V soft switch
L1 L1
6100-A L2 L2 6100-B
MAC C MAC C only uses (1) NIC
MAC A MAC B MAC A MAC B
 Fabric Failover
provides the missing
redundancy
Server w/
vNIC 1 MAC C  Everything to gain
Cisco VIC or Menlo
 Nothing to lose
HYPER V soft switch

VM VM

MAC A MAC B

Implicit MAC Learned MAC

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
Fabric Failover with Hypervisor Pass-Through
L1 L1
 Default setting 6100-A L2 L2 6100-B
MAC B MAC B
 Cannot be disabled
MAC A MAC A
 Single adapter per VM with
Redundancy
 Load Sharing
Dynamic Dynamic
Alternating fabric
vNIC 1 vNIC 2
Round Robin
Server w/
Cisco VIC
VM-FEX

VM 1 VM 2

MAC A MAC B

Implicit MAC Learned MAC

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
 Ethernet Switching Modes
Recommendations

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
Scalability
• Spanning Tree protocol is not run in EHM hence control plane
is unoccupied
• EHM is least disruptive to upstream network – BPDU
Filter/Guard, Portfast enabled upstream
• MAC learning does not happen in EHM on uplink ports.
Current MAC address limitation on the 6100 ~14.5K.

Recommendation: End Host Mode

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
Pinning for deterministic traffic flows
• Dynamic pinning
Server ports pinned to an uplink
port/port-channel automatically DEFINED:
PinGroup
• Static pinning Oracle

1 2 3 4
Specific pingroups created and 6100 A
associated with adapters Pinning
vEth 3 vEth 1
• Static pinning allows Switching
Fabric A
traffic management if
required for certain
applications / servers
APPLIED:
PinGroup
VNIC 0 VNIC 0 Oracle

Recommendation: Server X Oracle

End Host Mode
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
 Fabric Failover
L1 L1
6100-A L2 L2 6100-B

vEth vEth
• Fabric Failover is only 1 1
applicable in EHM.
• NIC teaming software Physical Cable IOM IOM
required to provide
Virtual
failover in Switch mode. Cable 10G 10G
E E

PHY Adapter
Cisco VIC – M81KR
Menlo – M71KR
vNIC VIRT
1 Adapter

OS / Hypervisor / VM
Recommendation: End Host Mode
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
Active/Active use of Uplinks
End Host Mode Switch Mode
Primary Root Secondary Root Primary Root Secondary Root

LAN LAN

Active/Active Blocking
Border Ports Border Ports

FI-A FI-B FI-A FI-B

Server Ports Server Ports

Recommendation: End Host Mode

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
Disjoint L2 Upstream
Backup Production

• EHM built on the premise

that the L2 upstream is NOT
disjoint.
External LAN
• Incoming
broadcast/multicast received
only on 1 uplink for ALL
VLANs Border Ports
Designated Bcast
Receiver

Fabric InterConnect

Recommendation: Switch Mode

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Application Specific Scenarios

• Certain application like MS-NLB (Unicast mode)

have the need for unknown unicast flooding which
is not done in EHM
• Certain network topologies provide better network
path out of the Fabric Interconnect due to STP
placement and HSRP L3 hop.
• Switch Mode is ―catch all‖ for different scenarios.

Recommendation: Switch Mode

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
 Upstream Connectivity (Storage)

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
334
SAN ―End Host‖ NPV Mode
N-Port Virtualization Forwarding
 Fabric Interconnect operates in N_Port Proxy
SAN A SAN B mode (not FC Switch mode)
Simplifies multi-vendor interoperation
FLOGI Simplifies management
FDISC
NPIV NPIV
F_Port F_Port  SAN switch sees Fabric Interconnect as an FC
End Host with many N_Ports and many FC IDs
VSAN 1 VSAN 1 assigned
N_Proxy N_Proxy
 Server facing ports function as F-proxy ports
6100-A 6100-B
 Server vHBA pinned to an FC uplink in the
vFC 1 vFC 2 vFC 1 vFC 2
same VSAN. Round Robin selection.
F_Proxy F_Proxy
 Provides multiple FC end nodes to one F_Port
off an FC Switch
N_Port N_Port
 Eliminates the FC domain on UCS Fabric
vHBA vHBA vHBA vHBA Interconnect
0 1 0 1
 One VSAN per F_port (multi-vendor)
Server 1 Server 2
VSAN 1 VSAN 1  F_Port Trunking and Channeling with MDS, 5K

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
 SAN ―End Host‖ NPV Mode
N-Port Virtualization Forwarding with MDS, Nexus 5000
SAN A SAN B  F_Port Channeling and Trunking from MDS or
Nexus 5000 to UCS

NPIV NPIV  FC Port Channel behaves as one logical uplink

F_ Port
Channel &
Trunk
F_Port  FC Port Channel can carry all VSANs (Trunk)
VSAN VSAN
1,2 1,2  UCS Fabric Interconnects remains in NPV end
N_Proxy host mode
6100-A 6100-B  Server vHBA pinned to an FC Port Channel
vFC 1 vFC 2 vFC 1 vFC 2
 Server vHBA has access to bandwidth on any
F_Proxy link member of the FC Port Channel

 Load balancing based on FC Exchange_ID

N_Port Per Flow

vHBA vHBA vHBA vHBA

0 1 0 1

Server 1 Server 2
VSAN 1 VSAN 2

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
 SAN FC Switch Mode
Direct Attach FC & FCoE Storage to UCS
 UCS Fabric Interconnect behaves like an FC FC FCoE SAN
fabric switch

 Storage ports can be FC or FCoE

MDS MDS
 Light subset of FC Switching features N_Port
VSAN 1 VSAN 2
Select Storage ports
TE_Port
Set VSAN on Storage ports
F_Port
Default zoning per VSAN

 No zoning configuration inputs in UCSM 6100-A FC Switch 6100-B FC Switch

vFC 1 vFC 2 vFC 1 vFC 2
 Connection to an external FC switch is reqd:
Zoning configured and pushed to UCS from MDS F_Port

 Fabric Interconnect uses a FC Domain ID

N_Port
vHBA vHBA vHBA vHBA
0 1 0 1

Server 1 Server 2
VSAN 1 VSAN 2

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Direct Attach IP Storage to UCS
NAS direct attached to ―Appliance port‖ Active/Standby
NAS LAN  IP Storage attached to ―Appliance Port‖
NFS, iSCSI
Volume Volume

 NAS appliance with single Controller head

A B

C1  Each NAS controller port owns the I/O for a

given Volume
Other port provides failover
Appliance Uplink
Port Port  Controller interfaces active/standby for a given
A U A U
volume when attached to separate 6100s
6100-A 6100-B
vEth 1 vEth 2 vEth 1 vEth 2  Controller interfaces Active/Active when each
handling their own volumes

 Sub-optimal forwarding possible if not careful

Insure vNICs are accessing Volumes local to its fabric
vNIC vNIC vNIC vNIC
0 1 0 1

Server 1 Server 2
Accessing Accessing
Volume A Volume A
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
 Upstream Connectivity (Ethernet)

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
 End Host Mode – Individual Uplinks
Dynamic Re-pinning of failed uplinks

6100 A
Sub-second re-pinning Pinning
vEth 3 vEth 1
VLAN 10 Switching
Fabric A

L2
VNIC stays up
All uplinks forwarding for all VLANs Switching
GARP aided upstream convergence vSwitch / N1K
No STP VNIC 0
Sub-second re-pinning MAC A ESX HOST 1
No server NIC disruption VM 1 VM 2
MAC B MAC C
VNIC 0 Server 2

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
 End Host Mode – Port Channel Uplinks
Recommended: Port Channel Uplinks No disruption

No GARPs
needed

6100 A Sub-second convergence Pinning

vEth 3 vEth 1
VLAN 10 Switching
Fabric A

L2
Switching NIC stays up
More Bandwidth per Uplink
vSwitch / N1K
Per flow uplink diversity VNIC 0
No Server NIC disruption ESX HOST 1
MAC A
Fewer GARPs needed
Faster bi-directional convergence VM 1 VM 2
Fewer moving parts MAC B MAC C
VNIC 0 Server 2
RECOMMENDED
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
 End Host Mode – vPC Uplinks
vPC uplinks hide uplink & switch failures from Server VNICs
vPC Domain

No disruption

No GARPs
Needed!

6100 A
Pinning
vEth 3 vEth 1
VLAN 10 Switching
Fabric A

More Bandwidth per Uplink L2 NIC stays up

No Server NIC disruption Switching
Switch and Link resiliency vSwitch / N1K
Per flow uplink diversity VNIC 0
ESX HOST 1
No GARPs
Faster Bi-directional convergence VM 1 VM 2
Fewer moving parts MAC B MAC C
VNIC 0 Server 2
vPC RECOMMENDED
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
Inter-Fabric Traffic Example (1)
Cisco UCS VM-FEX
Dynamic VNIC1
Primary fabric A
L2 Switching
Backup fabric B

Dynamic VNIC2
6100 A 6100 B Primary fabric B
EHM EHM
Backup fabric A

Dynamic Dynamic
VM1 on VLAN 10
VNIC 1 VNIC 2 VM2 on VLAN 10

PTS VM1 to VM2 traffic:

1) Leaves Fabric A
2) Gets L2 switched
VM VM 3) Enters Fabric B
1 2
ESX HOST

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
 Inter-Fabric Traffic Example (2)
VNIC 0 on Fabric A VM1 to VM4:
VNIC 1 on Fabric B 1) Leaves Fabric A
2) L2 switched
VM1 Pinned to VNIC0 L2 Switching
upstream
VM4 Pinned to VNIC1 3) Enters Fabric B
VM1 on VLAN 10
VM4 on VLAN 10 6100 A 6100 B
EHM EHM

VNIC 0 VNIC 1 VNIC 0 VNIC 1

ESX HOST 1 ESX HOST 2

vSwitch / N1K vSwitch / N1K

Mac Pinning Mac Pinning
VM VM VM VM
1 2 3 4

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
Singly Attached Uplinks

7K1 7K2

6100 A 6100 B
EHM EHM

1. Traffic destined for a vNIC on the Red Uplink enters 7K1

2. Same scenario vice-versa for Green
3. All Inter-Fabric traffic traverses Nexus 7000 peer link

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
Recommended Topology
vPC uplinks to L3 aggregation switch

7K1 vPC Domain 7K2

keepalive

vPC peer-link

6100 A 6100 B
EHM EHM

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
Recommended Topology w/o vPC
With 4 x 10G (or more) uplinks per 6100 – Port Channels

6100 A 6100 B
EHM EHM

All UCS uplinks forwarding

No STP influence on the topology
End Host Mode
Connect 6100’s with End Host Mode to Aggregation L3 switch

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
 Adapter Offerings

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
Gen1 ―Compatibility‖ Adapters
10GbE/FCoE
 NIC and HBA ASICs from:
Qlogic, Emulex, Intel
Dual 10GbE/FCoE ports
 Cisco ―Menlo‖ ASIC
IEEE DCB
VN-Tag
Fabric Failover

 Support for native drivers

and utilities
Customer certified stacks 10GbE FC

 21w
 4Gbps FC
 vNIC Fabric Failover
PCIe Bus

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
Gen2 ―Compatibility‖ Adapters

Emulex M72KR-E (CNA) QLogic M72KR-Q (CNA)

 Single Emulex ASIC design  Single QLogic ASIC design

 Low power – 13w  Lowest power – 4.5w
 8 Gbps FC  8 Gbps FC
 Emulex drivers, Eth & FC  QLogic drivers, Eth & FC

No vNIC Fabric Failover No vNIC Fabric Failover

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
Gen2 ―Cost‖ Adapters

Broadcom BCM57711 Intel M61KR-I

 iSCSI acceleration  SR-IOV compatible

 iSCSI offload (HBA)  iSCSI acceleration
 iSCSI boot (future)  PXE Boot
 TCP offload engine (TOE)  VMDq
 Low cost 10GE  IEEE DCB
 FCoE software (future)
No vNIC Fabric Failover No vNIC Fabric Failover

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 51
Cisco Virtual Interface Card (VIC) (Palo)
 Converged Network Adapter
 FCoE in hardware 10GbE/FCoE
 Single-OS and VM deployments
 Virtualize in hardware
 PCIe compliant
Up to 58 distinct PCIe devices
Eth
 Ethernet vNIC and FC vHBA FC FC Eth
 2nd Tier Fabric Extender
For virtualization environments User
Definable
 Bypass vSwitch to deliver VN-Link vNICs
in hardware
 Tight integration with 0 1 2 3 58
VMware vCenter
 vNIC as Hardware DVS-port
 QoS
 (8) COS based queues
 vNIC bandwidth guarantees 18w
PCIe x16
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 52
Cisco VIC: VM-FEX Logical View

Fabric Interconnect A Fabric Interconnect B

vfc vEth vEth vEth vEth vEth vEth vEth vEth vfc
1 1 2 3 4 5 6 7 8 2

IOM A IOM B

Cisco VIC

vhba VM vhba
VM 1 VM 2 VM 3 VM 4 VM 5 VM 6 VM 7
0 50 1

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 53
 UCS Generation-2 Hardware

© 2011 Cisco and/or its

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 54
Next Gen UCS Components and Capabilities
Additions to the UCS Fabric Portfolio

2x Fabric Capacity
40% Latency Reduction
6248UP Fabric Interconnect Unified Ports

2x Blade Chassis Bandwidth

160 Gb/s per Chassis
2208XP IO Module

4x Blade Server Bandwidth

Dual 40 Gb/s per Card
VIC 1280 256 virtual Interfaces

UCS Version 2.0 Platform Features

Production Public Backup
Vlan 10,20 Vlan 31,32 Vlan 40,41

L2 Disjoint Networks
End Host End Host
More Flexible Designs
Reduced Networking HW

iSCSI Boot Support in UCSM

Increased Customer Choice

VM-FEX for RedHat KVM

Additional Hypervisor Support

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55
UCS 6248: Unified Ports
Dynamic Port Allocation: Lossless Ethernet or Fibre Channel

FC Eth

Native Fibre Channel Lossless Ethernet:

1/10GbE, FCoE, iSCSI, NAS

Benefits Use-cases
 Simplify switch purchase -  Flexible LAN & storage convergence
remove ports ratio guess work based on business needs
 Increase design flexibility  Service can be adjusted based on the
demand for specific traffic
 Remove specific protocol
bandwidth bottlenecks

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 57
UCS 6248: Unified Ports
Dynamic Port Allocation: Lossless Ethernet or Fibre Channel
Ports on the base card or the Unified Port GEM Module can
either be Ethernet or FC

Only a continuous set of ports can be configured as

Ethernet or FC

Ethernet Ports have to be the 1st set of ports

Port type changes take effect after next reboot of switch for
Base board ports or power-off/on of the GEM for GEM
unified ports.
Base card – 32 Unified Ports GEM – 16 Unified Ports

Eth FC Eth FC

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 58
 UCS 2208XP I/O Module
Customer benefits
 Double the uplink bandwidth to the Fabric
 Quadruple the downlink bandwidth to the
server slots
 Lower latency and better QoS

Feature details
• PID: UCS-IOM-2208XP
• Double the uplinks
• 8x 10GE uplinks from each IOM/FEX
• Total 160 Gbps per chassis
• Quadruple the downlinks
• 32x 10GE or 4x 10GE from each
IOM/FEX to each blade slot*
• Total: 80 Gbps per server slot*
• *Requires VIC 1280 for full server bandwidth
• Increased support for 8 egress CoS queues
• Lower latency

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 59
IOM to Fabric Interconnect Port Pinning
Server-to-Fabric Port Pinning Configurations
160 Gb (Discrete Mode) 160 Gb (Port Channel Mode) FAN FAN FAN FAN

FAN1

FAN1
PS1

PS1
FAN FAN FAN FAN STAT STAT STAT STAT
FAN1

FAN1
PS1

PS1
STAT STAT STAT STAT FAIL FAIL FAIL FAIL

STAT

STAT
FAIL FAIL FAIL FAIL

FAN2

FAN2
STAT

STAT
FAN2

FAN2
OK OK OK OK
OK OK OK OK

PS2

PS2
PS2

PS2
N10-PAC1-550W N10-PAC1-550W N10-PAC1-550W N10-PAC1-550W
N10-PAC1-550W N10-PAC1-550W N10-PAC1-550W N10-PAC1-550W

Slot 1 Slot 2 Slot 1 Slot 2

UCS 5108
UCS 5108

!
!
SLOT SLOT
SLOT SLOT
1 2
1 2

SLOT
3
Slot 3 Slot 4 SLOT
4
SLOT
3
Slot 3 Slot 4 SLOT
4

SLOT
5
Slot 5 Slot 6 SLOT
6
SLOT
5
Slot 5 Slot 6 SLOT
6

SLOT
7
Slot 7 Slot 8 SLOT
8
SLOT
7
Slot 7 Slot 8 SLOT
8

OK FAIL OK FAIL OK FAIL OK FAIL

OK FAIL OK FAIL OK FAIL OK FAIL

• 6100 to 2208 • 6200 to 2208

• 6200 to 2208
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 60
 2208-XP Architecture
Components
Woodside ASIC 1-8
Aggregates traffic to/from 32 host- Fabric Ports
facing 10G Ethernet ports from/to 8 FLASH to
network-facing 10G Ethernet ports Interconnect
DRAM
CPU (also referred to as CMC)
EEPROM
Controls Redwood and perform other
chassis management functionality Control
Chassis
L2 Switch Management IO Woodside ASIC
Controller
Aggregates traffic from BMCs on the
server blades

Switch
Woodside Interfaces
HIF (Backplane ports)
NIF (FabricPorts)
BIF Chassis 32 Backplane Ports to
CIF Signals Blades

No local switching – All traffic from

HIFs goes upstream for Switching
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 61
 UCS 1280 VIC
Customer benefits UCS 2208 IOM UCS 2208 IOM

 Dual 4x 10 GE (80 Gb per host)

Feature details
• Dual 4x 10 GE port-channels to a single server slot
• Host connectivity PCIe Gen2 x16 Side A Side B
• HW Capable of 256 PCIe devices UCS 1280 VIC
• OS restriction apply
• PCIe virtualization OS independent (same as M81KR)
• Single OS driver image for both M81KR and 1280 VIC
256 PCIe devices
• FabricFailover supported
• Eth hash inputs : Source MAC Address,Destination MAC
Address,Source Pprt, Destination Port,Source IP
address,Destination IP address and VLAN
• FC Hash inputs: Source MAC Address
Destination MAC Address,FC SID and FC DID and OXID

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 62
 UCS Fabric Component Interoperability
Complete hardware inter-operability between Gen 1 and Gen 2

Fabric IOM Adapter Supported Min Software

Interconnect version required
6100 2104 UCS M81KR UCSM 1.4(1) or
earlier
6100 2208 UCS M81KR UCSM UCS 2.0

6100 2104 UCS1280 VIC UCSM UCS 2.0

6100 2208 UCS1280 VIC UCSM UCS 2.0

6200 2104 UCS M81KR UCSM UCS 2.0

6200 2208 UCS M81KR UCSM UCS 2.0

6200 2104 UCS1280 VIC UCSM UCS 2.0

6200 2208 UCS1280 VIC UCSM UCS 2.0

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 63
 End Host Mode – Disjointed L2 Domains
UCS ver 2.0 and beyond

Hardware/ FI independent Production Public Backup

Vlan 10,20 Vlan 31,32 Vlan 40,41

The ability to selectively assign VLANs to

uplinks
(NO overlapping VLANs)

Basing pinning decision on border port and

vNIC VLAN membership
End Host End Host

Allocating a designated
broadcast/multicast receiver on a per VLAN
rather than global basis

Max 31 disjoint Layer 2 domains supported

Recommendation:
Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential End Host Mode 64
 Complete Your Online
Session Evaluation
 Receive 25 Cisco Preferred Access points for each session
evaluation you complete.
 Give us your feedback and you could win fabulous prizes.
Points are calculated on a daily basis. Winners will be notified
by email after July 22nd.
 Complete your session evaluation online now (open a browser
through our wireless network to access our portal) or visit one
of the Internet stations throughout the Convention Center.
 Don’t forget to activate your Cisco Live and Networkers Virtual
account for access to all session materials, communities, and
on-demand and live activities throughout the year. Activate
your account at any internet station or visit
www.ciscolivevirtual.com.

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
665
 Visit the Cisco Store for
Related Titles
http://theciscostores.com

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
666
Thank you.

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 67