S e n d d o c u m e n t a t i o n c o m m e n t s t o m d s f e e d b a ck - d o c @ c i s c o . c o m .

C H A P T E R 33
Configuring FCIP

Cisco MDS 9000 Family IP storage (IPS) services extend the reach of Fibre Channel SANs by using
open-standard, IP-based technology. The switch can connect separated SAN islands using Fibre Channel
over IP (FCIP).

Note FCIP is specific to the IPS module and is available in Cisco MDS 9200 Switches or Cisco MDS 9500
Directors.

The Cisco MDS 9216I switch and the 14/2 Multiprotocol Services (MPS-14/2) module also allow you
to use Fibre Channel, FCIP, and iSCSI features. The MPS-14/2 module is available for use in any switch
in the Cisco MDS 9200 Series or Cisco MDS 9500 Series.

For information on configuring Gigabit Ethernet interfaces, see the “Configuring Gigabit Ethernet
Interfaces” section on page 37-4.

This chapter includes the following sections:

• About FCIP, page 33-2
• Enabling FCIP, page 33-4
• Basic FCIP Configuration, page 33-5
Advanced FCIP Profile Configuration, page 33-7
Advanced FCIP Interface Configuration, page 33-12
Configuring E Ports, page 33-18
Advanced FCIP Features, page 33-19
Displaying FCIP Information, page 33-24
FCIP High Availability, page 33-30
Ethernet PortChannels and Fibre Channel PortChannels, page 33-32
Default Settings, page 33-33

Cisco MDS 9000 Family Configuration Guide

OL-6973-03, Cisco MDS SAN-OS Release 2.x 33-1
 Chapter 33 Configuring FCIP
About FCIP

About FCIP
distributed Fibre Channel storage area networks (SAN islands) transparently over IP local area networks
(LANs), metropolitan area networks (MANs), and wide area networks (WANs). See Figure 33-1.

Figure 33-1 Fibre Channel SANs Connected by FCIP

Virtual (E)ISL
Switch 1 Switch 2 Switch 3 Switch 4
WAN
Fibre Fibre
Channel Channel

91556
fabric IP router fabric
IP router

FCIP uses TCP as a network layer transport.

For more information about FCIP protocols, refer to the IETF standards for IP storage at
http://www.ietf.org. Also refer to Fibre Channel standards for switch backbone connection at
http://www.t11.org (see FC-BB-2).

To configure IPS modules or MPS-14/2 modules for FCIP, you should have a basic understanding of the
following concepts:
FCIP and VE Ports, page 33-2
FCIP Links, page 33-3
FCIP Profiles, page 33-4
FCIP Interfaces, page 33-4

FCIP and VE Ports

(ISLs) and Cisco's enhanced ISLs (EISLs).

FCIP virtual E (VE) ports behave exactly like standard Fibre Channel E ports, except that the transport
in this case is FCIP instead of Fibre Channel. The only requirement is for the other end of the VE port
to be another VE port.
A virtual ISL is established over an FCIP link and transports Fibre Channel traffic. Each associated
virtual ISL looks like a Fibre Channel ISL with either an E port or a TE port at each end (see
Figure 33-2).

Cisco MDS 9000 Family Configuration Guide

33-2 OL-6973-03, Cisco MDS SAN-OS Release 2.x
 Figure 33-2 FCIP Links and Virtual ISLs

Switch A

FC Switch C
F

F FC
FC F
E IP
GE VE
FC F

FC
Switch B
FCIP link
VE GE

FC E
F Virtual ISL

FC

91557
See the “E Port” section on page 11-3.

FCIP Links

• One connection is used for data frames.

• The other connection is used only for Fibre Channel control frames, that is, switch-to-switch
protocol frames (all Class F). This arrangement provides low latency for all control frames.
To enable FCIP on the IPS module or MPS-14/2 module, an FCIP profile and FCIP interface (interface
FCIP) must be configured.
The FCIP link is established between two peers, the VE port initialization behavior is identical to a
normal E port. This behavior is independent of the link being FCIP or pure Fibre Channel, and is based
on the E port discovery process (ELP, ESC).
Once the FCIP link is established, the VE port behavior is identical to E port behavior for all inter-switch
communication (including domain management, zones, and VSANs). At the Fibre Channel layer, all VE
and E port operations are identical.
 Chapter 33 Configuring FCIP
Enabling FCIP

FCIP Profiles

•
•
The FCIP profile’s local IP address determines the Gigabit Ethernet port where the FCIP links terminate
(see Figure 33-3).

FCIP Profile and FCIP Links

Switch 2
Switch 1
Gigabit
Ethernet FCIP link 1

FCIP IP
interfaces network
Switch 4

FCIP
profile FCIP link 3

91558
FCIP Interfaces

•
•
• E port parameters—trunking mode and trunk allowed VSAN list.

Enabling FCIP

(SAN_EXTN_OVER_IP or SAN_EXTN_OVER_IP_IPS4) (see Chapter 3, “Obtaining and Installing

Licenses”).

Cisco MDS 9000 Family Configuration Guide

33-4 OL-6973-03, Cisco MDS SAN-OS Release 2.x
 Basic FCIP Configuration

Command Purpose
Step 1 switch# config t

switch(config)# fcip enable

no fcip enable

Basic FCIP Configuration

Step 1
Step 2
Step 3
Step 4
Step 5

Creating FCIP Profiles

You must assign a local IP address of a Gigabit Ethernet interface or subinterface to the FCIP profile to
create an FCIP profile (see Figure 33-4).

Figure 33-4 Assigning Profiles to Each Gigabit Ethernet Interface

Switch 1 Switch 2
IP router IP router
IP
Network
IP address of Gigabit Ethernet IP address of Gigabit Ethernet

91561
interface 3/1 = 10.100.1.25 interface 3/1 = 10.1.1.1

switch1# config terminal

fcip profile 10
switch1(config-profile)#

switch1(config-profile)# ip address
10.100.1.25

33-5
 switch2#
switch2(config)#
switch2(config)#
switch2(config-profile)#
switch2(config-profile)#

Figure 33-5 Assigning Profiles to Each Gigabit Ethernet Interface

Endpoint Endpoint
Interface FCIP = 51 Interface FCIP = 52
Profile = 10 Profile = 20
Connecting switch (peer) = 10.1.1.1 Connecting switch (peer) = 10.100.1.25
Switch 1 Switch 2

FCIP link Virtual (E) ISL

Switch 1 Switch 2
IP router IP router
IP
network

91562
IP address of Gigabit Ethernet IP address of Gigabit Ethernet
interface 3/1 = 10.100.1.25 interface 3/1 = 10.1.1.1

switch1#
switch(config)#
switch1(config)#
switch1(config-if)#
switch1(config-if)# use-profile 10

peer-info ipaddr 10.1.1.1

no shutdown

33-6
 Advanced FCIP Profile Configuration

Enters configuration mode.

Creates an FCIP interface (52).

Binds the profile (20) to the FCIP interface.

Assigns the peer IP address information (10.100.1.25
for switch 1) to the FCIP interface.
Enables the interface.

Advanced FCIP Profile Configuration

•
•

Enters configuration mode.

Creates the profile (if it does not already exist) and enters profile
configuration submode. The valid range is from 1 to 255.

Configuring TCP Listener Ports

port

Associates the profile with the local port number (5000).

Reverts to the default 3225 port.

Configuring TCP Parameters

•
• Keepalive Timeout, page 33-8
Maximum Retransmissions, page 33-9

33-7
 Chapter 33 Configuring FCIP
Advanced FCIP Profile Configuration

• Path MTUs, page 33-9

Selective Acknowledgments, page 33-9
Window Management, page 33-9
Buffer Size, page 33-11
Monitoring Congestion, page 33-10
Estimating Maximum Jitter, page 33-11

Minimum Retransmit Timeout

Command Purpose
Step 1

Keepalive Timeout

Note

Command Purpose
Step 1
keepalive-timeout 120

no tcp
keepalive-timeout 120

Cisco MDS 9000 Family Configuration Guide

33-8 OL-6973-03, Cisco MDS SAN-OS Release 2.x
Maximum Retransmissions

Command Purpose
Step 1
max-retransmissions 6

no tcp
max-retransmissions 6

Path MTUs

PMTU discovery is a mechanism by which TCP learns of the PMTU dynamically and adjusts the
maximum TCP segment accordingly (RFC 1191).
By default, PMTU discovery is enabled on all switches with a timeout of 3600 seconds. If TCP reduces
the size of the maximum segment because of PMTU change, the reset-timeout specifies the time after
which TCP tries the original MTU.
To configure PMTU, follow these steps:

no tcp pmtu-enable

tcp pmtu-enable

tcp pmtu-enable
reset-timeout 90

no tcp pmtu-enable
reset-timeout 600

Selective Acknowledgments

no tcp sack-enable

tcp sack-enable

Window Management

33-9
 round-trip-time

round-trip-time

min-available-bandwidth

max-bandwidth-mbps

Command Purpose
Step 1
900 min-available-bandwidth-mbps 300
round-trip-time-ms 10

no tcp max-bandwidth-mbps
900 min-available-bandwidth-mbps 300
round-trip-time-ms 10

tcp max-bandwidth-kbps
2000 min-available-bandwidth-kbps 2000
round-trip-time-us 200

Congestion

Note

33-10
 Tip

Command Purpose
Step 1

tcp cwm burstsize 30

no tcp cwm burstsize

25

Estimating Maximum Jitter

Note

Command Purpose
Step 1 no tcp max-jitter

tcp max-jitter

tcp max-jitter 300

no tcp max-jitter 2500

Buffer Size
switch(config-profile)# Configure the advertised buffer size to 5000 KB. The valid
range is from 0 to 16384 KB.
switch(config-profile)# Reverts the switch to its factory default. The default is 0 KB.

Quality of Service, page 33-18

To establish a peer connection, you must first create the FCIP interface and enter the config-if
submode.
To enter the config-if submode, follow these steps:

switch# Enters configuration mode.

switch(config)# Creates an FCIP interface (100).

To establish an FCIP link with the peer, you can use one of two options:
Peer IP address—Configures both ends of the FCIP link. Optionally, you can also use the peer TCP
port along with the IP address.
Special frames—Configures one end of the FCIP link when security gateways are present in the IP
network. Optionally, you can also use the switch WWN (sWWN) and profile ID along with the IP
address.
Peer IP Address

Command Purpose
Step 1

Step 2

Step 3

Special Frames

Note

Tip

Command Purpose
Step 1
12:12:34:45:ab:bc:cd:00

show wwn switch

 Command Purpose
Step 2

Step 3

Active Connections

Note

Command Purpose
Step 1

Step 2

Number of TCP Connections

Command Purpose
Step 1

Step 2

Time Stamp Control

arrives within a 2000 millisecond interval (+ or –2000 ms) from the network time, that packet is
accepted.

The default value for packet acceptance is 2000 microseconds.

If the option is enabled, be sure to configure NTP on both switches (see the “NTP
Configuration” section on page 4-18).

Do not enable time stamp control on an FCIP interface that has tape acceleration or write acceleration
configured.

To enable or disable the time stamp control, follow these steps:

Please enable NTP with a common time

source on both MDS Switches that are
on either side of the FCIP link
switch(config-if)#

switch(config-if)#

switch(config-if)#

switch(config-if)#
B Port Interoperability Mode

Figure 33-6 FCIP B Port and Fibre Channel E Port

E port

Switch A

F FC Switch C
FC F
F FC
F IP
GE VE
FC E

FC

Switch B FCIP link

VE GE
FC E
F
Virtual ISL
FC

B port FC bridge

B FC
Switch A B
access
F FC
FCIP link
FC F
FC bridge
E
Switch C
FC B
FC B access IP
F FC
FC E
Switch B E
B access ISL
FC
E FC
FC E ISL
F
ISL
91559

FC
 while VE
ports establish a virtual ISL over an FCIP link, B ports use a B access ISL

Figure 33-7 FCIP Link Terminating in a B Port Mode

Switch A 7200 router

F FC
Fibre Channel
FC F port adapter (1G)
E
B B
access FC
FC IP

Switch B
FCIP link
Switch C

F FC
FC bridge
FC E
B
VE B access GE E

FC
ISL
FC E B access ISL
E

91560
FC
FCIP interface
 Enables the reception of keepalive responses sent by a
remote peer.
Disables the reception of keepalive responses sent by a
remote peer (default).

Quality of Service

•
•

qos control 24 data 26

no qos control 24 data 26

Configuring E Ports

•
•
–
–

and Managing Zones”).

FCIP Write Acceleration

Note
 Chapter 33 Configuring FCIP
Advanced FCIP Features

Figure 33-8 FCIP Link Write Acceleration

Initiator MDS 9000 MDS 9000 Target

FC
FCIP over
WAN

Command

RTT1
Transfer ready
Without
Data transfer acceleration
RT2
Status

Initiator MDS 9000 MDS 9000 Target

FC
FCIP over
WAN

Command
Transfer ready
Data transfer
Transfer ready
RTT1 With
acceleration
Status

105224

Caution

Cisco MDS 9000 Family Configuration Guide

OL-6973-03, Cisco MDS SAN-OS Release 2.x
 Chapter 33 Configuring FCIP
Advanced FCIP Features

Note

Caution

Command Purpose
Step 1

Step 2

Step 3

FCIP Tape Acceleration

Note

Cisco MDS 9000 Family Configuration Guide

OL-6973-03, Cisco MDS SAN-OS Release 2.x
 Chapter 33 Configuring FCIP
Advanced FCIP Features

Figure 33-9 FCIP Link Tape Acceleration

Backup MDS with MDS with Tape

Server IPS IPS Drive
FC
WAN

Write Command 1 Write Command 1 Write Command 1

Status 1 Status 1

Write Command N Write Command N

Status N
WRITE FILEMARKS operation Write Command N
Status N

WRITE FILEMARKS operation

WRITE FILEMARKS status

120492

Cisco MDS 9000 Family Configuration Guide

OL-6973-03, Cisco MDS SAN-OS Release 2.x
 Enters configuration mode.

Creates an FCIP interface (5).

Enables tape acceleration (and write acceleration—if

not already enabled).
Enables tape acceleration with automatic flow
control (default)

2048
no write-accelerator
tape-accelerator

no write-accelerator
tape-accelerator flow-control-buffer-size
2048

no write-accelerator

mode1 is a fast compression mode for high bandwidth links (> 25 Mbps)
mode2

mode3 is a high compression mode for low bandwidth links (< 10 Mbps)
auto (default) mode picks the appropriate compression scheme based on the bandwidth of the link
(the bandwidth of the link configured in the FCIP profile’s TCP parameters)
The IP compression feature behavior differs between the IPS module(s) or MPS-14/2 module(s) and
MPS-14/2 module—while mode2 and mode3 perform software compression in both modules, mode1
performs hardware-based compression in MPS-14/2 modules, and software compression in IPS-4 and
IPS-8 modules.

The Cisco MDS 9216i Switch also supports IP compression feature. The integrated supervisor module
has the same hardware components that are available in the MPS-14/2 module.
 switch#
switch(config)#
switch(config)#
switch(config-if)#
switch(config-if)#

switch(config-if)#

switch(config-if)#

Displaying FCIP Information

show interface

Example 33-1 Displays the FCIP Summary

show fcip summary

Tun prof Eth-if peer-ip Status T W T Enc Comp Bandwidth rtt

E A A max/min (us)
-------------------------------------------------------------------------------
10 91 GE4/1 3.3.3.2 UP N N N N N 1000M/1000M 2000
11 11 GE3/1.601 30.1.1.2 DOWN N N N N N 1000M/500M 1000
12 12 GE3/1.602 30.1.2.2 DOWN N N N N N 1000M/500M 1000
13 0 0.0.0.0 DOWN N N N N N
14 0 0.0.0.0 DOWN N N N N N
15 0 0.0.0.0 DOWN N N N N N
16 0 0.0.0.0 DOWN N N N N N
17 0 0.0.0.0 DOWN N N N N N
18 0 0.0.0.0 DOWN N N N N N
19 0 0.0.0.0 DOWN N N N N N
20 92 GE4/2 3.3.3.1 UP N N N N N 1000M/1000M 2000
21 21 GE3/2.601 30.1.1.1 DOWN N N N N N 1000M/500M 1000
22 22 GE3/2.602 30.1.2.1 DOWN N N N N N 1000M/500M 1000

Specified Host End FCIP Link.

MAP TABLE (5 entries TOTAL entries 5)

OXID | RXID | HOST FCID| TARG FCID| VSAN | Index

------+------+----------+----------+------+---------
0xd490|0xffff|0x00690400|0x00620426|0x0005|0x0000321f
0xd4a8|0xffff|0x00690400|0x00620426|0x0005|0x00003220
0xd4c0|0xffff|0x00690400|0x00620426|0x0005|0x00003221
0xd4d8|0xffff|0x00690400|0x00620426|0x0005|0x00003222
0xd4f0|0xffff|0x00690400|0x00620426|0x0005|0x00003223

switch#

MAP TABLE (3 entries TOTAL entries 3)

OXID | RXID | HOST FCID| TARG FCID| VSAN | Index

------+------+----------+----------+------+---------
0xc308|0xffff|0x00690400|0x00620426|0x0005|0x00003364
0xc320|0xffff|0x00690400|0x00620426|0x0005|0x00003365
0xc338|0xffff|0x00690400|0x00620426|0x0005|0x00003366

Displays Information About Tapes for which Exchanges are Tape Accelerated at the Host
End FCIP Link

switch#

HOST TAPE SESSIONS (1 entries TOTAL entries 1)

Host Tape Session #1

FCID 0xef0001, VSAN 1, LUN 0x0002
Outstanding Exchanges 0, Outstanding Writes 0
Target End Buffering 0 Bytes, Auto Max Writes 1
Flags 0x0, FSM state Non TA Mode
First index 0xfffffff7, Last index 0xfffffff7
Current index=0xfffffffe, Els Oxid 0xfff7, Seq-Id 0x0000
Hosts 1
FCID 0x20300
switch#

TARGET TAPE SESSIONS (1 entries TOTAL entries 1)

Target Tape Session #1

FCID 0xef0001, VSAN 2, LUN 0x0002
Outstanding Exchanges 0, Outstanding Writes 0
Estimated IO Time 0x0
Flags 0x0, Timer Flags 0x0
First index 0xfffffff7, Last index 0xfffffff7
Current index=0xfffffffe, Els Oxid 0xfff7, Seq-Id 0x0000
Hosts 1
FCID 0x20300

switch#
fcip10 is up
Hardware is GigabitEthernet
Port WWN is 20:d0:00:0c:85:90:3e:80
Peer port WWN is 20:d4:00:0c:85:90:3e:80
Admin port mode is auto, trunk mode is on
Port mode is E, FCID is 0x720000
Port vsan is 91
Speed is 1 Gbps
Using Profile id 91 (interface GigabitEthernet4/1)
Peer Information
Peer Internet address is 3.3.3.2 and port is 3225
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is 256 KBytes
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
Time Stamp is disabled
QOS control code point is 0
QOS data code point is 0
B-port mode disabled
TCP Connection Information
50529025 Active TCP connections
Local 0.0.0.7:6, Remote 0.0.0.200:0
0 host table full 0 target entries in use
211419104 Attempts for active connections, 1500 close of connections
TCP Parameters
Path MTU 124160 bytes
Current retransmission timeout is 124160 ms
Round trip time: Smoothed 127829 ms, Variance: 14336
Advertized window: Current: 0 KB, Maximum: 14 KB, Scale: 14336
Peer receive window: Current: 0 KB, Maximum: 0 KB, Scale: 51200
Congestion window: Current: 14 KB, Slow start threshold: 49344 KB
Current Send Buffer Size: 206463 KB, Requested Send Buffer Size: 429496728
3 KB
CWM Burst Size: 49344 KB
5 minutes input rate 491913172779207224 bits/sec, 61489146597400903 bytes/se
c, 0 frames/sec
5 minutes output rate 491913175298921320 bits/sec, 61489146912365165 bytes/s
ec, 14316551 frames/sec
5702 frames input, 482288 bytes
5697 Class F frames input, 481736 bytes
 5 Class 2/3 frames input, 552 bytes
0 Reass frames
0 Error frames timestamp error 0
5704 frames output, 482868 bytes
5698 Class F frames output, 482216 bytes
6 Class 2/3 frames output, 652 bytes
0 Error frames

switch#
fcip4
TCP Connection Information
...
5 minutes input rate 207518944 bits/sec, 25939868 bytes/sec, 12471 frames/sec
5 minutes output rate 205340328 bits/sec, 25667541 bytes/sec, 12340 frames/sec
2239902537 frames input, 4658960377152 bytes
18484 Class F frames input, 1558712 bytes
2239884053 Class 2/3 frames input, 4658958818440 bytes
0 Reass frames
0 Error frames timestamp error 0
2215051484 frames output, 4607270186816 bytes
18484 Class F frames output, 1558616 bytes
2215033000 Class 2/3 frames output, 4607268628200 bytes
0 Error frames

Displays Detailed FCIP Interface Compression Information, if Enabled

switch#
fcip4
TCP Connection Information
...
IP compression statistics
208752 rxbytes, 208752 rxbytes compressed
5143584 txbytes
0 txbytes compressed, 5143584 txbytes non-compressed
1.00 tx compression ratio

Example 33-9 Displays Detailed FCIP Interface Write Acceleration Counter Information, if Enabled

switch#
fcip4
TCP Connection Information
...
Write Accelerator statistics
6091 packets in 5994 packets out
0 frames dropped 0 CRC errors
0 rejected due to table full
0 ABTS sent 0 ABTS received
0 tunnel synchronization errors
37 writes recd 37 XFER_RDY sent (host)
0 XFER_RDY rcvd (target)
37 XFER_RDY rcvd (host)
0 XFER_RDY not proxied due to flow control (host)
0 bytes queued for sending
0 estimated bytes queued on the other side for sending
 0 times TCP flow ctrl(target)
0 bytes current TCP flow ctrl(target)

Example 33-10 Displays Detailed FCIP Interface Tape Acceleration Counter Information, if Enabled

switch#
fcip4
TCP Connection Information
...
Tape Accelerator statistics
1 (host) tape sessions 0 (target) tape sessions
37 writes recd 33 STATUS proxied (host)
37 write good STATUS rcvd (host)
0 write good STATUS rcvd (target)
0 write bad STATUS rcvd (host)
0 write bad STATUS rcvd (target)
4 writes not TAed 8 queued flow ctrl (host)
0 recovery REC sent Got 0 ACCs 0 Rejects (host)
0 ABTS sent Got 0 ACCs (host)
0 REC Accs 0 REC Rjts 14 REC fwded (host)
0 SRR Accs 0 SRR Rjts 0 SRR fwded(host)
0 XferRdy retries 0 Status retries (host)
0 recovery REC sent Got 0 ACCs 0 Rejects (target)
0 recovery SRR sent Got 0 ACCs (target)
0 ABTS sent Got 0 ACCs (target)
0 tmf cmds rcvd (host)
0 tmf cmds rcvd (target)

Example 33-11 Displays the Compression Engine Statistics for the MPS-14/2 Module
Example 33-12 Displays Brief FCIP Interface Counter Information

Example 33-13 Displays the FCIP Interface Description

Example 33-14 Displays FCIP Profiles

Example 33-15 Displays the Specified FCIP Profile Information

show fcip profile 7

 FCIP High Availability

FCIP High Availability

•
•
•
•

Fibre Channel PortChannels

Ethernet
FCIP link switch

IP
FC fabric network FC fabric

Ethernet Ethernet
switch FCIP link switch

90857
PortChannel of
two FCIP links

•
•
•
 Chapter 33 Configuring FCIP
FCIP High Availability

FSPF

FCIP link

IP
FC fabric network FC fabric

90858
PortChannel of FCIP link
two FCIP tunnels

•
•
•

VRRP

VRRP-Based High Availability

FCIP link

IP
FC fabric FC fabric
network
90859

IP interfaces are in VRRP group

Cisco MDS 9000 Family Configuration Guide

OL-6973-03, Cisco MDS SAN-OS Release 2.x
Figure 33-13 Ethernet PortChannel-Based High Availability

FCIP link

IP
FC fabric FC fabric
network

90860
Ethernet ports are
in PortChannel
 Figure 33-14 PortChannels at the Fibre Channel and Ethernet Levels

Fibre Channel PortChannel

Fibre Channel Fibre Channel

FCIP FCIP

TCP TCP

IP IP

Ethernet Ethernet

Ethernet PortChannel

94176
Gigabit Ethenet link Gigabit Ethenet link Gigabit Ethenet link

Default Settings

Parameters Default

keepalive-timeout
max-retransmissions
Default FCIP Parameters (continued)