Flexpod Deploy

FlexPod Deployment Guide
Last Updated: February 23, 2012
Building Architectures to Solve Business Problems
Cisco Validated Design
About the Authors
About the Authors
David Antkowiak, Solutions Architect, Systems Development Unit, Cisco Systems

David Antkowiak is a Solutions Architect with the Systems Development Unit (SDU). With over 10 years of experience in various private and government organizations, his areas of focus have included virtual desktop infrastructure, server virtualization, cloud migration, and storage design. Prior to joining Cisco, David was Solutions Architect at JetBlue. David holds a masters degree from Florida State University and two VMware certifications.
David Antkowiak
Ramesh Isaac, Technical Marketing Engineer, Systems Development Unit, Cisco

Ramesh Isaac has worked in data center and mixed-use lab settings since 1995. He started in information technology supporting UNIX environments and focused on designing and implementing multi-tenant virtualization solutions in Cisco labs over the last couple of years. Ramesh holds certifications from Cisco, VMware, and Red Hat.
Ramesh Isaac
Jon Benedict, Reference Architect, Infrastructure and Cloud Engineering, NetApp

Jon Benedict is a reference architect in the Infrastructure & Cloud Engineering team at NetApp. Jon is largely focused on designing, building, and evangelizing cloud and shared storage solutions based around NetApp for enterprise customers. Prior to NetApp, he spent many years as a consultant, integrator, and engineer with expertise in Unix and Linux. Jon holds many industry certifications including several from Red Hat.
Jon Benedict
Chris Reno, Reference Architect, Infrastructure and Cloud Engineering, NetApp

Chris Reno is a reference architect in the NetApp Infrastructure and Cloud Enablement group and is focused on creating, validating, supporting, and evangelizing solutions based on NetApp products. Before being employed in his current role, he worked with NetApp product engineers designing and developing innovative ways to perform Q&A for NetApp products, including enablement of a large grid infrastructure using physical and virtualized compute resources. In these roles, Chris gained expertise in stateless computing, netboot architectures, and virtualization.
Chris Reno
Cisco Validated Design
About Cisco Validated Design (CVD) Program

The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit http://www.cisco.com/go/designzone.
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCBs public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at http://www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R) NetApp, the NetApp logo, Go further, faster, AutoSupport, DataFabric, Data ONTAP, FlexPod, MultiStore, NearStore, NOW, and vFiler are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. FlexPod Deployment Guide 2011 Cisco Systems, Inc. All rights reserved.
About Cisco Validated Design (CVD) Program

FlexPod Overview
Industry trends indicate a vast data center transformation toward shared infrastructures. Enterprise customers are moving away from silos of information and moving toward shared infrastructures to virtualized environments and eventually to the cloud to increase agility and reduce costs. FlexPod is a predesigned, base configuration that is built on the Cisco Unified Computing System (UCS), Cisco Nexus data center switches, and NetApp FAS storage components and includes a range of software partners. FlexPod can scale up for greater performance and capacity or it can scale out for environments that need consistent, multiple deployments. FlexPod is a baseline configuration, but also has the flexibility to be sized and optimized to accommodate many different use cases. Cisco and NetApp have developed FlexPod as a platform that can address current virtualization needs and simplify data center evolution to ITaaS infrastructure. FlexPod serves as a base infrastructure layer for a variety of IT solutions. A number of solutions have been built on the FlexPod architecture, including Microsoft SharePoint, VMware View, VMware vSphere, and Secure Multi-tenancy, among others. These FlexPod solutions can be found at:

http://www.netapp.com/us/technology/flexpod/ http://www.cisco.com/en/US/netsol/ns964/index.html
NetApp partners may access additional information at: https://fieldportal.netapp.com/.
Audience
This document describes the basic architecture of FlexPod as well as the general procedures for deploying the base FlexPod system. The intended audience for this document includes, but is not limited to, sales engineers, field consultants, professional services personnel, IT managers, partner engineering personnel, and customers who want to deploy the base FlexPod architecture.
Note
For more detailed deployment information, Cisco and NetApp partners should contact their local account teams or visit http://www.netapp.com/us/technology/flexpod/.
Corporate Headquarters: Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA 95134-1706 USA
Copyright 2011 Cisco Systems, Inc. All rights reserved.
FlexPod Overview
FlexPod Architecture
Cisco and NetApp have provided documentation around best practices for building the FlexPod shared infrastructure stack. As part of the FlexPod offering, Cisco and NetApp designed a reference architecture with a technical specifications sheet and bill of materials that is highly modular or pod-like. Although each customers FlexPod system may vary in its exact configuration, once a FlexPod unit is built it can easily be scaled as requirements and demand change. This includes scaling both up (adding additional resources within a FlexPod unit) and out (adding additional FlexPod units). Specifically, FlexPod is a defined set of hardware and software that serves as a foundation for data center deployments. FlexPod includes NetApp storage, Cisco networking, and the Cisco Unified Computing System in a single package in which the computing and storage fit in one data center rack and the networking resides in a separate rack. Due to port density the networking components can accommodate multiple instances of FlexPod systems. Figure 1 shows the FlexPod components. The solution can be scaled while still maintaining its integrity, either by adding more FlexPod units or by adding to the solution components. A number of solutions can be built on top of one or more FlexPod units, providing enterprise flexibility, supportability, and manageability. Figure 1 and Figure 2 outline the possible NetApp Filer interconnect choices. The first topology in Figure 1 is an FCoE-only implementation, while the second in Figure 2 adds the option of native FC connectivity. These interconnects are not interdependent and may be deployed together or separately to meet customer hypervisor or application support requirements. Both deployments are fully supported.
FlexPod Overview
Figure 1
Access
FlexPod FCoE and 10 GbE Based Architecture
CISCO NEXUS N5548P
10 11
12 13
14 15
16
17
18 19
20 21
22 23
24 25
26 27
28 29
30 31
32
1 /10 GIGABIT ET HERNET
1 /2/ 4/8 G F IBRE CHANNEL
CISCO NEXUS N5548P
10 11
12 13
14 15
16
17
18 19
20 21
22 23
24 25
26 27
28 29
30 31
32
3
ID STAT
N55-M8P8FP
3
ID
STAT
N55-M8P8FP
Qty 2: Cisco Nexus 5548 vPC vPC
1/ 2/4 G F IBRE CHANNEL
PS2
PS1
PS2
PS1
SLOT2
Unified Computing System
CONSOLE
L1 L2 MGMT 0 MGMT 1
10 11
12
13
14
15
16
17
18
19
20
SLOT2
Cisco UCS 6120XP
Cisco UCS 6120XP
N10-E0440
CONSOLE
L1 L2 MGMT 0 MGMT 1
10 11
12
13
14
15
16
17
18
19
20
N10-E0440
UCS B200 M1
! ! ! !
UCS B200 M1
UCS 5108
!
SLOT
SLOT
Console
Reset
Console
Reset
UCS B250 M1
SLOT
SLOT
Console
Reset
UCS B250 M1
SLOT
SLOT 6
! Console Reset
Qty 2: Cisco UCS 6120 Fabric Interconnect 2 10 GbE with FCoE per Fabric Extender Qty 6: Cisco UCS B200 M2 Qty 9: Cisco UCS B250 M2 Qty 6: Cisco UCS 2140 Fabric Extender Qty 3: Cisco UCS 5108 Chassis
UCS B250 M1
SLOT 7
SLOT
8
! Console Reset
OK
FAIL
OK
FAIL
OK
FAIL
OK
FAIL
UCS B200 M1
! ! ! !
UCS B200 M1
UCS 5108
!
SLOT
SLOT
Console
Reset
Console
Reset
UCS B250 M1
SLOT
SLOT
Console
Reset
UCS B250 M1
SLOT
SLOT 6
! Console Reset
UCS B250 M1
SLOT 7
SLOT
8
! Console Reset
OK
FAIL
OK
FAIL
OK
FAIL
OK
FAIL
UCS B200 M1
! ! ! !
UCS B200 M1
UCS 5108
!
SLOT
SLOT
Console
Reset
Console
Reset
UCS B250 M1
SLOT
SLOT
Console
Reset
UCS B250 M1
SLOT
SLOT 6
! Console Reset
UCS B250 M1
SLOT 7
SLOT
8
! Console Reset
OK
FAIL
OK
FAIL
OK
FAIL
OK
FAIL
Storage FCoE +2 x 10 GbE
10
11
12
13
14
15
16
17
18
19
20
21
22
23
DS 2 2 4 6
291691
600G B
0
600G B
1
600G B
2
600G B
3
600G B
4
600G B
5
600G B
6
600G B
600G B
8
600G B
9
600G B
10
600G B
11
12
600G B
13
600G B
14
600G B
15
600G B
16
600G B
600G B
17
18
600G B
600G B
19
20
600G B
600G B
21
22
600G B
600G B
23
DS 2 2 4 6
FCoE 2 x 13 GbE NetApp FAS3210 HA Pair w/Qty. 4 DS2246 Disk Shelves
600G B
0
600G B
1
600G B
2
600G B
3
600G B
4
600G B
5
600G B
6
600G B
600G B
8
600G B
9
600G B
10
600G B
11
12
600G B
13
600G B
14
600G B
15
600G B
16
600G B
600G B
17
18
600G B
600G B
19
20
600G B
600G B
21
22
600G B
600G B
FA S3270 FA S3270
23
DS 2 2 4 6
600G B
0
600G B
1
600G B
2
600G B
3
600G B
4
600G B
5
600G B
6
600G B
600G B
8
600G B
9
600G B
10
600G B
11
12
600G B
13
600G B
14
600G B
15
600G B
16
600G B
17
600G B
18
600G B
19
600G B
20
600G B
21
600G B
22
600G B
23
600G B
DS 2246
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
600G B
FlexPod Overview
Figure 2
Access
FlexPod FC and 10 GbE Based Architecture
CISCO NEXUS N5548P
10 11
12 13
14 15
16
17
18 19
20 21
22 23
24 25
26 27
28 29
30 31
32
CISCO NEXUS N5548P
10 11
12 13
14 15
16
17
18 19
20 21
22 23
24 25
26 27
28 29
30 31
32
3
ID STAT
N55-M8P8FP
3
ID
STAT
N55-M8P8FP
Qty 2: Cisco Nexus 5548 vPC vPC
PS2
PS1
PS2
PS1
SLOT2
Unified Computing System
CONSOLE
L1 L2 MGMT 0 MGMT 1
10 11
12
13
14
15
16
17
18
19
20
SLOT2
Cisco UCS 6120XP
Cisco UCS 6120XP
N10-E0440
CONSOLE
L1 L2 MGMT 0 MGMT 1
10 11
12
13
14
15
16
17
18
19
20
N10-E0440
UCS B200 M1
! ! ! !
UCS B200 M1
UCS 5108
!
SLOT
SLOT
Console
Reset
Console
Reset
UCS B250 M1
SLOT
SLOT
Console
Reset
UCS B250 M1
SLOT
SLOT 6
! Console Reset
Qty 2: Cisco UCS 6120 Fabric Interconnect 2 10 GbE with FCoE per Fabric Extender Qty 6: Cisco UCS B200 M2 Qty 9: Cisco UCS B250 M2 Qty 6: Cisco UCS 2140 Fabric Extender Qty 3: Cisco UCS 5108 Chassis
UCS B250 M1
SLOT 7
SLOT
8
! Console Reset
OK
FAIL
OK
FAIL
OK
FAIL
OK
FAIL
UCS B200 M1
! ! ! !
UCS B200 M1
UCS 5108
!
SLOT
SLOT
Console
Reset
Console
Reset
UCS B250 M1
SLOT
SLOT
Console
Reset
UCS B250 M1
SLOT
SLOT 6
! Console Reset
UCS B250 M1
SLOT 7
SLOT
8
! Console Reset
OK
FAIL
OK
FAIL
OK
FAIL
OK
FAIL
UCS B200 M1
! ! ! !
UCS B200 M1
UCS 5108
!
SLOT
SLOT
Console
Reset
Console
Reset
UCS B250 M1
SLOT
SLOT
Console
Reset
UCS B250 M1
SLOT
SLOT 6
! Console Reset
UCS B250 M1
SLOT 7
SLOT
8
! Console Reset
OK
FAIL
OK
FAIL
OK
FAIL
OK
FAIL
Storage Ether Channel: 2 x 10 GbE
10
11
12
13
14
15
16
17
18
19
20
21
22
23
DS 2 2 4 6
600G B
0
600G B
1
600G B
2
600G B
3
600G B
4
600G B
5
600G B
6
600G B
600G B
8
600G B
9
600G B
10
600G B
11
12
600G B
13
600G B
14
600G B
15
600G B
16
600G B
600G B
17
18
600G B
600G B
19
20
600G B
600G B
21
22
600G B
600G B
23
DS 2 2 4 6
Ether Channel: 2 x 10 GbE NetApp FAS3210 HA Pair w/Qty. 4 DS2246 Disk Shelves
600G B
0
600G B
1
600G B
2
600G B
3
600G B
4
600G B
5
600G B
6
600G B
600G B
8
600G B
9
600G B
10
600G B
11
12
600G B
13
600G B
14
600G B
15
600G B
16
600G B
600G B
17
18
600G B
600G B
19
20
600G B
600G B
21
22
600G B
600G B
FA S3270 FA S3270
23
Fiber Channel: 2 x 4 Gbps
DS 2 2 4 6
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Fiber Channel: 2 x 4 Gbps
The default hardware is detailed in the FlexPod technical specifications and includes two Cisco Nexus 5548 switches, two Cisco UCS 6120 fabric interconnects, and three chassis of Cisco UCS blades with two fabric extenders per chassis. Storage is provided by a NetApp FAS3210CC (HA configuration within a single chassis) with accompanying disk shelves. All systems and fabric links feature redundancy, providing end-to-end high availability. This is the default base design, but each of the components can be scaled flexibly to support a specific customers business requirements. For example, more (or different) blades and chassis could be deployed to increase compute capacity, additional disk shelves could be deployed to improve IO capacity and throughput, or special hardware or special hardware or software features may be added to introduce new features (such as NetApp FlashCache for dedupe-aware caching). The remainder of this document guides the reader through the steps necessary to deploy the base architecture as shown above. This includes everything from physical cabling to compute and storage configuration.
291692
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B 600G B
600G B
DS 2246
600G B
FlexPod Configuration Deployment

The following section provides detailed information on configuring all aspects of a base FlexPod unit. Because the FlexPod architecture is flexible, the exact configuration detailed below may vary from customer implementations depending on specific requirements. Although customer implementations may deviate from the information that follows, the practices, features, and configurations below should still be used as a reference for building a customized FlexPod deployment.
Cabling Information
The following information is provided as a reference for cabling the physical equipment in a FlexPod environment. The tables include both local and remote device and port locations to simplify cabling requirements.
Note
The following tables are for the prescribed and supported configuration of the FAS3210 running Data ONTAP 8.0.1. This configuration leverages a dual-port 10GbE add-on adapter and the on-board SAS ports for disk shelf connectivity. Onboard FC storage target ports are still supported for legacy implementations. For any modifications of this prescribed architecture, consult the currently available Interoperability Matrix Tool (IMT): http://now.netapp.com/matrix.
Note
See the Site Requirements guide when deploying a storage system to ensure power and cooling requirements are met: http://now.netapp.com/NOW/public/knowledge/docs/hardware/NetApp/site/pdf/site.pdf.
Note
The FlexPod deployment guide assumes that out-of-band management ports are plugged into existing management infrastructure at the deployment site.
Note
Be sure to cable as detailed below, because failure to do so will necessitate changes to the following deployment procedures as specific port locations are mentioned.
Note
It is possible to order a FAS3210A system in a different configuration than the one prescribed below. Ensure that your configuration matches the one described in the tables and diagrams below before starting.
Note
The tables below indicate recommended cabling for both FC- and FCoE-based architectures.
Note
For FCoE-based architectures, the Fibre Channel protocol is addressed with the FCoE storage target adapters as indicated in Table 1.
Note
For FC-based architectures, the Fibre Channel protocol is addressed with native FC storage target ports as indicated in Table 2.
Table 1
FlexPod Ethernet Cabling Information
Local Device Cisco Nexus 5548 A
Local Port Eth1/1 Eth1/2 Eth1/5 Eth1/6 Eth1/7 Eth1/8
Connection 10GbE or FCoE 10GbE or FCoE 10GbE 10GbE 10GbE 10GbE 10GbE or FCoE 10GbE or FCoE 10GbE 10GbE 10GbE 10GbE 100MbE 1GbE 10GbE or FCoE 10GbE or FCoE 100MbE 1GbE 10GbE or FCoE 10GbE or FCoE 10GbE 10GbE 10GbE 10GbE
Remote Device NetApp Controller A NetApp Controller B Cisco Nexus 5548 B Cisco Nexus 5548 B Cisco UCS Fabric Interconnect A Cisco UCS Fabric Interconnect B 100MbE Management Switch NetApp Controller A NetApp Controller B Cisco Nexus 5548 A Cisco Nexus 5548 A Cisco UCS Fabric Interconnect A Cisco UCS Fabric Interconnect B 100MbE Management Switch 100MbE Management Switch SAS shelves Nexus 5548 A Nexus 5548 B 100MbE Management Switch SAS shelves Nexus 5548 A Nexus 5548 B Nexus 5548 A Nexus 5548 B Nexus 5548 A Nexus 5548 B
Remote Port e2a e2a Eth1/5 Eth1/6 Eth1/7 Eth1/7 Any e2b e2b Eth1/5 Eth1/6 Eth1/8 Eth1/8 Any Any ACP port Eth1/1 Eth1/1 Any ACP port Eth1/2 Eth1/2 Eth1/9 Eth1/9 Eth1/9 Eth1/9
MGMT0 100MbE Cisco Nexus 5548 B Eth1/1 Eth1/2 Eth1/5 Eth1/6 Eth1/7 Eth1/8 e0M e0P NetApp Controller A e2a e2b e0M e0P NetApp Controller B e2a e2b Eth1/7 Eth1/7 Eth1/7 Eth1/8
MGMT0 100MbE
10
Table 1
FlexPod Ethernet Cabling Information
Local Device Cisco UCS Fabric Interconnect
Local Port Eth1/1 Eth1/2 Eth1/3 Eth1/4 Eth1/5 Eth1/6 L1 L2
Connection
Remote Device
Remote Port port 1 port 2 port 1 port 2 port 1 port 2 Any L1 L2 port 1 port 2 port 1 port 2 port 1 port 2 Any L1 L2 Eth1/7 Eth1/7 Eth1/8 Eth1/8
10GbE/FCoE Chassis 1 FEX A 10GbE/FCoE Chassis 1 FEX A 10GbE/FCoE Chassis 2 FEX A 10GbE/FCoE Chassis 2 FEX A 10GbE/FCoE Chassis 3 FEX A 10GbE/FCoE Chassis 3 FEX A 100MbE Management Switch UCS Fabric Interconnect B UCS Fabric Interconnect B 1GbE 1GbE
MGMT0 100MbE
Cisco UCS Fabric Interconnect
Eth1/1 Eth1/2 Eth1/3 Eth1/4 Eth1/5 Eth1/6 L1 L2 Eth1 Eth2 Eth1 Eth2
10GbE/FCoE Chassis 1 FEX B 10GbE/FCoE Chassis 1 FEX B 10GbE/FCoE Chassis 2 FEX B 10GbE/FCoE Chassis 2 FEX B 10GbE/FCoE Chassis 3 FEX B 10GbE/FCoE Chassis 3 FEX B 100MbE Management Switch UCS Fabric Interconnect A UCS Fabric Interconnect A Nexus 5548 A (only used with vSphere) Nexus 5548 B (only used with vSphere) Nexus 5548 A (only used with vSphere) Nexus 5548 B (only used with vSphere) 1GbE 1GbE 1GbE 1GbE 1GbE 1GbE
MGMT0 100MbE
Table 2
FlexPod Fibre Channel Cabling Information
Local Device Cisco Nexus 5548 A
Local Port Connection Remote Device FC2/1 FC2/2 FC2/3 FC2/4 FC FC FC FC FC FC FC FC FC FC NetApp Controller A NetApp Controller B Cisco UCS Fabric Interconnect A UCS Fabric Interconnect A NetApp Controller A NetApp Controller B Cisco UCS Fabric Interconnect B UCS Fabric Interconnect B Cisco Nexus 5548 A Cisco Nexus 5548 B
Remote Port 0c 0c FC2/1 FC2/2 0d 0d FC2/1 FC2/2 FC2/1 FC2/1
Cisco Nexus 5548 B
FC2/1 FC2/2 FC2/3 FC2/4
NetApp Controller A
0c 0d
11
Table 2
FlexPod Fibre Channel Cabling Information
Local Device NetApp Controller B Cisco UCS Fabric Interconnect Cisco UCS Fabric Interconnect
Local Port Connection Remote Device 0c 0d FC2/1 FC2/2 FC2/1 FC2/2 FC FC FC FC FC FC Cisco Nexus 5548 A Cisco Nexus 5548 B Cisco Nexus 5548 A Cisco Nexus 5548 A Cisco Nexus 5548 B Cisco Nexus 5548 B
Remote Port FC2/2 FC2/2 FC2/3 FC2/4 FC2/3 FC2/4
12
Figure 3
CISCO NEXUS N5548P
FlexPod Cabling
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
1 /10 GIGABIT ETHERNET 1 /2/ 4/8 G FIBRE CHANNEL
3
ID STAT
N55-M8P8FP
CISCO NEXUS N5548P
10 11
12 13
14 15
16
17
18 19
20 21
22 23
24 25
26 27
28 29
30 31
32
1 /10 GIGABIT ETHERNET
1 /2/ 4/8 G FIBRE CHANNEL
Qty 2: Cisco Nexus 5548

6 7 8
3
ID STAT
N55-M8P8FP
1/2/4G FIBRE CHANNEL
PS2
PS1
SLOT2
Cisco UCS 6120XP
CONSOLE
L1 L2 MGMT0 MGMT1
10 11
12
13
14
15
16
17
18
19
20
N10-E0440
1/2/4G FIBRE CHANNEL
PS2
PS1
SLOT2
Cisco UCS 6120XP
Qty 2: Cisco UCS 6120
CONSOLE
L1 L2 MGMT0 MGMT1
10 11
12
13
14
15
16
17
18
19
20
N10-E0440
UCS 2104XP
FAN STATUS FAN STATUS
UCS 2104XP
FAN STATUS FAN STATUS
FAN 1
FAN 5
FAN 2
FAN 6
FAN 3
FAN 7
FAN 4
FAN 8
FAN STATUS
FAN STATUS
FAN STATUS
FAN STATUS
Qty 3: Cisco UCS 5108
CHS A56
X1938A
1
c a 0 0 c e 0 a
0 a N K L
0 b N K L
2
c b 0 0 d e 0 b
Qty 1: NetApp 3210A

X1938A
1
c a 0 0 c e 0 a
0 a N K L
0 b N K L
2
c b 0 0 d e 0 b
= Used FCoE Port = Used 1 GbE Port = Used SAS Port (Disk Shelf Connectivity)
= Used 10 GbE Port = Used FC Port

291693
NetApp FAS3210A Deployment ProcedurePart I

This section describes the procedures for configuring the NetApp FAS3210A for use in a FlexPod environment. This section has the following objectives:

Establishment of a functional Data ONTAP 8.0.1 failover cluster with proper licensing Creation of data aggregates Creation of flexible volumes Configuration of NFS exports if using NFS for infrastructure volumes
13
Creation of infrastructure vFiler unit Assign the storage controller disk wwnership. Ensure Data ONTAP 8.0.1 is installed.
The following measures should be taken to meet these objectives:

Note
Upgrade or downgrade to Data ONTAP 8.0.1 if necessary. Set up Data ONTAP8.0.1. Install Data ONTAP to the on-board flash storage. Install required licenses. Start FCP service and enable proper FC port configuration. Enable active-active configuration between the two storage systems. Create the data aggregate aggr1. Enable 802.1q VLAN trunking and add the NFS VLAN. Harden storage system logins and security. Create SNMP requests role and assign SNMP login privileges. Create SNMP management group and assign SNMP request role to it. Create SNMP user and assign to SNMP management group. Enable SNMP on the storage controllers. Delete SNMP v1 communities from storage controllers. Set SNMP contact information for each of the storage controllers. Set SNMP location information for each of the storage controllers. Establish SNMP trap destinations. Re-Initialize SNMP on the storage controllers. Enable FlashCache. Create the necessary infrastructure volumes (flexible volumes) for infrastructure services. Create the infrastructure IP space. Create the infrastructure vFiler units. Map the necessary infrastructure volumes to the infrastructure vFiler units. Set the priority levels for the volumes.
Cisco Nexus 5548 Deployment ProcedurePart I

This section describes the procedures for deploying the Cisco Nexus 5548 platforms for use in a FlexPod system and achieves the following objectives:

Establish a functional pair of Cisco Nexus 5548 switches with proper licensing and features enabled. Establish connectivity between FlexPod elements including the use of traditional and virtual port channels. Establish connectivity to existing data center infrastructure.
14
The following actions are necessary to configure the Cisco Nexus 5548 switches for use in a FlexPod environment.

Execute the Cisco Nexus 5548 setup script. Enable the appropriate Cisco Nexus features and licensing. Set global configurations. Create necessary VLANs including NFS and management. Add individual port descriptions for troubleshooting. Create necessary port channels including the vPC peer-link. Add Port Channel configurations. Configure virtual Port Channels (vPCs) to UCS fabric interconnects and NetApp controllers. Configure uplinks into existing network infrastructure, preferably by using vPC. Save the configuration.
Cisco Unified Computing System Deployment Procedure

This section provides the procedure for configuring the Cisco Unified Computing System for use in a FlexPod environment. This workflow should achieve the following goals:

Creates a functional Cisco UCS fabric cluster Creates the logical building blocks for UCS management model including MAC, WWNN, WWPN, UUID and server pools, vNIC and vHBA templates, and VLANs and VSANs via UCSM Defines policies enforcing inventory discovery, network control, and server boot rules via UCSM Creates Service Profile templates Instantiates Service Profiles by association templates to physical blades Execute the initial setup of the Cisco UCS 6100 Fabric Interconnects. Log into the Cisco UCS Manager via Web browser. Edit the Chassis Discovery Policy to reflect the number of links from the chassis to the fabric interconnects. Enable Fibre Channel Server and Uplink Ports. Create an Organization that manages the FlexPod infrastructure and owns the logical building blocks. Create MAC Address Pools under infrastructure organization. Create global VLANs, including NFS and OS data VLANs. Create a Network Control Policy under infrastructure Organization. Set Jumbo Frames in UCS Fabric. Create global VSANs. Create WWNN Pool under infrastructure Organization. Create WWPN Pools under infrastructure Organization. Create vNIC Template under infrastructure Organization using previously defined pools.
The following process should be followed for proper configuration.

15
Create vHBA Templates for Fabric A and B under infrastructure Organization. Create necessary Ethernet and SAN uplink Port-Channels to the Cisco Nexus 5548 Switches. Create WWNN Pool under infrastructure Organization. Create WWPN Pools under infrastructure Organization. Create global VSANs. Create vHBA Templates for Fabric A and B under infrastructure Organization. Create Boot Policies under infrastructure Organization. Create Server Pools under infrastructure Organization. Create UUID Suffix Pools under infrastructure Organization. Create Service Profile Templates under infrastructure Organization. Create Service Profiles under infrastructure Organization. Add a block of IP Addresses for KVM access. Backup the configuration of the running system, taking into consideration the backup location, the types of backup operations, the methods of backing up the configuration, and the need for scheduled backups.
Gather Necessary Information

Once the Cisco UCS Service Profiles have been created above, the infrastructure blades in the environment each have a unique configuration. To proceed with the FlexPod deployment, specific information must be gathered from each Cisco UCS blade as well as from the Netapp controllers. Table 3 and Table 4 detail the information that is needed for later use.
Table 3 NetApp FAS3210A FC Portname Information
NetApp FAS3210 A NetApp FAS3210 B
0c or 2a 0d or 2b 0c or 2a 0d or 2b
Note
On each NetApp controller use the show fcp adapters command to gather the above information.
Table 4
Cisco UCS Blade WWPN Information
Cisco UCS Service Profile Name
vHBA_A WWPN
vHBA_B WWPN
16
Cisco Nexus 5548 Deployment ProcedurePart II

This section describes the procedures for additional Fibre Channel functionality on the Cisco Nexus 5548 platforms within the FlexPod environment and achieves the following objectives:

Creates VSANs and VFCs, assigns FC ports to SAN Port Channels and appropriate VSANs, and turns on FC ports Defines Fibre Channel aliases for Service Profiles and NetApp controller target ports Establishes Fibre Channel Zoning and working sets Create VSANs for fabric A or B on respective Nexus platform. Create necessary SAN port channels to be connected to UCS Fabric Interconnect. Assign to VSAN appropriate FC interfaces or, alternatively for FCoE use, create vFC ports and map to the defined VSANs. Create device aliases on each Cisco Nexus 5548 for each service profile using corresponding fabric PWWN. Create device aliases on each Cisco Nexus 5548 for each service NetApp controller using corresponding fabric PWWN. Create zones for each service profile and assign devices as members by using Fibre Channel aliases. Activate the zoneset. Save the configuration. Backup the configuration of the running system, taking into consideration the backup location, the types of backup operations, the methods of backing up the configuration, and the need for scheduled backups
The following measures should be taken on each Nexus platform:

NetApp FAS3210A Deployment ProcedurePart II

This section describes additional procedures necessary on the NetApp controllers to provide UCS stateless boot functionality. At the end of this workflow the following objectives should be met:

Fibre Channel target ports defined Fibre Channel interface groups (iGroups) defined for each service profile Boot LUNs allocated for each Cisco UCS service profile Boot LUN mapped to associated Cisco UCS service profile Create the necessary volume for boot of the UCS hosts. Create LUNs for booting of the UCS hosts and house them within the newly created volume. Create any necessary iGroups. For those OSes that support ALUA, NetApp recommends enabling ALUA on the iGroups for the host. Map the newly created iGroups to their respective LUNs in a 1:1 fashion. Following the necessary zoning, LUN creation, and mapping, you can boot the UCS host.
The following process outlines the steps necessary:

17
AppendixFlexPod Configuration Information
Backup the configuration of the running system, taking into consideration the backup location, the types of backup operations, the methods of backing up the configuration, and the need for scheduled backups.
NetApp Operations Manager Deployment Procedure

There are a variety of choices available to manage the storage in a FlexPod environment, including NetApp FilerView, NetApp Virtual Storage Console (VSC), NetApp Operations Manager, and Microsoft Windows System Manager. The following section provides the general procedures for configuring NetApp Operations Manager, which is part of the DataFabric Manager (DFM) 4.0 suite for use in a FlexPod environment. After completing this section the following should be available:
A Microsoft Windows 2008 virtual machine (VM or bare metal) running NetApp DataFabric Manager Suite including:
Operations Manager Provisioning Manager Protection Manager
NetApp Operations Manager monitoring both FlexPod storage controllers
The following section provides the procedures for configuring NetApp Operations Manager for use in a FlexPod environment.
Install DFM on the same Windows virtual machine hosting the virtual storage controller through a Web browser (Windows).
Note
DFM is available at: http://now.netapp.com/NOW/download/software/dfm_win/Windows/. Generate a secure SSL key for the DFM HTTPs server. Enable HTTPs. Add a license in DFM server. Enable SNMP v3 configuration. Configure AutoSupportTM information. Run diagnostics to verify DFM communication with FlexPod controllers. Configure an SNMP trap host. Configure Operations Manager to generate e-mails for every Critical or higher event and send e-mails

The following tables outline the information tht must be available to complete the setup and deployment of a FlexPod system.
Global Configuration Information

This information is used throughout the deployment across multiple layers in the environment.
18
Table 5
FlexPod Global Configuration Information
Name VLAN ID for NFS traffic
Customized Value
Description Provide the appropriate VLAN ID used for NFS traffic throughout the FlexPod environment Provide the network address for NFS VLAN traffic in CIDR notation (that is, 192.168.30.0/24) Provide the appropriate VLAN ID used for Management traffic throughout the FlexPod environment Provide the appropriate VLAN ID that will be used for the native VLAN ID throughout the FlexPod environment. Provide the default password that will be used in initial configuration of the environment. NOTE: It is recommended to change this password as needed on each device once the initial configuration is complete. Provide the IP address of the appropriate nameserver for the environment. Provide the appropriate domain name suffix for the environment. The VSAN ID that will be associated with Fabric A. This will be associated with both FC and FCoE traffic for Fabric A. The VSAN ID that will be associated with Fabric B. This will be associated with both FC and FCoE traffic for Fabric B. Provide the VLAN ID of the VLAN that will be mapped to the FCoE traffic on fabric A. Provide the VLAN ID of the VLAN that will be mapped to the FCoE traffic on fabric B. Provide the appropriate SSL country name code. Provide the appropriate SSL state or province name. Provide the appropriate SSL locality name (city, town, etc.). Provide the appropriate SSL organization name (company name). Provide the appropriate SSL organizational unit (division).
Network address for NFS traffic
VLAN ID for Management traffic
VLAN ID for native VLAN
Default password
DNS/Nameserver name Domain name suffix VSAN ID for Fabric A
VSAN ID for Fabric B
FCoE VLAN ID for Fabric A FCoE VLAN ID for Fabric B SSL country name code SSL state or province name SSL locality name SSL organization name SSL organizational unit
19
NetApp Configuration Information

The information in Table 6 through Table 9 is specific to the NetApp portion of the deployment only.
Table 6 NetApp FAS3210A Configuration Information
Name FAS3210 A hostname FAS3210 B hostname Netboot interface name
Customized Value
Description Provide the hostname for NetApp FAS3210 A. Provide the hostname for NetApp FAS3210 B. Designate the appropriate interface to use for initial netboot of each controller. Interface e0M is the recommended interface. Provide the full TFTP path to the 8.0.1 Data ONTAP boot image. Provide the IP Address for the management interface on NetApp FAS3210 A Provide the IP Address for the management interface on NetApp FAS3210 B Provide the subnet mask for the management interface on NetApp FAS3210 A Provide the subnet mask for the management interface on NetApp FAS3210 B. Provide the gateway IP address for the management interface on NetApp FAS3210 A. Provide the gateway IP address for the service processor interface on NetApp FAS3210 B. Provide the IP address of the host that will be used for administering the NetApp FAS3210A. Provide a description of the physical location where the NetApp chassis resides. Provide the IP address for the service processor interface on NetApp FAS3210 A. Provide the IP address for the service processor interface on NetApp FAS3210 B. Provide the subnet mask for the service processor interface on NetApp FAS3210 A. Provide the subnet mask for the service processor interface on NetApp FAS3210 B. Provide the gateway IP address for the service processor interface on NetApp FAS3210 A.
NetApp Data ONTAP 8.0.1 Netboot kernel location NetApp FAS3210 A management interface IP address NetApp FAS3210 B management interface IP address NetApp FAS3210 A management interface subnet mask NetApp FAS3210 B management interface subnet mask NetApp FAS3210 A management interface gateway IP address NetApp FAS3210 B management interface gateway IP address NetApp FAS3210A administration host IP address NetApp FAS3210A location NetApp FAS3210 A service processor interface IP address NetApp FAS3210 B service processor interface IP address NetApp FAS3210 A service processor interface subnet mask NetApp FAS3210 B service processor interface subnet mask NetApp FAS3210 A service processor interface gateway IP address
20
Table 6
NetApp FAS3210A Configuration Information
Name NetApp FAS3210 B service processor interface gateway IP address NetApp FAS3210A Mailhost name NetApp FAS3210A Mailhost IP address NetApp DataONTAP 8.0.1 flash image location NetApp FAS3210A administrators e-mail address NetApp FAS3210A infrastructure vFiler IP address
Customized Value
Description Provide the gateway IP address for the service processor interface on NetApp FAS3210 B. Provide the appropriate Mailhost name. Provide the appropriate Mailhost IP address. Provide the http or https Web address of the NetApp Data ONTAP 8.0.1 flash image to install the image to the on-board flash storage. Provide the e-mail address for the NetApp administrator to receive important alerts/messages via e-mail. Provide the IP address for the infrastructure vFiler unit on FAS3210A. Note: This interface will be used for the export of NFS datastores and possibly iSCSI LUNs to the necessary ESXi hosts.
NetApp FAS3210A infrastructure vFiler administration host IP
Provide the IP address of the host that will be used to administer the infrastructure vFiler unit on FAS3210A. This variable might have the same IP address as the administration host IP address for the physical controllers as well. Provide the IP address for the infrastructure vFiler unit on FAS3210B. Keep in mind that this interface will be used for the export of NFS datastores and possibly iSCSI LUNs to the necessary ESXi hosts. Provide the IP address of the host that will be used to administer the infrastructure vFiler unit on FAS3210B. This variable might possibly have the same IP address as the administration host IP address for the physical controllers as well.
NetApp FAS3210B infrastructure vFiler IP address
NetApp FAS3210B infrastructure vFiler administration host IP address
Table 7
NetApp Licensing Configuration Information
Name NetApp Cluster license code NetApp Fibre Channel license code NetApp Flash Cache license code
Customized Value
Description Provide the license code to enable cluster mode within the FAS3210 A configuration. Provide the license code to enable the Fibre Channel protocol. Provide the license code to enable the installed Flash Cache adapter.
21
Table 7
NetApp Licensing Configuration Information
Name NetApp NearStore license code
Customized Value
Description Provide the license code to enable the NearStore capability which is required to enable deduplication. Provide the license code to enable deduplication. Provide the license code to enable the NFS protocol. Provide the license code to enable MultiStore. Provide the license code to enable FlexClone.
NetApp Deduplication license code NetApp NFS license code NetApp MultiStore license code NetApp FlexClone license code
Table 8
NetApp Disk and Volume Configuration Information
Name NetApp FAS3210 A total disks attached
Customized Value
Description Number of disks assigned to controller A using software ownership. NOTE: Do not include the three disks used for the root volume in this number. Number of disks assigned to controller B using software ownership. NOTE: Do not include the three disks used for the root volume in this number. Number of disks to be assigned to aggr1 on controller A. Number of disks to be assigned to aggr1 on controller B. Each UCS server will boot using the FC protocol. Each FC LUN will be stored in a volume on either controller A or controller B. Choose the appropriate volume size depending on the environment. Each UCS server will boot using the FC protocol. Each FC LUN will be stored in a volume on either controller A or controller B. Choose the appropriate volume size depending on the environment.
NetApp FAS3210 B total disks attached
NetApp FAS3210 A total disks in Aggregate 1 NetApp FAS3210 B total disks in Aggregate 1 NetApp FAS3210 A ESXi boot volume size
NetApp FAS3210 B ESXi boot volume size
Table 9
NetApp Data Fabric Manager Configuration Information
Name NetApp DFM server hostname NetApp DFM server IP address
Customized Value
Description Provide the hostname for the NetApp DFM server instance. Provide the IP address to be assigned to the NetApp DFM server.
22
Table 9
NetApp Data Fabric Manager Configuration Information
Name NetApp DFM server license key Mailhost IP address or hostname SNMP community string SNMP username SNMP password SNMP Traphost SNMP request role SNMP managers SNMP site name Enterprise SNMP trap destination
Customized Value
Description Provide the license key for the NetApp DFM server. Provide address of the mailhost that will be used to relay AutoSupport e-mails. Provide the appropriate SNMP community string. Provide the appropriate SNMP username. Provide the appropriate SNMP password. Provide the IP address or hostname for the SNMP traphost. Provides the request role for SNMP. Users who have the ability to manage SNMP. Provides the site name as required by SNMP. Provides the appropriate enterprise SNMP trap destination.
Cisco Configuration Information

The information in Table 10 and Table 11 is specific to the Cisco portion of the deployment only.
Table 10 Cisco Nexus 5548 Configuration Information
Name Cisco Nexus 5548 A hostname Cisco Nexus 5548 B hostname Cisco Nexus 5548 A Management Interface IP Address Cisco Nexus 5548 B Management Interface IP Address Cisco Nexus 5548 A Management Interface Subnet Mask Cisco Nexus 5548 B Management Interface Subnet Mask Cisco Nexus 5548 A Management Interface Gateway IP Address Cisco Nexus 5548 B Management Interface Gateway IP Address Cisco Nexus 5548 Virtual Port Channel (vPC) Domain ID
Customized Value
Description Provide the hostname for the Cisco Nexus 5548 A. Provide the hostname for the Cisco Nexus 5548 B. Provide the IP address for the mgmt0 interface on the Cisco Nexus 5548 A. Provide the IP address for the mgmt0 interface on the Cisco Nexus 5548 B. Provide the subnet mask for the mgmt0 interface on the Cisco Nexus 5548 A. Provide the subnet mask for the mgmt0 interface on the Cisco Nexus 5548 B. Provide the gateway IP address for the mgmt0 interface on the Cisco Nexus 5548 A. Provide the gateway IP address for the mgmt0 interface on the Cisco Nexus 5548 B. Provide a unique vpc domain id for the environment.
23
Table 11
Cisco Unified Computing System Configuration Information
Name Cisco UCS Fabric Interconnect A hostname Cisco UCS Fabric Interconnect B hostname Cisco UCS Name
Customized Value
Description Provide the hostname for Fabric Interconnect A. Provide the hostname for Fabric Interconnect B. Both Cisco UCS Fabric Interconnects will be clustered together as a single Cisco UCS. Provide the hostname for the clustered system. Both Cisco UCS Fabric Interconnects will be clustered together as a single Cisco UCS. Provide the IP address for the clustered system. Provide the IP address for Fabric Interconnect As Management Interface. Provide the IP address for Fabric Interconnect Bs Management Interface. Provide the subnet mask for Fabric Interconnect As Management Interface. Provide the subnet mask for Fabric Interconnect Bs Management Interface. Provide the gateway IP address for Fabric Interconnect As Management Interface. Provide the gateway IP address for Fabric Interconnect Bs Management Interface. A Cisco UCS organization will be created for the necessary Infrastructure resources. Provide a descriptive name for this organization. A pool of MAC addresses will be created for each fabric. Depending on the environment, certain MAC addresses may already be allocated. Identify a unique MAC address as the starting address in the MAC pool for Fabric A. It is recommended, if possible, to use either 0A or 0B as the second to last octet in order to distinguish from MACs on fabric A or fabric B. A pool of MAC addresses will be created for each fabric. Depending on the environment, certain MAC addresses may already be allocated. Identify a unique MAC address as the starting address in the MAC pool for Fabric B. It is recommended, if possible, to use either 0A or 0B as the second to last octet in order to more easily distinguish from MACs on fabric A or fabric B.
Cisco UCS IP
Cisco UCS Fabric Interconnect A Management Interface IP Address Cisco UCS Fabric Interconnect B Management Interface IP Address Cisco UCS Fabric Interconnect A Management Netmask Cisco UCS Fabric Interconnect B Management Interface Netmask Cisco UCS Fabric Interconnect A Management Interface Gateway Cisco UCS Fabric Interconnect B Management Interface Gateway Cisco UCS Infrastructure Organization Starting MAC Address for Fabric A
Starting MAC Address for Fabric B
24
Table 11
Cisco Unified Computing System Configuration Information
Name Starting WWPN for Fabric A
Customized Value
Description A pool of wwpns will be created for each fabric. Depending on the environment, certain wwpns may already be allocated. Identify a unique wwpn as the starting point in the wwpn pool for Fabric A. It is recommended, if possible, to use either 0A or 0B as the second to last octet in order to more easily distinguish from wwpns on fabric A or fabric B. A pool of wwpns will be created for each fabric. Depending on the environment, certain wwpns may already be allocated. Identify a unique wwpn as the starting point in the wwpn pool for Fabric B. It is recommended, if possible, to use either 0A or 0B as the second to last octet in order to more easily distinguish from wwpns on fabric A or fabric B.
Starting WWPN for Fabric B
NetApp FAS3200 Sample Configuration

Appliance Sample Interface Configuration
ntap3200-1a> ifconfig -a c0a: flags=0x354a867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE inet 192.168.1.85 netmask-or-prefix 0xffffff00 broadcast 192.168.1.255 ether 00:a0:98:13:d2:d0 (auto-unknown-enabling) flowcontrol full c0b: flags=0x3d4a867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 PRIVATE inet 192.168.2.135 netmask-or-prefix 0xffffff00 broadcast 192.168.2.255 ether 00:a0:98:13:d2:d1 (auto-10g_kr-fd-up) flowcontrol full e0M: flags=0x694c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 1500 inet 10.61.185.144 netmask-or-prefix 0xffffff00 broadcast 10.61.185.255 partner e0M (not in use) ether 00:a0:98:13:d2:d2 (auto-100tx-fd-up) flowcontrol full e0P: flags=0x2d4c867<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 inet 192.168.2.48 netmask-or-prefix 0xfffffc00 broadcast 192.168.3.255 noddns ether 00:a0:98:13:d2:d3 (auto-100tx-fd-up) flowcontrol full e0a: flags=0x250c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 ether 00:a0:98:13:d2:ce (auto-unknown-cfg_down) flowcontrol full e0b: flags=0x250c866<BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 1500 ether 00:a0:98:13:d2:cf (auto-unknown-cfg_down) flowcontrol full e2a: flags=0x8bd0a867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000 ether 02:a0:98:13:d2:d0 (auto-10g_sr-fd-up) flowcontrol full trunked vif0 e2b: flags=0x8bd0a867<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000 ether 02:a0:98:13:d2:d0 (auto-10g_sr-fd-up) flowcontrol full trunked vif0 lo: flags=0x1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160 inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1 ether 00:00:00:00:00:00 (RNIC Provider) vif0: flags=0xa3d0a863<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000 ether 02:a0:98:13:d2:d0 (Enabled virtual interface) vif0-900: flags=0x394a863<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM> mtu 9000 inet 192.168.90.144 netmask-or-prefix 0xffffff00 broadcast 192.168.90.255 partner vif0-900 (not in use)
25
ether 02:a0:98:13:d2:d0 (Enabled virtual interface)
Sample Startup Information Configuration

ntap3200-1a> rdfile /etc/rc hostname ntap3200-1a vif create lacp vif0 -b ip e1a e1b vlan create vif0 3150 900 ifconfig e0M `hostname`-e0M netmask 255.255.255.0 mtusize 1500 -wins flowcontrol full partner e0M route add default 10.61.185.1 1 routed on options dns.domainname rtp.netapp.com options dns.enable on options nis.enable off savecore vlan create vif0 900 ifconfig vif0-900 mtusize 9000 ifconfig vif0-900 partner vif0-900 ifconfig vif0-900 192.168.90.144 netmask 255.255.255.0 vlan add vif0 3150 ifconfig vif0-3150 `hostname`-vif0-3150 netmask 255.255.255.0 mtusize 1500 -wins partner vif0-3150 ifconfig vif0-3150 192.168.150.1 netmask 255.255.255.0
Sample Volume Information

ntap3200-1a> vol status Volume State infrastructure_root online Status Options raid_dp, flex guarantee=none, fractional_reserve=0 vol0 online raid_dp, flex root infrastructure_datastore_1 online raid_dp, flex guarantee=none, sis fractional_reserve=0 esxi_boot_A online raid_dp, flex guarantee=none, sis fractional_reserve=0
Sample LUN Information

ntap3200-1a> lun show -m LUN path Mapped to LUN ID Protocol ----------------------------------------------------------------------/vol/esxi_boot_A/ucs2b-1-sc ucs2b-1-sc_A 0 FCP ucs2b-1-sc_B 0 FCP
Sample Initiator Group (iGroup) Information

ntap3200-1a> igroup show ucs2b-1-sc_A (FCP) (ostype: 20:00:00:25:b5:00:0a:9f ucs2b-1-sc_B (FCP) (ostype: 20:00:00:25:b5:00:0b:df "Hypervisor or (logged in on: "Hypervisor or (logged in on: Bare Metal": 0c) Bare Metal"): 0d)
Sample vFiler Unit Structure

ntap3200-1a> vfiler status vfiler0 running
26
infrastructure_1_vfiler
running
Sample List of Defined ipspaces and Interface Assignment

ntap3200-1a> ipspace list Number of ipspaces configured: 3 default-ipspace infrastructure (e0M e0P e0a e0b ) (vif0-900 )
Sample vFiler Unit Context Route Configuration

infrastructure_1_vfiler@ntap3200-1a> route -s Routing tables Internet: Destination 192.168.90 192.168.90.109 192.168.90.110 192.168.90.111 192.168.90.112 192.168.90.117
Gateway link#12 0:50:56:70:f8:9a 0:50:56:77:8a:ac 0:50:56:70:c0:80 0:50:56:7b:df:f9 0:50:56:a0:0:0
Flags UC UHL UHL UHL UHL UHL
Refs 0 2 2 2 2 0
Use 0 409 5181 9 9 18
Interface vif0-900 vif0-900 vif0-900 vif0-900 vif0-900 vif0-900
Sample vFiler Unit Context Exported Directories and Files

infrastructure_1_vfiler@ntap3200-1a> exportfs /vol/infrastructure_datastore_1-sec=sys,rw=192.168.90.109:192.168.90.110:192.168.90.111:19 2.168.90.112:192.168.95.10,root=192.168.90.109:192.168.90.110:192.168.90.111:192.168.90.11 2:192.168.95.10 /vol/infrastructure_root-sec=sys,rw,anon=0
Cisco Nexus 5548 Sample Running Configuration

!Command: show running-config !Time: Wed Aug 10 11:35:26 2011 version 5.0(3)N1(1c) feature fcoe feature npiv feature fport-channel-trunk no feature telnet no telnet server enable cfs eth distribute feature lacp feature vpc feature lldp username admin password 5 $1$JpsspOIX$jk0ujNUh8cxcm2b7lOEaG0 ip domain-lookup switchname ice5548-1 system jumbomtu 9000 logging event link-status default ip access-list classify_COS_4 10 permit ip 192.168.102.0/24 any 20 permit ip any 192.168.102.0/24 ip access-list classify_COS_5 10 permit ip 192.168.101.0/24 any 20 permit ip any 192.168.101.0/24 class-map type qos class-fcoe
role network-admin
27
class-map type qos match-all Silver_Traffic match access-group name classify_COS_4 class-map type qos match-all Platinum_Traffic match access-group name classify_COS_5 class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-all-flood match qos-group 2 class-map type queuing class-ip-multicast match qos-group 2 policy-map type qos Global_Classify class Platinum_Traffic set qos-group 2 class Silver_Traffic set qos-group 4 class class-fcoe set qos-group 1 class-map type network-qos class-fcoe match qos-group 1 class-map type network-qos class-all-flood match qos-group 2 class-map type network-qos Silver_Traffic_NQ match qos-group 4 class-map type network-qos class-ip-multicast match qos-group 2 class-map type network-qos Platinum_Traffic_NQ match qos-group 2 policy-map type network-qos Setup_QOS class type network-qos Platinum_Traffic_NQ set cos 5 mtu 9000 class type network-qos Silver_Traffic_NQ set cos 4 mtu 9000 class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default multicast-optimize system qos service-policy type qos input Global_Classify service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type network-qos Setup_QOS snmp-server user admin network-admin auth md5 0x91d2518e00e2d50e9e5d213bee818692 priv 0x91d2518e00e2d50e9e5d213bee818692 localizedkey snmp-server enable traps entity fru ntp server 10.61.185.11 use-vrf management vrf context management ip route 0.0.0.0/0 10.61.185.1 vlan 1 vlan 101 fcoe vsan 101 name FCoE_Fabric_A vlan 186 name MGMT-VLAN vlan 3101 name NFS-VLAN vlan 3102 name vMotion-VLAN vlan 3103 name Packet-Control-VLAN vlan 3104 name VM-Traffic-VLAN
28
spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default spanning-tree port type network default vpc domain 23 role priority 10 peer-keepalive destination 10.61.185.70 source 10.61.185.69 vsan database vsan 101 name "Fabric_A" device-alias database device-alias name ice3270-1a_2a pwwn 50:0a:09:81:8d:dd:92:bc device-alias name ice3270-1b_2a pwwn 50:0a:09:81:9d:dd:92:bc device-alias name esxi41_host_ice3270-1a_2a1_A pwwn 20:00:00:25:b5:00:0a:0f device-alias name esxi41_host_ice3270-1b_2b1_A pwwn 20:00:00:25:b5:00:0a:1f device-alias commit fcdomain fcid database vsan 1 wwn 20:42:00:05:9b:79:7a:80 fcid 0x800000 dynamic vsan 1 wwn 20:41:00:05:9b:79:7a:80 fcid 0x800001 dynamic vsan 101 wwn 50:0a:09:81:8d:dd:92:bc fcid 0x4e0000 dynamic ! [ice3270-1a_2a] vsan 101 wwn 50:0a:09:81:9d:dd:92:bc fcid 0x4e0001 dynamic ! [ice3270-1b_2a] vsan 101 wwn 20:42:00:05:9b:79:7a:80 fcid 0x4e0002 dynamic vsan 101 wwn 20:41:00:05:9b:79:7a:80 fcid 0x4e0003 dynamic vsan 101 wwn 20:00:00:25:b5:00:0a:0f fcid 0x4e0004 dynamic ! [esxi41_host_ice3270-1a_2a1_A] vsan 101 wwn 20:00:00:25:b5:00:0a:1f fcid 0x4e0005 dynamic ! [esxi41_host_ice3270-1b_2b1_A]
interface san-port-channel 1 channel mode active interface port-channel10 description vPC peer-link switchport mode trunk vpc peer-link switchport trunk native vlan 2 switchport trunk allowed vlan 186,3101-3104 spanning-tree port type network interface port-channel11 description ice3270-1a switchport mode trunk vpc 11 switchport trunk native vlan 2 switchport trunk allowed vlan 101-102,186,3101 spanning-tree port type edge trunk interface port-channel12 description ice3270-1b switchport mode trunk vpc 12 switchport trunk native vlan 2 switchport trunk allowed vlan 101-102,186,3101 spanning-tree port type edge trunk interface port-channel13 description iceucsm-2a-m switchport mode trunk vpc 13 switchport trunk native vlan 2 switchport trunk allowed vlan 186,3101-3104
29
spanning-tree port type edge trunk interface port-channel14 description iceucsm-2b-m switchport mode trunk vpc 14 switchport trunk native vlan 2 switchport trunk allowed vlan 186,3101-3104 spanning-tree port type edge trunk interface port-channel20 description Po20:iceds-1:Po12 switchport mode trunk vpc 20 switchport trunk native vlan 2 switchport trunk allowed vlan 186 spanning-tree port type !Command: show running-config !Time: Wed Aug 10 11:36:57 2011 version 5.0(3)N1(1c) feature fcoe feature npiv feature fport-channel-trunk no feature telnet no telnet server enable cfs eth distribute feature lacp feature vpc feature lldp username admin password 5 $1$PBq/n2.b$g8jK3jqj8MelNDQKGRBD50 ip domain-lookup switchname ice5548-2 system jumbomtu 9000 logging event link-status default ip access-list classify_COS_4 10 permit ip 192.168.102.0/24 any 20 permit ip any 192.168.102.0/24 ip access-list classify_COS_5 10 permit ip 192.168.101.0/24 any 20 permit ip any 192.168.101.0/24 class-map type qos class-fcoe class-map type qos match-all Silver_Traffic match access-group name classify_COS_4 class-map type qos match-all Platinum_Traffic match access-group name classify_COS_5 class-map type queuing class-fcoe match qos-group 1 class-map type queuing class-all-flood match qos-group 2 class-map type queuing class-ip-multicast match qos-group 2 policy-map type qos Global_Classify class Platinum_Traffic set qos-group 2 class Silver_Traffic set qos-group 4 class class-fcoe set qos-group 1 class-map type network-qos class-fcoe match qos-group 1 class-map type network-qos class-all-flood match qos-group 2
role network-admin
30
class-map type network-qos Silver_Traffic_NQ match qos-group 4 class-map type network-qos class-ip-multicast match qos-group 2 class-map type network-qos Platinum_Traffic_NQ match qos-group 2 policy-map type network-qos Setup_QOS class type network-qos Platinum_Traffic_NQ set cos 5 mtu 9000 class type network-qos Silver_Traffic_NQ set cos 4 mtu 9000 class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default multicast-optimize system qos service-policy type qos input Global_Classify service-policy type queuing input fcoe-default-in-policy service-policy type queuing output fcoe-default-out-policy service-policy type network-qos Setup_QOS snmp-server user admin network-admin auth md5 0x7021e5331f25b481ed3ad26b96ccd729 priv 0x7021e5331f25b481ed3ad26b96ccd729 localizedkey snmp-server enable traps entity fru ntp server 10.61.185.11 use-vrf management vrf context management ip route 0.0.0.0/0 10.61.185.1 vlan 1 vlan 102 fcoe vsan 102 name FCoE_Fabric_B vlan 186 name MGMT-VLAN vlan 3101 name NFS-VLAN vlan 3102 name vMotion-VLAN vlan 3103 name Packet-Control-VLAN vlan 3104 name VM-Traffic-VLAN spanning-tree port type edge bpduguard default spanning-tree port type edge bpdufilter default spanning-tree port type network default vpc domain 23 role priority 20 peer-keepalive destination 10.61.185.69 source 10.61.185.70 vsan database vsan 102 name "Fabric_B" device-alias database device-alias name ice3270-1a_2b pwwn 50:0a:09:82:8d:dd:92:bc device-alias name ice3270-1b_2b pwwn 50:0a:09:82:9d:dd:92:bc device-alias name esxi41_host_ice3270-1a_2a1_B pwwn 20:00:00:25:b5:00:0b:0f device-alias name esxi41_host_ice3270-1b_2b1_B pwwn 20:00:00:25:b5:00:0b:1f device-alias commit fcdomain fcid database vsan 1 wwn 20:42:00:05:9b:6f:7a:40 fcid 0x590000 dynamic vsan 1 wwn 20:41:00:05:9b:6f:7a:40 fcid 0x590001 dynamic vsan 102 wwn 50:0a:09:82:9d:dd:92:bc fcid 0xae0000 dynamic ! [ice3270-1b_2b]
31
vsan 102 wwn 50:0a:09:82:8d:dd:92:bc fcid 0xae0001 [ice3270-1a_2b] vsan 102 wwn 20:42:00:05:9b:6f:7a:40 fcid 0xae0002 vsan 102 wwn 20:41:00:05:9b:6f:7a:40 fcid 0xae0003 vsan 102 wwn 20:00:00:25:b5:00:0b:0f fcid 0xae0004 ! [esxi41_host_ice3270-1a_2a1_A] vsan 102 wwn 20:00:00:25:b5:00:0b:1f fcid 0xae0005 ! [esxi41_host_ice3270-1b_2b1_A] !
dynamic dynamic dynamic dynamic dynamic
interface san-port-channel 2 channel mode active interface port-channel10 description vPC peer-link switchport mode trunk vpc peer-link switchport trunk native vlan 2 switchport trunk allowed vlan 186,3101-3104 spanning-tree port type network interface port-channel11 description ice3270-1a switchport mode trunk vpc 11 switchport trunk native vlan 2 switchport trunk allowed vlan 101-102,186,3101 spanning-tree port type edge trunk interface port-channel12 description ice3270-1b switchport mode trunk vpc 12 switchport trunk native vlan 2 switchport trunk allowed vlan 101-102,186,3101 spanning-tree port type edge trunk interface port-channel13 description iceucsm-2a-m switchport mode trunk vpc 13 switchport trunk native vlan 2 switchport trunk allowed vlan 186,3101-3104 spanning-tree port type edge trunk interface port-channel14 description iceucsm-2b-m switchport mode trunk vpc 14 switchport trunk native vlan 2 switchport trunk allowed vlan 186,3101-3104 spanning-tree port type edge trunk interface port-channel20 description Po20:iceds-1:Po12 switchport mode trunk vpc 20 switchport trunk native vlan 2 switchport trunk allowed vlan 186 spanning-tree port type net
32
Cisco Unified Computing System Configuration Extracts

All configurations in this section occur after the initial UCS cluster setup scripts have completed and the UCS Manager is accessible to the administrator. Use the configuration information described above to execute the setup script and complete the deployment required in your environment. For more information on the initial setup of Cisco UCS Manager, go to: http://www.cisco.com/en/US/products/ps10281/products_installation_and_configuration_guides_list.h tml and select the appropriate release of the System Configuration documentation.
Cisco Unified Computing System Configuration Examples and TechNotes

The latest Cisco UCS configuration examples and TechNotes may be found at: http://www.cisco.com/en/US/products/ps10281/prod_configuration_examples_list.html.
Sample Cisco Unified Computing System Initial CLI 1.4

The following video provides an example of the initial configuration of UCS from the command line interface: http://www.youtube.com/watch?v=86H_4lOeXfA&feature=related
Sample Chassis Discovery Policy Configuration

Define the Chassis Discovery Policy to reflect the number of links from the chassis to the fabric interconnects. At a minimum FlexPod requires two links.
Figure 4 Chassis Discovery Policy Screen
Define and enable Fibre Channel, Server, and Uplink Ports.
33
Figure 5
Fibre Channel Server and Uplink Ports Screen
The physical display after completing this procedure is shown in Figure 6.

Figure 6 Physical Display after Procedure Completion
Create an Organization
The use of organizations allows the physical UCS resources to be logically divided. Each organization can have its own policies, pools, and quality of service definitions. Organizations are hierarchical in nature, allowing sub-organizations to inherit characteristics from higher organizations or establish their policies, pools, and service definitions. To create an Organization, go to the Main panel New menu drop-down list and select Create Organization to create an organization which manages the FlexPod infrastructure and owns the logical building blocks.
34
Figure 7
Create Organization Screen
Cisco Unified Computing SystemWorking with Pools, Policies, and Templates

The following video provides examples and instructions for working with pools, policies, and templates for the Cisco Unified Computing System: http://www.youtube.com/watch?v=obmenIF4ggU&feature=related. This UCS video covers the following topics.
Pools
Create MAC Address Create vNIC Create VLANs Create VSANs Create WWPNs Servers WWNN UUID
Templates
vNIC vHBA Service Profile
Polices
QoS
35
References
Network Control Pin Group Boot Power Control Firmware BIOS Adapter
FC Port Channel and Trunking

The following video provides instructions and examples for setting up Fiber Channel port channel and trunking: http://www.youtube.com/watch?v=PpzKPguRTXc&feature=related.
Cisco Unified Computing System Block of IP Addresses for KVM Access

The following video provides instructions and examples for setting up the KVM address pool: http://www.youtube.com/watch?v=d0KTYItdU6g&NR=1.
References

Cisco Nexus 5548 Switch: http://www.cisco.com/en/US/products/ps11215/index.html Cisco Unified Computing System: http://www.cisco.com/en/US/netsol/ns944/index.html NetApp FAS3210 Storage Controller: http://now.netapp.com/NOW/knowledge/docs/hardware/hardware_index.shtml#Storage%20applia nces%20and%20V-series%20systems/gFilers NetApp Support (formerly NetApp on the Web (NOW) site: http://.now.netapp.com
36

Flexpod Deploy

Uploaded by

Copyright:

Available Formats

Flexpod Deploy

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Flexpod Deploy

Uploaded by

Copyright:

Available Formats

FlexPod Deployment Guide

Last Updated: February 23, 2012

Building Architectures to Solve Business Problems

Cisco Validated Design

About the Authors

About the Authors

David Antkowiak, Solutions Architect, Systems Development Unit, Cisco Systems

Ramesh Isaac, Technical Marketing Engineer, Systems Development Unit, Cisco

Jon Benedict, Reference Architect, Infrastructure and Cloud Engineering, NetApp

Chris Reno, Reference Architect, Infrastructure and Cloud Engineering, NetApp

Cisco Validated Design

About Cisco Validated Design (CVD) Program

About Cisco Validated Design (CVD) Program

FlexPod Deployment Guide

NetApp partners may access additional information at: https://fieldportal.netapp.com/.

Copyright 2011 Cisco Systems, Inc. All rights reserved.

FlexPod Deployment Guide

FlexPod FCoE and 10 GbE Based Architecture

CISCO NEXUS N5548P

1 /10 GIGABIT ET HERNET

1 /2/ 4/8 G F IBRE CHANNEL

CISCO NEXUS N5548P

1 /10 GIGABIT ET HERNET

1 /2/ 4/8 G F IBRE CHANNEL

Qty 2: Cisco Nexus 5548 vPC vPC

1/ 2/4 G F IBRE CHANNEL

1/ 2/4 G F IBRE CHANNEL

Unified Computing System

Cisco UCS 6120XP

Cisco UCS 6120XP

Storage FCoE +2 x 10 GbE

FlexPod Deployment Guide

FCoE 2 x 13 GbE NetApp FAS3210 HA Pair w/Qty. 4 DS2246 Disk Shelves

FlexPod FC and 10 GbE Based Architecture

CISCO NEXUS N5548P

1 /10 GIGABIT ET HERNET

1 /2/ 4/8 G F IBRE CHANNEL

CISCO NEXUS N5548P

1 /10 GIGABIT ET HERNET

1 /2/ 4/8 G F IBRE CHANNEL

Qty 2: Cisco Nexus 5548 vPC vPC

1/ 2/4 G F IBRE CHANNEL

1/ 2/4 G F IBRE CHANNEL

Unified Computing System

Cisco UCS 6120XP

Cisco UCS 6120XP

Storage Ether Channel: 2 x 10 GbE

Fiber Channel: 2 x 4 Gbps

Fiber Channel: 2 x 4 Gbps

FlexPod Deployment Guide

FlexPod Configuration Deployment

FlexPod Configuration Deployment

FlexPod Deployment Guide

FlexPod Configuration Deployment

FlexPod Ethernet Cabling Information

Local Device Cisco Nexus 5548 A

Local Port Eth1/1 Eth1/2 Eth1/5 Eth1/6 Eth1/7 Eth1/8

FlexPod Deployment Guide

FlexPod Configuration Deployment

FlexPod Ethernet Cabling Information