DCI - Project Layer2 and Vmotion
DCI - Project Layer2 and Vmotion
DCI - Project Layer2 and Vmotion
A successful application migration through VMware VMotion heavily relies on the underlying network
infrastructure. Therefore it is extremely important that IP network be resilient, robust, and highly
available.. The IP network becomes more important when the applications have to be mobile across
data centers. Cisco has been the industry leader in IP network and routing technologies and has
®
been providing data center IP network extensions since the 1980s. Cisco switching and routing
technologies provide the robust and redundant network that is essential for VMware VMotion to
succeed.
This document discusses the VMware VMotion feature and Cisco networking technologies essential
for application mobility across data centers.
The application mobility discussed in this document provides the foundation necessary to enable
cloud computing—for example, cloud import and export—providing the flexibility to move virtual
machines into the cloud from an enterprise data center, to move them between different clouds, and
to move them back into the enterprise data center.
Challenges
Figure 2. Infrastructure Challenges to Vmware Vmotion Across Data Centers
Facilitating VMware VMotion migration across physical infrastructure boundaries and data centers
using the WAN poses specific challenges for the data center network and storage designs as shown
in Figure 2. In particular, the LAN and SAN designs have to be addressed.
● LAN (Layer 2 Domain Elasticity): The LAN must be extended across the physical locations
or data centers without compromising the availability, resilience, and security that exists
within a single physical location or data center. The best practices for the current network
In addition, WAN characteristics such as available bandwidth for VMware VMotion and SAN
extension and its latency are crucial factors in the eventual success of application mobility.
Solution Options
The challenges that arise from migrating applications across data centers are addressed by the joint
VMware and Cisco solution. The solution uses best practices for the LAN, data network, and WAN to
overcome the challenges and provide the IT staff with a tool for migrating applications between data
centers without downtime.
The solution can be designed in multiple ways based on the topologies used for the LAN and
storage extensions, as shown in Table 1.
If storage replication were in place between the data centers, the volume(s) containing the virtual
machine data could be readily available in real time at the secondary data center. The existing
Active-Passive storage replication techniques require a set of explicit control operations to make the
storage replica accessible to the servers in the secondary data center. Consequently, at present, this
is not a supported technology to perform virtual machine vMotion.
● High availability: The solution must help ensure that no link or device failure will cause the
Layer 2 extension to be disconnected and thus affect the ability to perform VMware VMotion
migration across the data center.
● Load balancing: The solution should fully utilize cross-sectional bandwidth between the data
centers; DCI connections are usually more expensive than the LAN, making effective use of
the available connection critical.
● Spanning Tree Protocol isolation and loop and broadcast storm prevention: The
solution must fully contain and isolate Spanning Tree Protocol within each data center with
Bridge Protocol data units (BPDUs) filtered at the boundary of each edge switch facing the
core. Network loop and broadcast storm avoidance features need to be available to prevent
disruption of applications.
● Scalability: The solution must be able to handle multiple VMware VMotion migrations
concurrently. To meet this requirement, the network must be able to scale. The available
bandwidth, the number of VLANs, and the number of data centers connected through the
solution should all be capable of expansion as needed.
● Encryption: IP Security (IPsec) or Layer 2 link encryption (IEEE 802.1AE) may be needed to
help ensure the privacy and confidentiality of the data traversing between the data centers.
This requirement is particularly important if the interconnection crosses a public network.
Encryption may also be a requirement for regulatory compliance.
● Hierarchical quality of service (HQoS): HQoS may be required to help ensure quality of
service (QoS) for VMware VMotion, particularly on WAN links acquired through a service
provider. HQoS is important on interconnecting devices when an enterprise subscribes to a
subrate service provider service.
Table 2. Cisco LAN Extension Solutions and Platforms for Various Transport Options
Multiprotocol Label Ethernet over MPLS (two data ● Cisco Catalyst 6500 Series Shared
Switching (MPLS) centers) Port Adapter (SPA) Interface
Processor 400 (SIP-400) and SIP-
600
● Cisco ASR 1000 Series
Aggregation Services Routers
VPLS over GRE (multiple data ● Cisco Catalyst 6500 Series SIP-
centers) 400
Additional services such as encryption and HQoS can be implemented on the Cisco hardware listed
in Table 3.
Table 3. Cisco Solutions and Platforms for Additional Services for LAN Extension Schedules
Multilevel QoS HQoS ● Cisco Catalyst 6500 Series SIP-400 and SIP-600
● Cisco ASR 1000 Series
Virtual SAN Isolation and security The VSAN technology provides secure hardware-based
network segmentation, similar to the VLAN technology that is
widely deployed in LANs. Fabric services such as zoning and
routing are independent per VSAN.
Management and Cisco MDS 9000 NX-OS Software management offers several
access control levels of role-based access control (RBAC). This feature
allows an administrator to be in charge of a specific VSAN
without having any visibility into other VSANs.
Inter-VSAN Isolation and security In a DCI solution, each data center can implement
Routing independent VSANs, preserving the fabric services
(IVR) segmentation, data isolation, and administration
independence. IVR allows selected devices from different
VSANs, even across different data centers, to communicate
without any fabric merging.
SAN Integrated solution The capability to plug long-wave and Coarse Wavelength
SAN Integrated solution Cisco MDS 9000 Series Switches provide Gigabit Ethernet
extension interfaces and support the FCIP protocol, to transparently
with FCIP extend the SAN over an IP network.
Port Availability Cisco MDS 9000 Series PortChannels are the aggregation of
channeling multiple physical Fibre Channel or FCIP links into one logical
link, to provide higher aggregated bandwidth, load balancing,
and link redundancy.
I/O Application IOA is an intelligent distributed fabric service built into Cisco
acceleration performances MDS 9000 Series Switches. IOA accelerates I/O performance
(IOA) across distances. This feature helps the overall application
performance remain relatively the same, even when the
application server and the storage are separated by
considerable distance. In this validated solution, I/O
performance has been enhanced over the FCIP link.
All the features listed in Table 4 make a Cisco MDS 9000 Family SAN resilient and highly available.
More information about the Cisco MDS 9000 Series Switches can be obtained from
http://www.cisco.com/en/US/products/hw/ps4159/ps4358/prod_white_papers_list.html.
The solutions described here have been jointly validated in the VMware and Cisco Joint Solutions
Lab. The validated topology used for the testing is shown in Figure 3.
The network topology used in the joint solution test simulates two data centers extended over
different types of DCIs. Data center 1 is a Cisco best-practices three-tier architecture with Cisco
Catalyst 6500 Series Switches forming the core and aggregation layers and a pair of Cisco Catalyst
4900M Switches forming the access layer. VSS technology is used to provide the extended (or
stretched) VLAN. Data center 2 is also designed using the Cisco three-tier architecture, but with the
Cisco Nexus 7000 Series Switches and the Nexus 5000 Series Switches. The functional architecture
at both data centers is similar, with VSS and vPCs used to provide the extended VLAN.
The storage for the solution is provisioned using either of two methods depending on the test being
performed:
● Shared storage: Storage is located in data center 1, and the SAN is extended to data center
2 using FCIP SAN extension. FCIP IOA is enabled to help ensure that the application
performance does not suffer when the application is accessing its storage across the DCI
switches.
● Active-passive storage using VMware Storage VMotion: Storage is provisioned at both
locations. The storage capacity is provisioned identically at both data centers, and the
storage is presented to the VMware ESX servers as unique data stores. VMware Storage
VMotion migration is performed from one data store to the other; later, the VMware VMotion
virtual machine migration is performed. The storage must be available at all times to both the
source and destination VMware vSphere servers for VMware VMotion migration to be
successful. The SAN extension method used is again FCIP with IOA enabled.
Table 5. Cisco and Vmware Vmotion Migration Across Data Center: Test Topology Summary
Microsoft SQL CPU: 4 virtual Dell DVD Store Orders per minute The DVD Store
Server 2005 CPUs (vCPUs) open source (OPM) benchmark is an
(64-bit) benchmark online transaction
Memory: 8 GB
processing
Storage: (OLTP)
Approximately 300 benchmark that
GB (EMC simulates the
CLARiiON) operation of a
DVD store.
OS: Microsoft Performance is
2008 64-bit server measured in
OPMs, indicating
the number of
orders
successfully
inserted into the
database per
minute.
Test Methodology
Test Results
The goal of the joint testing is to measure the duration of the overall VMware VMotion migration
process during VMware VMotion migration. The tests measure the time taken for the overall VMware
VMotion migration to be complete, beginning from the initiation of the VMware VMotion process. The
overall migration time is an important measure, and it becomes critical when multiple VMware
VMotion migrations are being performed. The amount of time for the overall migration depends on
the duration of each VMware VMotion migration. The duration of a VMware VMotion migration
largely depends on the distance between the source and destination VMware ESX servers and the
amount of bandwidth available between the data centers.
The application used to validate the solution is an e-commerce suite hosted on a Microsoft SQL
Server 2005. DVD Store Version 2 (DS2) is a complete online e-commerce test application with a
back-end database component, a web application layer, and driver programs. The virtual machine
hosting the back-end Microsoft SQL Server database is migrated across the data centers, and the
performance of the application in OPMs is captured.
Figure 4 shows the VMware VMotion migration times as a virtual machine is migrated from one
VMware ESX server to another, with the servers separated by different distances. The elapsed time
increases with distance, but it is a directly related to the network latency and the amount of network
bandwidth available for VMware VMotion as the distance increases. In the test scenario, the
application client maintained all the sessions, and a momentary drop in performance was observed
before the performance returned to steady-state values.
The performance of the application when the back-end database is moved away from the users to a
data center 200 km away is shown in Figure 5. The results show that the migration to a remote data
center is feasible not only in terms of the mobility of the application but also from a business
perspective since degradation is less than 15 percent. Shared storage for the application therefore is
a viable option; with advanced SAN extension technologies such as FCIP and IOA, application
performance is maintained well within the SLA of the application.
The performance graph shows the performance of the application within the local data center, in
which the storage is located. The graph then shows the performance of the application with
increasing distance as it migrates to the remote data center, demonstrating that performance is
within the range of the SLA, which makes the SAN topology of shared storage for application
workload balancing a viable option.
In addition to migrating one mission-critical application across the distance of 200 km, the joint
testing migrated more applications in the data center to validate evacuation of the data center for
disaster avoidance (Figure 6). The joint testing included a Microsoft Exchange Server 2007 with
1000 Microsoft Outlook 2007 LoadGen users. The application performance monitored by LoadGen is
shown in Figure 7. The Microsoft Exchange Server is a four-vCPU virtual machine with 8 GB of
memory; resource utilization was 80 to 90 percent for the CPU, with approximately 20 Mbps of data
being read and written to the disks. Simultaneously, the Microsoft SQL Server running the DVD
Store database was also migrated. The OPM values for the DVD Store application are shown in
Figure 5. Figure 6 shows the migration times when the two applications are migrated simultaneously.
The results clearly show that the elapsed time for the Microsoft SQL Server increases by a small
amount, which is acceptable since the VMware VMotion network is now being shared with an
extremely busy Microsoft Exchange Server workload. In spite of this increase, there was no
perceivable effect on the clients performing the benchmark.
Conclusion
VMware VMotion enables data centers to transparently implement virtual machine mobility using the
Cisco LAN and storage extension solutions. The VMware vSphere Virtual Data Center Operating
System (vDCOS) with the suite of features bundled with vSphere allows customers to transparently
migrate or evacuate data center applications with no downtime from a user perspective. This
technological capability gives IT departments tools to redefine the business continuance and disaster
recovery plans of the enterprise. The need for a more complex and expensive solution to meet the
recovery-point objective (RPO) of the business continuance plan can be reduced with the use of
VMware VMotion. The optimal architecture of the underlying transport infrastructure—the IP network
and SAN—enhances the solution to bring the RPO to near zero, and also reduces the recovery time
objective (RTO) to a very small number. The joint Cisco and VMware solution gives IT departments a
very powerful tool for better provisioning, utilizing, and maintaining a virtualized data center with
resources spread across multiple physical locations.
Copyright © 2009. VMware, Inc. All rights reserved. Protected by one or more U.S. Patent Nos. 6,397,242, 6,496,847, 6,704,925, 6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966,
6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377, 7,111,086, 7,111,145, 7,117,481, 7,149, 843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136,
7,277,998,7,277,999, 7,278,030, 7,281,102, 7,290,253, 7,356,679 and patents pending.
Cisco, the Cisco logo, and Cisco Systems are registered trademarks or trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks
mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.