White Paper Symmetrix Virtual Provisioning

Implementing Virtual Provisioning on
EMC Symmetrix with

VMware Infrastructure 3
Applied Technology
Abstract
This white paper provides a detailed description of the technical aspects and benefits of deploying VMware
Infrastructure version 3 on EMC® Symmetrix® devices using Virtual Provisioning™.
October 2010
Copyright © 2008, 2010 EMC Corporation. All rights reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is
subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION
MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE
INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable
software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com
All other trademarks used herein are the property of their respective owners.
Part Number H4285.5
Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Applied Technology 2
Table of Contents
Executive summary ............................................................................................ 4

Introduction ......................................................................................................... 4
Audience ...................................................................................................................................... 4
Terminology ................................................................................................................................. 5
EMC Symmetrix Virtual Provisioning ................................................................ 6
New features ................................................................................................................................ 8
Thin pool write balancing.......................................................................................................... 8
Virtual Provisioning space reclamation .................................................................................. 11
Considerations for VMware Infrastructure 3 on thin devices ....................... 13
Device visibility and access ....................................................................................................... 13
VMware file system on thin devices ........................................................................................... 13
VMware file system creation and formatting .......................................................................... 13
VMware file system block allocation and reuse algorithms .................................................... 15
Creating VMware virtual machines on thin devices ................................................................... 16
“Zeroedthick” allocation format ............................................................................................... 18
“Eagerzeroedthick” allocation format ..................................................................................... 19
Impact of guest operating system activities on thin pool utilization ........................................... 20
Changing the default virtual disk allocation mechanism policy.................................................. 21
Duplicating virtual machines ...................................................................................................... 22
Cloning virtual machines and the impact on virtually provisioned devices ............................ 22
Creating templates from existing virtual machines................................................................. 25
Deploying new virtual machines from templates .................................................................... 26
VMware DRS and Virtual Provisioning ...................................................................................... 27
Cold migration and Virtual Provisioning ..................................................................................... 27
Storage vMotion and Virtual Provisioning .................................................................................. 27
Nondisruptive expansion of VMFS datastore on thin devices ................................................... 27
Use of Enginuity 5874 SR1 thin pool space reclamation in VMware
environments .................................................................................................... 31
Virtual Provisioning space reclamation in VMware Infrastructure 3 environments.................... 31
Considerations for space reclamation in VMware environments............................................... 31
Local and remote replication between thin devices ...................................... 32
Local replication between thin and regular devices ...................................... 32
Performance considerations ........................................................................... 32
Thin pool management ..................................................................................... 32
Thin pool monitoring .................................................................................................................. 33
Exhaustion of oversubscribed pools .......................................................................................... 33
Conclusion ........................................................................................................ 34
References ........................................................................................................ 35

Executive summary
One of the biggest challenges facing storage administrators is provisioning storage for new applications.
Administrators typically allocate space based on anticipated future growth of applications. This is done to
mitigate recurring operational functions, such as incrementally increasing storage allocations or adding
discrete blocks of storage as existing space is consumed. Using this approach results in more physical
storage being allocated to the application than needed for a significant amount of time and at a higher cost
than necessary. The overprovisioning of physical storage also leads to increased power, cooling, and floor
space requirements. Even with the most careful planning, it may be necessary to provision additional
storage in the future, which could potentially require an application outage.
A second layer of storage overprovisioning happens when a server and application administrator
overallocate storage for their environment. The operating system sees the space as completely allocated
but internally only a fraction of the allocated space is used.
EMC® Virtual Provisioning™ can address both of these issues. Virtual Provisioning, available starting
with Enginuity™ release 5773, allows more storage to be presented to an application than is physically
available. More importantly, Virtual Provisioning will allocate physical storage only when the storage is
actually written to. This allows more flexibility in predicting future growth, reduces the initial costs of
provisioning storage to an application, and can obviate the inherent waste in overallocation of space and
administrative management of subsequent storage allocations.
EMC Symmetrix® DMX™ and Symmetrix VMAX™ storage arrays continue to provide increased storage
utilization and optimization, enhanced capabilities, and greater interoperability and security. The
implementation of Virtual Provisioning, generally known in the industry as “thin provisioning,” for these
arrays enables organizations to improve ease of use, enhance performance, and increase capacity utilization
for certain applications and workloads.
This white paper addresses the considerations for deploying VMware Infrastructure version 3 environments
on thinly provisioned devices. An understanding of the principles that are exposed here will allow the
reader to deploy VMware Infrastructure 3 environments with Virtual Provisioning in the most effective
manner.
Introduction
This white paper will discuss how to best leverage Symmetrix Virtual Provisioning in a VMware
Infrastructure 3 environment. The primary features and benefits of Symmetrix Virtual Provisioning are
discussed, but for more detailed information please refer to the document Best Practices for Fast, Simple
Capacity Allocation with EMC Symmetrix Virtual Provisioning Technical Note available on Powerlink®.
This paper primarily addresses the considerations and best practices for deploying VMware Infrastructure 3
environments on thinly provisioned devices.
An understanding of the principles that are exposed here will allow the reader to deploy VMware
Infrastructure 3 environments with Virtual Provisioning in the most effective manner.
Audience
This white paper is intended for storage architects, and server and VMware administrators responsible for
deploying VMware Infrastructure 3 on Symmetrix DMX using Enginuity 5773 and later operating
environments, as well as on Symmetrix VMAX using Enginuity 5874 and later operating environments.
Virtual Provisioning is now free for all supported Symmetrix platforms, new and existing, which includes
all Symmetrix VMAX arrays plus DMX3/4 arrays running Enginuity 5773.

Terminology
Table 1. Basic Symmetrix array terms
Term Description
Device A logical unit of storage defined within a Symmetrix array. In this paper this style of
device is also referred to as a traditional device and a non-thin device
Device Capacity The actual storage capacity of a device
Device Extent Specifies a quantum of logical contiguous block of storage
Host Accessible Device A device that is exported for host use
Internal Device A device used for internal function of the array
Metavolume An aggregation of host accessible devices, seen from the host as a single device
Pool A collection of internal devices for some specific purpose
Table 2. Virtual Provisioning terms
Term Description
Thin Device A host accessible device that has no storage directly associated with it
Data Device An internal device that provides storage capacity to be used by thin devices
Thin Device Extent The minimum quantum of storage that must be mapped at a time to a thin device
Thin Pool A collection of data devices that provide storage capacity for thin devices. It should be
noted that a thin pool can exist without any storage associated to it
Thin Pool Capacity The sum of the capacities of the member data devices
Bind The process by which one or more thin devices are associated to a thin pool
Unbind The process by which a thin device is diassociated from a given thin pool. When
unbound, all previous allocated extents are freed and returned to the thin pool
Enabled Data Device A data device belonging to a thin pool from which capacity is allocated for thin
devices. This state is under user control
Disabled Data Device A data device belonging to a thin pool from which capacity cannot be allocated for thin
devices. This state is under user control
Activated Data Device Reads and writes can be done from allocated and unallocated space on activated data
devices
Deactivated Data Device Reads and writes can be done from already allocated space on deactivated data devices.
No new allocations can be done on deactivated data devices
Data Device Drain The process of removing allocated extents off of one data device and moving them to
another enabled data device in the same thin pool
Thin Pool Enabled Capacity The sum of the capacities of enabled data devices belonging to a thin pool
Thin Pool Allocated A subset of thin pool enabled capacity that has been allocated to thin devices bound to
Capacity that thin pool
Thin Device Written The capacity on a thin device that was written to by a host. In most implementations
Capacity this will be a large percentage of the thin device allocated capacity
Thin Device Subscribed The total amount of space that the thin device would use from the pool if the thin device
Capacity were completely allocated
Oversubscribed Thin Pool A thin pool whose thin pool enabled capacity is less than the sum of the reported sizes
of the thin devices using the pool
Thin Device Allocation The capacity limit that a thin device is entitled to withdraw from a thin pool, which may
Limit be equal to or less than the thin device subscribed capacity

Term Description
Thin Device Subscription The ratio between the thin device subscribed capacity and the thin pool enabled
Ratio capacity. This value is expressed as a percentage
Thin Device Allocation The ratio between the thin device allocated capacity and the thin device subscribed
Ratio capacity. This value is expressed as a percentage
Thin Device Utilization The ratio between the thin device written capacity and thin device allocated capacity.
Ratio This value is expressed as a percentage
Thin Pool Subscription The ratio between the sum of the thin device subscribed capacity of all its bound thin
Ratio devices and the associated thin pool enabled capacity. This value is expressed as a
percentage
Thin Pool Allocation Ratio The ratio between the thin pool allocated capacity and thin pool enabled capacity. This
value is expressed as a percentage
Pre-allocating Allocating a range of extents from the thin pool to the thin device at the time the thin
device is bound. Sometimes used to reduce the operational impact of allocating extents
or to eliminate the potential for a thin device to run out of available extents
EMC Symmetrix Virtual Provisioning

Symmetrix thin devices are logical devices that can be used in many of the same ways that Symmetrix
devices have traditionally been used. Unlike traditional Symmetrix devices, thin devices do not need to
have physical storage completely allocated at the time the device is created and presented to a host. A thin
device is not usable until it has been bound to a shared storage pool known as a thin pool. Multiple thin
devices may be bound to any given thin pool. The thin pool is comprised of devices called data devices
that provide the actual physical storage to support the thin device allocations.
When a write is performed to a part of any thin device for which physical storage has not yet been
allocated, the Symmetrix allocates physical storage from the thin pool for that portion of the thin device
only. The Symmetrix operating environment, Enginuity, satisfies the requirement by providing a unit of
physical storage from the thin pool called a thin device extent. This approach reduces the amount of
storage that is actually consumed.
The thin device extent is the minimum amount of physical storage that can be reserved at a time for the
dedicated use of a thin device. An entire thin device extent is physically allocated to the thin device at the
time the thin storage allocation is made as a result of a host write operation. A round-robin mechanism is
used to balance the allocation of data device extents across all of the data devices in the pool that are
enabled and that have remaining unused capacity. The thin device extent size is 12 tracks (768 KB). Note
that the initial bind of a thin device to a pool causes one thin device extent, or 12 tracks, to be allocated per
thin device. If the thin device is a metavolume, then one thin device extent is allocated per metamember
device. So a four-member thin meta would cause 48 tracks (3078 KB) to be allocated when the device is
bound to a thin pool.
When a read is performed on a thin device, the data being read is retrieved from the appropriate data device
in the thin pool. If a read is performed against an unallocated portion of the thin device, zeros are returned
to the reading process.
When more physical data storage is required to service existing or future thin devices, for example, when a
thin pool is running out of physical space, data devices can be added to existing thin pools dynamically
without needing a system outage1. New thin devices can also be created and bound with existing thin
pools.
When data devices are added to a thin pool they can be in an enabled or disabled state. In order for the data
device to be used for thin extent allocation it needs to be enabled. For it to be removed from the thin pool,
it needs to be in a disabled state. Beginning with Enginuity 5874, active data devices can be disabled,
1
Enginuity operating environment 5874 SR1 on the Symmetrix VMAX and Solutions Enabler 7.1 allow
users to rebalance extents over the thin pool when new data devices are added. This feature is discussed in
detail in the next section.

which will cause any allocated extents to be drained to the other enabled devices in the pool. They can then
be removed from the pool when the drain operation has completed.
The following figure depicts the relationships between thin devices and their associated thin pools. There
are nine devices associated with thin Pool A and three thin devices associated with thin pool B.
Figure 1. Thin devices and thin pools containing data devices

The way thin extents are allocated across the data devices results in a form of striping in the thin pool. The
more data devices that are in the thin pool, the wider the striping is, and therefore the greater the number of
devices that can participate in application I/O.
The maximum size of a thin device on a Symmetrix DMX is approximately 60 GB, which has been
increased to 240 GB for the Symmetrix VMAX. If a larger size is needed, then a metavolume comprised of
thin devices can be created. It is recommended that the metavolume be concatenated rather than striped
since the data is already striped across the data devices in the thin pool, which may also be striped across
physical disks depending on the RAID protection of the data devices. Concatenated metavolumes also
support fast expansion capabilities, as new metavolume members can easily be added to the existing
concatenated metavolume. This functionality is required when the provisioned thin device has been
consumed by the host, and further storage allocations are required.
Concatenated metavolumes using regular volumes have the drawback of being limited by the performance
of a single metamember. In the case of a concatenated metavolume comprised of thin devices, each
member device is typically spread across the entire underlying thin pool, thus eliminating that drawback.
Concatenated metavolumes have two important operational advantages over striped metavolumes:
Concatenated metavolumes can be expanded without destroying existing data by adding
metavolume members to an existing metavolume.
Non-metavolumes can be converted to concatenated metavolumes without destroying existing
data.
In most cases, EMC recommends using concatenated rather than striped metavolumes with Virtual
Provisioning. However, there may be certain situations where better performance may be achieved using
striped metavolumes.

With synchronous SRDF®, Enginuity allows one outstanding write per thin device per path. With
concatenated metavolumes, this could cause a performance problem by limiting the concurrency of writes.
This limit will not affect striped metavolumes in the same way because of the small size of the metavolume
stripe (one cylinder or 1,920 blocks).
Enginuity allows eight read requests per path per thin device. This limits the number of read requests that
can be passed through to the thin pool regardless of the number of data devices that may be in it. This can
cause slower performance in environments with a high read miss rate.
Symmetrix Enginuity has a logical volume write pending limit to prevent one volume from monopolizing
writeable cache. Because each metamember gets a small percentage of cache, a striped meta is likely to
offer more writeable cache to the metavolume.
Before configuring striped metavolumes, please consult with an EMC performance specialist.
Caution: Striped thin metavolumes cannot be expanded while preserving data.

A more detailed discussion of Symmetrix Virtual Provisioning is beyond the scope of this paper. For more
information please refer to the technical note Best Practices for Fast, Simple Capacity Allocation with
EMC Symmetrix Virtual Provisioning available on Powerlink.
New features
Solutions Enabler 7.1 and Enginuity 5874 SR1 introduce two new features to Symmetrix Virtual
Provisioning– thin pool write balancing and zero space reclamation. Thin pool write balancing provides the
ability to automatically rebalance allocated extents on data devices over the entire pool when new data
devices are added. Zero space reclamation allows users to reclaim space from tracks of data devices that are
all zeros.
Thin pool write balancing

Whenever writes occur on a thin device, the actual data gets written to the data devices within the thin pool
to which the thin device is bound. Enginuity intelligently spreads out the I/O equally on all the data devices
within the thin pool, so at any given moment the used capacity of each data device is expected to be
uniform across the pool.
Prior to the availability of automated thin pool write balancing, as soon as new data devices were added to
the thin pool, the allocation spread immediately ceased to be uniform. If data devices were added in groups
that were smaller in number than what were previously in the thin pool, performance decreased due to the
fact that as soon as the original data devices became completely full, any new data written to the pool was
striped over fewer data devices.
Therefore, the best practice was to add a large number of data devices, typically equal to or greater than the
number of devices used when the pool was originally created to maintain expected performance levels.
This method of expanding pools using larger numbers of devices led to an increase in assigned, but unused,
storage as in most situations the thin pool did not need to be expanded by that much to fulfill the increased
thin pool storage requirements.
Automated thin pool write balancing alleviates both of these problems. It is now possible to add any
number of data devices and still have a pool that is balanced for capacity while preventing potential
performance problems due to narrow striping that previously would be caused by adding too few data
devices.
Thin pool write balancing works by normalizing the used capacity levels of data devices within a thin pool.
A background optimization process scans the used capacity levels of the data devices within a pool and
performs movements of groups from the most utilized devices to the least. The threshold for write
balancing of 10% has two implications — that there can be up to a 10% difference between the most used
and least used data devices after a rebalance operation completes and that if the thin pool is within this
range from the initiation of the balancing command, the balancing operation will not start.

The basic steps taken by thin pool write balancing are as follows:
• Initialization – the target resources are locked so that no other process can try to optimize them
• Scan – the pool is scanned for utilization statistics
• Move – the group movements are initiated between devices
• Wait – the task monitors the background data movements until the next period begins
The balancing procedure will start by calculating the minimum, maximum, and mean used capacity values
of the data devices. Afterward, a source/target data device pair is automatically created by Enginuity to
bring the source data device’s utilization down and the target data device’s utilization up toward the mean.
Once the data movements have finished, the process restarts by scanning the pool again. Data devices that
are currently a source or a target in the pool balance operation will be shown to have a state of “Balancing.”
While a data device is in this state, it cannot be disabled or drained. The “Balancing” state will be transient
as Enginuity will pick different devices periodically to accomplish the pool balancing operation.
While write balancing is going on, it is possible that the data can be moved from “deactivated” data devices
but data will never be moved to “deactivated” data devices. Solutions Enabler will report the state
“Balancing” in the pool state and once the task is completed, the pool state will revert to the default state of
“Enabled.” For more detailed information on how to perform thin pool write balancing, please refer to the
technical note Best Practices for Fast, Simple Capacity Allocation with EMC Symmetrix Virtual
Provisioning.
Symmetrix thin pool write balancing using Solutions Enabler
In the following example a thin pool containing four data devices is completely full and requires more data
devices to satisfy the storage needs of the bound thin devices. Figure 2 shows such a thin pool.
Figure 2. Full Symmetrix thin pool before a write balance operation

To rectify this, as seen in Figure 3, two more data devices are added to the pool. Since the four previous
data devices are completely full, any new writes to the thin pool will only be striped over the two recently
added data devices, possibly reducing performance. Initiating the rebalance procedure shown in Figure 4 on
page 10 will remediate this issue. In addition, the lower portion of Figure 4 shows the pool state change to
“Balancing,” which reports that the pool is now amidst being balanced.

Figure 3. A Symmetrix thin pool with two new empty data devices before a rebalance
operation
Figure 4. Initiating a thin pool write balancing operation with Solutions Enabler
As pictured in Figure 5, while the balancing operation executes, varying data devices will go into the
“Balancing” state temporarily as extents are drained off of them onto other less utilized data devices.

Figure 5. Thin pool write balancing and automatic data device draining
The thin pool write balancing procedure does not complete before the 10% differential threshold is met.
The balanced pool can be seen in Figure 6.
Figure 6. Completed thin pool write balancing operation
Virtual Provisioning space reclamation

Virtual Provisioning space reclamation provides the ability to free, also referred to as “de-allocate,” storage
extents found to contain all zeros. This feature is an extension of the existing Virtual Provisioning space
de-allocation mechanism. Previous versions of Enginuity and Solutions Enabler allowed for reclaiming
allocated (reserved but unused) thin device space from a thin pool. Administrators now have the ability to
reclaim both allocated/unwritten extents as well as extents filled with host-written zeros within a thin pool.
The space reclamation process is nondisruptive and can be executed with the targeted thin device ready and
read/write to operating systems and applications.
Starting the space reclamation process spawns a back-end disk director (DA) task that will examine the
allocated thin device extents on specified thin devices. As discussed earlier, a thin device extent is 768 KB
(or 12 tracks) in size and is the default unit of storage at which allocations occur. For each allocated extent,
all 12 tracks will be brought into Symmetrix cache and examined to see if they contain all zero data. If the
entire extent contains all zero data, the extent will be de-allocated and added back into the pool, making it
available for a new extent allocation operation. An extent that contains any non-zero data is not reclaimed.

Symmetrix Virtual Provisioning space reclamation using Solutions Enabler
In the following example, the thin device in Figure 7 hosts one virtual machine with a 25 GB virtual disk.
Only 1.5 GB of that virtual disk is actual data, while the remaining 23.5 GB is comprised completely of
zeros.
Figure 7. The allocation of a Symmetrix thin device before undergoing space reclamation
This situation is ideal for space reclamation as over 90% of the allocated and written space from this thin
device can be reclaimed and returned to the thin pool as free space. In Figure 8, the space reclamation
process is initiated through the Solutions Enabler command line interface. Enginuity is told to examine the
entire thin device for any extent full of zeros and to then deallocate them.
Figure 8. Initiating space reclamation on a Symmetrix thin device

As seen in Figure 9, the space reclamation process deallocates 23.7 GB of zeros from the thin pool. After
the space reclamation is done, thin device 0208 consumes only 1% of the thin pool instead of the 16% from
before the reclamation process.
Figure 9. Space reclamation on a Symmetrix thin device

Considerations for VMware Infrastructure 3 on thin
devices
There are certain considerations that VMware administrators must be aware of when deploying a VMware
Infrastructure 3 using Virtual Provisioning. How the various VMware Infrastructure 3 features work in this
kind of configuration depends on the feature, its usage, and the guest operating system running on the
virtual machine. This section describes how the various VMware Infrastructure 3 features interact with
virtually provisioned devices.
Device visibility and access

Bound thin devices2 appear like any other SCSI-attached device to the VMware Infrastructure 3. Thin
devices can be used to create a VMware file system, or assigned exclusively to a virtual machine as a raw
disk mapping (RDM). An example of this is shown in Figure 10. Using the EMC Storage Viewer for
VMware Infrastructure Client, a VMFS volume named VMax_ThinDS is highlighted. The VMFS volume
is shown to be residing on thin device (TDEV) 50C on a Symmetrix VMAX running Enginuity 5874.
One thing to remember is that when a thin device is presented as a RDM (either physical or virtual
compatibility) to a virtual machine, the VMware kernel does not play a direct role in the I/O to the thin
device generated by the guest operating system running in the virtual machine. In this configuration, the
considerations for using thin devices are no different from the ones for physical servers running the same
operating system and applications.
VMware file system on thin devices
VMware file system creation and formatting

The VMware file system (VMFS) has some interesting and valuable behavioral characteristics when
viewed in a Virtual Provisioning landscape. First, a minimal number of thin extents are allocated from the
thin pool when a VMFS is created on virtually provisioned devices. The amount of storage required to
store the file system metadata is a function of the size of the thin device.
The metadata for the VMware file system (VMax_Thin_DS) on the thin device, vmhba1:0:9:1, shown
in Figure 10, reserves 519 MB. This is shown in Figure 11 using Symmetrix Management Console. Figure
12 shows the tracks allocated by Enginuity in response to the write activity generated by the formatting of
the VMFS. It can be seen from this figure that only 108 MB were actually allocated from the thin pool.
Therefore, the VMFS does not write all of its metadata to disks on creation. The VMFS formats and uses
the reserved area for metadata as requirements arise.
2
Thin devices not bound to a thin pool are presented as not-ready to the host. However, the SCSI INQ
command to the device succeeds. The VMware ESX server kernel, therefore, cannot determine the correct
status of the unbound device. If the VMware ESX server kernel performs a read on unbound thin devices,
zeros would be returned back. Similarly, any attempt to create a datastore on an unbound thin device will
fail. Therefore, EMC strongly dissuades customers from presenting unbound thin devices to VMware ESX
servers.

Figure 10. Thin device viewed in EMC Storage Viewer 1.1 for VMware Infrastructure Client
Figure 11. Metadata area reserved on the VMware file system
Figure 12. Track allocation from a thin pool in response to VMware file system format
activity

VMware file system block allocation and reuse algorithms
The current implementation of Virtual Provisioning does not have a direct mechanism to reclaim storage
from the thin pool when an object is deleted by the application using the thin device. If the file system
created on a thin device does not reuse a previously allocated block, new tracks are continuously allocated
from the thin pool in response to new write activity3. This behavior reduces the benefit of using virtually
provisioned devices.
Figure 13 shows the results from a series of tests conducted to determine the behavior of the VMFS. The
size of the thin device used in the test was 539.30 GB in size. The thin pool to which the thin device was
bound had a total enabled capacity of 134.80 GB. The graph shows the percentage utilization of the VMFS
and the thin pool providing the storage for the thin device used in the experiment. The graph also shows the
normalized utilization percentage of the VMFS. This value, obtained by scaling the VMFS utilization
percentage by the ratio of thin device to thin pool size, enables effective evaluation of the VMFS behavior.
It can be seen from the figure that the VMFS utilization drops when the 30 GB virtual disk is deleted in file
operation step number 4. As expected, the percentage utilization of the thin pool does not change.
However, from that point on, the VMFS reuses the freed blocks as new files are created and deleted on the
file system. The VMFS thus exhibits a behavior that is beneficial when used with virtually provisioned
devices.
120
100
80
% VMFS Used
60
%VMFS Used (Normalized)
% Pool Allocation
40
20
Figure 13. VMware file system allocation mechanisms
3
Previously allocated but unused blocks can be de-allocated if a utility is used to zero unused blocks out
(such as the sDelete utility for Windows) and then a zero-space reclamation operation is performed on the
thin device. For more information on zero-space reclamation, please refer to the section, “Virtual
Provisioning space reclamation.” Note that zero-space reclamation requires a Symmetrix VMAX running
Enginuity 5874 or later.

Creating VMware virtual machines on thin devices
The New Virtual Machine wizard provided by Virtual Infrastructure Client is not impacted by the use of
virtually provisioned devices. This is due to the fact that a VMware datastore on the thin devices appears as
any other datastore to the wizard. Figure 14 shows the final step of the New Virtual Machine wizard to
create a virtual machine with a 16 GB virtual disk on a VMFS datastore hosted on a thin device.
When the Finish button shown in Figure 14 is clicked, the Virtual Infrastructure Client performs a number
of actions including creation of the virtual disk required to support the virtual machine. Figure 15 shows the
storage utilization of the VMFS and the thin pool supporting the datastore after the New Virtual Machine
wizard has completed. The figure clearly shows that a small number of tracks are initialized when a new
virtual machine is created. However, as shown in Figure 16, the VMware kernel reserves the storage
requirement for the virtual machine on the VMFS for future use. This behavior prevents the VMFS from
running out of space required for metadata.
Figure 14. Using the Virtual Infrastructure Client to create a new virtual machine

Figure 15. Thin pool utilization when creating a new virtual machine

Figure 16. VMware datastore utilization on creation of a new virtual machine
The behavior of VMware Infrastructure 3 can be synergistic with the benefits offered by Virtual
Provisioning because of the number of different allocation mechanisms for creating virtual disks. These are
listed in Table 3.
Table 3. Allocation policies when creating new virtual disks on a VMware datastore
Allocation mechanism VMware kernel behavior

(Virtual Disk format)
zeroedthick All space is allocated at creation but is not initialized with zeros. However, the allocated
space is wiped clean of any previous contents of the physical media. This is the default
policy when creating new virtual disks.
eagerzeroedthick This allocation mechanism allocates all of the space and initializes all of the blocks with
zeros. This allocation mechanism performs a write to every block of the virtual disk, and
hence results in equivalent storage use in the thin pool.
thick A thick disk has all the space allocated at creation time. If the guest operating system
performs a read from a block before writing to it, the VMware kernel may return stale
data. The use of this format is dissuaded.
thin This allocation mechanism does not reserve any space on the VMware file system on
creation of the virtual disk. The space is allocated and zeroed on demand.
rdm The virtual disk created in this mechanism is a mapping file that contains the pointers to
the blocks of SCSI disk it is mapping. However, the SCSI INQ information of the
physical media is virtualized. This format is commonly known as the “Virtual
compatibility mode of raw disk mapping.”
rdmp This format is similar to the rdm format discussed above. In this format, the SCSI INQ
information of the physical media is not virtualized. This format is commonly known as
the “Pass-through raw disk mapping.”
raw This mechanism can be used to address all SCSI devices supported by the kernel except
for SCSI disks.
2gbsparse The virtual disk created using this format is broken into multiple sparsely allocated
extents (if needed), with each extent no more than 2 GB in size.
“Zeroedthick” allocation format

This is the default allocation format when creating virtual disks and is the recommended disk format for use
with Symmetrix Virtual Provisioning. In this allocation scheme, the storage required for the virtual disks is
reserved in the datastore but the VMware kernel does not initialize all the blocks. The blocks are initialized
by the guest operating system as write activities to previously uninitialized blocks are performed. The
VMFS will return zeros to the guest operating system if it attempts to read blocks of data that it has not
previously written to. This is true even in cases where information from a previous allocation is available—
the VMFS will not present stale data to the guest operating system when the virtual disk is created using the
“zeroedthick” format. Since the VMFS volume will report the virtual disk as fully allocated, the risk of

oversubscribing and running out of space in the thin pool is reduced. This is shown in Figure 17, which
displays a single 10 GB virtual disk (with only 1400 MB of actual data on it) that uses the “zeroedthick”
allocation method. The datastore browser reports the virtual disk as consuming the full 10 GB on the
volume.
Figure 17. Zeroedthick virtual disk allocation size as seen in a VMFS datastore browser
However, since the VMware kernel does not actually initialize unused blocks, the full 10 GB is not
consumed on the thin device. This virtual disk resides on thin device 0482 and, as seen in Figure 18, only
consumes 1600 MB of space on it (1400 MB for the data and 200 for the VMFS metadata).
Figure 18. Zeroedthick virtual disk allocation on Symmetrix thin devices
“Eagerzeroedthick” allocation format

With the “eagerzeroedthick” allocation mechanism, space required for the virtual disk is completely
allocated and written to at creation time. This leads to a full reservation of space on the VMFS datastore
and on the underlying Symmetrix device. Accordingly, it takes longer to create disks in this format than to
create other types of disks. Because of this behavior, “eagerzeroedthick” format is not ideal for use with
virtually provisioned devices.
Note: Eagerzeroedthick is the default allocation mechanism for cloning virtual machines, deploying virtual
machines from template, and Storage VMotion. In VMware vCenter Server update 6 this behavior can be changed
so the default for all of these operations is the Virtual Provisioning-friendly “zeroedthick” format. For information
on implementing this change refer to the section “Changing the default virtual disk allocation mechanism policy”
in this paper.

A single 10 GB virtual disk (with only 1.4 GB of actual data on it) that uses the “eagerzeroedthick”
allocation method is shown in Figure 19. The datastore browser reports the virtual disk as consuming the
entire 10 GB on the volume.
Figure 19. Eagerzeroedthick virtual disk allocation size as seen in a VMFS datastore
browser
Unlike with “zeroedthick” the VMware kernel does initialize all the unused blocks, so the full 10 GB is
reserved on the VMFS and consumed on the thin device. The “eagerzeroedthick” virtual disk resides on
thin device 0484 and, as seen in Figure 20, consumes 10.2 GB of space on it (10 GB for the data and 200
MB for the VMFS metadata).
Figure 20. Eagerzeroedthick virtual disk allocation on Symmetrix thin devices
It is important to note that VMware vCenter Server 2.5 does not offer an option to use non-default behavior
when creating virtual disks for virtual machines. Creating virtual disks using a “thin” allocation mechanism
requires the use of the CLI utility, vmkfstools, on the service console or remote CLI for VMware ESX
Server version 3i.
Impact of guest operating system activities on thin pool

utilization
As discussed in the previous section, a small number of thin extents are allocated in response to the creation
of a new virtual machine. However, the thin pool utilization rapidly grows as the user performs various
activities in the virtual machine. For example, the installation of an operating system in the virtual machine

triggers write I/Os to previously uninitialized blocks of data. These I/Os in turn result in allocation of new
thin extents in the thin pool associated with the thin device.
The amount of storage used by the virtual machine depends on the behavior of the operating system, the
logical volume manager, file systems, and applications running inside the virtual machine. Poor allocation
and reuse of blocks freed from deleted files can quickly result in thin pool allocation that is equal to the size
of the virtual disks presented to the virtual machines. Nevertheless, the thin pool allocation in support of a
virtual machine can never exceed the size of the virtual disks. Therefore, users should consider the guest
operating system and application behavior when configuring virtual disks for new virtual machines.
A number of additional documents listed in the “References” section discuss the behavior of various
operating systems, file systems, and applications in a Virtual Provisioning environment. Since the behavior
does not change inside a virtual machine, readers should consult these documents to understand the
anticipated behavior of the virtual machines in a VMware Infrastructure 3 environment.
Changing the default virtual disk allocation mechanism policy

Prior to VMware vCenter Server 2.5 update 6, all virtual disk movement/duplication operations used the
default allocation mechanism of “eagerzeroedthick.” These operations include:
Cloning a virtual machine
Deploying a virtual machine from a template
Performing a Storage VMotion operation on a virtual machine
This behavior could not be altered and resulted in fully allocated virtual disks and accordingly could not
benefit from the inherent space-saving features of Symmetrix Virtual Provisioning. With the introduction
of VMware vCenter Server 2.5 update 6, the default allocation mechanism for these operations could be
changed from “eagerzeroedthick” to the Virtual Provisioning-friendly “zeroedthick” format.
In addition to upgrading to VMware vCenter Server 2.5 update 6 and ESX 3.5 update 5 a manual
modification is required on each ESX server to implement this change. Refer to the following VMware KB
article for more information:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externa
lId=1017666
1. Log in to the service console of an ESX 3.5 host.
2. Open the file vpxa.cfg located at /etc/opt/vmware/vpxa/vpxa.cfg.
3. Find the <nfc> tag and enter the following line (exactly as seen in Figure 21):
<eagerzeroedthick>false</eagerzeroedthick>

Figure 21. Changing the default virtual disk allocation mechanism policy
4. Restart the vCenter agent on the ESX host by issuing the command service vmware-vpxa
restart on the service console.
5. Repeat the process on all ESX hosts.
Duplicating virtual machines

In a large VMware Infrastructure 3 environment it is impractical to manually install and configure the guest
operating system on every new virtual machine. To address this, VMware provides multiple mechanisms to
simplify the process to create preconfigured new virtual machines:
 Cloning: This wizard-driven method allows users to select a virtual machine4 and clone it to a new
virtual machine. The wizard allows users to change some characteristics of the cloned virtual
machine.
 Deploy from template: The users can use a virtual machine to create a template. Once the template
has been created, new virtual machines can be deployed from the template and customized to meet
specific requirements. VMware vCenter Server 2.5 provides wizards for both activities.
Detailed discussion of these options is beyond the scope of this paper. Readers should consult VMware
documentation for details.
Cloning virtual machines and the impact on virtually provisioned devices

The wizard to clone virtual machines offers users a number of customizations including the VMware ESX
server cluster on which the cloned virtual machine will be hosted, the resource pool to associate the virtual
machine to, and the VMware datastore on which to deploy the cloned virtual machine. The only option that
impacts the virtually provisioned devices is the process that is utilized to deploy the virtual disk for the
cloned virtual machine.
4
All virtual machines cannot be cloned or converted to a template. Please consult the VMware
documentation for further details.

The cloning process, by default, converts the virtual disk on the source virtual machines that had been
created using the “zeroedthick” format to the “eagerzeroedthick” format on the cloned virtual machine. In
VMware vCenter Server 2.5 update 5 and earlier VMware Infrastructure does not provide a mechanism to
change the allocation policy for target virtual disks resulting from a virtual machine cloning operation,
hence the deployment process is inherently detrimental to the use of virtually provisioned devices. This
default behavior can be changed by upgrading to VMware vCenter Server 2.5 update 6. This version allows
the behavior to be changed so that the preferred mechanism “zeroedthick” is used for the cloning operation.
This is explained in the section “Changing the default virtual disk allocation mechanism policy.” Please
refer to that section for more information on implementation.
Figure 22 shows the cloning process as visualized in the VMware Infrastructure Client. The example shows
the cloning of the virtual machine with the virtual disk allocation mechanism policy set to
“eagerzeroedthick.” The target virtual machine is named “Windows_Clone_NoFix” and is placed on
Symmetrix thin device 50E. It can be seen from Figure 22 that the cloning operation took just over 13
minutes to complete.
Figure 22. Cloning virtual machines using Virtual Infrastructure Client with
“eagerzeroedthick” as default policy
The virtual machine has a 40 GB virtual disk with only approximately 1.5 GB of data. Since the allocation
policy is “eagerzeroedthick” the resulting allocation on the Symmetrix thin pool is the full 40 GB, as the
remaining 38.5 GB is filled with zeros written by the ESX kernel. This is shown in Figure 23.
Figure 23. Thin device allocation with the "eagerzeroedthick" virtual machine

Figure 24 also shows the cloning process as visualized in the VMware Infrastructure Client. This second
example shows the cloning of the virtual machine with the virtual disk allocation mechanism policy set to
“zeroedthick.” The target virtual machine is named “Windows_Clone_Fix” and is placed on Symmetrix
thin device 510. It can be seen from Figure 24 that the cloning operation only took 1.5 minutes to complete.
Since the “zeroedthick” does not zero out the virtual disk, the time to complete the cloning operation was
considerably shorter.
Figure 24. Cloning virtual machines using Virtual Infrastructure Client with “zeroedthick”
as the default policy
The virtual machine has a 40 GB virtual disk with only approximately 1.5 GB of data. Since the allocation
policy is “zeroedthick” the resulting allocation on the Symmetrix thin pool is only the amount of data
written by the guest OS of the virtual machine—the remaining 38.5 GB is not written out by the ESX
kernel until written to by the guest OS. This is shown in Figure 25.
Figure 25. Thin device allocation with the "zeroedthick" virtual machine
As shown by contrasting these two examples, EMC highly recommends upgrading to VMware vCenter
Server 2.5 update 6 and implementing this change. The savings in physical storage space and also in time
to clone the virtual machines are dramatic.

Creating templates from existing virtual machines
As discussed earlier, VMware Infrastructure 3 provides two mechanisms to simplify the provisioning of
new virtual machines. The first option, cloning, was discussed in the previous section. The second
alternative, deploy from template, requires a predefined template describing the virtual machine
configuration. The template can be created by either cloning an existing virtual machine or by converting
an existing virtual machine in place.
The process to clone an existing virtual machine to a new template involves copying the virtual disks
associated with the source virtual machine. The problem described in the previous section also occurs
during the process to clone a virtual machine to a template. Therefore, if the datastore holding the templates
is a thin device, the user will not benefit from the storage optimization provided by Virtual Provisioning
technology. However, unlike the cloning wizard, the Clone to Template wizard, as seen in Figure 26, offers
the option of using the Compact format. When this option is utilized, the cloned virtual disk is allocated
using the “thin” format described in Table 3 on page 18. Figure 27 shows the metadata of the virtual disk
created by using the Compact option.
Figure 26. Using the Compact option when creating templates

Figure 27. Metadata for cloned virtual disks created using the Compact option
The Compact option of the Clone to Template wizard thus offers an attractive alternative since the thin pool
allocation is equal to the physical storage used by the virtual disk. The risk of overprovisioning is higher
when the Compact option is used. However, if a separate datastore is dedicated for templates, the contents
of the datastore tend to be relatively static. Furthermore, once the templates have been created, the virtual
disks do not grow. Therefore, in the worst-case scenario, if the capacity of the thin pool supporting the
datastore holding the templates is exceeded, the only impact would be to the process of creating new
templates. Existing templates can still be used to provision new virtual machines. This behavior, although
inconvenient, has a minimal impact on the production environment.
An alternative to the Clone to Template process is the in-place conversion of an existing virtual machine
into a template. This process does not modify the virtual disks associated with the virtual machine. The
only modification that occurs is to the configuration file. Therefore, an in-place conversion to a template
does not unnecessarily increase allocations from the thin pool.
Based on these observations, EMC recommends the use of virtually provisioned devices for templates. A
dedicated template datastore on a thin device should be used in conjunction with the Compact option of the
Clone to Template wizard. Alternatively, the in-place conversion of virtual machines on thin devices
created using the New Virtual Machine wizard can be used.
Independent of the process that is used to create the template, the size of the virtual disk on the source
virtual machine should not be much larger than the space required to hold the operating system and the
applications included in the standard build. The reasons for this recommendation are discussed in the
following section.
Deploying new virtual machines from templates

When a new virtual machine is deployed from a template, the virtual disk on the new virtual machine is
created using the eagerzeroedthick format by default. Therefore, a significant benefit afforded by the
virtually provisioned devices on the Symmetrix storage array is lost. In VMware vCenter Server update 5
and earlier versions VMware Infrastructure does not provide a mechanism to change the allocation policy
for the virtual disks on a virtual machine deployed from a template, hence the deployment process is
inherently detrimental to the use of virtually provisioned devices.
This default behavior can be changed by upgrading to VMware vCenter Server update 6, which allows the
behavior to be changed so that the preferred mechanism “zeroedthick” is used for deploying from a
template. This is explained in the section, “Changing the default virtual disk allocation mechanism policy.”
Please refer to that section for more information on implementation.

VMware DRS and Virtual Provisioning
EMC Symmetrix thin devices behave like any other SCSI disk attached to the VMware ESX server kernel.
If a thin device is presented to all nodes of a VMware DRS cluster group, Virtual Center allows live
migration of viable virtual machines on the thin device from one node of the cluster to another using
VMware vMotion.
Automatic migrations of virtual machines occur if the “automatic” policy is enabled for the DRS cluster
group. Virtual Center will suggest migration recommendations for virtual machines if the VMware DRS
cluster group automation level is set to Partially Automated or Manual. These behaviors are not different
from any other VMware DRS cluster using shared storage on non-thin devices.
Cold migration and Virtual Provisioning

VMware Infrastructure 3 supports cold migration of virtual machines from one VMware ESX server to
another. The cold migration process can also be utilized to change the datastore hosting the virtual
machine.
There is no impact to the cold migration process when a virtual machine is moved from one ESX server to
another while maintaining the location of the datastore containing the virtual machine files. However,
changing the datastore location as part of the migration process can have a negative impact. The migration
of the data to a datastore on a thin device is performed using the “eagerzeroedthick” format and results in
unnecessary allocation from the thin pool. Therefore, the cold migration functionality should not be used
for migrating virtual machines from non-thin devices to thin devices unless the default allocation
mechanism is changed to “zeroedthick.”
Storage vMotion and Virtual Provisioning

VMware Storage vMotion supports hot migration of virtual machines from one VMware ESX server to
another while changing the datastores hosting the virtual machine.
Similar to the cold migration process discussed above, changing the datastore location can have a negative
impact. The migration of the data to a datastore on a thin device is performed using the “eagerzeroedthick”
format and results in unnecessary allocation from the thin pool. Therefore, Storage vMotion should not be
used for migrating virtual machines from non-thin devices to thin devices unless the default allocation
mechanism is changed to “zeroedthick.” Furthermore, to minimize impact to the business, VMware
administrators have to be careful utilizing the technology in the VMware Infrastructure 3 environments
when Symmetrix Virtual Provisioning is in use.
Nondisruptive expansion of VMFS datastore on thin devices

Thin devices presented to VMware Infrastructure 3 should be configured to address the storage requirement
for the lifetime of the infrastructure. This approach is viable since thin devices do not require equivalent
physical storage initially. Physical capacity can be added to the thin pool as the environment grows.
A properly configured thin device presented to VMware Infrastructure 3 should not have a need for
expansion. Nevertheless, it is possible to expand a thin device nondisruptively while maintaining existing
data on the devices. To exploit this functionality, the thin devices need to be presented as a concatenated
metavolume to VMware Infrastructure 3. Figure 28 shows a VMware file system datastore on a thin device.
The figure also shows the data that is present in the datastore. Figure 29 shows the process to expand the
concatenated metavolume by adding additional thin devices. The existing data is preserved during the
expansion operation. After the expansion is complete, as seen in Figure 30, VMware Infrastructure 3
recognizes the additional storage available on the thin device. The VMware file system, as seen in Figure
31, can then be expanded to occupy the additional storage as a second extent. The figure also shows the
original data that was preserved during the expansion process.

Figure 28. VMware file system on a thin device

Figure 29. Using symconfigure to expand a concatenated metavolume of thin devices

Figure 30. Expanding the VMware file system to occupy additional space on a thin device
Figure 31. An expanded VMware file system with preserved data

Use of Enginuity 5874 SR1 thin pool space reclamation
in VMware environments
In a VMware environment, the zeroing out of virtual disks can lead to unnecessary physical storage
allocation on Symmetrix thin pools, a situation that is inherently detrimental to the space-saving benefit of
Symmetrix Virtual Provisioning. Introduced in Enginuity 5874 SR1 for the Symmetrix VMAX is the
ability to perform space reclamation on thin devices hosting, for example, zeroed-out virtual disks. In the
next few subsections, Symmetrix Virtual Provisioning space reclamation and the considerations for it in a
VMware Infrastructure 3 environment are discussed in detail.
Virtual Provisioning space reclamation in VMware Infrastructure

3 environments
In VMware Infrastructure 3, the usefulness of zero reclaim comes into play for three main scenarios: virtual
machine cloning, deploying a virtual machine from a template, and importing a virtual machine using
VMware Converter. All three actions result in a completely zeroed-out virtual disk, as they all use the
virtual disk allocation mechanism called, “eagerzeroedthick.” This allocation mechanism allocates all of
the space and initializes all of the blocks with zeros. When “eagerzeroedthick” is used, every block of the
virtual disk is written to, and hence results in equivalent storage use in the thin pool. Consequently, the use
of these three types of virtual machine deployment methods causes the new virtual machines to consume
far more space than is used by actual data. Since the extra space is taken up by zeros, it can be reclaimed to
free space up on the thin pool.
The fourth method of deploying virtual machines is through the use of the New Virtual Machine wizard.
This method will use the “zeroedthick” allocation mechanism for virtual disks by default. With the
“zeroedthick” allocation scheme, the storage required for the virtual disks is reserved in the datastore but
the VMware kernel does not initialize all the blocks. The blocks are initialized by the guest operating
system as write activities to previously uninitialized blocks are performed. Therefore, the space consumed
on the thin pool is only actual data, not zeros. For virtual machines created with this method, zero reclaim
is not typically necessary.
Considerations for space reclamation in VMware environments

For VMware Infrastructure 3, there are important considerations that should be evaluated before
performing reclaims.
If a thin device stores multiple virtual machines, a space reclamation function cannot be performed on just
one of the virtual disks. It is currently not possible to determine the specific blocks each virtual disk
consumes on the thin device and therefore it is not possible to limit a reclaim to a single virtual disk out of
many on a given VMFS 5. Consequently, a reclaim should only be performed on a thin device if all virtual
machines hosted by the VMware file system volume on that thin device can and should be reclaimed.
Additionally, zeroing out by guest operating systems must be taken into account before performing space
reclamation. If the virtual machines’ operating system or its hosted applications zero out files for a
particular reason, any implications of removing those zeros should be considered before reclaiming them.
EMC Storage Viewer 1.16 is highly recommended in order to easily map virtual machines and their
corresponding VMFS volumes to the correct underlying Symmetrix thin device. Double-checking storage
mapping information with EMC Storage Viewer 1.1 will eliminate the possibility of performing space
reclamation on an incorrect thin device.
5
EMC is actively working with VMware on the vStorage initiative that may allow in the future for space
reclamation to be performed on a single virtual disk.
6
For more information on the EMC Storage Viewer version 1.1 please refer to the white paper on
Powerlink entitled Using EMC Storage Viewer for VMware Infrastructure Client.

Local and remote replication between thin devices
Organizations will be able to perform “thin to thin” replication with Symmetrix thin devices by using
standard TimeFinder®, SRDF, and Open Replicator operations. This includes TimeFinder/Snap,
TimeFinder/Clone, SRDF/Synchronous, SRDF/Asynchronous, and SRDF/Data Mobility.
In addition, thin devices can be used as control devices for hot and cold pull and cold push Open Replicator
copy operations. If a push operation is done using a thin device as the source, zeros will be sent for any
regions of the thin device that have not been allocated, or that have been allocated but have not been
written to.
Open Replicator can also be used to copy data from a regular device to a thin device. If a pull or push
operation is initiated from a regular device that targets a thin device, then a portion of the target thin device,
equal in size to the reported size of the source volume, will become allocated.
Local replication between thin and regular devices

TimeFinder/Clone supports the creation of clone sessions between regular devices and thin devices on
Symmetrix VMAX arrays as of the Enginuity 5874 Q4 2009 service release.
When a clone copy is made between a regular source device and a thin target device, thin device extents
that have never been written to by a host will not be copied to the target volume. Following the clone copy,
any thin device extents that were allocated on the clone target that contain all zeros can be reclaimed and
added back to the target thin device’s pool. This is accomplished using Virtual Provisioning space
reclamation.
Performance considerations
The architecture of Virtual Provisioning creates a naturally striped environment where the thin extents are
allocated across all volumes in the assigned storage pool. The larger the storage pool for the allocations is,
the greater the number of devices that can be leveraged for the VMware Infrastructure 3 I/O.
One of the possible consequences of using a large pool of data devices and potentially sharing these devices
with other applications is variability in performance. In other words, possible contention with other
applications for physical disk resources may cause inconsistent performance levels. If this variability is not
desirable for a particular application, that application could be dedicated to its own thin pool. Symmetrix
supports up to 512 thin pools.
When a new data extent is required from the thin pool there is an additional latency introduced to the write
I/O while the thin extent location is assigned and formatted. This latency is approximately 1 millisecond.
Thus, for a sequential write stream, a new extent allocation will occur on the first new allocation, and again
when the current stripe has been fully written and the writes are moved to a new extent. If the application
cannot tolerate the additional latency it is recommended to preallocate storage to the thin device when the
thin device is bound to the thin pool.
Thin pool management

When storage is provisioned from a thin pool to support multiple thin devices there is usually more
“virtual” storage provisioned to hosts than is supported by the underlying data devices. This is one of the
main reasons for using Virtual Provisioning. However, there is a possibility that applications using a thin
pool may grow rapidly and request more storage capacity from the thin pool than is actually there. This
condition is known as oversubscription. This is an undesirable situation. The next section discusses the
steps necessary to avoid oversubscription.

Thin pool monitoring
Along with Virtual Provisioning come several methodologies to monitor the capacity consumption of the
thin pools. The Solutions Enabler 6.5 symcfg monitor command can be used to monitor pool
utilization, as well as display the current allocations through the symcfg show pool option. There are
also event thresholds that can be monitored through the SYMAPI event daemon, thresholds that can be set
with the Symmetrix Management Console, and SNMP traps that can be sent for monitoring by EMC
Ionix™ ControlCenter® or any data center management product.
System administrators and storage administrators must put processes in effect to monitor the capacity for
thin pools to make sure that they do not get filled. The pools can be dynamically expanded to include more
data devices without application impact. These devices should be added in groups long before the thin pool
approaches a full condition. If devices are added individually, hot spots on the disks can be created when
much of the write activity is suddenly directed to the newly added data devices because other data devices
in the pool are full.
Exhaustion of oversubscribed pools

If a thin pool is oversubscribed and has no available space for new extent allocation, different behaviors
can be observed depending on the activity that caused the thin pool capacity to be exceeded. If the capacity
was exceeded when a new virtual machine was being deployed using the Virtual Center wizard, the error
message shown in Figure 32 is posted. An I/O error message appears when the thin pool capacity is
exceeded while using command line utilities on the service console.
The behavior of virtual machines when the thin pool is oversubscribed depends on a number of factors
including the guest operating system, the application running in the virtual machine, and also the format
that was utilized to create the virtual disks. The virtual machines behave no differently than the same
configuration running in a nonvirtual environment. The white papers referenced in the “References” section
should be consulted for further details. In general the following comments can be made for VMware
Infrastructure 3 environments:
1. Virtual machines configured with virtual disks using the eagerzeroedthick format continue to operate
without any disruption.
2. Virtual machines that do not require additional storage allocations continue to operate normally.
3. If any virtual machine in the environment is impacted due to lack of additional storage, other virtual
machines continue to operate normally as long as those machines do not require additional storage.
4. Some of the virtual machine may need to be restarted after additional storage is added to the thin pool.
In this case, if the virtual machine hosts an ACID-compliant application (such as relational databases),
the application performs a recovery process to achieve a transactionally consistent point in time.
5. The VMkernel continues to be responsive as long as the VMware ESX server is installed on a device
with sufficient free storage for critical processes.

Figure 32. Error message displayed by Virtual Infrastructure Client on thin pool full
conditions
Conclusion
EMC Symmetrix Virtual Provisioning provides a simple, noninvasive, and economical way to provide
storage for VMware Infrastructure 3 environments. The VMFS behavior dovetails very nicely into this
type of configuration, while certain operations may not allow VMware administrators to realize the full
benefits of the product. The following scenarios are optimal for use of Virtual Provisioning with VMware
Infrastructure 3 by default:
Deployment of new virtual machines using a VMware Infrastructure 3 client
Creation of virtual machine templates on virtually provisioned devices using the Compact option of the
Clone to Template wizard
Systems where overallocation of storage is typical
Systems where rapid growth is expected over time but downtime is limited
The following are situations where Virtual Provisioning may not be optimal by default (unless a change in
default allocation mechanism policy is made using VMware vCenter Server 2.5 update 6):
Provisioning new virtual machines from templates using Virtual Infrastructure
Cloning virtual machines to thin devices using Virtual Infrastructure
Performing a Storage VMotion operation
The following are situations where Virtual Provisioning may not be optimal:
VMware Infrastructure 3 environments that cannot tolerate an occasional response time increase of
approximately 1 millisecond due to writes to uninitialized blocks
Virtual machines in which data is deleted and re-created rapidly and the underlying data management
structure does not allow for reuse of the deleted space

References
White papers
New Features in EMC Enginuity 5773 for Symmetrix Open Systems Environments
New Features in EMC Enginuity 5874 for Symmetrix Open Systems Environments
Implementing Virtual Provisioning on EMC Symmetrix VMAX with Oracle 10g and 11g
Implementing Virtual Provisioning on EMC Symmetrix with Microsoft Exchange Server
Implementing Virtual Provisioning on EMC Symmetrix with Microsoft SQL Server
Implementing Virtual Provisioning on EMC Symmetrix with DB2 LUW Databases
Implementing Virtual Provisioning on EMC Symmetrix DMX with Sybase ASE
Using EMC Storage Viewer for VMware Infrastructure Client
Technical note
Best Practices for Fast, Simple Capacity Allocation with EMC Symmetrix Virtual Provisioning
Feature sheet
Symmetrix Virtual Provisioning Feature Specification
Product documentation
EMC Solutions Enabler Symmetrix Array Management CLI Product Guide
EMC Solutions Enabler Symmetrix Array Controls CLI Product Guide
EMC Solutions Enabler Symmetrix CLI Command Reference


White Paper Symmetrix Virtual Provisioning

Uploaded by

Copyright:

Available Formats

White Paper Symmetrix Virtual Provisioning

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

White Paper Symmetrix Virtual Provisioning

Uploaded by

Copyright:

Available Formats

Implementing Virtual Provisioning on

EMC Symmetrix with

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Executive summary ............................................................................................ 4

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Table 2. Virtual Provisioning terms

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

EMC Symmetrix Virtual Provisioning

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 1. Thin devices and thin pools containing data devices

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Caution: Striped thin metavolumes cannot be expanded while preserving data.

Thin pool write balancing

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 2. Full Symmetrix thin pool before a write balance operation

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 6. Completed thin pool write balancing operation

Virtual Provisioning space reclamation

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 8. Initiating space reclamation on a Symmetrix thin device

Figure 9. Space reclamation on a Symmetrix thin device

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Device visibility and access

VMware file system on thin devices

VMware file system creation and formatting

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 11. Metadata area reserved on the VMware file system

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 13. VMware file system allocation mechanisms

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Allocation mechanism VMware kernel behavior

“Zeroedthick” allocation format

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 18. Zeroedthick virtual disk allocation on Symmetrix thin devices

“Eagerzeroedthick” allocation format

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 20. Eagerzeroedthick virtual disk allocation on Symmetrix thin devices

Impact of guest operating system activities on thin pool

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Changing the default virtual disk allocation mechanism policy

1. Log in to the service console of an ESX 3.5 host.

2. Open the file vpxa.cfg located at /etc/opt/vmware/vpxa/vpxa.cfg.

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

5. Repeat the process on all ESX hosts.

Duplicating virtual machines

Cloning virtual machines and the impact on virtually provisioned devices

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Figure 26. Using the Compact option when creating templates

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Deploying new virtual machines from templates

Implementing Virtual Provisioning on EMC Symmetrix with VMware Infrastructure 3

Cold migration and Virtual Provisioning

Storage vMotion and Virtual Provisioning

Nondisruptive expansion of VMFS datastore on thin devices