Fr

ee

VMware
Virtual
SAN
is
a
radically
simple,
hypervisor-converged storage system designed and
optimized for vSphere virtual infrastructure. Traditional
storage solutions such as monolithic storage arrays and
complex management tools are often ill-suited to the needs
of the modern data center. Software-defined storage
solutions, such as VMware Virtual SAN, integrate the
storage side of the infrastructure with the server side and
can simplify management and improve flexibility.
You will start with an introduction to VSAN and object
storage, before moving on to hardware selection. Later,
you will go through the preparation that must be performed
in your existing infrastructure to support the pending
VSAN deployment. Next, you will touch base with Storage
Policy-based Management and also cover policy changes,
maintenance, validation, and troubleshooting VSAN. Finally,
you will move on to recipes which will help expedite the
problem resolution process that will provide you with all the
information required to find a rapid resolution.

Who this book is written for

Prepare your infrastructure for VMware

Virtual SAN
Plan and build a solution that will suit
your needs
Implement VMware Virtual SAN
Harness the power of policy-based
management
Increase or decrease the scale of your
Virtual SAN as needs change
Monitor your Virtual SAN infrastructure
effectively
Respond to and troubleshoot problems
Understand the technical underpinnings
of VMware Virtual SAN

$ 39.99 US
25.99 UK

professional expertise distilled

P U B L I S H I N G

Jeffrey Taylor

If you are an administrator of a VMware vSphere

infrastructure who wants to simplify storage delivery by
integrating storage into vSphere, Virtual SAN is for you.
No extensive storage background is needed as VMware
Virtual SAN integrates into the existing vSphere solutions
with which you are already familiar.

What you will learn from this book

VMware Virtual SAN Cookbook

VMware Virtual SAN

Cookbook

Sa

pl
e

Q u i c k

a n s w e r s

t o

c o m m o n

p r o b l e m s

VMware Virtual SAN

Cookbook
The perfect guide to successful VMware Virtual SAN implementation
and operations, with recipes to guide you through the process

Prices do not include

local sales tax or VAT
where applicable

Visit www.PacktPub.com for books, eBooks,

code, downloads, and PacktLib.

Jeffrey Taylor

professional expertise distilled

P U B L I S H I N G

In this package, you will find:

The author biography

A preview chapter from the book, Chapter 1 'Hardware Selection for Your
VSAN Cluster'
A synopsis of the books content
More information on VMware Virtual SAN Cookbook

About the Author

Jeffrey Taylor is an IT professional with deep knowledge of VMware vSphere and storage
infrastructures. He has been working with VMware's global support services for nearly 5
years and is currently a staff engineer with a focus on Virtual SAN and storage infrastructure.
Before his current role, he worked on a variety of software platforms for a global financial
processor, with an emphasis on Unix/Linux midrange systems and mission-critical distributed
applications.

Preface

Preface
VMware Virtual SAN (VSAN) is the converged storage solution for VMware vSphere.
Integrated directly into the hypervisor, this is a native solution with no VM-based
intermediaries for storage delivery. As a result, VSAN is fast, simple to deploy and
manage, and integrates tightly with the existing VMware vSphere product suite.
As the solution is native to ESXi and runs on the hypervisor itself, its architecture is simplified
and the storage-delivery mechanism is tightly integrated and uniquely suited to the needs
of vSphere infrastructures. VSAN is an object-oriented storage solution, where each VM
is comprised of a number of objects physically distributed across the ESXi cluster. This
object-oriented nature means that access to your data happens natively, without the need
for traditional intermediate storage protocols like iSCSI or Fibre Channel.
VSAN's object-oriented nature also means that your VMs exist as a series of distributed
objects rather than a series of monolithic files. Whereas, with traditional storage, you have a
series of files (configuration files, virtual disks, swap, snapshots, and so on), with VSAN you
have a coordinated and related series of objects. These objects are a container for small
configuration files (the namespace object, also called VM Home), objects for each virtual
disk, objects for each snapshot, and an object for the VM swap space.
The key point when discussing VSAN's architecture, is how it integrates into the existing
hypervisor infrastructure to deliver the final service of any hypervisor: production virtual
machines. Behind this integration of the storage and compute virtualization layers is the
notion of hyper-converged infrastructure. VSAN is one element of converged infrastructure.
The goal of hyper-converged infrastructure is to abstract the traditional demarcations within
the environment (compute, network, and storage) by converging all aspects of the datacenter
into a software-defined model with a centralized control plane and a distributed data/IO plane.

Preface
VSAN is the storage element of the software-defined datacenter. User data (VMs) is abstracted
and distributed across the compute cluster. Each VM exists as a collection of related objects,
distributed optimally within the compute resources. VSAN natively integrates with vCenter
and its associated management tools. As a result, VSAN brings truly native, truly integrated
management of the storage system into the existing and familiar vSphere operating structure.
A major benefit to this new operating model of the storage system is its simplicity and
centralized management. VSAN obviates the need for traditional monolithic storage arrays
connected via traditional protocols. It also significantly improves on existing virtualization-centric
distributed storage solutions, typically delivered as virtual storage appliances (VSAs). Embedding
the distributed storage system into the hypervisor allows for gains in performance and
management. Native management through the vCenter Server means that storage for the
virtual infrastructure can now be deployed and managed by the virtualization engineer.

What this book covers

Chapter 1, Hardware Selection for Your VSAN Cluster, provides guidance on how to build your
own VSAN cluster hardware, or how to select from a VSAN Ready Node.
Chapter 2, Initial Configuration and Validation of Your VSAN Cluster, provides the step-by-step
procedure to configure your new VSAN cluster and ensure that it is working properly.
Chapter 3, Storage Policy-based Management, introduces the concept of policy-based
management, outlines how and why it is useful, and shows how to implement and use
storage policies.
Chapter 4, Monitoring VSAN, outlines how to monitor the VSAN cluster following deployment
and into production.
Chapter 5, VSAN Maintenance Operations, describes how VSAN maintenance is performed,
from cluster expansion to patching to modifying virtual machines.
Chapter 6, Ruby vSphere Console, covers the VSAN-related aspects of the powerful RVC utility
that is bundled with the vCenter Server.
Chapter 7, Troubleshooting VSAN, discusses how to identify and resolve various problems that
may occur in the Virtual SAN environment.
Chapter 8, Support Success, should you need to contact VMware Support to pursue issue
resolution, this chapter sets you up for success by outlining the steps you should take
to ensure a smooth and rapid support engagement.
Chapter 9, VSAN 6.0, covers specific changes in VSAN 6.0 and calls out procedural
differences between VSAN 6.0 and VSAN 5.5.

Preface
Appendix A, Chapter-specific Expansion, highlights a number of VSAN concepts that we
discussed throughout this book, which can be expanded with supplementary information
to improve comprehension. This information is presented on a chapter-by-chapter basis.
Appendix B, Additional VSAN Information, highlights additional technical details about Virtual
SAN and provides information about useful third-party tools.

Hardware Selection for

Your VSAN Cluster
In this chapter, we will discuss the following topics, with a recipe for each:

Using the VMware Compatibility Guides

Selecting a server platform

Selecting a storage controller

Selecting a solid-state drive (SSD) for the cache tier

Selecting hard disk drives/magnetic disks

Deciding on a network standard

Choosing a VSAN Ready Node (an alternative option)

Introduction
VSAN can only be as good as the hardware on which it runs. VSAN has special requirements
for its hardware, and so elements of your VSAN hardware will need to be selected against
a stricter subset of the overall VMware Compatibility Guide. Each element of your future
VSAN node will need to be compliant with the vSphere and VSAN Compatibility Guides to be
production-ready and ensure that the configuration will be supported by VMware.
Depending on your specific needs, you may find that the VSAN Ready Node will better fit your
needs. VSAN Ready Nodes are preconfigured systems built by the VMware hardware partners
to be VSAN ready, so the machines can simply be purchased and deployed.
You will need to decide whether specifying your own system configuration, or selecting a VSAN
Ready Node, is the best choice for your infrastructure. This chapter will prepare you for either
choice. VSAN Ready Nodes are covered in the last recipe in this chapter.
1

Hardware Selection for Your VSAN Cluster

Using the VMware Compatibility Guides

While most system vendors offer hardware that is compatible with VMware vSphere, only a
subset of the hardware in the VMware Compatibility Guide is applicable for use for VSAN. It is
important to make sure that all applicable hardware exists in both guides. The storage-specific
components like hard-disk controllers and disks must meet the more-exhaustive performance
and reliability requirements for VSAN.

Getting ready
You should have some idea of your hardware requirements. This includes an estimate of your
needs for system memory (RAM), along with processor power, networking needs, and storage
requirements in terms of capacity and performance.

How to do it
Go to your web browser and navigate to both the regular VMware vSphere Compatibility
Guide at http://www.vmware.com/resources/compatibility/search.php
and the VMware VSAN Compatibility Guide at http://www.vmware.com/resources/
compatibility/search.php?deviceCategory=vsan.
The first link will take you to the standard VMware Compatibility Guide, and the second
will take you to the VSAN sub-guide. By default, you will land on the system/server page,
to help you select a server platform.

Selecting a server platform

For VSAN, the only requirement in terms of the server platform is that it needs to appear in the
regular VMware Compatibility Guide for vSphere 5.5 or 6.0, as applicable to your deployment.
Any compatible/certified server is acceptable for use with VSAN.

Getting ready
You should be on the VMware Compatibility Guide Systems/Server page.

How to do it
You will need to filter your selection to restrict output to only the relevant results. To do this,
carry out the following steps:
1. Within Product Release Version:, select the most recent vSphere ESXi 5.5 or
ESXi 6.0 update release.
2

Chapter 1
2. If you have a brand preference, select it within Partner Name.
3. While most systems in the Compatibility Guide are rack mount systems, if you have
other form-factor needs, select those within System Type.
4. If you need dedicated or expanded functionality (such as graphics acceleration for
VMware Horizon View or I/O redirection), make that selection within Features.
5. Select your CPU socket/core and/or brand needs within the Sockets:, Max Cores
per Socket:, and CPU Series sections.
6. VSAN requires at least two CPU cores across one or more sockets. Any field left blank
assumes an inclusive search and all subcategories will be included.
7.

Once you have selected all of your requirements, click on Update and View Results
to see a list of all the compatible systems meeting your criteria.

Example of the compatibility guide once filters are applied:

Hardware Selection for Your VSAN Cluster

There's more
VSAN does also impose additional resource needs. Depending on the size of the disks you
choose and how many, VSAN will consume additional compute and memory resources. If you
typically run your systems close to the margins, in terms of compute/memory resources (or if
you are planning to with your new cluster), consider specifying a modest amount of additional
CPU and memory in the configuration. VSAN is capped at 10 percent resource utilization for
CPU and memory as a maximum, so adding at least 10 percent to your assumed consumption
should leave you with comfortable operating margins.
Additionally, VSAN requires a local boot device, either an SD card, or other low-performance
solid-state memory, or a dedicated hard disk drive, or SSD. The boot device should be at least
16GB, in accordance with standard VMware recommendations. If your server configuration
equals/exceeds 512 GB of system RAM, then you must use a hard disk drive or SSD as your
boot device to ensure supportability, otherwise, a core dump cannot be written in the event
of a system crash.

Selecting a storage controller

Selecting an appropriate storage controller is one of the most important decisions you will
make when creating a VSAN server configuration. The storage controller has tremendous
weight in terms of I/O performance and reliability. Because of the importance of the storage
controller on the overall performance and reliability of your VSAN cluster the storage controller
must be selected from the VSAN-specific subset of the overall VMware Compatibility Guide.

Getting ready
You should be on the VMware VSAN Compatibility Guide page.

How to do it
The initial landing page for the VSAN Compatibility Guide will allow you to browse through
VSAN Ready Node configurations. As we will be discussing the case of VSAN Ready Nodes
later in this chapter, for now please click the "Build Your Own based on Certified Components"
link from the main page of the VSAN Compatibility Guide:

Chapter 1

Unlike the regular VMware Compatibility Guide, the VSAN-specific component guide is laid out
as a branching tree that you can use to drill down to your desired configuration, after which
time hardware choices will be displayed.
1. In the leftmost pane, select I/O Controller.
2. In the next pane, select the most-recent vSphere 5.5 or 6.0 update release.
3. In the next pane, select a brand name if desired.
4. In the next pane, specify whether you wish to view the SAS, SAS-RAID, or SAS/SATA
RAID controller types.
Some controllers for VSAN support the pass-through (JBOD) mode,
and some require you to create single-disk RAID-0 sets. Pass-through
controllers tend to be easier to configure and make drive-replacement
simpler. RAID-0 controllers are more common and typically have larger
queue depths. When selecting controllers and disks, please keep in mind
the performance and reliability differences between drive technologies.
Generally speaking, SAS and Nearline SAS (NL-SAS) disks have deeper
queue depths and are higher-performance. SAS disks are typically more
reliable than their SATA and NL-SAS counterparts.

Hardware Selection for Your VSAN Cluster

After making your selections, choose Update and View Results to get a list of hardware that
matches your specifications.

There's more
Aside from simply ensuring that you select a supported storage controller, the VSAN-specific
Compatibility Guide also provides you with additional detail that will be crucial to selecting
your storage controller.
This view will tell you whether or not the controller supports the pass-through/JBOD mode and,
vitally, it will tell you the controller's command queue depth. The controller's queue depth is a
vital consideration. If each node is relatively small with one disk group, any queue depth above
256 commands is acceptable. If you have two disk groups, please consider 512 commands to
be the minimum queue depth.
For large or high-performance configurations with several disk groups and extremely
large capacities, and/or very high-performance SSDs, you should choose the highest queue
depth you can. Standard controllers with maximum queue depths usually have around 1000
commands, like in the preceding example.
Queue depth can have a tremendous effect on the performance of streaming data. Opting for
controllers and disks with deeper queues will provide better overall performance, particularly
when VSAN is reconfiguring or resynchronizing data.

Chapter 1

Selecting a solid-state drive (SSD) for the

cache tier
In combination with the storage controller, the SSD for the cache tier is the most important
choice you can make in terms of the long-term reliability and performance of your VSAN
deployment. The cache-tier SSD is used for caching reads and writes in VSAN hybrid
configurations, and for caching writes only in all-flash configurations. SSDs are graded in
many ways, and these grades for performance, write resiliency, and fabrication technology will
all affect your selection. The VSAN Compatibility Guide gives you an overview of all of these
factors to help you make the best choice.
For the SSD:

The write-performance class is on a scale of A-F, with the F class

being the fastest (class A is technically deprecated).

Write-resiliency is on a scale of A-D, with D being the most resilient.

Fabrication technology is either multi-layer cell (MLC), or single-layer

cell (SLC). SLC is usually more performant and resilient, at the cost
of lower capacities and higher price.

SSDs come in either SATA, SAS, or PCIe connections. PCIe SSD cards are
usually at the top end in terms of performance. Given that SAS and PCIe
interfaces permit significantly higher device command queues, it is a good
idea to use SAS or PCIe type SSDs, especially if you are using a pass-through/
JBOD controller that directly leverages device-level command queues.

Getting ready
You should be on the VMware VSAN Compatibility Guide component page.

How to do it
The Compatibility Guide for SSD is navigated in the same way as for the I/O controller.
1. In the leftmost pane, select SSD.
2. In the next pane, select the most recent vSphere 5.5 or 6.0 update release.
3. In the next pane, select a brand name if desired.
4. Specify whether are searching for All Flash or Hybrid cache tier.
5. In the various other fields and drop-down menus, you can select the interface type,
manufacturer, performance class, capacity, and form-factor.

Hardware Selection for Your VSAN Cluster

6. After making your selections, choose Update and View Results to get a list of
hardware that matches your specifications:

There's more
Determining the size of your capacity-tier (magnetic or SSD) storage will directly affect the
sizing decision for your cache-tier SSD. In general, the SSD should be 10 percent of the size
of the magnetic disks in the underlying disk group, for example, if your disk group will consist
of four 1.5 TB capacity-tier disks, you will have 6 TB of bulk storage. To accommodate this,
the SSD should be about 600 GB. Alternatively, you may choose to opt for two disk groups
consisting of three 1TB capacity disks each. In this case, each disk group should have a
cache-tier SSD of 300GB. The ratio of cache-tier SSD to capacity-tier disks or SSDs should
be approximately 1:10.

Chapter 1

See also
For additional guidance regarding how to appropriately size the VSAN capacity and
cache tiers, please see the section Chapter 1 VSAN Capacity Planning of Appendix A,
Chapter-specific Expansions.

Selecting capacity tier disks

The magnetic disks or SSDs you choose will be used for storage capacity and persistent data
that is destaged from cache. This is the capacity tier within VSAN, whereas the caching tier
SSD will act as the performance caching layer.
In general, you will want to select magnetic disks or SSDs that have adequate capacity to fit
your needs. For highly dynamic workloads where data will be frequently destaged from the
SSD write buffer and fetched into the SSD read cache, HDD performance is important and you
may wish to go with faster disks and/or SAS disks. Only SAS and SATA disks are supported for
use with VSAN.

Getting ready
You should be on the VMware VSAN Compatibility Guide component page.

How to do it
The Compatibility Guide for SSD is navigated in the same way as for the I/O controller
1. In the leftmost pane, select HDD or SSD.
2. In the next pane, select the most recent vSphere 5.5 or 6.0 update release.
3. In the next pane, select a brand name if desired.
4. In the various other fields and drop-down menus, you can select the interface type,
manufacturer, disk speed (RPM), capacity, and form-factor.

If we are pursuing an all-flash VSAN configuration, ensure that we select

Virtual SAN All Flash Capacity Tier from the Tier: pane.

Hardware Selection for Your VSAN Cluster

5. After making your selections, choose Update and View Results to get a list of
hardware that matches your specifications.

There's more
In general, SAS disks outperform SATA disks of equivalent capacities and/or rotational speeds
because SAS drives use more robust recording technique, deeper queues or both. When cost
is a concern, slower SAS drives (typically 7200 RPM; also called near-line SAS or NL-SAS) are
usually built on cheaper SATA platforms but include enterprise-grade features like deeper
command queues, error-correction, dual-channel connections and native SCSI support.
Low-end SAS drives are typically better than high-end SATA drives despite the shared
technology platform and costs are usually not significantly higher. NL-SAS is a great alternative
to SATA for building out a cost-conscious capacity tier when HDD performance is a factor.

10

Chapter 1

See also
Before settling on a Cache + Capacity disk combination, please review the Chapter 1
VSAN Capacity Planning section of Appendix A, Chapter-specific Expansions for a verbose
description of the capacity expectations and recommended maximums to help you build your
VSAN cluster to an appropriate scale.

Deciding on a network standard

For smaller clusters, network speed is typically only a forefront concern during times of data
reconstruction in the event of a node/disk failure, rebalancing, or user-invoked configuration
changes. These are generally fairly rare operations and, for the most part, 1GbE networking
will be adequate for clusters with fewer than 5 nodes. 10GbE networking is recommended as
a VMware best practice for all clusters. 10GbE networking should be considered mandatory
for larger clusters, especially clusters of >8 nodes. If 1GbE network interfaces are being used
for VSAN, those interfaces need to be dedicated to use by VSAN. 10GbE interfaces can be
shared between VSAN and other workloads.
Unlike disks and storage controllers, there is no special compatibility
guide for network controllers. Any vSphere-compatible network
controller is acceptable for use with VSAN.

Getting ready
You should be on the VMware Compatibility Guide IO Devices page.

How to do it
1. Within Product Release Version:, select the most recent vSphere 5.5 or 6.0
update release.
2. If you have a brand preference, select it within Brand Name.
3. Select Network, within I/O Device Type.

11

Hardware Selection for Your VSAN Cluster

4. After making your selections, choose Update and View Results to get a list of
hardware that matches your specifications.

See also
Before settling on a networking standard, please take a look at a detailed description of the
VSAN networking options and recommendations in the VSAN Network Considerations section
of Appendix B, Additional VSAN Information.

Choosing a VSAN Ready Node

(an alternative option)
To help simplify the hardware selection process, numerous VMware partners have
preconfigured "VSAN Ready Nodes" that are compliant with the various aspects of the
vSphere and VSAN compatibility guides. With VSAN Ready Nodes, the sizing decisions, disk/
SSD technology, network standards, and so on, have already been determined by the systems
vendor. You may find that a VSAN Ready Node will simplify your purchasing and designing
decisions for new VSAN build-outs and/or if you have a preferred system vendor.
12

Chapter 1

Getting ready
You should be on the VMware VSAN Compatibility Guide landing page.

How to do it
VSAN Ready Nodes are configured by the manufacturer and are designed
to target specific infrastructure scales, or for specific use cases, such as
virtual desktops (VDI). These scales are known as Ready Node Profiles.
A matrix that defines these profiles can be found by clicking on the
Virtual SAN Hardware Quick Reference Guide link at the top of the
VSAN Compatibility Guide page.

1. The VSAN Compatibility Guide for VSAN Ready Nodes is navigated in a similar manner
to the component-oriented Compatibility Guide. The guide is presented as a series of
drill-down categories.
2. Select the parameters that fit your needs from the Ready Node type (All Flash or
Hybrid), preferred manufacturer, performance/scale profile, form-factor, etc.
3. Once your choices have been selected, click the "Update and View Results"
button to populate a list of compliant ready-node configurations:

13

Hardware Selection for Your VSAN Cluster

4. The initial output will be a collapsed list of the various Ready Node configurations that
are compliant with your selected filters. The output is initially grouped by vendor.
5. To get additional information about the specific Ready Node configuration, twirl
down the arrow next to the vendor that you wish to use and then you can expand
configuration information about the specific server platform that is recommended:

6. You can then use the SKU number (pictured above) when you contact your preferred
vendor to get pricing and ordering information for the Ready Node.

See also
As with user-specified build-outs, please review the Chapter 1 VSAN Capacity Planning
section of Appendix A, Chapter-specific Expansions for a verbose description of the capacity
expectations and recommended maximums, to help you select an appropriate VSAN
Ready Node.

14

Get more information VMware Virtual SAN Cookbook

Where to buy this book

You can buy VMware Virtual SAN Cookbook from the Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.

www.PacktPub.com

Stay Connected: