NetApp Product Analysis FAS3100 V3100

Download as pdf or txt
Download as pdf or txt
You are on page 1of 26

ES/OL

Evaluator Series On-Line


Network Attached Storage
Product Analysis
June 2010
Page 1 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
NetApp provides a family of products called the FAS and V series, which are targeted at the mid-tier
enterprise. These systems include the wealth of features offered by the Data ONTAP operating environment.
The FAS3100 storage systems and the V3100 gateway series offer file and block based storage in a common
platform. NetApp provides families or series of products targeted with different performance and capacities to
meet customer requirements in different market segments. The general division of the series is:
FAS6000 / V6000 high end storage systems
FAS3100 / V3100 mid-tier storage systems
FAS2000 entry level storage systems
This document provides an overview and analysis of the major features and functions of the NetApp FAS3100
and V3100 storage systems. NetApp systems use the Data ONTAP operating environment, which is detailed
in a separate Evaluator Group analysis document.
HIGHLIGHTS
Networked storage access for blocks and files via multiple protocols
Virtualized storage pools
File access via NFS and CIFS
Block SCSI access via FCP over FC and iSCSI over IP
Utilizes NetApps proprietary Data ONTAP operating system
Primary storage data deduplication using Data ONTAP capabilities
Snapshots, Fast Boot, FlexVol, telnet, e-mail alerts, NIS, DNS, LDAP, SNMP, FilerView, FlexShare,
FlexClone, NDMP and AntiVirus support
Up to 840 TB storage capacity
Up to 840 disk drives, FC or SATA
Up to 14 fibre channel ports per controller
Up to 14 gigabit Ethernet ports per controller

OVERVIEW
The 3100 line represents a third generation of the 3000 series, with evolutionary upgrades to the previous
products in performance, capacity and expandability. The upgrades are coupled with evolutionary updates to
the operating environment that deliver many of the features, Data ONTAP.
The controllers for the 3100 series are now integrated into a single 6U chassis. While single controller systems
are still available, they are enclosed within the 6U chassis, a design that now more closely matches
competitors systems.
New with the introduction of the FAS3100 series is the release of the gateway V-series 3100 models.
Previously, NetApp had maintained separate release and qualification cycles for their integrated FAS series
and the V-series. References to 3100 series will indicate the feature is applicable to both FAS and V-series
models, while differences are noted with specific reference to either a FAS or V moniker.
ES/OL
Evaluator Series On-Line
Network Attached Storage June 2010
Product Analysis Page 2 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
The FAS3100 fabric-attached storage (FAS) series supports fibre channel SAN, IP SAN (iSCSI) and network-
attached storage (NAS) environments via NFS and CIFS. NetApp refers to fabric-attached storage systems as
storage connected to hosts via any network. The FAS series is part of NetApps strategy to offer a network
storage system that is capable of supporting both NAS and storage-area network (SAN) accessibility from a
single storage pool.
The NetApp FAS systems are nearly identical to the V-series systems, the primary difference being the V-
series do not include NetApp disk storage, instead permitting attachment of third party storage systems.
NetApp uses the term fabric-attached storage (FAS) for their integrated systems, with the standalone
controller-only systems known as the V for virtualization series. NetApps FAS systems support three primary
types of host attachment, including fibre channel and iSCSI SAN attach and via NAS.
NetApp uses the terminology Unified Storage Architecture when describing their primary storage systems,
which includes the FAS and V series systems. For both, the underlying storage processing and intelligence
are the same, with the difference being V-Series are offered as standalone devices or gateways to third party
backend fibre channel (FC) storage systems.
Availability of the FAS3100 products is provided via NetApps standard channel resellers as well as their
OEMs, with IBM being the only current OEM partner.
The FAS3100 series from NetApp is a family of unified SAN and NAS devices targeted to cover the mid-tier
and large enterprise storage environments. The available models are:
FAS3140 / V3140 small to mid-tier enterprise environments
FAS3160 / V3160 and FAS3170 / V3170 mid-tier enterprise environments



FAS and V Series 3140, 3160, and 3170
Figure 1: NetApp 3100 Product Line

Clustered models (denoted by both an a and c suffix) are active-active configurations. In recent
announcements, NetApp utilizes product nomenclatures that are consistent with other traditional SAN storage
vendors. Specifically, dual controller configurations are not differentiated from single controller systems with
the default limits, capacity and connectivity provided for dual controller configurations. NetApps use of an a
or c suffix is still utilized in technical specifications although its use for other purposes has largely been
ended.
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 3 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
The FAS3100 series support both fibre channel and SATA disk drives. The FAS3170 supports up to 840 TB of
storage on 840 drives. All dual controller FAS3100 systems include 8 fibre channel and 8 gigabit Ethernet
ports. The primary difference between models of the 3100 series is memory and processor configurations, as
well as capacity limitations, all of which provide differentiated levels of performance. The FAS3100 series
provides high availability with support for clustered failover, RAID storage and redundant components.
Additional integrity features include disk scrubbing of data to look for unreadable locations before a disk failure
occurs and NVRAM for protection of volatile write data.

FAS and V Series Features Technical Highlights
SAN Protocol Support Fibre Channel Protocol (FCP) for SCSI; fabric-attached and direct-attached;
iSCSI
File Service Protocol Support NFS V2/V3/V4 over UDP or TCP, PCNFSD V1/V2 for (PC) NFS client
authentication, Microsoft CIFS
Other Protocol Support HTTP 1.0, HTTP 1.1 virtual hosts
LUNs 2,048
FlexVols 500 per controller
Volume/Aggregate Size 16TB
Snapshots 127,000 (maximum is 255 / Volume)
Up to 4 ports per controller for front-end (host) connectivity
Up to 4 directly connected servers (per controller or active/active system)
Number of Supported Hosts
Up to 256 SAN connected servers (per controller or active/active system)
Highly Available Controller Configurations:
Active/active controller with Controller Failover, active/active controller with
stretch (non-switch) MetroCluster, active/active controller with Fabric
Attached MetroCluster



HA Supported Configurations
Highly Available Back-End (Disk) Configurations:
Dual-path, Multipath HA Storage
Reliability Redundant hot-swappable controllers, cooling fans, power supplies, optics,
and RJ-45 ports

Management
Full-duplex 10/100/1000Base-T Ethernet onboard console, diagnostic
LED/LCD, Remote LAN Management (RLM) Card5, SNMP, telnet, SSH,
HTTP, Web (SSL), host scripting, e-mail alerts



Security
Virus Protection
Authentication: Kerberos (MIT/Active Directory)
Authorization: LDAP, NIS
Auditing: File access logging
Encryption: IPSec, SSL, SSH, SMB signing
Role Based Access Control (RBAC)

Table 1: NetApp FAS and V Series Overview

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 4 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

FAS Series Storage Features Technical Highlights
RAID-6 (RAID-DP)
FC28 (26 data disks plus 2 parity disks)
SATA16 (14 data disks plus 2 parity disks)
RAID-4
FC14 (13 data disks plus 1 parity disk)
SATA7 (6 data disks plus 1 parity disk)




Maximum RAID Group Sizes
RAID-6+RAID1 or RAID4+RAID1 (SyncMirror)

Table 2: NetApp FAS Storage Overview
Evaluator Group Comment: NetApp quotes the amount of capacity as the raw amount and
recommends using RAID-DP (also known as RAID-6) for protection. The problem with that is
the amount of actual usable capacity can be close to half that amount of real disk space.
Between the data protection usage of additional space and the allocation method used for the
file system in NetApp, much of the capacity is not available for real customer storage. The
capacity used for protection and system overhead is valuably consumed but it can be very
misleading for customers.
Detailed Model Description
The 3100 family supports both block and file host access to a combined storage pool with file connectivity
available via NFS, CIFS, HTTP and FTP via Ethernet connections. Block storage access is available over both
fibre channel and Ethernet connections, via the SCSI protocol. Provided below are specifications for the
enterprise HA cluster controller configurations for the 3100 line.
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 5 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

NetApp Model Comparison 3140a 3160a 3170a
Controllers / Config 2 / HA
active-active
2 / HA
active-active
2 / HA
active-active
Max SAN Host Connections 256 256 256
RAID Levels 4, 6, 4+1*, 6+1* 4, 6, 4+1, 6+1 4, 6, 4+1, 6+1
Max. Raw Capacity 420 TB 672 TB 840 TB
Max. # Disk Drives 420 672 840
ECC Memory 8 GB 16 GB 32 GB
NVRAM 1 GB 4 GB 4 GB
Max # LUN's 2048 2048 2048
Drive Type FAS series FC & SATA FC & SATA FC & SATA
Storage FC Interface 4 Gb 4 Gb 4 Gb
Spec NFS Performance (ops/s) SPEC sfs2008 Results Not Published
Expansion Slots: (PCI-E) 8 8 8
Onboard IO Ports:
GbE Ports 4 4 4
FC Ports 8 @ 4 Gb 8 @ 4 Gb 8 @ 4 Gb
Optional FC Connectivity:
2 port FC Disk connect HBA 8 @ 4 Gb 8 @ 4 Gb 8 @ 4 Gb
2 port FC Target HBAs 4 @ 4 Gb 4 @ 4 Gb 4 @ 4 Gb
4 port FC Target HBAs 6 @ 4 Gb 6 @ 4 Gb 6 @ 4 Gb
4 port FC Disk or Tape 8 @ 4 Gb 8 @ 4 Gb 8 @ 4 Gb
Optional 2-port Ultra320 SCSI Tape
HBA
8 8 8
Optional Network Connectivity:
4 port 1 GbE TOE 8 8 8
4 port 10 GbE TOE 8 8 8
2 port 10 Gb iSCSI HBA 8 8 8
Dual 10 GbE FCoE Target Adapter 8 8 8
4 port SAS HBA 8 8 8
Performance Acceleration Module 4 4 8
Performance Acceleration Module II 0 4 4
* RAID 4+1 and 6+1 available with SyncMirror option only

Table 3: NetApp 3100 System Overview
ES/OL
Evaluator Series On-Line
Network Attached Storage June 2010
Product Analysis Page 6 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
PRODUCT ARCHITECTURE
NetApps 3100 systems may be used for a variety of purposes, including providing file access to hosts, and
providing virtualized block data to hosts via both FC and IP networks. The NetApp implementation of the FAS
product line operates with several basic functions identified below along with specialized elements to improve
performance and add functionality.


Figure 2: NetApp FAS Architectural Components
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 7 of 26

s NetApp FAS3100 and V3100 Storage System


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Figure 3: NetApp V Series Architectural Components


ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 8 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
SOFTWARE ARCHITECTURE
NetApp software performs the RAID functionality to attached disk storage located in NetApp storage trays or
attached disk storage in the case of the V-series. Licensed software includes NetApps SnapManager,
SnapMirror, SnapVault, SnapRestore, SnapDrive, SnapLock, Clustered Failover, MetroCluster, MultiStore,
DataFabric Manager, FlexVol, FlexClone and SecureAdmin. The FAS and V series can be integrated with
ApplianceWatch for HP OpenView and Tivoli.
The software architecture of the 3100 product series is very similar to all other NetApp Data ONTAP based
systems. The Evaluator Group provides a detailed product analysis of Data ONTAP. Information on the
software features and functions of the 3100, the reader should refer to the Data ONTAP analysis.
I/O Connectivity
NetApp provides both block and file access to data with the 3100 products offering block I/O via both fibre
channel and IP networks and file access via IP networks. NetApp supports a variety of host based multi-
pathing and load balancing options including Symantecs VERITAS VxVM with DMP. Initially, SCSI was both a
block protocol and connectivity standard, although the protocol has successfully been used with other
connectivity methods including both Fibre Channel (FC) and TCP/IP. The SCSI protocol transmitted over FC
is known as FCP and SCSI over IP is known as iSCSI.
Additionally, 3100 systems may be connected to SAN attached tape devices using fibre channel interfaces for
backup and recovery. Typically, data backup is accomplished via the NDMP protocol in conjunction with
backup software from providers including EMC, IBM and Symantec. The lists of supported backup
applications and tape devices are in tables later in this document.

Evaluator Group Comment: The ability to use the same storage system for block and file storage
appeals to many customers. What is not clear are the tradeoffs in the usage when there are combined
types of I/O. What effect does streaming of files have on the cache hit ratios for block storage? It
would be useful to understand these tradeoffs before deploying a particular solution.
File I/O Connectivity
NetApp products originated providing remote file connectivity via standard protocols. Initially, NFS was the
primary protocol supported, and over time additional protocols such as CIFS, FTP, HTTP, and others have
been added. By their design, remote file connectivity protocols provide a means of ensuring data coherency
among multiple hosts. The two primary methods of remote file connectivity deployed are NFS and CIFS.
NetApp systems provide implementations of both protocols along with NetApp proprietary software in order to
provide the ability to share data in both environments simultaneously. For more information, refer to the
Evaluator Groups NetApp Data ONTAP product analysis.
Block I/O Connectivity
NetApps 3100 series systems support both FCP (SCSI over fibre-channel) and iSCSI (SCSI over TCP/IP)
protocols for block storage connectivity. NetApp supports the new ANSI T10 standard for multi-path access
known as ALUA, (Asymmetric Logical Unit Access). There is confusion in that the controllers are active/active,
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 9 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
which means each may takeover for a failed controller. However, data access via a LUN may not occur via
both controllers. In this sense, it is an asymmetric controller design similar to most other competing midrange
systems such as the EMC Clariion, HP EVA and LSI based systems.
NetApp supports iSCSI connectivity from a variety of hosts including Windows, Linux and several versions of
UNIX. For Windows hosts, NetApp supports iSCSI pathing via Microsofts MPIO framework and for Linux;
NetApp supports the iSCSI multi-pathing provided within the iSCSI distributions from RedHat and SuSE. For
UNIX distributions, NetApp supports HPs HP-UX, IBMs AIX, and Suns Solaris operating systems for the
native multi-pathing for iSCSI. NetApps support for iSCSI is detailed in subsequent tables.
Reliability and Performance
NetApp has changed the nomenclature for their cluster configurations, which were previously denoted by a c
in the model name. In place of the c denotation, NetApp now uses an a in order to denote active-active
controller configurations. However, for block data access, these systems continue to operate as asymmetric
systems as previously denoted.
In symmetric mode, file access via the alternate node can occur without a failover and timeout of several
seconds. Asymmetric systems cannot typically be used for accessing data via alternate paths due to the
associated performance penalty.
The unified SAN and NAS functionality allows the NetApp filers to serve blocks from LUNs for servers that may
be either direct attached or SAN-fabric connected. Each of the 3100 systems have two or more Ethernet
adapters installed with the system providing access via either depending on network configuration. In addition,
the Data ONTAP OS provides fabric login and other fabric access management capabilities. NetApp supports
link aggregation at the Ethernet level (802.3ad), commonly known as trunking, providing increased
performance and increased connectivity reliability in cases of line failures.
The 3100 systems offer both bundled and optional RAS features including the following:
HA Cluster, with alternate paths for access to disk via an InfiniBand interface
Redundant hot-plug power supplies and cooling fans standard on all models with load sharing to
extend life
Multiple IO connectivity which may be dynamically reconfigured
Dedicated service Ethernet port provides a service interface
Battery-backed NVRAM to maintain data consistency of WAFL filesystem
RAID protection of the data storage performed by Data ONTAP software with support for both
RAID-4 and RAID-DP (dual parity) with raid group sizes dependent upon RAID level and disk
drive type utilized.
Disk scrubbing to protect from an unreadable data block, which would be uncorrectable if read
while a drive had failed.
Dynamic online expansion of file systems allows for continued availability while increasing the file
system size.
Auto Support feature is a pro-active maintenance mechanism NetApp systems use to report
status to a NetApp global service center or an internal IT department
Filesystem snapshots (point in time copy) provide data protection and availability.
For serviceability, the 3100 line has a web-based GUI called FilerView to perform the prescribed administration
and service tasks. FilerView is included with all NetApp storage appliances using Data ONTAP.
ES/OL
Evaluator Series On-Line
Network Attached Storage June 2010
Product Analysis Page 10 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
When configuring 3100 systems for an active-active cluster arrangement, several points should be considered:
Network connections to hosts must be redundant. The cluster configuration is active-active, although
data access is tied to a particular controller. In this arrangement, sometimes known as active-passive
data access, each of the controllers may concurrently serve and manage the file systems over which it
has primary control. In the case of a failover, the remaining appliance will take over, assume the IP and
MAC address of the failed appliance and assume ownership of the file systems.
The recommended practice is to use multi-path clusters, in which there are two paths between disks
and controllers. This is accomplished utilizing the dual fibre channel loops where each appliance has
active path access to both sets of disks. With standard cluster configurations, there may only be one
connection active at a time. The minimum Data ONTAP level required for FAS3100 systems to utilize
multi-path clustering is ONTAP 7.2.1.
Clustering support requires an appliance-to-appliance link that is used for heartbeat messages and for
mirroring the NVRAM data between the two appliances. NVRAM cards utilize a private InfiniBand connection
in order to maintain their mirroring NVRAM data to the partner controller NVRAM card.


Figure 4: NetApp Cluster Configurations (Source: NetApp)
The figure above shows a graphical depiction of 3100 systems in the two primary types of cluster
configurations.
HARDWARE ARCHITECTURE
The ability to provide both block and file storage eliminates the need to deploy separate systems to address
requirements in some environments. The FAS3100 series systems support both SATA and Fibre Channel disk
drives providing users with more choice in meeting their storage requirements. The hardware platforms all run
NetApps common software platform, Data ONTAP.
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 11 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
The 3100 systems have extensive onboard Fibre Channel (FC) and Gigabit Ethernet connectivity, which can
be augmented using included PCI Express expansion slots and integrated remote LAN management. The
FAS3170 scales to 840 drives, offering 840 terabytes of raw capacity per system.
FAS3100 Architecture Overview
CPUs Dual core AMD Opterons
System memory (RAM) used for OS and data caching functions
Non-volatile RAM (NVRAM) is used as a data write cache, to help ensure data consistency
Cluster interconnect utilizes InfiniBand connections
On-board 10/100/1000 Base-T Ethernet for IP network connectivity
On-board FC connections for block SAN connectivity
PCI 32-bit and 64-bit slots for additional network and SAN storage connections
Contains an add-in card for non-volatile RAM for file system integrity
PCIe and PCI-X adapter cards:
Dual fibre channel adapters for disk device attachment.
Dual and Quad fibre channel host bus adapters for SAN attachment
Gigabit Ethernet adapter cards for Ethernet connection
SCSI adapter card for tape attachment
Dual fibre channel adapters for SAN tape backup capabilities
Cluster interconnect adapter to support clustering connection to companion
Power and cooling:
Redundant load-sharing power supplies to handle failure of a supply and to extend the life of the
power supplies by sharing the load in a non-failure condition
Dual AC power cords to allow for failure of one power source
On-board redundant battery backup for the NVRAM to allow for file system coherency data in the
event of a power loss
Redundant cooling fans to handle the failure of a fan
Packaging:
The 3100 systems use a system cabinet or rack mount for all of the components
Floppy disk drive (installed for specific service updates to 3100 systems)
Compact Flash
3100 systems have a compact flash (CF) card which contains system boot information, which
provides fast system starts
The CF can hold two versions of the Data ONTAP operating system as well as other support
software.

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 12 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Architectural Elements 3140 3160 3170
Processor Sockets 2 4 4
Processor Cores 2 2 2
Processor Type 2.5 GHz
AMD
Opteron 250
2.5 GHz
AMD
Opteron 250
2.5 GHz
AMD
Opteron 250
Memory Type DDR DDR2 DDR2
Memory Amount 4 GB 8 GB 16 GB
Chip / Memory Connect Hyper
Transport
Hyper
Transport
Hyper
Transport
Processor to Memory & IO
Interconnect
Crossbar
Switch
Crossbar
Switch
Crossbar
Switch
Disk Controllers 2 2 2
Memory Interface
Processor I/O Speed Up to 24
GB/s
Up to 24
GB/s
Up to 24
GB/s

Table 4: Detailed NetApp Single Controller Hardware Design
Evaluator Group Comment: NetApps choice of AMD processors, along with their HyperTransport I/O
provides a superior platform for multi-CPU, high I/O workloads such as high end Data ONTAP
environments. As processor capabilities have advanced, one of the greatest limitations on
performance gains has been the system architecture's ability to deliver sufficient IO and memory
bandwidth to the processor. AMD Opteron based products reduce data bottlenecks by integrating a
memory controller into the processor, as opposed to utilizing an external bridge I/O chip as Intel Xeon
chips. Additionally, IO is accessed via a separate interface, which AMD refers to as the
HyperTransport, able to transfer of over 4GB/s per link. As a result, processors are able to access
both memory and I/O via separate high-speed channels, greatly increasing bandwidth while decreasing
IO and memory access times. Use of AMD Opterons in NetApps design has improved the platforms
capabilities while still utilizing off the shelf components, thereby maintaining low design and
production costs.
SAN and Tape Connectivity
The 3100 product line also includes the capability of direct attachment to a tape drive (utilized for backup) via
SCSI. Additionally, a fibre channel adapter may be used for attachment for SAN tape backup.
SAN Connectivity
Supported HBAs include Emulex and the Qlogic HBAs for UNIX, Linux and Windows environments. A list of
supported devices is provided below.
ES/OL
Evaluator Series On-Line
Network Attached Storage June 2010
Product Analysis Page 13 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

OS Vendor HBA Multipath Host Cluster Volume Mgr File System

Emulex
Qlogic
MPIO
DMP*
MSCS
VCS*
LDM
VxVM
NTFS
VxFS

Native
Emulex
MPxIO*
DMP
Sun Cluster*
VCS
SVM*
VxVM
UFS*
VxFS

Native SANpath
MPIO*
DMP
HACMP
VCS
LVM
VxVM
JFS/2
Raw
VxFS

Qlogic
Emulex
DM-MP
Qlogic
Oracle 9i, 10g RAC
RH Cluster Suite
LVM ext2, ext3
Reiser
GFS

Qlogic DM-MP
Qlogic
Oracle 9i, 10g RAC LVM ext2, ext3
Reiser

Native
Emulex*
HP PVLinks
DMP
MC ServiceGuard
VCS
LVM
VxVM
JFT/HFS
Raw
VxFS

Emulex
Qlogic
VMware MSCS
Virtual Center
(Vmotion)
VMware VMFS 2.x
Raw

Qlogic Qlogic Novell Clusters - NSS

Table 5: NetApp SAN HBA and multi-pathing support

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 14 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
The SAN switches supported with the 3100 systems are listed below.

Storage Array SAN Connectivity
Brocade Silkworm 2400/2800, (Tape only)
3200, 3250, 3800, 3850, 3900, 4012, 4020,
4100, 4900, 12000, 24000, 48000
Qlogic 5600, 5602
Cisco MDS 9120, 9140, 9216, 9506, 9509, 9513,
9020, 9124, 9222i, 9134
Sphereon 32XX, 4300, 4400, 4500, 4700
Intrepid 6064, 6140, ED-5000
McDATA
ES 3016, ES 3032

Table 6: NetApp SAN Switch connectivity
Tape Connectivity
Backup offerings are offered in partnerships with leading third party ISVs, SAN switch vendors, tape drive and
tape library vendors.
Backup and restore over a FC SAN provide the following features when used with NetApp 3100 systems:

Library and tape sharing to improve utilization of tape resources
Extended distances from data to centralized tape backup libraries
Minimal impact from backups on servers on the network
Tape drive swap
Dynamic tape configuration changes without bringing down the NetApp system
Up to eight (8) concurrent dumps to shared tape resources.

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 15 of 26

stems NetApp FAS3100 and V3100 Storage Sy


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com


Figure 5: NetApp SAN tape Support (Source: NetApp)

Utilizing fibre channel fabric tape backups provides more flexibility than conventional direct-attached methods
by allowing multiple NetApp filers and tape devices to be shared on the same SAN. Up to 15 NetApp systems
in combination with up to 64 tape devices each with up to 16 media changers are supported in a fabric. Some
of the connectivity options are outlined above.
Because each system has visibility to all tape devices in the SAN, these configurations enable dynamic tape
sharing and optimal utilization of resources, when used with Network Data Management Protocol (NDMP) and
software applications that support dynamic tape sharing.
FC SAN tape backup enables hot swapping and dynamically changing tape configurations after the filer is
running. A Fibre Channel tape drive or library may be connected and disconnected from the Fibre Channel
switch, and the switches rescans the ports to determine whether devices were added or removed. The NetApp
system detects these configuration changes without requiring a restart. Drives that have been removed are
disabled with new drives automatically configured and exported for use by applications and users. A listing of
supported tape drives is provided below.
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 16 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Tape Drive Support Interfaces
Certance LTO2, LTO3 Ultrium SCSI
DLT 7000, DLT 8000 SCSI
Exabyte Mammoth (I, II) SCSI
Exabyte Mammoth II FC
HP LTO SCSI
HP LTO 2 SCSI/FC
HP LTO 3 , 4 SCSI/ 2&4Gb FC
IBM 3590 SCSI/FC
IBM 3590B, 3590E, 3590H,
3590E05
FC
IBM 3592J1A , TS1130 FC
IBM LTO FC
IBM LTO 2, 3, 4 Ultrium SCSI/FC
IBM LTO 3 ULT3580 SCSI/FC
IBM ULT3580-TD1--LTO SCSI
IBM Ultrium-TD1LTO SCSI
Quantum DLT-S4 FC
Quantum DLT-V4 SCSI
Quantum SDLT 320 SCSI
Quantum SDLT 600 SCSI/FC
Quantum Super DLT 220 SCSI
Seagate Ultrium (LTO) SCSI
Sony AIT 1, AIT 2, AIT 3,
AIT-3Ex
SCSI
Sony AIT4, SDX-900V SCSI
Sony AIT-5, SDX-1100 SCSI
Sony GY-2120--DTF SCSI
Sony GY-8240--DTF2 SCSI/FC
Sony S-AIT SCSI/FC
Sun STK 9840, 9940, 9840B,
9940B
SCSI/FC
STK T9840C, D FC
STK T10000A, B FC
Sun LTO 3 FC
Tandberg Data LTO 2, 3, 4 SCSI

Table 7: NetApp Tape Drive Support
Evaluator Group Comment: Many, if not most, customers are moving beyond using tape as a data
protection solution. It would be useful for NetApp to explain the alternative data protection enabled
using NetApp systems for those customers. This may provide motivation to move to better data
protection systems and could benefit NetApp with usage of additional software functionality.
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 17 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
Device naming convention: After a tape drive is discovered by the filer, the drive is allocated a name using
the format stn, where n is a number assigned to the drive based on when it was discovered. Tape libraries
(also known as media changers mc) follow a similar naming convention. The following are some limitations
with earlier naming conventions:
Data ONTAP assigns device addresses on a first-come first-served basis.
Assigned stn addresses do not move when the device is moved.
Addresses might not be reassigned in the same order after reboot.
Unlike direct attached SCSI devices, the order of device discovery on a SAN is not fixed. New tape drives can
be easily added or removed from the system. Names assigned to one device can be assigned to a different
device after a reboot. This dynamic naming scheme is problematic with tape drives and libraries on a SAN. To
solve this naming scheme problem, the tape aliasing method was developed.
Tape aliasing provides a correspondence between the logical names of a backup device, for example (st0,
mc1), and a name permanently assigned to a port or to a tape device. The following are three naming
schemes used to assign permanent names to logical names of tape devices:
Logical names provide software assigned name given to the tape device by the filer. The format for tape drive
logical name is stn, where n is an integer greater than or equal to 0. The value of n is assigned to the drive
based on when it was discovered. Tape libraries (also known as media changers mc) follow a similar naming
convention.
Example: Logical name - dump 0f /dev/nrst0a /vol/vol0
Electrical or Physical path names (PPN): A PPN is the numerical address sequence that Data ONTAP
assigns to a tape drive and library based on which SCSI-2/3 adapter or switch they are connected to on the
filer. A PPN is also known as an electrical name. Note: Different filers can have different PPNs for the same
tape device.
Example: Electrical name - dump 0f /dev/nr.MY_SWITCH:5.6.a /vol/vol0
World wide names (WWN): A WWN is assigned to tape devices at the time of manufacture. WWN applies
only to Fibre Channel devices and not to SCSI devices. Note: Accessing a tape device through the WWN
enables multiple filers to track the same device. In addition, using WWNs eliminates confusion in naming
conventions sometimes caused by using PPNs.
Backup Application Support
Depicted below, a simple backup configuration is shown that allows a backup application residing on a host, to
move data from a NDMP capable NetApp system to a tape device via a fibre channel SAN. This type of
backup is often referred to a as a LAN free backup, as data does not move across an Ethernet LAN.

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 18 of 26

stems NetApp FAS3100 and V3100 Storage Sy


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Figure 6: Backup Application Deployment (Source: NetApp)
Additionally, the applications listed below have been certified to interoperate with both NetApp Data ONTAP
7.2+ based systems, and the tape devices listed previously.

Company Backup Application
Atempo Time Navigator
BakBone NetVault
CommVault Galaxy
CA BrightStor ArcServe
CA BrightStor Enterprise
HP OpenView Data Protector
Oracle w/ SnapMirror
Oracle w/ SnapShot
Syncsort Backup Express
Tivoli TSM (Tivoli Storage Mgr.)
Symantec Veritas NetBackup

Table 8: NetApp Backup Application Support


ES/OL
Evaluator Series On-Line
Network Attached Storage June 2010
Product Analysis Page 19 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
iSCSI Connectivity
The internet SCSI (or iSCSI) protocol provides raw (or block) data access over a network. NetApp has been
one of the most vocal proponents for this technology due in part to fact that iSCSI utilizes TCP/IP networks,
which NetApp systems have traditionally utilized for file connectivity. There is a variety of management,
connectivity and multi-pathing issues that must be solved when deploying iSCSI solutions, much as when more
traditional fibre channel networks are used to provide block data access.

Figure 7: iSCSI SAN Deployment (Source: NetApp)

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 20 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
There are a number of options for providing multiple connections between clients (hosts) and storage for
iSCSI. By providing multiple paths, it is possible to improve performance as well as increase system reliability
and redundancy. Provided in the table below is a summary of the benefits of the various options.

iSCSI Multi Path
Features
Link
Aggregation
MPIO - Multi
pathing
MCS - Multiple
Connection
Sessions
Where implemented Network Link
(Ethernet
802.3ad) layer
Host device
driver layer
iSCSI initiator
layer
Improved bandwidth
between servers and
storage
Yes Yes Yes
Provides completely
physically independent
paths
No Yes Yes
Supports HBA initiators No Yes No
Provides multi-pathing
benefits to other TCP/IP
traffic
Yes No No
Only basic iSCSI initiator
required
Yes Yes No
Only basic iSCSI target
required
Yes Yes No
Only basic device driver
required
Yes No Yes

Table 9: Summary of iSCSI Multi-path Options
The tables on the following pages provide details for iSCSI HBA and driver support by host operating system
platform, for Windows, UNIX and Linux hosts.
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 21 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Operating System Supported Initiator (OS Revisions)
Microsoft - Software Initiator
(Microsoft Windows 2000, Microsoft Windows 2000 Advanced Server, Microsoft
Windows Server 2003)
Microsoft - Software Initiator with Integrated iSCSI Boot for IBM BladeCenter
with Integrated NICs
(Microsoft Windows Server 2003)
Intel Pro 1000 T IP HBA - Hardware Initiator
(Microsoft Windows 2000 Server, Microsoft Windows 2000 Advanced Server,
Microsoft Windows Server 2003 )
Adaptec HBA 7211C/F - Hardware Initiator
(Microsoft Windows 2000 Server, Microsoft Windows NT 4.0 Server)
Alacritech HBA SES1001 - Accelerator for Microsoft Software Initiator
(Microsoft Windows 2000 Server, Microsoft Windows Server 2003)
Alacritech HBA SES2001/SES2002 - Accelerator for Microsoft Software Initiator
(Microsoft Windows 2000 Server, Microsoft Windows Server 2003)
Alacritech HBA SES2102/SES2104 - Accelerator for Microsoft Software Initiator
(Microsoft Windows 2000 Server, Microsoft Windows Server 2003)
QLogic HBA QLA4010 - Hardware Initiator
(Windows 2000 Server, Windows 2000 Advanced Server, Windows Server
2003)
QLogic HBA QMC4052 - Hardware Initiator
(Windows 2000 Server, Windows 2000 Advanced Server, Windows Server
2003)
QLogic HBA QLA405X - Hardware Initiator













Microsoft Windows Server
Platforms
(Windows 2000 Server, Windows 2000 Advanced Server, Windows Server
2003)

Table 10: NetApp iSCSI support for Windows
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 22 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Operating System Supported Initiator (OS Revisions)
Red Hat Enterprise Linux 2.1 - Software Initiator
Red Hat Enterprise Linux 3.0 - Software Initiator
Red Hat Enterprise Linux 3.0 Update 4 - Software Initiator
Red Hat Enterprise Linux 4.0 Update 2 - Software Initiator
Red Hat Enterprise Linux 4.0 Update 3 or higher - Software Initiator
SuSE Enterprise Server 8 and 9 - Software Initiator
Adaptec HBA 7211C/F - Hardware Initiator
(Red Hat Enterprise Linux 2.1)
QLogic HBA QLA4010 - Hardware Initiator
(Red Hat Enterprise Linux 2.1 Update 3, Red Hat Enterprise Linux 3.0 Update
3, SuSE Linux Enterprise Server 8 SP3)
QLogic HBA QLA405x - Hardware Initiator







Linux (Red Hat & SuSE)
(Red Hat Enterprise Red Hat Enterprise Linux 3.0 Update 3, Red Hat Enterprise
Red Hat Enterprise Linux 4.0 Update 3, SuSE Linux Enterprise Server 8 SP3,
SuSE Linux Enterprise Server 9 SP3)
Novell - Software Initiator Novell NetWare
(Novell NetWare 5.1, 6.0, & 6.5)
HP-UX - Software Initiator HP-UX
(HP-UX11iv1, HP-UX11iv2, & HP-UX 11.23 v2)
Solaris - Software Initiator
(Solaris 10)
QLogic HBA QLA4010 - Hardware Initiator
(Solaris 8, 9, 10 Update 1)
QLogic HBA QLA405x - Hardware Initiator



Solaris
(Solaris 9, 10 Update 1)
AIX - Software Initiator IBM AIX
(AIX 5.2ML3-ML7, AIX 5.3 ML2-ML3)
VMWare ESX ESX 3.0, 3.0.1- Software Initiator

Table 11: NetApp iSCSI support for UNIX
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 23 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
Physical Specifications
The following table lists the physical specifications for the FAS and V series 3100 controllers with the table on
the next page providing the physical specifications for the disk shelves used in the FAS series. The cluster
models have slightly higher requirements but from an environmental planning point should be viewed as two of
the base models. Physically, the clustered versions are equivalent to two controllers.

FAS/V 3100 System Specifications Dual Controller
Active/Active Configuration
Single Controller
Configuration
AC and DC Power/Current (line voltage for
standalone systems dependent on local power
distribution; system cabinets are 200 - 240 VAC)
100 to120VAC, 4.07 to 9.74A,
50/60Hz
200 to240VAC, 2.02 to 4.69A,
50/60Hz
100 to240VAC, 2.37 to 5.07A,
50/60Hz 200 to 240VAC, 1.19
to 2.52A, 50/60Hz
Thermal Rating 3,305 Btu/hr 1,722 Btu/hr
Weight 122 lb (68kg) 95 lb (34kg)
Height 10.24"(26 cm), fits into 6U
space
10.24"(26 cm), fits into 6U
space
Width 19" IEC rack-compliant (17.6", 44.7 cm)
Depth 24.3"(61.7 cm), 28" (71.1 cm) with cable management bracket
Operating Temperature, Altitude, and Relative
Humidity
10 C to 40 C (50 F to 104 F); at </= 3,000 m (at </=
10,000') elevation; 20%to 80% relative humidity, non-
condensing (28 C wet bulb temperature)
Non-operating Temperature and Relative
Humidity
-40 C to 79 C (-40 F to 158 F); 5% to 95% relative
humidity, non-condensing, in original container
Operating Acoustic Noise 61 dBA sound pressure (LpA) @normal operating conditions
(at 22C and at sea level)
Min. Cabinet Clearances for Airflow 10"(25.4 cm) in front, 12" (30.5 cm) in rear
Min. Cabinet Clearances for Service 30"(76.2 cm ) in front, 30" (76.2 cm) in rear
Compliance RoHS-compliant
Safety/Emissions/Immunity Safety: EN 60950, CE,CSA 60950, UL 60950, CB IEC60950-1
(all national deviations), EN60825-1,IRAM, GOST-R, BSMI
Emissions/Immunity: FCC Part 15 Class A, ICES-03, CE,
MIC,VCCI, AS/NZS CISPR 22, EN55022, EN55024, EN61000-
3-2, EN61000-3-3, CoC (South Africa), BSMI

Table 2: Physical Specifications of the 3100




ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 24 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

Disk Storage Shelf Specifications
Disk Shelves Supported
(Fibre Channel, SATA,
SAS)
DS14, DS14mk2 FC, DS14mk2AT Each w/14 low-
profile slots for FC and SATA disk drives
DS4243
Disks Supported on New Configurations:
- FC144GB, 300GB disks
- SATA250GB, 500GB disks
- SAS 300GB, 450GB
Disks Supported for Legacy (Controller Upgrade)
Configurations:
- FC36GB, 72GB, 144GB, 300GB disks



Disk Drive Support
- SATA250GB, 320GB, 500GB disks
Shelves Supported on New Configurations:
- Embedded Switched Hub (ESH)
- Embedded Switched Hub2 (ESH2)
- AT-FCX

Shelves Supported for Legacy (Controller Upgrade)
Configurations:
- Embedded Switched Hub (ESH)
- Embedded Switched Hub2 (ESH2)
- AT-FCX





Disk Shelf Support
- LRC
Disk Drive Storage Shelf
Interface
Fibre Channel-Arbitrated Loop (FC-AL)
Power Supply/Cooling Fans Dual, redundant, hot-pluggable, integrated power
supply/fan assemblies (220V/110V)
AC Power/Max. Current 100 to 120VAC/3.95A; 200 to 240VAC/1.9A
Thermal Rating 1,215 Btu/hr (fully loaded shelf)
Operating Acoustic Noise 49 dBA sound pressure (LpA) @ normal operating
conditions (at 22C and at sea level)
Dimensions
(height/width/depth)
3 EIA U (5.25", 13.3 cm)/19" IEC rack-compliant
(17.6", 44.7 cm)/20" (50.85 cm)
Weight 77 lb (35 kg) fully loaded

Table 14: Disk Shelf Specifications

ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 25 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com

EVALUATOR GROUP COMMENTS
The current line of NetApp 3100 storage systems are competitive storage systems in the mid-range
enterprise marketplace. These systems perform well, are scalable, highly reliable and provide a wide
variety of features. NetApp has continually updated and improved their hardware and software
capabilities while successfully expanding their products into new market segments. The NetApp FAS
and V series products provide a single point of access to both block and file storage, a concept that
appeals to many customers across vertical industries and application deployments.
In the FAS3100 and V3100, NetApp is able to provide their customers with products that are accessible
via nearly every protocol and physical connection commonly in use in the open systems storage
market. NetApp has built a loyal base of customers by continually delivering products that meet their
expectations while adding business value.
Strengths:
NetApps Data ONTAP OS provides a wealth of features, functions and serves as a solid platform for
the FAS3100 and V3100 midrange product lines. The features, performance and reliability of Data
ONTAP coupled with the hardware are well matched for many enterprise deployments.
NetApp has significantly upgraded the hardware capabilities on their newest systems, which borrow
many architectural elements with the larger 6000 product line. By leveraging the AMD HyperTransport
architecture, NetApp has the future ability to scale to multi CPU core systems along with sufficient IO
and memory processing capabilities. These hardware upgrades provide a capable platform for future
Data ONTAP upgrades capable of fully utilizing the hardware platform improvements.
The 3100 products are well-designed, stable, storage systems capable of handling a wide variety of
storage needs. The NetApp products are flexible storage systems, capable of providing multi protocol
access, advanced data replication features, all of which provide unique capabilities in an integrated
package that delivers many enterprise storage features.
NetApp has a proven record of delivering well-engineered products that customers value and will
undoubtedly continue to enhance the capabilities to address greater scalability in the future. NetApp
has continually differentiated their products well, especially in light of the minimal architectural
differences that exist between the models.
Potential Concerns:
There are few issues with the 3100 product line. The systems are well positioned and targeted at a
market space which NetApp understands well and has had a great deal of success in servicing.
If any challenges remain for NetApp, it is with their service and support capabilities. NetApp has been
focusing on broadening their services portfolio, both organically and through partnerships. To this
end, NetApp has a very capable partner in IBM, who offers the vast majority of NetApps new systems
via an OEM agreement, along with their well known business acumen. In addition, NetApp has been
expanding their professional services capabilities to meet the needs of enterprise customers. While
ES/OL
Evaluator Series On-Line
Network Attached Storage
Product Analysis
June 2010
Page 26 of 26

NetApp FAS3100 and V3100 Storage Systems


2010 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.
(303) 221-7867 www.evaluatorgroup.com
their offerings are still somewhat limited, with a product focus, these limitations are gradually
disappearing, with continual service improvements.
A challenge that NetApp is addressing with both their FAS and V-Series systems is the acceptance of
these systems as viable alternatives to more traditional block storage products. The current FAS3100
systems coupled with Data ONTAP enhancements move NetApp to a higher performance level than
with previous systems. One of the strengths of NetApp has been in continually refining their hardware
architecture while adding a wealth of high-feature software functions in addition to their embedded
operating system.



Copyright 2010 Evaluator Group, Inc. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and
recording, or stored in a database or retrieval system for any purpose without the express written consent of Evaluator Group Inc. The information
contained in this document is subject to change without notice. Evaluator Group assumes no responsibility for errors or omissions. Evaluator Group
makes no expressed or implied warranties in this document relating to the use or operation of the products described herein. In no event shall Evaluator
Group be liable for any indirect, special, inconsequential or incidental damages arising out of or associated with any aspect of this publication, even if
advised of the possibility of such damages. The Evaluator Series is a trademark of Evaluator Group, Inc. All other trademarks are the property of their
respective companies.

You might also like