KNOWLEDGE BASE

Knowledge Base Article: 000090041

OneFS: Best practices for NFS client settings (000090041)
Version:9

Audience: Level 30 = Customers

Article Type: Break Fix

Last Published: Thu May 14 17:29:10 GMT 2015

Validation Status: Final Approved

Summary:

Impact:

Introduction
This article describes the best practices and recommendations for client-side settings and mount options when using the
NFS protocol to connect to an Isilon cluster.
This article applies to all versions of OneFS.

Resolution:

Details
Use NFS over TCP, not UDP
When you use the User Datagram Protocol (UDP), if the network speed of the client and server are not identical,
dropped packets and retransmissions might cause performance to be extremely slow when the faster entity tries to send
data to the slower entity. For better performance and resilience, use the Transmission Control Protocol (TCP) instead. If
security is a concern, consider adding Kerberos.
The recommendation for clients (see examples below) enables TCP, instead of the NFS protocol, to handle timeouts and
retransmissions.
Use NFS version 3 or 4
NFS version 2 should not be used for mounting because it uses UDP and does not support files larger than 2 GB.
NFS version 3 is the standard for NFS and is the most stable and mature version of the NFS protocol.
NFS version 4 is still considered new as the time of this writing (February 2012), but it is becoming more adopted in
recent years and is also relatively stable. However, it does not support the failover that NFS version 3 can, because
version 3 is stateless and version 4 is stateful.
Understand the differences between the sync and async options
There are sync and async options for NFS exports and NFS client mounts:
Use sync exports from the cluster. Using async exports causes the cluster to immediately reply to write requests. This
restricts the cluster from signaling a write failure. Additionally, NFS versions 3 and 4 enable safe asynchronous writes
within the protocol. As such, async exports are considered unsafe and unnecessary.
Use async mounts from the client. Using sync as a client mount option causes all write operations to be synchronous,
which usually results in very poor write performance. Sync mounts should only be used in exceptional circumstances
where a client program does not specify synchronous writes itself, but relies on that behavior.
Configure read and write RPC sizes to be as large as possible
In general, you should configure the read and write remote procedure call (RPC) sizes (rsize, wsize) to be as large as
possible.
On OneFS 5.0 and later, the server-side limits are:
rsize: 128 KB
wsize: 512 KB
The client implementation might limit you or default to lower values. For more information about the client
implementation, consult your client documentation.
Use "hard" mounts
On most clients, it is recommended to use the hard and intr mount options so that input/output (I/O) errors are retried
indefinitely, but can still be interrupted by signals. The hard option retries I/O errors indefinitely; the intr option allows
signals to interrupt. In the event that an NFS client cannot access the mounted resource, a hard-mounted client
continues to request access to the resource until the request is satisfied.
Using soft mounts (as opposed to hard mounts) can create major timeouts (timeouts of 60 seconds or longer) and cause
I/O errors.
Considerations when multiple clients access the same files
If there are multiple clients accessing (writing to) the same files, it is advisable to use locking (NLM) and disable attribute
caching by the client.
If you need to disable attribute caching, use the actimeo=0 mount option.
Avoid using the noac mount option. This option turns off all attribute caching, which causes a large increase in
GETATTR calls. Additionally, it also turns on sync mode for the mount and makes all writes synchronous, which can
drastically reduce write performance (as described in the "Understand the differences between the sync and async
options" section above).
Use the lock mount option to ensure that clients can obtain exclusive access to the files when needed. The OneFS
implementation uses advisory locks, as opposed to mandatory locks, so all clients must be configured to use locking to
ensure consistent locking behavior.
If multiple clients are not accessing the same files, it should be safe to enable attribute caching and use local locks or no
locks.
Enable READDIRPLUS

If your client supports the rdirplus option, you can use it to enable the use of the READDIRPLUS call, which can improve
performance, especially for Mac OS X clients.
For more information about READDIRPLUS, see Directory listing is slow with large amount of files using Linux clients,
88890.
Recommended client settings
Following are the recommended settings for optimizing different client operating systems using NFS over TCP. The
commands and syntax might vary depending on the operating system.
Linux

Modern Linux distributions support NFS read/write block sizes up to 1 MB, and will automatically negotiate the optimal
block size with the NFS server. The negotiated values are ideal for properly configured high-performance, low-latency
networks. To limit the negotiated read/write record size for all clients, use these sysctls on the cluster:
NOTE
To make these changes permanent, follow the instructions for editing the /etc/sysctl.conf file in Configuring sysctls and
making sysctl changes persist through node and cluster reboots and upgrades, 89232.
sysctl vfs.nfsrv.max_rsize
sysctl vfs.nfsrv.max_wsize
Like most operating systems, the default maximum network buffer sizes are too small. The network buffer size
parameters must be set on the client before mounting NFS exports.
Recommended network buffer settings for 1 GbE Ethernet connections:
echo "262144" > /proc/sys/net/core/wmem_max
echo "262144" > /proc/sys/net/core/wmem_default
echo "262144" > /proc/sys/net/core/rmem_max
echo "262144" > /proc/sys/net/core/rmem_default
For a boot-time change, /etc/sysctl.conf should be changed as follows:
net.core.wmem_max = 262144
Recommended network buffer settings for 10 GbE connected clients by setting these sysctls:
net.core.wmem_max = 16777216
net.core.wmem_default = 524287
net.core.rmem_max = 16777216
net.core.rmem_default = 524287
Recommended mount settings for 10 GbE Ethernet connections:
mount -t nfs -o nfsvers=3,tcp,rw,hard,intr,timeo=600,retrans=2 isicluster:/ifs /mnt
If this is a non-public server, you can turn off TCP timestamps (to reduce CPU processing) by setting this sysctl:
net.ipv4.tcp_timestamps=0
Make sure window scaling is enabled by setting this sysctl:
net.ipv4.tcp_window_scaling=1
Set the number of task request slots for RPC calls to 16 by setting this sysctl:
sunrpc.tcp_slot_table_entries=16
See also: Directory listing is slow with large amount of files using Linux clients, 88890.
For more information about Linux NFS, see Linux NFS Overview, FAQ and HOWTO Documents on SourceForge or view
the nfs(5) man page.
Mac OS X

For optimal settings when using Mac OS X, refer to Using Mac OS X Clients with Isilon OneFS 6.5 and Using Mac OS X
Clients with Isilon OneFS 7.x.
FreeBSD (and other BSD variants)

Recommended mount settings:

mount -t nfs -o hard,nfsv3,rdirplus,proto=tcp isicluster:/ifs /mnt
Recommended sysctl settings:
Increase network buffer space from the 32K and 64K defaults by setting these sysctls:
net.inet.tcp.recvspace=524288
net.inet.tcp.sendspace=524288
Enable window scaling by setting this sysctl:
net.inet.tcp.rfc1323=1
Turn off bandwidth delay product limiting for TCP connections by setting this sysctl:
net.inet.tcp.inflight.enable=0
See also FreeBSD 8.0 and 8.1 clients stall under high load when mounted via NFS , 88899
For more information about NFS mount options, see the FreeBSD man page for mount_nfs(8).
For more information about tuning and sysctls, see the man pages for tuning(7), sysctl(8), and sysctl.conf(5).
Solaris

Recommended mount settings:

mount -o vers=3,proto=tcp isicluster:/ifs /mnt
Irix

Recommended mount settings:

mount -t nfs -o vers=3,proto=tcp isicluster/ifs /mnt
Product:

Isilon, Isilon OneFS,Isilon OneFS6.5,Isilon OneFS7.0,Isilon OneFS7.1

External Source:

Primus

Primus/Webtop solution ID:

emc14001361