STRSW Ilt Perfcdot Exerciseguide

NETAPP UNIVERSITY
Performance Analysis on Clustered Data ONTAP

Exercise Guide
Course ID: STRSW-ILT-PERFCDOT
Catalog Number: STRSW-ILT-PERFCDOT-EG
Content Version: 1.0
NetApp University - Do Not Distribute
ATTENTION
The information contained in this course is intended only for training. This course contains information and activities that,
while beneficial for the purposes of training in a closed, nonproduction environment, can result in downtime or other
severe consequences in a production environment. This course material is not a technical reference and should not,
under any circumstances, be used in production environments. To obtain reference materials, refer to the NetApp product
documentation that is located at http://now.netapp.com/.
COPYRIGHT
2013 NetApp, Inc. All rights reserved. Printed in the U.S.A. Specifications subject to change without notice.
No part of this document covered by copyright may be reproduced in any form or by any meansgraphic, electronic, or
mechanical, including photocopying, recording, taping, or storage in an electronic retrieval systemwithout prior written
permission of NetApp, Inc.
U.S. GOVERNMENT RIGHTS

Commercial Computer Software. Government users are subject to the NetApp, Inc. standard license agreement and
applicable provisions of the FAR and its supplements.
TRADEMARK INFORMATION
NetApp, the NetApp logo, Go further, faster, AdminNODE, Akorri, ApplianceWatch, ASUP, AutoSupport, BalancePoint,
BalancePoint Predictor, Bycast, Campaign Express, ChronoSpan, ComplianceClock, ControlNODE, Cryptainer, Data
ONTAP, DataFabric, DataFort, Decru, Decru DataFort, DenseStak, Engenio, E-Stack, FAServer, FastStak, FilerView,
FlexCache, FlexClone, FlexPod, FlexScale, FlexShare, FlexVol, FPolicy, GatewayNODE, gFiler, Imagine Virtually
Anything, Infinivol, Lifetime Key Management, LockVault, Manage ONTAP, MetroCluster, MultiStore, NearStore, NetApp
Select, NetCache, NetCache, NOW (NetApp on the Web), OnCommand, ONTAPI, PerformanceStak, RAID DP,
SANscreen, SANshare, SANtricity, SecureAdmin, SecureShare, Securitis, Service Builder, Simplicity, Simulate ONTAP,
SnapCopy, SnapDirector, SnapDrive, SnapLock, SnapManager, SnapMirror, SnapMover, SnapProtect, SnapRestore,
Snapshot, SnapValidator, SnapVault, StorageGRID, StorageNODE, StoreVault, SyncMirror, Tech OnTap, VelocityStak,
vFiler, VFM, Virtual File Manager, WAFL, and XBB are trademarks or registered trademarks of NetApp, Inc. in the United
States and/or other countries.
All other brands or products are either trademarks or registered trademarks of their respective holders and should be
treated as such.
E-2
Performance Analysis on Clustered Data ONTAP: Welcome
2013 NetApp, Inc. This material is intended only for training. Reproduction is not authorized.
TABLE OF CONTENTS
WELCOME..................................................................................................................................................... E-1
MODULE 1: HOW A NETAPP STORAGE SYSTEM WORKS................................................................... E1-1
MODULE 2: PERFORMANCE OVERVIEW ................................................................................................ E2-1
MODULE 3: CLUSTERED STORAGE SYSTEM WORKLOADS AND BOTTLENECKS ......................... E3-1
MODULE 4: CLUSTER PERFORMANCE MONITORING AND ANALYSIS ............................................. E4-1
MODULE 5: ONCOMMAND MANAGEMENT TOOLS ............................................................................... E5-1
MODULE 6: STORAGE QOS...................................................................................................................... E6-1
MODULE 7: SUMMARY .............................................................................................................................. E7-1
APPENDIX A: ANSWERS............................................................................................................................. A-1
E-3
Performance Analysis on Clustered Data ONTAP: Welcome
MODULE 1: HOW A NETAPP STORAGE SYSTEM WORKS

There is no exercise associated with Module 1.
E1-1
Performance Analysis on Clustered Data ONTAP: How a NetApp Storage System Works
MODULE 2: PERFORMANCE OVERVIEW

EXERCISE
In this exercise, you identify your exercise equipment, log in to the exercise environment, and verify the
equipment.
OBJECTIVES
By the end of this exercise, you should be able to:

Identify the exercise environment
Log in to the exercise environment
Add a cluster to the OnCommand System Manager
Configure the SNMP public community name
Identify clustered Data ONTAP components
Set the clustered Data ONTAP command-line system timeout value (optional)
TASK 1: IDENTIFY THE EXERCISE ENVIRONMENT
In this task, you log in to your assigned exercise environment. You perform all subsequent exercises from this
assigned machine.
STEP ACTION
1.
With the assistance of your instructor, identify your main Windows server. This machine might
be a virtual machine.
Windows Server
IP address: _______________________________________________
Domain: _________________________________________________
Domain administrator password: Netapp123
2.
With the assistance of your instructor, identify your clustered NetApp Data ONTAP operating
system nodes.
Clustered Data ONTAP
Node 1 Management LIF IP address: 192.168.0.91
Node 2 Management LIF IP address: 192.168.0.92
Cluster Management LIF IP address: 192.168.0.101
Cluster Administrator (admin) password: Netapp123
3.
With the assistance of your instructor, identify your Linux machine. This machine might be a
virtual machine.
Linux Server
IP address: 192.168.0.10
Root password: Netapp123
E2-1
Performance Analysis on Clustered Data ONTAP: Performance Overview
STEP ACTION
4.
With the assistance of your instructor, identify your secondary Windows server. This machine
might be a virtual machine.
Windows Server
IP address: 192.168.0.6
Root password: Netapp123
TASK 2: LOG IN TO THE EXERCISE ENVIRONMENT
In this task, you use Remote Desktop Connection to log in to your assigned exercise environment. You
perform all subsequent tasks from this assigned machine.
STEP ACTION
1.
On your local Windows machine desktop, click the Remote Desktop Connection link to log in
to the remote Windows machine through the Remote Desktop Connection tool.
If this link is not available, ask your instructor where to find the tool.
2.
Enter the IP address of your remote Windows server, and then click Connect.
3.
Verify that the desktop of the remote machine is displayed.
4.
If you are asked for authentication, enter the user name and password that your instructor gave
you.
E2-2
TASK 3: ADD A CLUSTER TO ONCOMMAND SYSTEM MANAGER
In this task, you add your cluster management port to the local hosts file, launch System Manager, and add
your assigned cluster.
NOTE: For more information about configuring a storage system with System Manager, see the Clustered
Data ONTAP Administration course.
STEP ACTION
1.
E2-3
On the Windows Server desktop, open Windows Explorer.
STEP ACTION
2.
Navigate to C:\Windows\System32\Drivers\etc.
3.
Double-click hosts.
E2-4
STEP ACTION
4.
Verify that the operation dialog box appears.
5.
Select Notepad to open the hosts file in this tool.
6.
Append the following information to the end of the file:

192.168.0.101 cluster1
NOTE: Written as variables, the information that you appended is:
cluster_management_LIF_IP_address cluster_name
7.
Save the file.
8.
Exit Notepad.
9.
Close Windows Explorer.
10.
On your Windows Server desktop, double-click the NetApp OnCommand System Manager
icon.
E2-5
STEP ACTION
11.
Verify that System Manager starts.
12.
In the upper left, click the Login button, and then enter the following:
User name: admin
Password: Netapp123
Save my credentials: selected
13.
E2-6
Click Sign in.
STEP ACTION
14.
In the left pane of System Manager, expand cluster1.

NOTE: There are two nodes in the cluster.
E2-7
TASK 4: CONFIGURE SNMP PUBLIC COMMUNITY NAME
In this task, you configure the SNMP public community name so that System Manager and OnCommand
Balance can properly discover your cluster.
NOTE: In some highly secure environments, this might not be appropriate.
STEP ACTION
1.
Navigate to Cluster > cluster1 > Configuration > System Tools > SNMP.
2.
Click Edit.
E2-8
STEP ACTION
3.
E2-9
Click Add.
STEP ACTION
4.
Enter the community name: public
5.
Click OK.
E2-10
STEP ACTION
6.
E2-11
Click OK to close the Edit SNMP Settings dialog box.
STEP ACTION
7.
E2-12
Verify that public is now in the community names area.
TASK 5: IDENTIFY CLUSTERED DATA ONTAP COMPONENTS
In this task, you use either System Manager or the clustered Data ONTAP CLI to identify key clustered Data
ONTAP components and revert any LIFs that are not on their home ports.
STEP ACTION
1.
Analyze and identify the following list of clustered Data ONTAP components:
Aggregates ______________________________________________________________
Storage virtual machines ___________________________________________________
Volumes ________________________________________________________________
LUNs __________________________________________________________________
Licenses (which features are installed?) ________________________________________
CIFS Shares ______________________________________________________________
LIFs (are all LIFs home?)____________________________________________________
2.
Revert any LIFs that are not on their home ports.
TASK 6: SET THE CLUSTERED DATA ONTAP COMMAND LINE SYSTEM TIMEOUT VALUE
(OPTIONAL)
In this optional task, you set the clustered Data ONTAP command-line system timeout value.
STEP ACTION
1.
Open a PuTTY session with cluster1.
2.
Check the current system timeout value.

system timeout show
3.
Set the system timeout to 0 (no timeout).

system timeout modify -timeout 0
END OF EXERCISE
E2-13
MODULE 3: CLUSTERED STORAGE SYSTEM WORKLOADS AND BOTTLENECKS

EXERCISE
In this exercise, you examine the different variations of the statistics command, and identify storage
system workloads and potential bottlenecks.
OBJECTIVES

Use the statistics catalog command
Use the statistics start and statistics show commands
Define workload characteristics
TASK 1: EXAMINE THE STATISTICS CATALOG COMMAND
In this task, you issue the three statistics catalog commands and exercise the supporting parameters,
using different privilege levels.
STEP ACTION
1.
2.
Examine the available statistics objects at the admin privilege level.

statistics catalog object show
3.
Change to the advanced privilege level and reissue the command.
4.
What command syntax would you use to display only the advanced level statistics objects
without their descriptions?
5.
What command syntax would you use to display statistics objects that are associated with
storage virtual machines (SVMs)?
NOTE: You should still be in the advanced privilege level.
How many statistics objects are there in the list?
6.
Change to the admin privilege level and examine the instance names that are available for the
statistics object volume.
statistics catalog instance show -object volume
7.
What command syntax would you use to display the instance names that are available for the
statistics object that represents SVMs that have LIFs associated with them?
How many instance names are there in the list?
8.
statistics object that represents SVMs that have volumes associated with them?
How many instance names are there in the list?
E3-1
Performance Analysis on Clustered Data ONTAP: Clustered Storage System Workloads and Bottlenecks
STEP ACTION
9.
statistics object that represents the disk that is associated with the second node in the cluster
(cluster1-02)?
10.
Examine the counters that are available for the statistics object disk and show the detailed
information.
statistics catalog counter show -object disk describe
11.
Change command privilege level to advanced and rerun the command.

NOTE: Remember to set the privilege back to admin when you are done.
12.
Examine the counters that are available for the statistics object aggregate and show the
detailed information.
statistics catalog counter show -object aggregate -describe
13.
Examine the counters that are available for the statistics object volume and show the detailed
information.
statistics catalog counter show -object volume -describe
TASK 2: EXAMINE THE STATISTICS START AND STATISTICS SHOW COMMANDS
In this task, you issue the statistics start and statistics show commands and exercise the
supporting parameters, using different privilege levels.
STEP ACTION
1.
Using the admin privilege level, start statistics data collection on the statistics object nfsv3.
statistics start -object nfsv3 -sample-id sample_nfsv3_adm
2.
Using the advanced privilege level, start statistics data collection on the statistics object nfsv3.
set advanced
statistics start -object nfsv3 -sample-id sample_nfsv3_adv
3.
Display the counters that are associated with the statistics object nfsv3 instance vs2 for both
samples.
statistics show -object nfsv3 -instance vs2 -counter * -sample-id
sample_nfsv3_adm
sample_nfsv3_adv
Was there any difference in the displays?

4.
Using the advanced privilege level, start statistics data collection on the statistics object disk.
statistics start -object disk -sample-id sample_disk
E3-2
STEP ACTION
5.
What command syntax would you use to display all of the latency related counters for disk
v4.20?
How many counters are there in the list?
6.
What command syntax would you use to display all of the user_read_latency counters for all
disks in the cluster?
How many counters are there in the list?
7.
Display only the disks with the user_read_latency counter between 1000us and 10000us.
statistics show -object disk -instance * -counter user_read_latency
-sample-id sample_disk -value 1000..10000
8.
What command syntax would you use to display all of the user_read_latency counters that are
greater than 10000us for all disks in the cluster, limiting the display to only the counter and
value fields?
9.
Using the advanced privilege level, start statistics data collection on the statistics object
aggregate.
statistics start -object aggregate -sample-id sample_aggr
10.
Display nonzero user_reads and user_writes counters for each aggregate.

statistics show -object aggregate -instance * -counter
user_reads|user_writes -sample-id sample_aggr -value >0
11.
volume.
statistics start -object volume -sample-id sample_volume
12.
Display all of the volume activity by displaying all of the data counters for each volume.
statistics show -object volume -instance * -counter *data -sample-id
sample_volume
13.
What command syntax would you use to display all of the latency counters for all volumes in
the cluster? How about all nonzero latency counters?
14.
processor.
statistics start -object processor -sample-id sample_processor
15.
E3-3
What command syntax would you use to display all of the counters for all processors in the
cluster?
STEP ACTION
16.
workload.
statistics start -object workload -sample-id sample_workload
17.
Display all of the latency counters for each workload and limit the display to the counter and
value fields.
statistics show -object workload -instance * -counter *latency -sampleid sample_workload -fields counter,value
18.
Stop all statistics data collection.

statistics stop -sample-id sample_nfsv3_adm
statistics stop -sample-id sample_nfsv3_adv
statistics stop -sample-id sample_disk
statistics stop -sample-id sample_aggr
statistics stop -sample-id sample_volume
statistics stop -sample-id sample_processor
statistics stop -sample-id sample_workload
TASK 3: DEFINING WORKLOAD CHARACTERISTICS
In this task, you gather statistical data and evaluate the data to determine the workload characteristics.
STEP ACTION
1.
2.
Turn off the Snapshot policy for volume vs2_vol01.

volume modify -vserver vs2 -volume vs2_vol01 -snapshot-policy none
3.
On your Windows Server desktop, double-click the CourseFiles shortcut:
4.
Copy the file 300mfile to the z: drive.
5.
Normally you would start data collection for all of the protocols that are being served by the
cluster; however, for this exercise you use only CIFS.
Using the diagnostic privilege level, start statistics data collection on the objects cifs,
volume, and readahead.
NOTE: Diagnostic privilege level commands are required to capture
cifs_read_size_histo, cifs_write_size_histo, rand_read_req, and
seq_read_req.
set diagnostic
statistics start -object cifs|volume|readahead -sample-id sample_cifs1
E3-4
STEP ACTION
6.
On the Windows Server desktop, double-click the Command Prompt.
7.
Drag the SIO.exe shortcut to the Command Prompt window.
8.
Press the Spacebar, type the following list of parameters, and then press Enter:
0 0 32k 300m 30 1 z:\300mfile
9.
After the command ends and the command prompt returns, go to the PuTTY session and turn off
data collection.
statistics stop -sample-id sample_cifs1
E3-5
STEP ACTION
10.
Start statistics data collection again on the same objects, but this time change the sample name to
sample_cifs2.
statistics start -object cifs|volume|readahead -sample-id sample_cifs2
11.
In the Command Prompt window, press the Up arrow, change the SIO parameters to the
following, and then press Enter:
100 0 32k 300m 30 4 z:\300mfile
12.
After the command ends and the command prompt returns, go to the PuTTY session and turn off
data collection.
statistics stop -sample-id sample_cifs2
13.
Run through the same sequence (Steps 1012) one more time for sample_cifs3, using the
following SIO parameters:
sample_cifs3
50 100 4k 300m 30 32 z:\300mfile
NOTE: After the command ends and the command prompt returns, remember to go to the
PuTTY session and turn off data collection.
14.
After all three samples are collected; analyze the counters to determine the workload
characteristics of each sample.
The following commands will help to define the CIFS workload in each sample (sample_cifs1
through sample_cifs3).
NOTE: Change the X in sample_cifsX to the sample being analyzed (1 through 3).
statistics
-sample-id
statistics
-sample-id
show -object cifs -instance * -counter cifs_read_ops

sample_cifsX
show -object cifs -instance * -counter cifs_write_ops
sample_cifsX
statistics
-sample-id
statistics
-sample-id
show -object volume -instance * -counter cifs_read_latency

sample_cifsX
show -object volume -instance * -counter cifs_write_latency
sample_cifsX
statistics show -object cifs -instance * -counter cifs_read_size_histo

-sample-id sample_cifsX
statistics show -object cifs -instance * -counter cifs_write_size_histo
The following statistics commands are specific to read workloads. If you have already
determined that the workload is a write workload, you can skip these commands.
statistics show -object readahead -instance * -counter rand_read_reqs
statistics show -object readahead -instance * -counter seq_read_reqs
E3-6
STEP ACTION
15.
Record your observations for sample_cifs1 below:

CHARACTERISTIC
OBSERVED VALUE
Read
Write
Throughput
Latency
Operation Size
Concurrency
Randomness
16.
What did the command-line output look like for throughput of sample_cifs1?
17.
What did the command-line output look like for latency of sample_cifs1?
18.
What did the command-line output look like for read operation size of sample_cifs1?
19.
What did the command-line output look like for write operation size of sample_cifs1?
20.
What did the calculation (throughput * latency) look like for concurrency of sample_cifs1?
21.
What did the command-line output look like for randomness (rand_read_reqs) of
sample_cifs1?
22.
What did the command-line output look like for randomness (seq_read_reqs) of
sample_cifs1?
23.
What workload did you conclude from your data collection of sample_cifs1?
E3-7
STEP ACTION
24.

CHARACTERISTIC
OBSERVED VALUE
Read
Write
Throughput
Latency
Operation Size
Concurrency
Randomness
25.
26.
27.
28.
29.
30.
sample_cifs2?
31.
sample_cifs2?
32.
E3-8
STEP ACTION
33.

CHARACTERISTIC
OBSERVED VALUE
Read
Write
Throughput
Latency
Operation Size
Concurrency
Randomness
34.
35.
36.
37.
38.
39.
sample_cifs3?
40.
sample_cifs3?
41.
E3-9
STEP ACTION
42.
The previous examples used CIFS as the basis for the workloads. What statistics objects and
counters would be needed for NFSv3?
END OF EXERCISE
E3-10
MODULE 4: CLUSTER PERFORMANCE MONITORING AND ANALYSIS

EXERCISE
In this exercise, you identify storage system performance baselines, analyze and isolate bottlenecks, and work
with the Perfstat utility.
OBJECTIVES

Perform initial health checks on a cluster
Perform baseline performance monitoring from the cluster shell
Perform performance monitoring and analysis from the cluster shell
Unlock the diag userid
Use the Performance and Statistics Collector (Perfstat)
TASK 1: PERFORM INITIAL HEALTH CHECKS ON THE CLUSTER
In this task, you query the cluster to assess the initial health status.
STEP ACTION
1.
2.
Check to see if all of the nodes are healthy.

cluster show
3.
Check to see if the replication rings have the same (or consistent) masters.
NOTE: Remember to set advanced privilege before you execute this command.
cluster ring show
4.
Using the advanced privilege level, check to see if the cluster connectivity is healthy.
cluster ping-cluster -node local
5.
Check to see if the storage virtual machines (SVMs) are healthy.

dashboard health vserver show
6.
If any of the SVMs return a nonzero status, add the all parameter for additional information.
dashboard health vserver show-all
7.
Check to see if any volumes are not online.

volume show -state !online
8.
Check to see if any aggregates are not online.

storage aggregate show -state !online
E4-1
Performance Analysis on Clustered Data ONTAP: Cluster Performance Monitoring and Analysis
STEP ACTION
9.
Check to see if any disks are in a broken state.

storage disk show -state broken
10.
Normally at this point in the health check, you would check the performance status by using the
commands that follow; however, because these commands will also be run as part of the next
task, they do not have to be run at this time.
To check the performance status (which is optional at this time), enter these commands:
a. Check the current level of activity on each node of the cluster.
statistics show-periodic -node cluster1-01 -instance node -iterations 4
b. Check the current level of activity on the entire cluster.

dashboard performance show
TASK 2: BASELINE PERFORMANCE MONITORING FROM THE CLUSTER SHELL
In this task, you use cluster shell commands to monitor cluster performance.
STEP ACTION
1.
Check the performance status:

statistics show-periodic -node cluster1-01 -instance node
-iterations 4
-iterations 4
b. Check the current level of activity on the entire cluster and notice the overall latency
column.
2.
Check the current NAS operations level on the entire cluster.

dashboard performance show operations
3.
Use the instance parameter to show everything and notice the overall latency by protocol
fields.
dashboard performance show instance
4.
Check the overall latency for the entire cluster.

statistics show-periodic -instance latency -iterations 4
E4-2
STEP ACTION
5.
Check the overall latency for each node in the cluster.

statistics show-periodic -node cluster1-01 -instance latency
-iterations 4
-iterations 4
6.
Check the throughput by cluster.

statistics show-periodic -iterations 4
7.
Check the throughput by volume for volume vs2_vol01.

statistics show-periodic -object volume -instance vs2_vol01 -iterations
4
How can you make the output more readable?

8.
Normally at this point in the baseline check, you would check the ifnet status by using the
command that follows; however, because the lab kits do not have any ifnets defined, the
command does not have to be run at this time.
If you want to do so at this time, check the throughput by interface.
statistics show-periodic -object ifnet -instance a0a -node cluster1-02
-iterations 4
9.
Check the throughput by LIF (IP).

statistics show-periodic -object lif -instance vs2:vs2_cifs_nfs_lif1
-iterations 4
10.
Using the advanced privilege level, start statistics data collection on the objects volume,
aggregate, disk, ext_cache_obj, port, and lif.
set advanced
statistics start -object volume|aggregate|disk|port|lif -sample-id
sample_baseline1
11.
Check the latency by volume for the entire cluster.

statistics show -object volume -counter *latency -sample-id
sample_baseline1
12.
Check volume latency for a specific volume.

statistics show-periodic -object volume -instance vs2_vol01
-iterations 4
13.
Check volume activity.

statistics show -object volume -counter *data -sample-id
sample_baseline1
E4-3
STEP ACTION
14.
Check aggregate latency.

statistics show -object aggregate -counter user_reads|user_writes
-sample-id sample_baseline1
15.
Check disk latency.

statistics show -object disk -counter *latency -sample-id
sample_baseline1
16.
Check port throughput.

statistics show -object port -counter *data -sample-id sample_baseline1
17.
Check the LIF or IP throughput.

statistics show -object lif -counter *data -sample-id sample_baseline1
18.
After completing the task, turn off the statistics data collection.
statistics stop -sample-id sample_baseline1
19.
Clean up any old samples.

statistics samples show
statistics samples delete -sample-id *
TASK 3: PERFORMANCE MONITORING FROM THE CLUSTER SHELL
STEP ACTION
1.
Copy sio_ntap.tar.gz to the z: drive.

2.
Using PuTTY, log in to the Linux server as root (password: Netapp123).
3.
Mount the vs2 export via NFS, using the following IP:
mount -t nfs 192.168.0.122:/vs2_vol1 /mnt/path01
4.
Change directory to /usr/tmp, make a new directory sio, and change directory into it.
cd /usr/tmp
mkdir sio
cd sio
E4-4
STEP ACTION
5.
Extract (untar) the contents of sio_ntap.tar.gz into the current directory.

tar xvfz /mnt/path01/sio_ntap.tar.gz
6.
In your cluster1 PuTTY session, verify that your LIFs that are associated with SVM vs2 are
home and, if they are not, send them home.
network interface show -vserver vs2
network interface revert -vserver vs2 -lif *
7.
Normally you would start data collection for all of the protocols being served by the cluster;
however, for this exercise you use only NFSv3.
In your cluster1 PuTTY session, using the diagnostic privilege level, start statistics data
collection on the objects nfsv3, volume, aggregate, disk, port, lif, and readahead.
NOTE: Diagnostic privilege level commands are required to capture rand_read_req and
seq_read_req.
set diagnostic
statistics start -object nfsv3|volume|aggregate|disk|port|lif|readahead
-sample-id sample_nfs1
8.
On the Linux server, start SIO with a 0% read workload, 0% random, 32-KB block size, 300MB file size, run for one hundred seconds, one thread, and point it at the file on the NFS mount.
/usr/tmp/sio/sio_ntap_linux 0 0 32k 300m 100 1 /mnt/path01/300mfile
9.
Using the analysis commands that you learned in the previous task, analyze the data and record
or save the results. It is recommended that you increase the Lines of scrollback in your
cluster1 PuTTY session to at least 2000.
This information will be used to complete questions later in this task.
HINT: Use the baseline analysis commands.
10.
After the SIO command ends and the command prompt returns, go to the PuTTY session and
turn off data collection.
statistics stop -sample-id sample_nfs1
11.
12.
On the Linux server, start SIO with a 0% read workload, 0% random, 32-KB block size, 300MB file size, run for one hundred seconds, four threads, and point it at the file on the NFS
mount.
13.
E4-5
or save the results.
STEP ACTION
14.
15.
16.
On the Linux server, start SIO with a 100% read workload, 0% random, 32-KB block size, 300MB file size, run for one hundred seconds, one thread, and point it at the file on the NFS mount.
17.
18.
19.
20.
mount.
21.
22.
23.
24.
mount. The 20-MB file will create a cached workload.
25.
E4-6
STEP ACTION
26.
27.
28.
On the Linux server, start SIO with a 50% read and 50% write workload, 100% random, 4-KB
block size, 300-MB file size, run for one hundred seconds, 32 threads, and point it at the file on
the NFS mount.
29.
30.
31.
Using the data collected in sample_nfs1 through sample_nfs6 and the analysis done on this data,
did the throughput and I/Os increase when the number of threads increased?
32.
which workload had the highest throughput in terms of KBps?
33.
which workload had the highest throughput in terms of IOPS?
34.
which workload showed the lowest latencies on the storage system?
35.
which workload showed the highest disk utilization on the storage system?
36.
which workload showed the highest CPU utilization on the storage system?
37.
38.
On the Linux server, reissue the same SIO workload by starting SIO with a 50% read and 50%
write workload, 100% random, 4-KB block size, 300-MB file size, run for one hundred seconds,
32 threads, and point it at the file on the NFS mount.
E4-7
STEP ACTION
39.
While the SIO is running, in your cluster1 PuTTY session, migrate your NAS LIF that is
associated with /mnt/path01.
network interface migrate -vserver vs2 -lif vs2_cifs_nfs_lif2 -destnode cluster1-02
40.
Using the analysis commands that you have learned, analyze the data and record or save the
results.
41.
Compare your results with the results from sample_nfs6 which uses the same workload
without a migrating LIF.
42.
43.
44.
On the Linux server, start SIO with a 50% read and 50% write workload, 100% random, 4-KB
block size, 300-MB file size, run for one hundred seconds, 32 threads, and point it at the file on
the NFS mount.
45.
Perform a volume move of volume vs2_vol01 from aggregate n01_aggr1 to n02_aggr2.

volume move start -vserver vs2 -volume vs2_vol01 -destination-aggregate
n02_aggr1
46.
Using the analysis commands that you have learned, analyze the data and record or save the
results.
HINT: Analyze the cluster interconnect statistics.
NOTE: You can view the status of the volume move with the following command:
volume move show -vserver vs2 -volume vs2_vol01
47.
After the volume move command ends and the command prompt returns, go to the PuTTY
session and turn off data collection.
48.
After you finish your analysis, clean up any old samples.

TASK 4: UNLOCK DIAG USERID
In this task, you unlock the diag userid for use with Perfstat.
E4-8
STEP ACTION
1.
Unlock the diag userid.

security login unlock -username diag
2.
Change the diag userids password.

security login password -username diag
3.
At the Enter a new password prompt, type Netapp123.
4.
At the Enter it again prompt, type Netapp123 again.
TASK 5: USING THE PERFORMANCE AND STATISTICS COLLECTOR (PERFSTAT)
In this task, you use Perfstat to collect detailed profile and troubleshooting data samples from a cluster.
STEP ACTION
1.
NOTE: Normally, you would open a browser and navigate to My Support > Downloads >
Utility Toolchest on the Support site and download the Perfstat Converged compressed file for
either Windows or Linux. This process has already been completed on your Windows Server.
2.
In the CourseFiles directory, create a directory called Perfstat, and then copy the perfstat8.exe
file from the zip file into the Perfstat directory.
3.
In the Perfstat directory, create two directories called workload1 and workload2.
4.
On the Windows Server desktop, double-click the Command Prompt icon and navigate to
C:\Users\Administrator\Desktop\CourseFiles\Perfstat\workload1.
5.
On the Linux server, start SIO with a 0% read workload, 0% random, 32-KB block size, 300MB file size, run indefinitely, 32 threads, and point it at the file on the NFS mount.
6.
In the first Command Prompt window, start Perfstat with the following command parameters:
192.168.0.101 (the cluster management LIF of cluster1)
--mode=cluster (to signify that this is a clustered Data ONTAP storage
environment)
--time 1 (to signify one minute iterations)
..\perfstat8 192.168.0.101 --mode="cluster" --time 1
7.
When prompted for the SSH passphrase, press Enter, and then press Enter again for same
passphrase.
8.
When prompted for filer password, type Netapp123, and when prompted for system shell
password (this may take a few seconds), type Netapp123 again.
After a few seconds you will see . where the curser was positioned. Perfstat runs one
iteration for each controller in the cluster and one for the host machine (in this case Windows).
E4-9
STEP ACTION
9.
After the Perfstat operation is complete, issue Ctrl-C in the Linux server session to stop the SIO
run.
10.
Look for the protocol latencies, CPU utilization, network utilization, and average per disk
utilization in the Perfstat data.
11.
In the first Command Prompt window, navigate to

C:\Users\Administrator\Desktop\CourseFiles\Perfstat\workload2.
12.
On the Linux server, start SIO with a 100% read workload, 100% random, 4-KB block size,
300-MB file size, run indefinitely, 32 threads, and point it at the file on the NFS mount.
13.
In the first Command Prompt window, start Perfstat with the same parameters as the first run.
14.
After the Perfstat operation is complete, issue Ctrl-C in the Linux server session to stop the SIO
run.
15.
Find the protocol latencies, CPU utilization, network utilization, and average per disk utilization
in the Perfstat data.
END OF EXERCISE
E4-10
MODULE 5: ONCOMMAND MANAGEMENT TOOLS

E5-1
Performance Analysis on Clustered Data ONTAP: OnCommand Management Tools
MODULE 6: STORAGE QOS

EXERCISE
In this exercise, you work with the storage quality of service (QoS) feature to prevent problem workloads and
tenants from impacting other workloads and tenants.
OBJECTIVES

Reactively limit throughput to a workload by associating the workload with a QoS policy group
Proactively monitor workload performance by associating a workload with a QoS policy group
Isolate a tenant workload by associating the workload with a QoS policy group
TASK 1: REACTIVELY LIMIT THROUGHPUT TO A WORKLOAD BY ASSOCIATING THE WORKLOAD
WITH QOS POLICY GROUP
In this task, you use Iometer to generate a 50% write and 50% read workload on a LUN and set up a storage
QoS limit to throttle the workload that is associated with the LUN.
STEP ACTION
1.
On the taskbar, click the Iometer icon and, if prompted, approve the Iometer EULA.
The Iometer window should open.
E6-1
Performance Analysis on Clustered Data ONTAP: Storage QoS
STEP ACTION
2.
In the left pane, select the WIN2K8-01 node.
3.
On the Disk Targets tab, specify the following:

Targets: D:lun_vsSCSI1_1 drive
Maximum Disk Size: 80000
Starting Disk Sector: 0
# of Outstanding I/Os: 16
E6-2
STEP ACTION
4.
Click the Access Specifications tab.
5.
Under Global Access Specifications, select 512B; 50% Read; 0% random.
E6-3
STEP ACTION
6.
Click the Add button.
7.
Click the Test Setup tab.
8.
On the Test Setup tab, specify the following:

Test Description: QoS Exercise
Ramp Up Time: 4 seconds
Record Results: None
E6-4
STEP ACTION
9.
Click the Results Display tab.
10.
Change the Update Frequency to 4 seconds.

This reduces the load on the Windows machine to update the Iometer interface.
E6-5
STEP ACTION
11.
In the toolbar, click the Start Tests button
(a green flag).
12.
Open Windows Explorer to the D:lun_vsISCSI1_1 drive and notice a new file, iobw.tst.
This is the test file that is growing until it is 40,000 KB (80,000 x 512 B).
13.
Notice that, when the file reaches its maximum size, the ramp up time begins to count down for
four seconds.
The ramp up time ensures that the storage is stable before the tests begin.
E6-6
STEP ACTION
14.
When the 50% read and 50% write test results begin to appear, notice that the total I/Os per
second (IOPS) are recorded on the top row of the display output.
NOTE: On the test system, the total IOPS will probably fall in a range below that of a clustered
system running on nonvirtualized equipment. Remember that this is a Windows machine and a
cluster running in a shared virtualized environment.
15.
Leave the test running.
16.
17.
Using the statistics command, identify the volume or LUN that is doing the most IOPS.
18.
Show the current storage QoS policies.

qos policy-group show
E6-7
STEP ACTION
19.
Switch back to the Iometer to see the current total IOPS.
20.
Create a new storage QoS policy.

NOTE: If Iometers current total IOPS is over 1000, set the -max-throughput parameter to
1000iops; otherwise, set it to 500iops.
qos policy-group create -policy-group maxtp_policy -vserver vsISCSI1
-max-throughput 1000iops
21.

22.
Associate the policy to the hot volume or LUN.

NOTE: You can associate a storage QoS policy to a storage virtual machine (SVM) with
FlexVol volumes, LUNs, volumes, or files.
23.
Verify the current throughput from the storage.

qos statistics performance show iterations 4
24.
Wait a few moments and watch the Iometers total IOPS drop.
NOTE: You can continue to run the previous command to observe the current storage readings.
It will take a few minutes for the IOPS to drop.
25.
When your testing is complete, stop the Iometer test by clicking the Stop button
toolbar.
26.
In the PuTTY window, delete the policy.

qos policy-group delete -policy-group maxtp_policy
27.
E6-8
Close Iometer.
on the
TASK 2: PROACTIVELY MONITOR WORKLOAD PERFORMANCE BY ASSOCIATING A WORKLOAD

WITH A QOS POLICY GROUP
In this task, you set up a storage QoS limit for a volume and use Iometer to generate a 50% write and 50%
read workload on the volume.
STEP ACTION
1.
2.
Create a new storage QoS policy.

qos policy-group create -policy-group prevent_policy -vserver vs2 -maxthroughput 500iops
3.

4.
You have identified vs2_vol01 as a new and potentially runaway workload.

Associate the storage QoS policy to the vs2_vol01 volume.
NOTE: You can associate a storage QoS policy to an SVM with FlexVol volumes, LUNs,
volumes, or files.
volume modify vs2_vol01 -vserver vs2 -qos-policy-group prevent_policy
5.

6.
On the taskbar, click the Iometer icon.
E6-9
STEP ACTION
7.
8.

Targets: Z drive
E6-10
STEP ACTION
9.
10.
Under Global Access Specifications, select 4K; 50% Read; 0% random.
E6-11
STEP ACTION
11.
12.
13.

E6-12
STEP ACTION
14.
15.

E6-13
STEP ACTION
16.
(a green flag).
17.
Open Windows Explorer to the Z drive and notice a new file, iobw.tst.
This is the test file that is growing until it is 80,000 KB (20,000 x 4 KB).
18.
four seconds.
E6-14
STEP ACTION
19.
When the 50% read and 50% write test results begin to appear, notice that the total IOPS are
recorded on the top row of the display output.
NOTE: Remember that this is a Windows machine and a cluster running in a shared virtualized
environment.
20.
21.
Notice the total IOPS:
E6-15
STEP ACTION
22.
In the PuTTY window, verify the current throughput from the storage.
23.
toolbar.
24.
In the PuTTY window, delete the policy.
on the
qos policy-group delete -policy-group prevent_policy
25.
Close Iometer.
TASK 3: ISOLATE A TENANT WORKLOAD BY ASSOCIATING THE WORKLOAD WITH A QOS

POLICY GROUP
In this task, you set up a storage QoS tenant limit at the SVM level and use Iometer to generate a 50% write
and 50% read workload on one of the tenants.
STEP ACTION
1.
2.
Create three new storage QoS policies.

qos policy-group create -policy-group tenant1_policy -vserver vs1
qos policy-group create -policy-group tenant2_policy -vserver vs2
qos policy-group create -policy-group tenant3_policy -vserver vsISCSI1
3.

4.
Associate each policy to the corresponding SVM.

NOTE: You can associate a storage QoS policy to an SVM with FlexVol volumes, LUNs,
volumes, or files.
vserver modify -vserver vs1 -qos-policy-group tenant1_policy
vserver modify -vserver vs2 -qos-policy-group tenant2_policy
vserver modify -vserver vsISCSI1 -qos-policy-group tenant3_policy
5.

E6-16
STEP ACTION
6.
On the taskbar, click the Iometer icon.
7.
E6-17
STEP ACTION
8.

Targets: Z drive
9.
E6-18
STEP ACTION
10.
Under Global Access Specifications, select 4K; 50% Read; 0% random.
11.
12.
E6-19
STEP ACTION
13.

14.
E6-20
STEP ACTION
15.

16.
E6-21
(a green flag).
STEP ACTION
17.
Open File Explorer to the Z drive and notice a new file, iobw.tst.
This is the test file that is growing until it is 80,000 KB (20,000 x 4 KB).
18.
four seconds.
E6-22
STEP ACTION
19.
When the 50% read and 50% write test results begin to appear, notice that the total IOPS are
recorded on the top row of the display output.
NOTE: Remember that this is a Windows machine and a cluster running in shared virtualized
environment.
20.
21.
In the PuTTY window, verify the current throughput from the storage.
E6-23
STEP ACTION
22.
Switch back to the Iometer to see the current total IOPS.
23.
toolbar.
24.
Close Iometer.
END OF EXERCISE
E6-24
on the
MODULE 7: SUMMARY
E7-1
Performance Analysis on Clustered Data ONTAP: Summary
NETAPP UNIVERSITY
Performance Analysis on Clustered Data ONTAP

Appendix A: Answers
Course ID: STRSW-ILT-PERFCDOT
Catalog Number: STRSW-ILT-PERFCDOT-EG
Content Version: 1.0
A-1
Performance Analysis on Clustered Data ONTAP: Appendix A: Answers
MODULE 2: PERFORMANCE OVERVIEW

EXERCISE
TASK 5: IDENTIFY CLUSTERED DATA ONTAP COMPONENTS
In this task, you use either System Manager or the clustered Data ONTAP CLI to identify key clustered Data
ONTAP components and revert any LIFs that are not on their home ports.
STEP ACTION
1.
Analyze and identify the following list of clustered Data ONTAP components:
Aggregates __aggr0_n1, aggr0_n2, n01_aggr1,
n02_aggr1____________________________________________________________
Storage virtual machines __cluster1, cluster1-01, cluster1-02, vs1, vs2,
vsISCSI1_________________________________________________
Volumes ________________________________________________________________vol0,
vol0, vs1_root, vs2_root, vs2_vol01, lun_ vsISCSI1_1 _vol, vsISCSI1_root
LUNs __/vol/lun_vsISCSI1_1_vol/lun_vsISCSI1_1
________________________________________________________________
Licenses (which features are installed?) __Base, Insight Balance, CIFS, NFS,
iSCSI______________________________________
CIFS Shares __admin$, ipc$, rootdir, vol1, ~%w
____________________________________________________________
LIFs (are all LIFs home?)__No__________________________________________________
A-2
MODULE 3: CLUSTERED STORAGE SYSTEM WORKLOADS AND BOTTLENECKS

EXERCISE
TASK 1: EXAMINE THE STATISTICS CATALOG COMMAND
In this task, you issue the three statistics catalog commands and exercise the supporting parameters,
using different privilege levels.
STEP ACTION
4.
What command syntax would you use to display only the advanced level statistics objects
without their descriptions?
statistics catalog object show -privilege advanced -fields object
5.
What command syntax would you use to display statistics objects that are associated with
storage virtual machines (SVMs)?
NOTE: You should still be in the advanced privilege level.
statistics catalog object show -object *vserver*
How many statistics objects are there in the list? 11 (in Clustered Data ONTAP 8.2)
7.
statistics object that represents SVMs that have LIFs associated with them?
statistics catalog instance show -object lif:vserver
How many instance names are there in the list? 5

8.
statistics object that represents SVMs that have volumes associated with them?
statistics catalog instance show -object volume:vserver
How many instance names are there in the list? 3

9.
statistics object that represents the disk that is associated with the second node in the cluster
(cluster1-02)?
statistics catalog instance show -object disk -node cluster1-02
TASK 2: EXAMINE THE STATISTICS START AND STATISTICS SHOW COMMANDS
In this task, you issue the statistics start and statistics show commands and exercise the
supporting parameters, using different privilege levels.
STEP ACTION
3.
Display the counters that are associated with the statistics object nfsv3 instance vs2 for both
samples.
sample_nfsv3_adm
sample_nfsv3_adv
Was there any difference in the displays? Yes

A-3
STEP ACTION
5.
What command syntax would you use to display all of the latency related counters for disk
v4.20?
statistics show -object disk -instance v4.20 -counter *latency*
-sample-id sample_disk
How many counters are there in the list? 10

6.
What command syntax would you use to display all of the user_read_latency counters for all
disks in the cluster?
-sample-id sample_disk
How many counters are there in the list? 28

8.
What command syntax would you use to display all of the user_read_latency counters that are
greater than 10000us for all disks in the cluster, limiting the display to only the counter and
value fields?
-sample-id sample_disk -fields counter,value -value >10000
13.
What command syntax would you use to display all of the latency counters for all volumes in
the cluster? How about all nonzero latency counters?
statistics show -object volume -instance * -counter *latency
-sample-id sample_volume
statistics show -object volume -instance * -counter *latency
-sample-id sample_volume -value >0
15.
What command syntax would you use to display all of the counters for all processors in the
cluster?
statistics show -object processor -instance * -sample-id
sample_processor
A-4
TASK 3: DEFINING WORKLOAD CHARACTERISTICS
In this task, you gather statistical data and evaluate the data to determine the workload characteristics.
STEP ACTION
15.

CHARACTERISTIC
A-5
OBSERVED VALUE
Read
Write
Throughput
0/sec
99/sec
Latency
6.32ms
Operation Size
32K
Concurrency
0.63
Randomness
STEP ACTION
16.
cluster1::*> statistics show -object cifs -instance * -counter cifs_read_ops -sample-id
sample_cifs1
Object: cifs
Instance: vs2
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_ops
cifs_read_ops
2 entries were displayed.
cluster1::*> statistics show -object cifs -instance * -counter cifs_write_ops -sample-id

sample_cifs1
Object: cifs
Instance: vs2
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_write_ops
99
cifs_write_ops
A-6
STEP ACTION
17.
cluster1::*> statistics show -object volume -instance * -counter cifs_read_latency sample-id sample_cifs1
...
Object: volume
Instance: vs2_vol01
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_latency
...
cluster1::*> statistics show -object volume -instance * -counter cifs_write_latency sample-id sample_cifs1
...
Object: volume
Instance: vs2_vol01
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_write_latency
6324us
Object: volume
Instance: vsISCSI1_root
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
A-7
STEP ACTION
18.
cluster1::*> statistics show -object cifs -instance * -counter cifs_read_size_histo sample-id sample_cifs1
Object: cifs
Instance: vs2
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_size_histo
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
cifs_read_size_histo
<= 32 KB
<= 64 KB
> 64 KB
A-8
STEP ACTION
19.
cluster1::*> statistics show -object cifs -instance * -counter cifs_write_size_histo sample-id sample_cifs1
Object: cifs
Instance: vs2
Start-time: 9/22/2013 21:56:21
End-time: 9/22/2013 21:58:18
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_write_size_histo
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
23284
<= 64 KB
> 64 KB
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
cifs_write_size_histo
20.
99/sec * 0.00632 (6.32ms) = 0.63
21.
sample_cifs1?
100% write workload, no reason to run this command
A-9
STEP ACTION
22.
sample_cifs1?
100% write workload, no reason to run this command
23.
All writes, large operations, virtually no concurrency
24.

CHARACTERISTIC
A-10
OBSERVED VALUE
Read
Write
Throughput
715/sec
0/sec
Latency
3.20ms
Operation Size
32K
Concurrency
2.29
Randomness
Sequential
STEP ACTION
25.
sample_cifs2
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_ops
715
cifs_read_ops

sample_cifs2
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_write_ops
cifs_write_ops
A-11
STEP ACTION
26.
...
Object: volume
Instance: vs2_vol01
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
3196us
...
...
Object: volume
Instance: vs2_vol01
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
Object: volume
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
A-12
STEP ACTION
27.
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
137368
<= 64 KB
> 64 KB
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
A-13
STEP ACTION
28.
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
29.
715/sec * 0.00320 (3.20ms) = 2.29
A-14
STEP ACTION
30.
sample_cifs2?
cluster1::*> statistics show -object readahead -instance * -counter rand_read_reqs sample-id sample_cifs2
Object: readahead
Instance: readahead
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------rand_read_reqs
UNUSED
4K
100
8K
98
12K
100
16K
100
20K
100
24K
28K
50
32K
40K
48K
56K
MAX
...
rand_read_reqs
UNUSED
4K
100
8K
100
12K
100
16K
20K
100
24K
100
28K
100
32K
40K
48K
56K
MAX
...
A-15
STEP ACTION
31.
sample_cifs2?
cluster1::*> statistics show -object readahead -instance * -counter seq_read_reqs sample-id sample_cifs2
Object: readahead
Instance: readahead
Start-time: 9/22/2013 22:21:55
End-time: 9/22/2013 22:23:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------seq_read_reqs
UNUSED
4K
8K
12K
16K
20K
24K
28K
50
32K
99
40K
48K
56K
MAX
...
seq_read_reqs
UNUSED
4K
8K
12K
16K
20K
24K
28K
32K
40K
48K
56K
MAX
...
A-16
STEP ACTION
32.
All reads, large operations, some concurrency, mostly random
33.

CHARACTERISTIC
A-17
OBSERVED VALUE
Read
Write
Throughput
159/sec
159/sec
Latency
168.37ms
0.14ms
Operation Size
4K
4K
Concurrency
26.77
0.02
Randomness
Random
STEP ACTION
34.
sample_cifs3
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_ops
159
cifs_read_ops

sample_cifs3
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_write_ops
159
cifs_write_ops
A-18
STEP ACTION
35.
...
Object: volume
Instance: vs2_vol01
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
168371us
...
...
Object: volume
Instance: vs2_vol01
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
143us
Object: volume
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
A-19
STEP ACTION
36.
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
30356
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
A-20
STEP ACTION
37.
Object: cifs
Instance: vs2
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
30288
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
0 bytes
<= 256 bytes
<= 512 bytes
<= 1 KB
<= 2 KB
<= 4 KB
<= 8 KB
<= 16 KB
<= 32 KB
<= 64 KB
> 64 KB
38.
Read:
159/sec * 0.16837 (168.37ms) = 26.77
Write: 159/sec * 0.00014 (0.14ms)
A-21
0.02
STEP ACTION
39.
sample_cifs3?
cluster1::*> statistics show -object readahead -instance * -counter rand_read_reqs sample-id sample_cifs3
Object: readahead
Instance: readahead
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------rand_read_reqs
UNUSED
4K
82
8K
50
12K
97
16K
20K
83
24K
100
28K
32K
13
40K
48K
56K
MAX
...
rand_read_reqs
UNUSED
4K
100
8K
12K
100
16K
100
20K
100
24K
100
28K
100
32K
100
40K
48K
56K
MAX
...
A-22
STEP ACTION
40.
sample_cifs3?
cluster1::*> statistics show -object readahead -instance * -counter seq_read_reqs sample-id sample_cifs3
Object: readahead
Instance: readahead
Start-time: 9/22/2013 22:25:56
End-time: 9/22/2013 22:27:31
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------seq_read_reqs
UNUSED
4K
17
8K
50
12K
16K
100
20K
16
24K
28K
100
32K
86
40K
48K
56K
MAX
...
seq_read_reqs
UNUSED
4K
8K
12K
16K
20K
24K
28K
32K
40K
48K
56K
MAX
...
A-23
STEP ACTION
41.
50% reads 50% writes, small operations, some read concurrency minimal write
concurrency
42.
The previous examples used CIFS as the basis for the workloads. What statistics objects and
counters would be needed for NFSv3?
set diagnostic
statistics start -object nfsv3|volume|readahead -sample-id sample_nfsv3
statistics
-sample-id
statistics
-sample-id
show -object nfsv3 -instance * -counter nfsv3_read_ops

sample_nfsv3
show -object nfsv3 -instance * -counter nfsv3_write_ops
sample_nfsv3
statistics
-sample-id
statistics
-sample-id
show -object volume -instance * -counter nfs_read_latency

sample_nfsv3
show -object volume -instance * -counter nfs_write_latency
sample_nfsv3
statistics show -object nfsv3 -instance * -counter

nfsv3_read_size_histo -sample-id sample_nfsv3
statistics show -object nfsv3 -instance * -counter
nfsv3_write_size_histo -sample-id sample_nfsv3
statistics
-sample-id
statistics
-sample-id
A-24
show -object readahead -instance * -counter rand_read_reqs

sample_nfsv3
show -object readahead -instance * -counter seq_read_reqs
sample_nfsv3
MODULE 4: CLUSTER PERFORMANCE MONITORING AND ANALYSIS

EXERCISE
TASK 1: PERFORM INITIAL HEALTH CHECKS ON THE CLUSTER
In this task, you query the cluster to assess the initial health status.
STEP ACTION
2.
Check to see if all of the nodes are healthy.

cluster show
cluster1::> cluster show
Node
Health
Eligibility
--------------------- ------- -----------cluster1-01
true
true
cluster1-02
true
true
3.
Check to see if the replication rings have the same (or consistent) masters.
NOTE: Remember to set advanced privilege before you execute this command.
cluster ring show
cluster1::> set advanced
Warning: These advanced commands are potentially dangerous; use them
only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y
cluster1::*> cluster ring show
Node
UnitName Epoch
DB Epoch
--------- -------- -------- -------cluster1-01 mgmt
3
3
cluster1-01 vldb
3
3
cluster1-01 vifmgr 3
3
cluster1-01 bcomd 3
3
cluster1-02 mgmt
3
3
cluster1-02 vldb
3
3
cluster1-02 vifmgr 3
3
cluster1-02 bcomd 3
3
A-25
DB Trnxs
-------3910
11
20
22
3910
11
20
22
Master
Online
--------- --------cluster1-01 master
cluster1-01 master
cluster1-01 master
cluster1-01 master
cluster1-01 secondary
STEP ACTION
4.
Using the advanced privilege level, check to see if the cluster connectivity is healthy.
cluster ping-cluster -node local
cluster1::*> cluster ping-cluster -node local
Host is cluster1-02
Getting addresses from network interface table...
Local = 169.254.173.221 169.254.230.66
Remote = 169.254.224.211 169.254.185.5
Ping status:
....
Basic connectivity succeeds on 4 path(s)
Basic connectivity fails on 0 path(s)
................
Detected 1500 byte MTU on 4 path(s):
Local 169.254.173.221 to Remote 169.254.185.5
Local 169.254.173.221 to Remote 169.254.224.211
Local 169.254.230.66 to Remote 169.254.185.5
Local 169.254.230.66 to Remote 169.254.224.211
Larger than PMTU communication succeeds on 4 path(s)
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)
5.
Check to see if the storage virtual machines (SVMs) are healthy.

dashboard health vserver show
cluster1::> dashboard health vserver show
EMS Issues
Vserver
Status
Health
Crit Warn Info
----------------- -------- ------- ----- ----- ----vs1
offline ok
0
0
0
Issues: There are no data lifs configured.
The filesystem protocols are not configured.
vs2
online
ok
0
Issues: LIF vs2_cifs_nfs_lif2 is not home.
vsISCSI1
offline ok
0
0
0
Issues: The filesystem protocols are not configured.
6.
If any of the SVMs return a nonzero status, add the all parameter for additional information.
dashboard health vserver show-all
cluster1::> dashboard health vserver show-all
There are no Vserver health issues reported.
7.
Check to see if any volumes are not online.

volume show -state !online
cluster1::> volume show -state !online
There are no entries matching your query.
8.
Check to see if any aggregates are not online.

storage aggregate show -state !online
cluster1::> storage aggregate show -state !online
A-26
STEP ACTION
9.
Check to see if any disks are in a broken state.

storage disk show -state broken
cluster1::> storage disk show -state broken
TASK 2: BASELINE PERFORMANCE MONITORING FROM THE CLUSTER SHELL
STEP ACTION
1.
Check the performance status:

-iterations 4
cluster1::> statistics show-periodic -node cluster1-01 -instance node -iterations 4
cluster1-01: node.node: 10/15/2013 13:11:44
cpu
total
data
data
data cluster cluster cluster
disk
disk
busy
ops nfs-ops cifs-ops busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- -------3%
0
0
0
0%
0B
0B
0%
3.35KB
35.0KB
0B
0B
3%
0
0
0
0%
192B
0B
0%
2.28KB
6.11KB
337KB
294KB
3%
0
0
0
0%
97B
0B
0%
2.99KB
7.87KB
0B
0B
2%
0
0
0
0%
96B
0B
0%
2.03KB
6.01KB
0B
0B
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- -------Minimums:
2%
0
0
0
0%
0B
0B
0%
2.03KB
6.01KB
0B
0B
Averages for 4 samples:
2%
0
0
0
0%
96B
0B
0%
2.66KB
13.8KB
84.4KB
73.7KB
Maximums:
3%
0
0
0
0%
192B
0B
0%
3.35KB
35.0KB
337KB
294KB

-iterations 4
cluster1::> statistics show-periodic -node cluster1-02 -instance node -iterations 4
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- -------3%
0
0
0
0%
412B
487B
0%
29.8KB
2.10KB
217KB
341KB
2%
0
0
0
0%
826B
329B
0%
4.02KB
9.79KB
0B
0B
3%
0
0
0
0%
315B
99B
0%
1.11KB
600B
0B
0B
1%
0
0
0
0%
31B
99B
0%
1.96KB
1.39KB
0B
0B
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- -------- -------Minimums:
1%
0
0
0
0%
31B
99B
0%
1.11KB
600B
0B
0B
2%
0
0
0
0%
396B
253B
0%
9.22KB
3.47KB
54.4KB
85.3KB
Maximums:
3%
0
0
0
0%
826B
487B
0%
29.8KB
9.79KB
217KB
341KB
b. Check the current level of activity on the entire cluster and notice the overall latency
column.
cluster1::> dashboard performance show
Average
---Data-Network--Total Latency CPU Busy
Recv
Sent
Ops/s in usec Busy Util
MB/s
MB/s
------ ------- ---- ---- ------ -----cluster1-01
0
0
3%
0%
0
0
cluster1-02
0
0
3%
0%
0
0
cluster:summary
0
0
3%
0%
0
0
A-27
-Cluster--Network- ----Storage--Busy
Recv
Sent
Read Write
Util
MB/s
MB/s
MB/s
MB/s
---- ------ ------ ------ -----0%
0%
0%
STEP ACTION
2.
Check the current NAS operations level on the entire cluster.

dashboard performance show operations
cluster1::> dashboard performance show -operations
Total Average
NFS
NFS
CIFS
CIFS
Node
Ops/s Latency
Ops/s Latency
Ops/s Latency
------- ------- -------- ------- -------- ------- -------cluster1-01
0
0us
0
0us
0
0us
cluster1-02
0
0us
0
0us
0
0us
cluster:summary 0
0us
0
0us
0
0us
A-28
STEP ACTION
3.
Use the instance parameter to show everything and notice the overall latency by protocol
fields.
dashboard performance show instance
cluster1::> dashboard performance show -instance
A-29
Node:
Average Latency (usec):
CPU Busy:
Total Ops/s:
NFS Ops/s:
CIFS Ops/s:
Data Network Utilization:
Data Network Received (per sec):
Data Network Sent (per sec):
Cluster Network Utilization:
Cluster Network Received (per sec):
Cluster Network Sent (per sec):
Storage Read (per sec):
Storage Write (per sec):
CIFS Average Latency:
NFS Average Latency:
cluster1-01
0us
2%
0
0
0
0%
0
0
0%
0
0
0
0
0us
0us
Node:
CPU Busy:
Total Ops/s:
NFS Ops/s:
CIFS Ops/s:
cluster1-02
0us
2%
0
0
0
0%
0
0
0%
0
0
0
0
0us
0us
Node:
CPU Busy:
Total Ops/s:
NFS Ops/s:
CIFS Ops/s:
cluster:summary
0us
2%
0
0
0
0%
0
0
0%
0
0
0
0
0us
0us
STEP ACTION
4.
Check the overall latency for the entire cluster.

cluster1::> statistics show-periodic -instance latency -iterations 4
cluster:summary: cluster.cluster: 10/15/2013 13:19:18
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- ------- -------3%
0
0
0
0%
371B
279B
0%
98.2KB
97.8KB
0B
0B
10%
0
0
0
0%
222B
100B
0%
34.1KB
33.6KB
0B
0B
4%
0
0
0
0%
413B
100B
0%
11.3KB
11.5KB
333KB
413KB
3%
0
0
0
0%
31B
100B
0%
7.80KB
7.71KB
238KB
335KB
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- ------- -------Minimums:
3%
0
0
0
0%
31B
100B
0%
7.80KB
7.71KB
0B
0B
5%
0
0
0
0%
259B
144B
0%
37.9KB
37.7KB
142KB
187KB
Maximums:
10%
0
0
0
0%
413B
279B
0%
98.2KB
97.8KB
333KB
413KB
A-30
STEP ACTION
5.
Check the overall latency for each node in the cluster.

-iterations 4
cluster1::> statistics show-periodic -node cluster1-01 -instance latency -iterations 4
cluster1-01: node.latency: 10/15/2013 13:20:57
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- ------- -------3%
0
0
0
0%
192B
0B
0%
3.35KB
34.9KB
0B
0B
2%
0
0
0
0%
96B
0B
0%
2.25KB
6.50KB
0B
0B
3%
0
0
0
0%
96B
0B
0%
2.82KB
7.12KB
0B
0B
4%
0
0
0
0%
194B
0B
0%
3.80KB
7.02KB
275KB
329KB
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- ------- -------Minimums:
2%
0
0
0
0%
96B
0B
0%
2.25KB
6.50KB
0B
0B
3%
0
0
0
0%
144B
0B
0%
3.05KB
13.9KB
69.0KB
82.3KB
Maximums:
4%
0
0
0
0%
194B
0B
0%
3.80KB
34.9KB
275KB
329KB

-iterations 4
cluster1::> statistics show-periodic -node cluster1-02 -instance latency -iterations 4
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- ------- -------7%
0
0
0
0%
412B
458B
0%
30.8KB
3.41KB
0B
0B
3%
0
0
0
0%
31B
99B
0%
1.19KB
831B
0B
0B
3%
0
0
0
0%
219B
99B
0%
2.08KB
1.04KB
0B
0B
3%
0
0
0
0%
125B
99B
0%
1.25KB
892B
570KB
764KB
cpu
total
data
data
disk
disk
busy
recv
sent
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- -------- ------- -------- -------- ------- -------Minimums:
3%
0
0
0
0%
31B
99B
0%
1.19KB
831B
0B
0B
4%
0
0
0
0%
196B
188B
0%
8.84KB
1.53KB
142KB
191KB
Maximums:
7%
570KB
A-31
0
764KB
0%
412B
458B
0%
30.8KB
3.41KB
STEP ACTION
6.
Check the throughput by cluster.

cluster1::> statistics show-periodic -iterations 4
cpu
total
data
data
data
disk
disk
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- --------- -------10%
0
0
0
0%
9.37KB
279B
0B
0B
2%
0
0
0
0%
3.02KB
100B
0B
0B
4%
0
0
0
0%
796B
100B
290KB
349KB
12%
0
0
0
0%
3.84KB
100B
0B
0B
cpu
total
data
data
data
disk
disk
busy
recv
sent
read
write
---- -------- -------- -------- ---- -------- --------- -------Minimums:
2%
0
0
0
0%
796B
100B
0B
0B
7%
0
0
0
0%
4.25KB
144B
72.5KB
87.3KB
Maximums:
12%
0
0
0
0%
9.37KB
279B
290KB
349KB
A-32
cluster
cluster
cluster
busy
recv
sent
------- -------- -------- -----0%
89.5KB
89.1KB
0%
8.33KB
8.22KB
0%
8.45KB
8.34KB
0%
11.1KB
10.9KB
cluster
cluster
cluster
busy
recv
sent
------- -------- -------- ------
0%
8.33KB
8.22KB
0%
29.3KB
29.1KB
0%
89.5KB
89.1KB
STEP ACTION
7.
Check the throughput by volume for volume vs2_vol01.

statistics show-periodic -object volume -instance vs2_vol01 -iterations
4
cluster1::> statistics show-periodic -object volume -instance vs2_vol01 -iterations 4
cluster1: volume.vs2_vol01: 10/15/2013 13:24:54
cifs_protocol cifs_protocol cifs_protocol fcp_protocol fcp_protocol fcp_protocol
iscsi_protocol
iscsi_protocol iscsi_protocol nfs_protocol nfs_protocol nfs_protocol
total_protocol total_protocol total_protocol
avg
other
read
write
other
read
write instance
other
read
write
other
read
write
node
node
other
other process
read
read
total
other
read
write vserver vserver
write
write
write
latency
latency
latency
latency
latency
latency
latency
name
latency
latency
latency
latency
latency
latency
name
uuid latency
ops
name
data
latency read_ops
ops
latency
latency
latency
name
uuid
data latency
ops
-------- ------------- ------------- ------------- ------------ ------------ ------------ -------- -------------- ------------- -------------- ------------ ------------ ------------ -------- -------- -------- -------- -------- ------- -------- -------- -------- -------------- -------------- -------------- -------- -------- -------- -------- -------0us
- vs2_vol01
- cluster1-01 dc52885e-cd79-11e2-923d-418872c5fb78 0us 0 - 0B
0us
0
0
vs2 ae9c5c97-107e-11e3-823d-123478563412 0B 0us
0
0us
- vs2_vol01
0us
0
0
vs2 ae9c5c97-107e-11e3-823d-123478563412 0B 0us
0
0us
- vs2_vol01
0us
0
0
vs2 ae9c5c97-107e-11e3-823d-123478563412 0B 0us
0
0us
- vs2_vol01
0us
0
0
vs2 ae9c5c97-107e-11e3-823d-123478563412 0B 0us
0
cifs_protocol cifs_protocol cifs_protocol fcp_protocol fcp_protocol fcp_protocol
iscsi_protocol
iscsi_protocol iscsi_protocol nfs_protocol nfs_protocol nfs_protocol
avg
other
read
write
other
read
write instance
other
read
write
other
read
write
node
node
other
other process
read
read
total
other
read
write
write
write
latency
latency
latency
latency
latency
latency
latency
name
latency
latency
latency
latency
latency
latency
name
uuid latency
ops
name
data
latency read_ops
ops
latency
latency
latency
name
uuid
data latency
ops
-------- ------------- ------------- ------------- ------------ ------------ ------------ -------- -------------- ------------- -------------- ------------ ------------ ------------ -------- -------- -------- -------- -------- ------- -------- -------- -------- -------------- -------------- -------------- -------- -------- -------- -------- -------Minimums:
0us
0us
0
0B
0us
0
0
0B
0us
0
0us
0us
0
0B
0us
0
0
0B
0us
0
Maximums:
0us
0us
0
0B
0us
0
0
0B
0us
0
How can you make the output more readable? Specify only the fields that you want to see
with the -fields parameter.
A-33
STEP ACTION
9.
Check the throughput by LIF (IP).

statistics show-periodic -object lif -instance vs2:vs2_cifs_nfs_lif1
-iterations 4
cluster1::> statistics show-periodic -object lif -instance
vs2:vs2_cifs_nfs_lif1 -iterations 4
cluster1: lif.vs2:vs2_cifs_nfs_lif1: 10/15/2013 13:36:26
instance instance
node
node process
recv
recv
recv
sent
sent
sent
vserver vserver
name
uuid
name
uuid
name
data
errors
packet
data
errors
packet up_time
id
name
-------- -------- -------- -------- -------- -------- -------- ------- -------- -------- -------- -------- -------- -------vs2:vs2_cifs_nfs_lif1 1026 cluster1-02 dc82ec7d-cd68-11e2-96922d8687fb65b7 - 0B 0 0 0B 0
0 438684778
4
vs2
vs2:vs2_cifs_nfs_lif1 1026 cluster1-02 dc82ec7d-cd68-11e2-96922d8687fb65b7 - 0B 0 0 0B 0
0 438686875
4
vs2
0 438688952
4
vs2
0 438691049
4
vs2
cluster1: lif.vs2:vs2_cifs_nfs_lif1: 10/15/2013 13:36:35
instance instance
node
node process
recv
recv
recv
sent
sent
sent
vserver vserver
name
uuid
name
uuid
name
data
errors
packet
data
errors
packet up_time
id
name
-------- -------- -------- -------- -------- -------- -------- ------- -------- -------- -------- -------- -------- -------Minimums:
0B
0
0
0B
0
0
0B
0
0
0B
0
0
Maximums:
0B
0
0
0B
0
0
-
10.
Using the advanced privilege level, start statistics data collection on the objects volume,
aggregate, disk, ext_cache_obj, port, and lif.
set advanced
statistics start -object volume|aggregate|disk|port|lif -sample-id
sample_baseline1
cluster1::> set advanced
Warning: These advanced commands are potentially dangerous; use them
only when directed to do so by NetApp personnel.
cluster1::*> statistics start -object volume|aggregate|disk|port|lif sample-id sample_baseline1
Statistics collection is being started for Sample-id: sample_baseline1
A-34
STEP ACTION
11.
Check the latency by volume for the entire cluster.

statistics show -object volume -counter *latency -sample-id
sample_baseline1
cluster1::*> statistics show -object volume -counter *latency -sampleid sample_baseline1
Object: volume
Instance: lun_vsISCSI1_1_vol
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:43:43
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------avg_latency
34us
cifs_other_latency
cifs_protocol_other_latency
<20us
0
<40us
0
<60us
0
<80us
0
<100us
0
...
A-35
STEP ACTION
12.
Check volume latency for a specific volume.

statistics show-periodic -object volume -instance vs2_vol01
-iterations 4
cluster1::*> statistics show-periodic -object volume -instance vs2_vol01 -iterations 4
cifs
cifs cifs_protocol cifs_protocol cifs_protocol
cifs
cifs
cifs
cifs
cifs
cifs fcp_protocol fcp_protocol fcp_protocol
iscsi_protocol iscsi_protocol iscsi_protocol
nfs
nfs
nfs_protocol nfs_protocol nfs_protocol
nfs
nfs
nfs
nfs
nfs
avg
other
other
other
read
write
read
read
read
write
write
write
other
read
write instance
other
read
write
other
other
other
read
write
read
read
read
write
write
node
node
other
other
process
read
read
total
other
read
write
write
write
latency latency
ops
latency
latency
latency
data latency
ops
data latency
ops
latency
latency
latency
name
latency
latency
latency latency
ops
latency
latency
latency
data latency
ops latency
ops
name
uuid latency
ops
name
data latency read_ops
ops
latency
latency
latency
name
uuid
data
latency
ops
-------- -------- -------- ------------- ------------- ------------- -------- -------- -------- -------- -------- ------- ------------ ------------ ------------ -------- -------------- -------------- -------------- -------- -------- ----------- ------------ ------------ -------- -------- -------- -------- -------- -------- -------- -------- --------------- -------- -------- -------- -------- -------------- -------------- -------------- -------- -------- -------- ------- -------0us
0us
0
0B
0us
0
0B
0us
0
- vs2_vol01
0us
0
0B
0us
0
0us
0 cluster1-01 dc52885e-cd79-11e2-923d418872c5fb78 0us 0 - 0B 0us
0
0
vs2 ae9c5c97-107e-11e3823d-123478563412 0B 0us 0
0us
0us
0
0B
0us
0
0B
0us
0
- vs2_vol01
0us
0
0B
0us
0
0us
0
0
vs2 ae9c5c97-107e-11e3823d-123478563412 0B 0us 0
0us
0us
0
0B
0us
0
0B
0us
0
- vs2_vol01
0us
0
0B
0us
0
0us
0
0
vs2 ae9c5c97-107e-11e3823d-123478563412 0B 0us 0
0us
0us
0
0B
0us
0
0B
0us
0
- vs2_vol01
0us
0
0B
0us
0
0us
0
0
vs2 ae9c5c97-107e-11e3823d-123478563412 0B 0us 0
cifs
cifs cifs_protocol cifs_protocol cifs_protocol
cifs
cifs
cifs
cifs
cifs
cifs fcp_protocol fcp_protocol fcp_protocol
iscsi_protocol iscsi_protocol iscsi_protocol
nfs
nfs
nfs_protocol nfs_protocol nfs_protocol
nfs
nfs
nfs
nfs
nfs
avg
other
other
other
read
write
read
read
read
write
write
write
other
read
write instance
other
read
write
other
other
other
read
write
read
read
read
write
write
node
node
other
other
process
read
read
total
other
read
write
write
write
latency latency
ops
latency
latency
latency
data latency
ops
data latency
ops
latency
latency
latency
name
latency
latency
latency latency
ops
latency
latency
latency
data latency
ops latency
ops
name
uuid latency
ops
name
data latency read_ops
ops
latency
latency
latency
name
uuid
data
latency
ops
-------- -------- -------- ------------- ------------- ------------- -------- -------- -------- -------- -------- ------- ------------ ------------ ------------ -------- -------------- -------------- -------------- -------- -------- ----------- ------------ ------------ -------- -------- -------- -------- -------- -------- -------- -------- --------------- -------- -------- -------- -------- -------------- -------------- -------------- -------- -------- -------- ------- -------Minimums:
0us
0us
0
0B
0us
0
0B
0us
0
0us
0
0B
0us
0
0us
0
0us
0
0B
0us
0
0
0B
0us
0
0us
0us
0
0B
0us
0
0B
0us
0
0us
0
0B
0us
0
0us
0
0us
0
0B
0us
0
0
0B
0us
0
Maximums:
0us
0us
0
0B
0us
0
0B
0us
0
0us
0
0B
0us
0
0us
0
0us
0
0B
0us
0
0
0B
0us
0
A-36
STEP ACTION
13.
Check volume activity.

statistics show -object volume -counter *data -sample-id
sample_baseline1
cluster1::*> statistics show -object volume -counter *data -sample-id sample_baseline1
Object: volume
Instance: lun_vsISCSI1_1_vol
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:49:01
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_data
0B
cifs_write_data
0B
nfs_read_data
0B
read_data
0B
write_data
0B
Object: volume
Instance: vol0
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:49:01
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_data
0B
cifs_write_data
0B
nfs_read_data
34.2KB
read_data
34.2KB
write_data
7.61KB
cifs_read_data
0B
cifs_write_data
0B
nfs_read_data
20.0KB
read_data
21.3KB
write_data
8.43KB
...
Object: volume
Instance: vs2_vol01
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:49:01
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_data
0B
cifs_write_data
0B
nfs_read_data
0B
read_data
0B
write_data
0B
Object: volume
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:49:01
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cifs_read_data
0B
cifs_write_data
0B
nfs_read_data
0B
read_data
0B
write_data
0B
A-37
STEP ACTION
14.
Check aggregate latency.

statistics show -object aggregate -counter user_reads|user_writes
-sample-id sample_baseline1
cluster1::*> statistics show -object aggregate -counter
user_reads|user_writes -sample-id sample_baseline1
Object: aggregate
Instance: aggr0_n1
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:53:07
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------user_reads
0
user_writes
2
Object: aggregate
Instance: aggr0_n2
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:53:07
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------user_reads
0
user_writes
2
Object: aggregate
Instance: n01_aggr1
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:53:07
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------user_reads
0
user_writes
0
Object: aggregate
Instance: n02_aggr1
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:53:07
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------user_reads
0
user_writes
0
A-38
STEP ACTION
15.
Check disk latency.

statistics show -object disk -counter *latency -sample-id
sample_baseline1
cluster1::*> statistics show -object disk -counter *latency -sample-id
sample_baseline1
Object: disk
Instance: v4.16
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:54:21
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------cp_read_latency
3630us
guarenteed_read_latency
guarenteed_write_latency
user_read_latency
user_write_latency
cp_read_latency
3886us
guarenteed_read_latency
guarenteed_write_latency
user_read_latency
3235us
user_write_latency
3034us
Object: disk
Instance: v4.17
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:54:21
Cluster: cluster1
Counter
Value
...
A-39
STEP ACTION
16.
Check port throughput.

statistics show -object port -counter *data -sample-id sample_baseline1
cluster1::*> statistics show -object port -counter *data -sample-id
sample_baseline1
Object: port
Instance: e0a
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:56:47
Node: cluster1-01
Counter
Value
-------------------------------- -------------------------------recv-data
1.05GB
sent-data
1.44GB
Object: port
Instance: e0b
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:56:47
Node: cluster1-01
Counter
Value
-------------------------------- -------------------------------recv-data
656MB
sent-data
810MB
...
Object: port
Instance: e0f
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:56:47
Node: cluster1-02
Counter
Value
-------------------------------- -------------------------------recv-data
53.6MB
sent-data
1.35MB
A-40
STEP ACTION
17.
Check the LIF or IP throughput.

statistics show -object lif -counter *data -sample-id sample_baseline1
cluster1::*> statistics show -object lif -counter *data -sample-id
sample_baseline1
Object: lif
Instance: cluster1-01:clus1
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:59:39
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------recv_data
258B
sent_data
490B
Object: lif
Instance: cluster1-01:clus2
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:59:39
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------recv_data
213B
sent_data
335B
...
Object: lif
Instance: vsISCSI1:cluster1_02_iscsi_lif_2
Start-time: 10/15/2013 13:38:10
End-time: 10/15/2013 13:59:39
Cluster: cluster1
Counter
Value
-------------------------------- -------------------------------recv_data
0B
sent_data
0B
18.
After completing the task, turn off the statistics data collection.
statistics stop -sample-id sample_baseline1
cluster1::*> statistics stop -sample-id sample_baseline1
Statistics collection is being stopped for Sample-id: sample_baseline1
A-41
STEP ACTION
19.
Clean up any old samples.

cluster1::*> statistics samples show
Vserver
Sample ID
Start Time
Stop Time
Status
---------------- -------------------- -------------- -------------- --------cluster1
sample_baseline1
10/15 13:38:10 10/15 14:01:22
Ready
cluster1::*> statistics samples delete -sample-id *
1 entry was acted on.
TASK 3: PERFORMANCE MONITORING FROM THE CLUSTER SHELL
STEP ACTION
5.
Extract (untar) the contents of sio_ntap.tar.gz into the current directory.

tar xvfz /mnt/path01/sio_ntap.tar.gz
[root@CM-CentOS-001 sio]# tar xvfz /mnt/path01/sio_ntap.tar.gz
Make_win32.bat
Makefile
README
sio_unx.h
sio_win32.h
sio_ntap.c
sio_ntap.htm
sio_ntap_aix
sio_ntap_hpux_ia64
sio_ntap_hpux_parisc
sio_ntap_linux
sio_ntap_sol
sio_ntap_sol64
sio_ntap_win32.exe
A-42
STEP ACTION
7.
Normally you would start data collection for all of the protocols being served by the cluster;
however, for this exercise you use only NFSv3.
NOTE: Diagnostic privilege level commands are required to capture rand_read_req and
seq_read_req.
set diagnostic
cluster1::*> set diagnostic
Warning: These diagnostic commands are for use by NetApp personnel
only.
cluster1::*> statistics start -object nfsv3|volume|readahead -sample-id
sample_nfs1
Statistics collection is being started for Sample-id: sample_nfs1
A-43
STEP ACTION
9.
or save the results. It is recommended that you increase the Lines of scrollback in your
cluster1 PuTTY session to at least 2000.
This information will be used to complete questions later in this task.
HINT: Use the baseline analysis commands.
dashboard performance show -operations
dashboard performance show -instance
statistics show-periodic -node cluster1-01 -instance latency -iterations 4
statistics show-periodic -node cluster1-02 -instance latency -iterations 4
statistics show-periodic -object volume -instance vs2_vol01 -iterations 4
statistics show-periodic -object lif -instance vs2:vs2_cifs_nfs_lif1 iterations 4
statistics show -object volume -counter *latency -sample-id sample_nfs1
statistics show-periodic -object volume -instance vs2_vol01 -iterations 4
statistics show -object volume -counter *data -sample-id sample_nfs1
statistics show -object aggregate -counter user_reads|user_writes -sample-id
sample_nfs1
statistics show -object disk -counter *latency -sample-id sample_nfs1
statistics show -object port -counter *data -sample-id sample_nfs1
statistics show -object lif -counter *data -sample-id sample_nfs1
The following can be run from the cluster shell as a single command:
row 0; set diagnostic; statistics start -object
nfsv3|volume|aggregate|disk|port|lif|readahead -sample-id sample_nfs1;
statistics show-periodic -node cluster1-01 -instance node -iterations 4;
statistics show-periodic -node cluster1-02 -instance node -iterations 4;
dashboard performance show; dashboard performance show -operations; dashboard
performance show -instance; statistics show-periodic -instance latency iterations 4; statistics show-periodic -node cluster1-01 -instance latency iterations 4; statistics show-periodic -node cluster1-02 -instance latency iterations 4; statistics show-periodic -iterations 4; statistics show-periodic
-object volume -instance vs2_vol01 -iterations 4 ; statistics show-periodic object lif -instance vs2:vs2_cifs_nfs_lif1 -iterations 4; statistics show object volume -counter *latency -sample-id sample_nfs1; statistics showperiodic -object volume -instance vs2_vol01 -iterations 4; statistics show object volume -counter *data -sample-id sample_nfs1; statistics show -object
aggregate -counter user_reads|user_writes -sample-id sample_nfs1; statistics
show -object disk -counter *latency -sample-id sample_nfs1; statistics show object port -counter *data -sample-id sample_nfs1; statistics show -object lif
-counter *data -sample-id sample_nfs1; statistics stop -sample-id sample_nfs1
A-44
STEP ACTION
31.
did the throughput and I/Os increase when the number of threads increased? Yes, when all of
the other parameters stayed the same.
32.
which workload had the highest throughput in terms of KBps? 100% read workload, 0%
random, 32-KB block size, 20-MB file size, and four threads.
33.
which workload had the highest throughput in terms of IOPS? 50% read and 50% write
workload, 100% random, 4-KB block size, 300-MB file size, 32 threads.
34.
which workload showed the lowest latencies on the storage system? 100% read workload, 0%
random, 32-KB block size, 20-MB file size, and four threads.
35.
which workload showed the highest disk utilization on the storage system? 50% read and 50%
write workload, 100% random, 4-KB block size, 300-MB file size, 32 threads.
36.
which workload showed the highest CPU utilization on the storage system? 0% read workload,
0% random, 32-KB block size, 300-MB file size, run for one hundred seconds, four
threads.
A-45

STRSW Ilt Perfcdot Exerciseguide

Uploaded by

Copyright:

Available Formats

STRSW Ilt Perfcdot Exerciseguide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

STRSW Ilt Perfcdot Exerciseguide

Uploaded by

Copyright:

Available Formats

What is the purpose of this document?

What is the purpose of this document?

What are some of the key performance metrics discussed in this document?

What are some of the key performance metrics discussed in this document?

NETAPP UNIVERSITY

Performance Analysis on Clustered Data ONTAP

NetApp University - Do Not Distribute

U.S. GOVERNMENT RIGHTS

Performance Analysis on Clustered Data ONTAP: Welcome

NetApp University - Do Not Distribute

Performance Analysis on Clustered Data ONTAP: Welcome

NetApp University - Do Not Distribute

MODULE 1: HOW A NETAPP STORAGE SYSTEM WORKS

NetApp University - Do Not Distribute

MODULE 2: PERFORMANCE OVERVIEW

By the end of this exercise, you should be able to:

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

TASK 2: LOG IN TO THE EXERCISE ENVIRONMENT

Verify that the desktop of the remote machine is displayed.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

TASK 3: ADD A CLUSTER TO ONCOMMAND SYSTEM MANAGER

On the Windows Server desktop, open Windows Explorer.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Verify that the operation dialog box appears.

Select Notepad to open the hosts file in this tool.

Append the following information to the end of the file:

Save the file.

Close Windows Explorer.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Verify that System Manager starts.

Click Sign in.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

In the left pane of System Manager, expand cluster1.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

TASK 4: CONFIGURE SNMP PUBLIC COMMUNITY NAME

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Enter the community name: public

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Click OK to close the Edit SNMP Settings dialog box.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

Verify that public is now in the community names area.

Performance Analysis on Clustered Data ONTAP: Performance Overview

NetApp University - Do Not Distribute

TASK 5: IDENTIFY CLUSTERED DATA ONTAP COMPONENTS

Revert any LIFs that are not on their home ports.

Open a PuTTY session with cluster1.