Bottlenecks

Pinpointing and Exploiting
Specific Performance Bottlenecks

Revised for:
Software Test & Performance Conference
November 2005 New York, NY
First Presented for:

IBM Rational Users Conference, 2004
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc.
www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 1
© 2005 PerfTestPlus All rights reserved.
Agenda
Introduction
Analyzing Results to Identify Bottlenecks
What the Development Team Needs to Know
Identify Tier of Detected Issue
Identify Component of Detect Issue
Develop Test To Exploit Issue
Available Tools
Examples (Time Permitting)
Want More Information?
Summary/Questions

Introduction
This presentation is adapted from User Experience,
not Metrics: Parts 6, 8, 9 and10 and Beyond
Performance Testing: Parts 6, 7, 8, 9, 10 located at
http://www-106.ibm.com/developerworks/rational/library/
(RDW) and http://www.perftestplus.com/.

Introduction
One part of the system is always slowest (the bottleneck). Until it is
remedied, no other tuning will actually improve the overall
performance of the application along that path. Before that
bottleneck can be tuned, it must first be conclusively identified.
Once a bottleneck is identified, resolution can be reached more
quickly if your existing tests are modified to eliminate distraction
from ancillary issues. Pinpointing exactly where the bottleneck is
an art all its own.
After determining where the bottleneck is architecturally, a new test
will likely be needed to exploit it in order to help the development
team with tuning. These bottleneck exploiting tests needn’t bear
any resemblance to real user activity but rather focus on the
bottleneck alone. In fact, these tests often don’t even interact
with the system in ways users could and may include direct
interaction with back-end tiers.
Intro – Scott’s Rules of Bottlenecks
A bottleneck is a slowdown, not a stoppage. A stoppage is a failure.
Bottlenecks don’t only exist under load.
The symptoms of the bottleneck are (virtually) never observed at the actual
location of the bottleneck.
The critical bottleneck is the one bottleneck along a particular user path the
removal of which will improve both performance and the ability to find
other bottlenecks.
If you have multiple paths through a system and think there’s a bottleneck,
you should isolate each path and evaluate it separately.

Intro – Scott’s Rules of Bottlenecks
The bottleneck is more likely to be found in the hardware than
in the network, but the network is easier to check.
Unless other activities and/or users are affected by the

observed slowness or its cause, it’s not a bottleneck but a
slow spot.
When reporting bottleneck suspects, don’t assume you know
the cause, just report the symptoms.

Analyzing Results to ID Bottlenecks
Examine Response vs. Time Charts/Tables
Help identify bottleneck suspects
Parts 6, 7, 8, and 9 of the “User Experience, Not Metrics” series

Study Scatter Charts
Any pattern that shows more than one dot outside of your predefined
acceptable performance levels is a potential bottleneck
Part 6 of the “Beyond Performance Testing” series

Rely on Personal Observation
Listen to Third Party Comments
Confirm Suspects
Reproduce Results
Exactly
Manually
With Similar Tests
With Minimalist Tests
With Not-So-Similar Tests
Report Suspects
Verbally
Visually
Via Demonstration

What the Dev Team Needs to Know
Which related activities produce the same symptoms?
Which other activities are affected by the bottleneck?
What were the load characteristics of the test yielding
the symptoms?
What data did you use to create the symptoms?
What’s the configuration of the environment you’re
testing?
Other metrics the developers wanted you to collect.

Logical
Architecture
Physical Architecture

Physical Architecture with Logical Overlay

Design Tests to Determine Tier
Ask “What if…? Questions.
Ask Developers to Speculate
Evaluate Commands with Slow Responses
Think in Terms of Distinguishing Failures, Slow Spots
and Bottlenecks
Visualize and Prioritize
Modify Existing Tests
Create New Tests
Use Same Tool
Use Different Tool
Use Test Harnesses

Speak Intelligently with the Development Team
Capture Metrics by Tier
Resource Utilization
Response Times
Others Identified by Developers
Interpret Metrics
Look for the Obvious
Consult Development Team
Change Tests to Prove (or Disprove) Theories

Identify Component of Detected Issue
Once Tier is Identified…
Further Narrowing may be Required
Same Principles as Identifying Tier
Speak Intelligently with the Development Team
Capture Metrics by Component
Resource Utilization
Response Times
Others Identified by Developers
Interpret Metrics
Look for the Obvious
Consult Development Team
Change Tests to Prove (or Disprove) Theories

Lessons from Hydrodynamics
Pools/Queues
Flows/Threads/Processes
Pipe Size/Throughput

Exploitation Methods
Bounds Conditions
Breakpoints
Resource Constraints
Hand off to Development Team
Following Development Team’s Lead (Case Studies, BPT 10)
Different Testing Styles
Black-Box
Grey-Box
White-Box
Knowing When to put the Load Generation Tool Away
Available Tools
Common
LoadRunner, Silk Performer, Performance Tester,
OpenSTA, Visual Studio Test System, eLoad
Purify, Quantify, Performassure
J-Meter, Perfmon, Perfmeter, Top
WebTrends, WebLogic, Tivoli
Other Performance Test/Monitoring Tools
Load Generation
Performance Monitoring
Performance Analysis
OS/System Specific Tools
Application Specific Tools
Available Tools
Other Analysis Tools
Spreadsheets
Statistical Calculators
Mathematical Graphing
Graphical Presentation
Most Important Tools
Your Brain
Your Development Team

Examples

Want More Information?
Information adapted from User Experience, not Metrics: Parts 6, 8,
9 and10 and Beyond Performance Testing: Parts 6, 7, 8, 9, 10
located at http://www-106.ibm.com/developerworks/rational/library/
(RDW) and http://www.PerfTestPlus.com
Ask me directly on the Performance and Load Testing forums on
RDW (http://www-106.ibm.com/developerworks/forums/dw_rforums.jsp) or
http://QAForums.com (Huge QA Forum)
Good sources for additional information about Performance
Testing:
http://www.PerfTestPlus.com (Methodology, Templates, Articles,
Presentations)
http://www.loadtester.com (Good articles and links)
http://www.keynote.com/resources/resource_library.html (Good articles
and statistics)
Graphical Presentation of Information – Edward Tufte, PhD.
http://www.edwardtufte.com (Books and seminars)

Summary
Report Symptoms, not Solutions/Reasons
Verify Observations
Analyze Results Collaboratively
Don’t Over-depend on your Initial Tests
Determine Related and Unrelated Activities
Analyze, Analyze, Analyze (Collaboratively)
Know When to Hand-off to the Development Team
Yours is a Support Role
Document Conclusions
Document Recommendations

Questions

Contact Info
Scott Barber
Chief Technology Officer
PerfTestPlus, Inc
E-mail: Web Site:

[email protected] www.PerfTestPlus.com


Bottlenecks

Uploaded by

Copyright:

Available Formats

Bottlenecks

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bottlenecks

Uploaded by

Copyright:

Available Formats

Pinpointing and Exploiting

Specific Performance Bottlenecks

First Presented for:

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 2

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 3

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 5

Unless other activities and/or users are affected by the

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 6

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 7

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 8

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 9

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 10

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 11

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 12

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 13

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 14

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 15

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 16

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 19

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 20

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 21

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 22

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 23

E-mail: Web Site:

www.PerfTestPlus.com Pinpointing and Exploiting Specific Bottlenecks Page 24

You might also like