Measuring Usability: The Basics

Summary: Usability is not a luxury. If your website drives revenue, no

matter how big or small, you have a vested interest in turning motivated
users into buyers, and that requires understanding the needs and
expectations of those users. Fortunately, you don’t need a big budget
or a team of men in white coats wielding eye-tracking lasers to get a
handle on your website’s usability.

©2007 User Effect • w w w. u s e r e f f e c t . c o m • D r. Pe t e r J. Meyers

 Measuring Usability: The Basics
Usability is not a luxury. If your website drives revenue, no matter how big or small, you have a
vested interest in turning motivated users into buyers, and that requires understanding the needs and
expectations of those users. Fortunately, you don’t need a big budget or a team of men in white coats
wielding eye-tracking lasers to get a handle on your website’s usability.

Assessing your current usability means knowing where to look and creating measurable benchmarks
and goals. This article will help you assess your current usability, quickly and easily, as well as give you
some basic tools for improving your user experience and, with it, your bottom line. I’m going to break
it down into three major areas: (1) Web analytics, (2) Conversion rates, and (3) Split (A/B) testing.
...............................................................................................
(1) Web Analytics
If you have a commercial website of any kind, you’re already collecting data
in the form of traffic logs. Odds are good that you also have an analytics
package to analyze those logs, either through your host or a third party,
like Google Analytics. If you don’t, talk to your hosting company or see the
Resources section at the end of this paper.

You can get quite a few clues about usability from your existing traffic logs
and web analytics. Although analytics have been around just about as long
as websites, they’ve matured a lot in the last few years, and we’re still just
beginning to tap into their value.

It’s All Relative

Whenever you’re taking a fresh look at your web analytics, it’s important to keep in mind that there
are seldom “right” answers. Getting caught up in absolutes and what any given number should be is a
good way to make yourself crazy. Websites vary wildly, and you need to have a good understanding of
your own site’s baselines. For any given metric, focus on improvement and gradually sorting out the
story behind the number.

So, what are some of the numbers that matter? Following are a few of the metrics that speak to
usability issues.

Metric: Pages per Visit

One of your first clues to your website’s usability is whether or not your visitors stay on your site long
enough to see what you have to offer. Web surfers are notoriously impatient, and this is probably the
biggest battle of website usability. Tracking the average number of pages per visit is a good starting
point to understanding whether or not your visitors are sticking around. Again, don’t focus too much
on what a “good” number is. If you have a blog where most of the visitors hit the home page, getting
them to 2-3 pages/visit could be fantastic. If you’ve got an e-commerce site where the product
ordering page is 5 layers deep, that same 2-3 page/visit average is probably a bad sign.

Metric: Time on Site

Similarly, the average time each visitor spends on your site is a decent indicator of how positive that
visitor’s experience is. Practically speaking, the more time people spend on your site the better,
especially if you’re trying to sell a product directly.

1
©2007 User Effect • w w w. u s e r e f f e c t . c o m • D r. P e t e r J . M e y e r s
 Metric: Bounce Rate
Your “bounce rate” is an indicator of how many people leave immediately after hitting your site (or
a particular page on your site). It should come as no surprise that you want to drive the bounce rate
down. It’s important to note that many factors can conspire to inflate the bounce rate. If a lot of
people hit your site and immediately leave, you may have general usability issues. You may also have
marketing campaigns or search engine results that violate user expectations. In other words, it’s
important that the expectations that you create in an ad are matched when a visitor clicks through, or
you can expect a high bounce rate.

Tool: Exit Pages

This is less of a metric and more of a tool for finding your trouble spots, but most analytics packages
will tell you where your visitors are exiting. Specifically, look for points in your process that seem
like unnatural exit points. It’s not uncommon for people to exit at major entry points (like your home
page), but if a large number of users are dropping out at the second page of your shopping cart, for
example, you probably have a usability problem. Finding and fixing these problem points can have a
major impact on your bottom line.

Don’t Panic
If you haven’t looked at your analytics for a while or haven’t looked at any of these particular numbers
before, don’t panic. Website visitors are a fickle bunch, and your bounce rate and other numbers may
make your heart skip a beat or two the first time you see them. Focus on the big picture and work
towards improvement. Website usability is, ideally, an ongoing process of optimization, and today’s
numbers are just a benchmark for figuring out where you’d like to be tomorrow.
...............................................................................................
(2) Conversion Rates
Let’s pretend you’ve dug through your analytics and are getting a handle on
how usable your site currently is. It’s a good starting point, and will help you
establish some benchmarks, but the next step is to start thinking about your
goals. What does usability success look like? In the world of website usability,
the gold standard has become something we call the Conversion Rate (CR).

Simply put, the Conversion Rate is just a measure of how many visitors convert
into buyers, where “buyers” can be loosely defined to mean people who
reach any target goal, including making a purchase, filling out a contact form,
downloading a document, etc. If you’re operating a commercial website, you
have an action you want people to take, and the role of strategic usability is to
make it as easy as possible for them to achieve that goal.

The Long Version

Of course, that’s just the quick and dirty definition. Technically, conversion rate is the ratio of
conversions to visitors, represented by a percentage. Let’s say you’re tracking actual online buyers of
a product. If, in a given month, your website had 5,000 visitors and 125 of them purchased a product,
your conversion rate for that month would be 2.5% (125/5000).

There are a couple of details that are worth mentioning. When pulling the visitor side of the equation
from analytics, I generally prefer Unique Visitors. It may inflate your CR slightly, but for tracking over
time, it’s a much less noisy number than overall visitors. For conversions, you’ll probably have to look
outside of your analytics, although many packages are starting to support tracking conversions/goals.

2
©2007 User Effect • w w w. u s e r e f f e c t . c o m • D r. P e t e r J . M e y e r s
 If you have a commercial website, you may already have a mechanism to track purchases. Bear in mind
that you should measure the total number of visitors who converted, not how many purchases they
made (i.e. one person buying 14 items is still just one conversion).

Tracking Changes in CR
Like any measurement, as soon as you start tracking CR, you’re probably going to get a bit carried
away. Let me try to head that off with some advice. First, as with just about any website metric, try to
track changes over longer time periods. Comparing daily numbers is an exercise in futility. Personally,
I like to track both weekly and monthly numbers (separately). Note that some analytics packages
compute unique visitors differently depending on the time period, so only compare like time periods
(months to months and weeks to weeks).

Just as importantly, let me share the mantra my experimental psychology professors drilled into me:
“correlation does not imply causation”. If you change something on your website, and CR goes up or
down, don’t assume it’s because of what you did. Just like overall traffic, many factors can effect
conversions, including weekends and holidays, consumer confidence, marketing campaigns, etc.

So, how do you determine if changes to your website actually do have an impact on conversions? For
that, you’re going to need to run an experiment, using something we like to call the A/B or “split” test.
...............................................................................................
(3) Split (A/B) Testing
Most of the time, people make changes to their websites and never look
back. They take an educated guess and hope for the best. Ideally, though,
you’d like to know whether a change was for the better or worse, in a real,
quantifiable sense. Enter the split test, sometimes called the “A/B” test.
The name is pretty self-explanatory; you have two versions of something
(copy, a graphic, a layout element, etc.) and want to split them to your
visitors and find out whether A or B performs better.

Why Split Test?

So, why not just run version A for a while, then run version B for a while, and see what works better
(sometimes called “sequential” testing)? The main reason is that something may have happened during
those two different time periods to make visitors behave differently. People may go on vacation, the
market may slump, you may have a major marketing campaign, the Fed might raise rates, Starbucks
might introduce a size bigger than “Venti”, etc. In the end, you want to have some confidence that any
difference you measure between groups A and B is because of what you changed, not because of some
external factors muddying up your results.

What Should You Test?

Split-testing originated in the advertising world and is often used on the web to test subtle differences
in advertisements and landing pages (the pages people arrive on when they click on an ad). You might
test changes in copy, layout, colors, button shapes, or just about any page element that potentially has
an impact on your visitors.

As for whether you should test big changes or small ones, there seem to be two camps on this subject.
One side, which I tend towards, believes in starting small. That way, if you do find that B is better
than A (or vise-versa) you’ll have some idea of why. In the long-term, that will allow you to make more
educated guesses about future changes. The other side says that you should go for big changes, as
you’ll get more bang for your buck (and your time). I can’t completely deny the logic of that; it often

3
©2007 User Effect • w w w. u s e r e f f e c t . c o m • D r. P e t e r J . M e y e r s
 comes down to what you need to accomplish and how long you have to accomplish it. A stable site that
does pretty well may want to stick to an evolutionary process of incremental changes. A site that’s
having trouble or needs a major overhaul may require more radical methods.

Finding Significance
A split-test is essentially an experiment; you present two options to two groups of people and measure
how those groups react. Let’s say you observe the following outcome: Version A converts at 1.5% and
version B converts at 2.5%. That sounds good, but are the two groups really different? Simply put,
maybe not: most often, this is because the groups aren’t big enough for the difference to be reliable.
In statistics, we call this “significance”. A significant difference is one that, to the best of our ability
to measure it, represents a real difference and not just poor measurement or individual differences
between the people who happen to be in the groups.

Understanding all of the mathematics of significance is well beyond the scope of this article, but your
best defense is to collect plenty of data. Make sure that your groups are: (1) split roughly 50/50 (in a
straight A/B test), and (2) that you measure a decent chunk of conversions. The reality is that it may
take thousands of visitors for you to get a reliable result, depending on the size of the difference that
you observe. For a clear answer, you’ll need to consult an expert or at least use a split-test calculator
(see the Resources section below).

Some Advice Before You Split

From my days as an experimental psychologist, I know how tempting it can be to peek at the data
every chance you get. Don’t. Bias is a powerful force, and it’s best to let the test run its course and
not interpret the data until you’re done. Too often, people start to see the results trending the way
they’d like, decide they’ve got enough information, and cut the experiment off early. Also, try not to
make changes during the test, no matter how small they seem. If you make a mistake in the testing
procedure (that would bias the results), start over, as painful as that may be.
...............................................................................................
Beyond The Basics
Hearing the voices of your users across the internet isn’t easy, but hopefully I’ve given you the
incentive and a few of the tools to start listening better. Pay attention to your web analytics, define
your goals, use conversion rates to measure those goals, and start testing changes that may have an
impact on potential buyers. Good usability is a process and a habit; you may make a few mistakes along
the way, but if you consistently make an effort to improve your user experience, you’ll eventually be
rewarded.

Dr. Peter J. Meyers is a website usability specialist and President of User Effect (www.usereffect.
com). “Dr. Pete” works with clients with commercial websites to help them exceed user expectations
and attract and motivate potential buyers. He can be reached at [email protected].
...............................................................................................
Additional Resources
1. Google Analytics
http://www.google.com/analytics

2. Split Test Calculator

http://www.usereffect.com/split-test-calculator

4
©2007 User Effect • w w w. u s e r e f f e c t . c o m • D r. P e t e r J . M e y e r s