Student-Run Usability Testing

Nancy J. Wahl
Middle Tennessee State University
Computer Science
Murfreesboro, TN 37132
[email protected]

Abstract
Usability testing is performed by software development companies to determine whether
their software systems are usable. Not only development companies but also end-user
companies run usability studies. Given the importance of usability testing in industry, it
seems equally important to teach students the basic elements of usability testing. Usability
testing can be incorporated into a capstone course.
Students can learn about user-centered design and what makes software usable by
running usability tests. Usability testing consists of three phases—preparing for the testing,
running the actual test, and analyzing test results. Each phase is described in detail in this
article, which also contains a description of how usability testing can be incorporated as an
assignment in a capstone course or a software engineering sequence.

1. Introduction

John Karat wrote, “We write software to meet system requirements, but we must consider
how we ensure system acceptance” [4]. Students write software to meet professors'
requirements and do not consider whether or not the software would be accepted for use by
anyone else. When students reach their junior/senior software engineering courses, they
think their programs are well-designed and easy to use. It is an enlightening and sometimes
humbling experience for students to watch other people try to use their software.
Incorporating usability testing into a software engineering sequence or capstone course is an
excellent way to help students focus on usability issues and user-centered design.
Students are not the only ones who become enamored with their code. Professional
developers can be too wrapped up in their programs to evaluate them well. IBM
incorporated human factors engineering into their development process forty years ago to
make their products more usable. Currently user-centered design (UCD) is a strategic focus
in design of hardware, software and services for IBM. IBM has twenty-five UCD labs
worldwide [9].
Microsoft’s usability staff celebrated their 10 th anniversary in 1999 [5]. Microsoft has 100
people involved in the usability process including usability engineers, usability specialists,
recruiters, schedulers, and support staff. Usability staff people are assigned to a product
development team to help the team understand user behavior and to measure success of
product design [5]. Microsoft product teams have a financial incentive to develop user-
centered products because each team pays part of the costs of customer support calls relating
to its product [5].
 Like IBM and Microsoft most large software development companies have their own
usability labs. Independent usability testing vendors provide services to small companies
that do not have labs and to large companies that want their products tested independently
against the competitions’ products [3].
Usability is so important an issue to user companies such as State Farm, Boeing, Fidelity
Investments, Eastman Kodak, and Northwestern Mutual Life Insurance Company that they
asked the National Institute of Standards and Technology to help in the development of a
standard reporting form for evaluating the usability of software [7]. As one can see, usability
testing is viewed as extremely important in industry. Thus, it is equally important to teach
this concept in the classroom.
Usability testing can aid in the goal of producing quality software that meets its
requirements by determining if the software is easy to learn and contains the functionality
the user wants. Participating in a usability test can help students learn the importance of
designing with the end-user in mind.
Computer science students need to understand what makes software usable. One way to
do this is to have students develop prototypes and run usability tests on them. This article
contains a description of usability testing run by students. The usability testing process
consists of three phases—preparing for the testing, the actual testing, and analyzing the
results. In the following sections the three phases are discussed, but first usability is defined
in the next section.

2. Usability

The ultimate goal in software development is that the user community accepts the
software. To accomplish the goal, the software must be easy to learn, easy to remember, and
allow users to accomplish their tasks in a reasonable amount of time. In other words, the
software must be usable. According to ISO 9241-11, usability is “the extent to which a
product can be used by specified users to achieve specific goals with effectiveness,
efficiency, and satisfaction in a specified context of use.”
ANSI/Human Factors Ergonomics Society and ISO have been working on standards of
usability [4]. Part of the ISO standards are based on the following seven principles of
usability:

• Self-descriptiveness
• Controllability
• Suitability for task
• Conformity with user expectations
• Error tolerance
• Suitability for individualization
• Suitability for learning

Usability principles would have us organize the user interface in a way that is natural to
the user, makes simple tasks easy to do, makes important tools visible, reduces the cost of
making mistakes by implementing the undo command, and maintains “consistency with
purpose” [2] to reduce user memory load. All command and textbox labels should be self-
descriptive.
3. An evaluation continuum
Usability testing can be part of a continuum of software evaluation techniques that starts
with walkthroughs and continues on to assessment of the final product by end-users.
Important information can be gained from walkthroughs of specification documents and
design models. Walkthroughs are used to verify the work done in several phases of the
software development life cycle.
Reviews by experts have also proved useful [8]. Expert reviews can be scheduled at
different times during the development process. An expert can be asked to perform a
heuristic evaluation, guideline review, cognitive walkthrough, or consistency inspection [8].
A danger in using an expert is that the expert may have difficulty knowing how typical or
first-time users will react.
Next in the evaluation continuum is usability testing. Usability testing and laboratories
appeared in the early 1980’s [8]. Companies found many benefits to usability testing
including shortened product development time. Studies at IBM indicated that changes made
to products due to usability testing saved the company 6.8 million dollars per year [1].
Usability testing is an important part of the software development life cycle in industry.
The last part of the evaluation continuum is acceptance testing and assessment during
active use. Acceptance testing is performed by the end-user and often is focused on
measurable goals [8]. Assessment during active use occurs informally every time someone
uses the product. Formal assessment during active use can include interviews and continuous
user performance data capture [8]. Such assessment data can be used to improve the next
version of the software.
The goal of most of this evaluation is to catch usability problems and/or errors in
implementation of requirements as early in the software development process as possible.
Problems found early in the software life cycle are easier and less costly to correct than
problems found during the operations phase. Usability testing is an important technique in
the evaluation continuum.

4. Testing laboratories

Usability testing can be performed in formal usability laboratories or in ordinary rooms.

General Electric Information Systems (GEIS) has formal usability laboratories at its
headquarters. A typical laboratory consists of two rooms connected by a one-way mirror. In
one room are a computer, two cameras (one directed at the test participant’s face and one at
the keyboard and desktop), the test participant, and the test administrator. In the second
room are one or more observers with a computer. A video mixer is used to mix the video
tapes [6]. Advantages of formal usability laboratories are that there are large quantities of
data to analyze and that observers are hidden from the participants. On the other hand, the
large quantity of data generated can be a disadvantage just because of the difficulty in
analyzing all of it.
Economy usability testing can be done with one room, one PC, one video camera, and
two chairs. Advantages of this type of testing are that you get fairly good data, results can be
determined quickly, and testing can be done at the users’ site. Disadvantages include getting
less data, being observed may change the participant’s behavior, and more subtle problems
with the software may be overlooked [6].
Student-run usability tests can be run in a university computer lab with no other special
equipment than the computers. If time and resources permit, students from a mass
 communications department could videotape the testing, but this would add considerably to
the complexity of the testing process.

5. Usability testing

Usability testing is performed to gather data about a product under development in order
to evaluate how usable the product is and if it contains the required functionality. One
purpose of usability testing is to collect quantitative data by measuring time to learn, speed
of performance, and rate of human error. Another purpose of usability testing is to gather
qualitative date relating to user satisfaction with the product under test. Qualitative data
include comments made by the test participants, answers to questions on questionnaires, and
responses to exit interviews.
Usability testing has been shown to reduce development time, decrease the number of
bugs, and produce a more usable product thereby increasing sales [1]. In turn a more usable
product should generate fewer help desk calls than a less usable product.
Usability testing should be done early and often. For instance, usability specifications can
be incorporated into the software specifications. In the early stages of development usability
testing can be performed on paper prototypes of the user interface. Later, usability testing
involves users and the system prototype.
Given the importance of usability testing in industry, it seems important to teach students
the basic elements of usability testing. Students are accustomed to writing software that
meets teacher expectations. They could benefit from watching other not-so-expert users
attempt to perform tasks on code they have developed.

6. Student-run usability testing

The following is a description of an assignment on usability testing that was given in the
spring of 1999. Students were assigned a project and divided into groups of three based
largely on their personal choices. The project was to develop software to automate library
functions. The software was to allow the user to perform searches based on title, author, and
subject; to check the availability of specified books; to check books in and out; and to update
database files on books and borrowers.
Students started the library project by documenting the tasks performed by the software
and identifying who the users of the software would be. Next, each team developed a paper
prototype of the user interface. The prototypes were evaluated by the professor, then each
team evaluated another team's prototype. Based on professor's and students' evaluations, the
paper prototypes were reworked.
Each team developed a functional prototype of their project that was evaluated by other
classmates. This evaluation consisted of watching a team demonstrate the product and was
largely geared toward the user interface. After this evaluation the prototypes were
redesigned and readied for user testing.
There are three phases in the process of usability testing—preparing for the test, running
the actual tests, and analyzing results. The three phases are explained in the following
sections.
7. Preparing for usability testing

This section contains a description of the steps the student should take to prepare for
usability testing. These steps represent the first assignment for the students. (Note: this
approach to usability testing is largely based on an article by Russell Branaghan that appears
on his web site [1].)

Step One. Identify the purpose of the test.

Students were asked to identify why they were running the usability test. Reasons for
running a test include any of the following: the software is new, significant changes have
been made in functionality, or the user interface has been upgraded.

Step Two. Define the user population.

A realistic profile of the intended end-users is developed. The prototype the students
developed was a library system for a university. This project was chosen so that it would be
easy for the teams to identify representative end-users for their usability tests. It is
interesting to note that a member of one of the teams actually went to the university’s
administration to get statistics on our student population so that he could give valid figures
for some of the characteristics in the list of personal characteristics.
Given the library project, the teams developed three user profiles based on the following
characteristics:

• Age
• Gender
• Education
• Attitude toward technology
• Computer literacy
• Use of similar systems
• Familiarity with the operating system
• Native language

They developed a profile for a librarian, library student worker, and borrower. After they
had determined the user profiles, they were asked to develop a questionnaire to gather this
personal data from the actual users in their tests.

Step Three. Specify problem statements.

The problem statements identify the questions that will be answered by the usability test.
The questions relate to various functions of the software under test. Sample questions
relating to the library software are as follows:

• Can users successfully search for all books by a certain author?

• Can users determine if a given book is checked out?
• Can users search for all books on a given subject?

Given the time constraints, each team only had to specify three problem statements. The
usability testing of one participant was to take fifteen minutes or less.
Step Four. Identify user tasks.
The tasks that test participants will be asked to perform are identified. The tasks are
designed to answer the questions posed in Step Three. Branaghan suggests that a task table
be constructed that consists of the following for each task: a task description, conditions that
must exist before the task can be started, what constitutes successful completion of the task,
and maximum allowable time to complete the task [1].

Step Five. Specify the performance data to be collected.

The end-user’s performance will be measured to ascertain quantitative data. Students
were asked to collect the following data:

• Time to complete each task provided the maximum time of 15 minutes was not
taken.
• Number of incorrect commands selected and other user errors.
• Number of questions asked soliciting help.
• Whether or not a task was completed correctly.

Step Six. Exit questionnaire

The purpose of the exit questionnaire is to determine the user’s subjective satisfaction
with the program. Students designed a questionnaire that was to be administered to each test
participant after finishing the test session. The students were instructed to use a five point
Likert scale and to ask participants to rate their software on the following areas:

• Aesthetic appeal of the program’s user interface.

• Ease of use.
• Ease of learning.
• Consistency of terminology.
• Appropriateness of terminology.
• Do display layouts simplify the tasks?
• Is it easy to correct errors?
• Overall satisfaction with the software.

In addition to questions requiring numerical answers, open-ended questions were

included. Sample questions included the following:

• What specific suggestions would improve the program?

• What specifically did you like about the software?

Step Seven. Consent form

Each test participant was to be asked to sign a consent form. The form contained
statements similar to the following statements [8]:

• I have volunteered to participate in the test.

• I have been informed in advance of my responsibilities and how the test will
proceed.
• I have been able to ask questions.
• I am aware that I may withdraw from the test at any time.
 Each team devised their own particular consent form based on their reading of the
Branaghan article [1] and the Shneiderman textbook [8].

Step Eight. Roles for team members

Each team member must agree to play certain roles during the testing process. A variety
of activities takes place during the testing process and a team member must be responsible
for each activity. Someone must greet the test participant, explain the procedure, and have
the user sign the consent form and fill out the personal data questionnaire. Another team
member must give each user the list of tasks to perform, discuss the think-aloud technique of
verbalizing what the user is thinking as a task is performed, and monitor the actual test. A
third team member must write down everything the user says and does, have the user fill out
the exit questionnaire, and thank the user for participating.
Not every team followed the division of labor as specified above. The teams decided how
to cover all activities. One team had two team members monitoring the test and writing
down what the user said and did.
These eight steps must be completed before the actual testing begins. Thus the students’
assignment in the first phase of usability testing was to turn in the following:

• Purpose of the test

• Typical user profiles
• Three problem statements
• Three user tasks
• Performance data to be collected
• Personal data form
• Consent form
• Exit questionnaire

The teams turned in their assignments, which were evaluated and returned to them before
the actual test. Some groups had to revise forms and rethink what tasks the test participants
would perform. It was important to devise tasks that were not too simple or too hard.

8. The actual usability test

The assignment for the second phase of usability testing was to actually run the test. A
computer lab with thirty-two stations was reserved for 1½ hours for the usability testing.
Each team was assigned to a different corner of the lab. A table for filling out forms was
allocated for each team’s use.
Test participants were volunteers from another computer science class. They were given
the incentive of adding extra points to their total number of quiz points in the other class. It
was hoped that both the student test participants and the students administering the tests
would learn from the experience. Using computer science students to act as end-users was
realistic to a point. Computer science students would use the university library software, but
they would only represent one small part of the overall borrower profile. Computer science
students were used as end-users largely as an expedient.
Each team conducted the usability test on a minimum of three and maximum of six test
participants. Research explained in Branaghan’s article shows that three users can detect
approximately 65% of the usability bugs and that five users can detect 83% of the usability
bugs [1].
 Test participants were brought in one at a time by each of the teams. After one test
participant finished, a team member would have to reset all data files so that the same
conditions existed for each participant.

9. Analysis of data

The assignment for the third phase of usability testing was for the students to summarize
and tabulate the data collected during their usability tests. The goal of this assignment was to
determine how their software could be improved. The following items were included in the
assignment:

A.) For each of the three tasks

1.) Briefly describe each task
2.) Summarize the following:
a.) comments made by the user while performing a task
b.) performance data (see Step Five in Section 7)
c.) answers to any ad-hoc questions about the task

B.) Overall Observations

1.) Summarize the following:
a.) General comments made by the user about the software.
b.) Results of the personal data questionnaire (i.e., create a profile
of their actual test participants)
c.) Results of the exit questionnaire (Note: they tabulated answers
to the questions with numerical answers and listed responses to
open-ended questions.)
2.) Identify weaknesses of the software
3.) Identify strengths of the software
4.) Describe changes that the team thinks should be made based on the
results of the usability test.

10. Conclusions

Overall the students felt that they were well prepared to give the usability test. They had
all of their questionnaires and task scenarios ready prior to the test, and they each knew what
roles they were to play. When asked to evaluate the class at the end of the semester, more
than half the class mentioned the usability testing as a highlight of the class. Several students
wrote that they would have liked to have been a test participant.
It would have been better to include test participants that were not computer science
majors for two reasons. First, the computer science students were more familiar with
software and computers in general than the average university student, faculty or staff.
Second, some of the test participants were friends of the students giving the test. However, it
simplified finding test participants to use computer science students.
Given the importance of usability testing in the software development community and the
end-user community, it seems equally important to teach the basic elements of usability
testing to computer science majors. Students can learn about user-centered design and what
makes software usable by running usability tests.
11. References
[1] Branaghan, R., Testing, One—Two—Three: Fundamentals of Usability Testing,
http://www.branaghan.com/fun_utesting.htm.
[2] Constantine, L. and L. Lockwood, Software for Use, Addison Wesley, Reading MA, 1999, 579 pages.
[3] Dolan, W. and J. Dumas, “A Flexible Approach to Third Party Usability,” Communications of the ACM,
42, 5, May 1999, pp. 83-85.
[4] Karat, J., “Evolving the Scope of User-Centered Design,” Communications of the ACM, 40, 7,
July 1997, pp. 33-38.
[5] Muller, M. and M. Czerwinski, “Organizing Usability Work,” Communications of the ACM, 42, 5,
May 1999, pp. 87-90.
[6] Ohrt, D., Personal Communication, Murfreesboro, TN, January 27, 1999.
[7] Orenstein, D., Is Software too Hard to Use?, August 25, 1999,
http://www.cnn.com/TECH/computing/9908/25/easyuse.ent.idg/
[8] Shneiderman, B. Designing the User Interface, 3rd Edition, Addison-Wesley, Reading MA, 1998, 639 pages.
[9] Vredenburg, K., “Increasing Ease of Use,” Communications of the ACM, 42, 5, May 1999, pp. 67-71.