Usability Goals Setting Tool

Anirudha Joshi 1
1
IIT Bombay
Mumbai, India
[email protected], +91-9820345569

Abstract. Usability goals are often the basis on which usability evaluations are
done. Multi-disciplinary software development teams, interaction designers,
software engineers and business analysts among them, need to agree on a
common set of goals before they design and evaluate interactive products.
However, teams often struggle to set clear, measurable usability goals and all
members may not be on the same page even late in the project. This paper
proposes Usability Goal setting Tool (UGT), a tool meant to help design teams
set, prioritise and evaluate usability goals. UGT helps the designer to create
profiles for the product and its users. Based on these inputs, UGT provides
examples and guidelines to prioritize the usability goals. UGT also suggests
evaluation ideas for each goal. Prototypes of UGT were improved through
formative evaluations by using it for 15 interaction design projects. Summative
evaluations were done by applying UGT on 49 industry projects. Participants
found UGT to be useful and systematic and said they would use it in their next
project.

Keywords: Usability goals, design tools

1 Introduction

Setting goals is an important step early in the design process. Setting goals before
design gives the team a target to achieve. Goals help guide the design process, make
the design activity tangible and help evaluate the designs. In interaction design, often
multi-disciplinary teams are involved, so setting goals early and getting an agreement
from all stakeholders is all the more important.
In this paper, I differentiate between user goals (“enjoy a peaceful vacation” or
“ensure a secure future for children”), business goals (“capture the youth market”, or
“save distribution costs”) and product goals (“the website will support planning a
vacation end-to-end” or “the electronic voting machine should enable all voters to
vote without help”). Product goals set out benchmarks against which the design will
be evaluated. This paper is about product goals. Further, product goals may be about
functionality, costs, reliability, time as well as about usability and user experience.
This paper is about the subset of product goals related to the usability and user
experience of the product.
In my interactions with software development teams, I have observed that
designers are often unclear about the usability goals of the product they are designing.
In some cases designers are clear, but other stakeholders are not. Several teams need
help in setting goals. This happens not only in new, unfamiliar projects, but also in
cases where goals are being set retrospectively.
The second section of this paper discusses prior work related to usability, user
experience and goals. The third section introduces the initial design of the UGT. A
low-fidelity prototype of the UGT was developed and evaluated in 15 think-aloud
sessions. The fourth section describes the formative evaluations and the fifth section
describes UGT in its current form. After the UGT had stabilized, two summative
evaluations were done with help of 49 industry projects. The sixth section describes
these, followed by the last section on conclusions and future work.

2 Related Work

There is so much related work about goals, usability and user experience that any
‘one-page’ review is bound to be incomplete. Here I mention only the work that has
directly influenced UGT.
Design for a ‘need’ has been a part of traditional industrial design thinking. Charles
Eames reportedly said, “Design is a plan for arranging elements in such a way as best
to accomplish a particular purpose” [5]. This emphasis on goals probably dates back
to the days of Bauhaus school for design (1920s), and certainly from the days of the
Ulm school (1950s). Archer [1] explains that since design is necessarily associated
with change, identifying goals means “defining the needs and pressures which
constitute the driving force for change”. The first step is to determine the goals of the
design effort together with “the essential criteria by which a ‘good’ solution will be
distinguished from a ‘not so good’ solution”.
It may not be possible to meet all goals and it is necessary to prioritize. Archer [1]
talks about ‘rank ordering’ sub-problems as a method of prioritizing goals and
resolving conflicts. More recently, Cross [4] talks about an objective tree method –
organizing objectives into a hierarchy of higher and lower level objectives.
The closely related fields of usability, interaction design and information
architecture also emphasise the importance of goals. ISO 9241 defines usability as the
extent to which a product can be used by specified users to achieve specified goals
with effectiveness, efficiency and satisfaction in a specified context of use [7].
Shneiderman [13] and Nielsen [12] agree on five high-level usability goals of a
product – learnability, speed of use, error-free use, retention over time and subjective
satisfaction. Mayhew [10] categorizes usability goals as qualitative and quantitative,
and also as performance goals, preference goals and satisfaction goals. ISO 9126-1
describes usability in terms of understandability, learnability, operability and
attractiveness [6]. Cooper and Reimann [3] emphasise the importance of goals in
usability and interaction design. Bevan [2] summarises several other ways of
organising usability measures which could be looked upon as goals.
Relatively recent research interest in human emotions has broadened the traditional
focus of researchers from ‘usability’ to ‘user experience’. But approaches to user
experience goals are not universal. There is no single accepted definition of what user
experience is, and some researchers wonder whether it could be designed at all.
McCarthy and Wright [11] prefer to look at people’s experience in terms of ‘felt life’.
They believe that people actively construct their experiences and that each person’s
experience is unique, rich and difficult to communicate, let alone design. It is unclear
in this approach, what the goals of the design efforts that create technology-mediated
experiences ought to be. In a recent (2009) survey, Law et al [9] concluded that the
concept of user experience is dynamic, context-dependent and subjective. While there
may be no agreement on a complete list of user experience goals, most researchers
and practitioners are for a ‘goal-driven’ approach in design. While the first concern of
this UGT is to set usability goals, it also helps set a few common user experience
goals.

3 Initial Design of UGT

A goal is a commitment that a designer makes to his client or his management. The
designer approaches UGT when he / she is clear about the product brief and has
sufficient understanding about the domain, the problems, the context and users.
UGT is envisioned as a part recommender system and a part social-networking
tool. It helps break down high-level goals into more concrete, measurable goal
parameters by assigning each goal parameter a weightage. UGT has four steps:
initiate, set goals, review goals and set guidelines and share.

3.1 Initiate

UGT begins by helping the designer create a product and user profiles. These form
the input for UGT. The designer creates a product profile by answering the following:
• Name, version and industry domain of the product.
• Work practice domain: Life critical; business critical; goal-oriented; casual;
enabling technology; e-learning; information product; entertainment product.
• Expected cost to the user: Premium product; low-end product; not free but I don't
pay; free with another product / service; free to use.
• Platform(s): Desktop; web; mobile phone; IVR / call centre; TV; custom.
• How many types of users will use the product? Which ones?

The designer creates a user profile per user type, by answering the following:
• Age (pick a range)
• Tech-savvyness: (pick the lowest level targeted) Low (e.g. can't save a contact,
but can make a call); moderate-low (e.g. can save a contact on a phone);
moderate (e.g. checks mail every day); moderate-high (e.g. can install
applications); high (e.g. writes programs, loves Linux).
• Frequency of use: (pick a range) Once; once a year; once a month; once a week;
once a day; few times a day; continuously.
• Product complexity: Very complex; complex; moderate; simple; very simple.
• Nature of the market: Mass market; wide market; niche market; internal
application for trained staff; thick domain product for experts.
• Value addition of this product in user's life / Motivation to use the product: (pick
all that apply) no intrinsic motivation; I have many options; socialize; it
entertains me; it's my hobby; I have been using it; saves my time, informs me,
helps me do my job better; makes things accessible; makes / saves money for me;
I have no choice, I have to use it as a part of my job.

3.2 Set Goals

The designer then sets goals by assigning weightage to a list of goals. Depending on
the context of the project, some goals may be very important for the project, while
others might be irrelevant. UGT presents a list of potential goals and goal parameters.
In the first version of UGT, the suggested goals and goal parameters were the ones
shown in Table 1. These were derived from Shneiderman [13] and Nielsen [12],
supplemented by the authors’ experience and brainstorming with practitioners [8].
The goals were not meant to be comprehensive and are not meant to cover all aspects
of user experience. They were suggestive of the kind of goals that the designer could
consider. At all times the designer has the freedom to add goal parameters, edit or re-
word the suggested goal parameters, and to group goal parameters differently if they
preferred.

Table 1. Initial set of goals and goal parameters used for UGT.

Learnability Ease of use

Conceptual model clarity Minimal user task load
Language understandability Automation of routine tasks
Minimal learning time Error free use
Consistency with earlier version Good feedback
Visibility of choices and data Error tolerance
Consistency with other products Error recovery
Speed of use Retention
User control and freedom Retention / memorability
No memory and cognitive load Subjective satisfaction
Internal consistency Visceral appeal
Customization Behavioural appeal
Automation and shortcuts Reflective appeal

The designer sets goals by assigning one of the following weightages: Irrelevant
(0); Somewhat relevant (1); Usual relevance / hygiene factor (2); Important (3); Very
important (4); Extremely important / unique selling proposition (5). In the first
version of UGT, all goal parameters had the default relevance of 2. As UGT collected
data from projects, its later versions started giving recommendations for weightages
based experience with similar product and user profiles. Of course, the designer may
use the recommendations or choose to ignore them. These decisions are captured and
fed back to enrich UGT further.
3.3 Review Goals and Set Evaluation Guidelines

The list of goal parameters is re-presented to the user, this time sorted by their
weightages. Goal parameters with weightages that deviate substantially from the
recommended range are highlighted. The designer reviews the weightages and tweaks
them. This step was introduced because it was observed during the formative
evaluation that designers over-assigned weightages in the first pass, but preferred to
tone them down during a review. At this step, the designer is also asked to suggest
ideas for how the eventual design will be evaluated against each goal parameter.
These decisions are mined and fed back into UGT. The granularity of the goal
parameters is such that it is easy to define how the product will be evaluated against it
and yet generic enough to be applicable in a wide range of projects. Sometimes, this is
called “sea-level granularity” (as against fish-level or bird-level).

3.4 Share

Once the designer is satisfied with his goal parameters and evaluation guidelines, he
can share this data with others in his team. They give their inputs and after
negotiations, the goals are finalised. This step of UGT has been envisaged, but not
prototyped or evaluated at this time.

4 Formative Evaluation

A paper prototype of UGT was developed and is available at [Error! Reference

source not found.]. Designers were invited to use the prototype to define the product
and user profiles and set and review goals for a project that they had recently
completed, while thinking aloud. Designers were given the freedom to add or edit
choices at each step. Once the designers had finalized the goal parameters and
weightages, they were asked to describe a usability test that they will use to evaluate
their design against the important (3+) goal parameters. Then, the designers were
asked to comment about UGT and probed on whether UGT expressed their ideas
sufficiently and whether the step was useful.
A total of 15 designers participated in the formative evaluation of UGT. Of these, 4
were industry projects by professional designers and 11 were masters thesis projects
by interaction design students. Designers suggested many changes. UGT was
continuously modified during and after each session to incorporate these changes.
Goal parameters were added, re-worded, split, merged or regrouped to fit the
contexts. Yet, goal parameters were kept general enough to apply to a wide range of
products and expressive enough to suit the needs of the individual products.
The high-level goals had two important changes from Table 1. Retention /
memorability did not turn out to be important enough for most projects for it to be an
independent high-level goal. So it was merged as a goal parameter under the high-
level goal of learnability. A new high-level goal was introduced – “ease of
communication”. This emerged to be important in cases of the information and
communication oriented products such as web sites and learning tools. After the
formative evaluation, the total number of goal parameters went up from 20 to 30.
Most changes were discovered in the early projects. No major changes happened in
the last two projects and only two new changes happened in the last four projects.
Table 2 lists the modified goal parameters that emerged from the formative
evaluation.
The formative evaluation also generated qualitative feedback about UGT. Many
participants had a positive response to the activity. Participants agreed that
information required for product and user profiling is almost always available early.
One participant remarked “These were some of the things that I should have done
earlier. Had I done them, the product might have been better.” Several participants
also thought that the activity helped them prioritize goals, made many things clear and
gave many insights. It helped them consider goal parameters that they had not
considered earlier. A participant remarked “though I had set goals for these projects
earlier, it was not so systematic”.

Table 2. The set of goal parameters after the formative evaluation of UGT.
Learnability
1 Findability: options / data / information should be visible / easy to find
2 User should take less time to learn: (e.g. in < 10 minutes, in < 2 hours practice, in < 2nd attempt)
3 Users should be able to learn on their own
4 Product should be internally consistent
5 Product should be consistent with other products, older methods / past habits of users
6 Product should be consistent with earlier version
7 User should remember / retain critical, but infrequent tasks
Speed of use
8 User must be able to do the primary task / the most frequent tasks quickly, easily, at all times
9 User should be able to navigate quickly and easily
10 Product should not load user’s memory / product should not put cognitive load on user
11 Flexibility: User should control the sequence of tasks
12 User should be able to complete frequent / critical tasks in specific time / no. of steps / in less efforts
13 Product should be personalised for the user automatically
14 Product should be localised for specific market segments
15 User should be able to customise the product for himself
Ease of use
16 Interface should clearly communicate the conceptual model
17 Intuitiveness: User should be able to predict the next step / task
18 No entry barrier: user must be able to complete critical first tasks
19 Product should require no unnecessary tasks
20 Product should automate routine tasks / minimise user task load
21 Product should be always on, always accessible
Ease of Communication
22 Information architecture: Information should be well aggregated, categorised, presented
23 Communication should be clear / user should easily understand text, visuals
Error-free use
24 Product should give good feedback / display its current status
25 Product should not induce errors
26 Product should tolerate user’s errors / forgiving interface / should prevent errors
27 Product should help user recover from errors / help users troubleshoot problems
Subjective Satisfaction
28 User should feel in control of the product / behavioural appeal
29 User should feel emotionally engaged with product / brand / product should be fun / reflective appeal
30 User should find the product aesthetically appealing / product should have a visceral appeal
 Participants also liked the activity of setting evaluation guidelines for each goal
early. It made them think through the goal carefully at the initial stage and gave them
better control over the project. None of the participants had thought of setting
evaluation guidelines in this way. One participant was undecided about if he could
share the goals with non-designers as he felt it required a lot of background about
HCI. On the other hand, another participant remarked that “this will be helpful to
bring the team on the same page”. Another participant thought she could use UGT to
pitch for new projects, as it makes relevant interaction design activities tangible and
communicates “why we are charging so much”.

5 Modified UGT

After the formative evaluation, product profiles, user profiles and goal parameters of
the 15 projects were manually inspected to identify patterns that predict the weightage
of goal parameters. Though not all, many goal parameters were found to be somewhat
related to the product and user profiles. A detailed analysis of each goal parameter is
not presented in this paper due to lack of space. However, qualitative
recommendations for assigning weightage were developed for each goal parameter. A
subset of 3 such recommendations is presented in Table 3. The entire set of
recommendations for the 30 goal parameters is available at [Error! Reference source
not found.].

Table 3. Recommended weightages to goal parameters arising from formative evaluation.

Goal Parameters 0 1 2 (usual 3 4 (very 5

(irrelevant) (somewhat relevance) (important) important) (extremely
relevant) important /
USP)
5 Products should
be consistent Products for continuous use,
with other Games for if there is a high motivation Products targeted to higher age groups (50+),
products, older children for users to use the products products targeted to low-tech savvy users
methods / past (e.g. save money and time)
habits of users
12 User should be Products whose main
Continuous
able to complete promise is that they save
Most other products than of the type listed on use products
frequent / time, complex products with
the right – it tends to be on the lower side if with a few,
critical tasks in frequent usage, products
the user doesn’t care about saving time. repetitive
specific time / with repetitive micro-tasks
tasks
number of steps (e.g. clicking input device)
30 User should find
the products Products that is invisible to
Branding oriented
aesthetically the users, products with Business critical or goal
information website,
appealing / captive audience, where oriented products targeted to
socializing or entertainment
products should users have no alternative but wide or mass markets
products
have a visceral to use it
appeal

During the formative evaluation, participants were asked to describe a usability

evaluation that they will use to test the design against goal parameters. Brainstorming
was done to generate more evaluation ideas for each goal parameter. At least one
user-based test idea and one review-based evaluation idea was generated for each goal
parameter. Table 4 lists 4 of these ideas. The entire set is available at [Error!
Reference source not found.].

Table 4. Ideas to evaluate the design against goal parameters derived from an analysis of the
evaluations suggested for the 15 projects.

5 Product should be consistent with other products, past habits

Ask users to perform a set of tasks and probe for inconsistencies with what people do in real world or older habits.
During a review, look for design elements in the product that might be inconsistent with users’ current habits. These
problems are hard to spot in a review unless the reviewers are very familiar with the users’ current habits.
12 Complete frequent / critical tasks in specific time / no. of steps
In a performance test, measure time of completion of benchmark tasks.
In a review, look for number of steps involved, and whether each step is taking the user closer to achieving his goal.
For detailed analysis, use GOMS.
17 Intuitiveness: User should be able to predict the next step / task
During a think-aloud test, probe users for their expectation of the next task / step. Look for confusions,
misunderstandings.
In a review, look for steps that are not intuitive or are difficult to predict.
22 Information architecture: Well aggregated, categorised, presented
During a think-aloud test, probe to see if users are lost or if users look for data in the ‘wrong’ place.
During a performance test, find out if users can locate data / options without help. Count errors induced due to
information architecture problems.
In a review, look for potential information architecture problems.

6 Summative Evaluations of UGT

Two summative evaluations were done to evaluate UGT. The first was during a nine-
day professional course on interaction design was conducted by the author. It was
attended by 35 participants from mixed backgrounds such as graphic design, product
design, web design, user interface design, e-learning, engineering, product
management and ergonomics. Most had no prior formal education and relatively less
experience in HCI. Among other concepts and techniques, participants were shown
the modified UGT during the course. Each participant was then asked to set goals for
an industrial project that they had worked on. After the goal-setting exercise,
participants were asked to fill out a short post-test questionnaire. Participants were
also asked to suggest more goal parameters relevant to their projects.
A second, similar summative evaluation was done with 15 participants who had
more experience and formal background in HCI. These participants had an overall
average experience of 7 years and interaction design related experience of 3.5 years.
Only the goal-setting exercise and the post-test questionnaire were done with these
participants. Table 5 summarizes responses to the two post-test questionnaires.
The D values of Kolmogorov-Smirnov test for all questions for inexperienced
participants and questions 1, 2, 3, 5, 7, 8 for experienced participants are significant
beyond α = .01. While the mean ratings to all questions is positive, the inexperienced
group consistently gave more positive mean ratings. We can conclude that both
groups agreed that the exercise was useful (Q1), that it helped them understand the
context of the project better (Q2) and that the exercise made them think about goals
that they had not considered earlier (Q3). The inexperienced group wished that they
had done the goal-setting exercise during their projects (Q4) and it would have led to
a better user experience (Q6), but the experienced group was inconclusive on these
questions. Both groups believed that UGT could have helped them do their job better
(Q5), wanted to use the technique in their next project (Q7) and felt that they could
involve their colleagues in the exercise (Q8).

Table 5. Responses by 34 relatively inexperienced participants (lighter rows) and 15 more

experienced participants (darker rows).

Strongly Can't Strongly

disagree say agree
(-2) (-1) (0) (1) (2) Average n D α<
0 0 2 3 29 1.79 34 .65 .01
1. This was a useful exercise.
0 0 1 4 10 1.60 15 .53 .01

2. This exercise helped me understand the 1 0 1 5 27 1.68 34 .59 .01

context of my project better. 0 0 0 8 7 1.47 15 .60 .01
3. This exercise made me think about 1 0 2 10 21 1.47 34 .51 .01
relevant user experience goals that I
had not considered earlier. 1 0 1 5 8 1.27 15 .47 .01

4. I wish I had done this goal-setting 0 0 0 8 25 1.76 33 .60 .01

exercise while I was doing my project. 0 2 6 2 5 0.67 15 .27 --
5. Had I done this exercise during the 0 1 2 6 24 1.61 33 .52 .01
project, it would have helped me do my
job better. 0 0 2 4 9 1.47 15 .47 .01
6. Had I done this goal-setting exercise 0 0 4 5 24 1.61 34 .52 .01
during the project, it could have led to
better user experience. 0 2 3 4 6 0.93 15 .27 --

7. I will do this exercise when I am doing 0 0 0 10 24 1.71 34 .60 .01

my next project. 0 0 2 5 8 1.40 15 .47 .01
8. I think it is possible to involve my 0 0 4 9 21 1.50 34 .48 .01
colleagues / boss / clients in this
exercise. 0 1 1 8 5 1.13 15 .47 .01

The overall reliability of the weightages set for goal parameters by the 34
participants in the first summative evaluation using Cronbach's alpha was 0.796. Each
of the 30 goal parameters was deleted by turns and Cronbach alpha was calculated.
The resulting alpha did not vary much (0.777 to 0.806). Thus we can conclude that all
30 goal parameters are internally consistent and therefore essential to measure the
same construct.

7 Conclusions and Future Work

UGT provides goal setting guidance based on inputs from product and user profiles in
terms of a weightage range for goal parameters that design teams can use to prioritise
goals. While the 30 goal parameters presented in this paper are not claimed to be
comprehensive, they provide a substantial starting point for goal setting. Participants
found UGT useful and ‘more systematic’ than their current methods. It helped
designers prioritize goals and pushed them to consider goals that they had not
considered earlier. Setting evaluation guidelines helped them think through their goals
more thoroughly. While both experienced and inexperienced designers found the tool
valuable; the inexperienced found it more so.
UGT could have several applications in the design process. It can bring a multi-
disciplinary team on the same page as far as usability goals are concerned. Teams can
focus efforts on design and evaluation of high-weighted goal parameters to optimise
the design process. UGT can help estimate interaction design efforts – if there are too
many high-weighted goal parameters, efforts required for the project might be higher.
Another application is to use UGT to compute metrics [8]. Evaluations presented in
this paper were done on projects retrospectively. Prospective evaluations are planned.
At present UGT exists as a low-fidelity prototype. UGT will be developed as an
online tool and automated to the extent possible. While collecting, sharing and mining
data manually to identify patterns was the first step, automation in data mining to
enrich UGT on an ongoing basis could prove to be challenging.

Acknowledgements

I thank all the participants who helped me evaluate the UGT. I also thank Dr. Sanjay
Tripathi for his help in statistics and my guide Prof. NL Sarda for his constant support
and encouragement.

References
1. Archer, B., Systematic Method for Designers, Council of Industrial Design (1965).
2. Bevan, N., Classifying and Selecting UX and Usability Measures in Law, E., Bevan, N.,
Christou,G., Springett, M., Lárusdóttir, M. (eds.) International Workshop on Meaningful
Measures: Valid Useful User Experience Measurement (2008) 13-18
3. Cooper, A., Reimann, R., About Face 2.0 – The Essentials of Interaction Design, Wiley
(2003).
4. Cross, N., Engineering Design Methods 3rd ed., John Wiley & Sons (2000).
5. Eames, C., http://www.brainyquote.com/quotes/quotes/c/charleseam169187.html, (accessed
2008).
6. International Organization for Standardization, ISO/IEC 9126-1:2001 Software
Engineering - Product Quality (2001).
7. International Organization for Standardization, ISO 9241-1:1997 Ergonomic requirements
for office work with visual display terminals (VDTs) (1997).
8. Joshi, A., Tripathi, S., User Experience Metric and Index of Integration: Measuring Impact
of HCI Activities on User Experience, International Workshop on the Interplay between
Usability Evaluation and Software Development, Pisa (2008).
9. Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., Kort, J., Understanding, Scoping and
Defining User eXperience: A Survey Approach, In Proceedings of CHI (2009) 719-728
10. Mayhew, D.J., The Usability Engineering Lifecycle, Morgan Kaufmann (1999).
11. McCarthy, J., Wright, P., Technology as Experience, The MIT Press (2004).
12. Nielsen, J., Usability Engineering, Morgan Kaufmann (1993).
13. Shneiderman, B., Designing the User Interface: Strategies for Effective Human-Computer
Interaction 1st edition, Addison-Wesley (1987).
14. http://www.idc.iitb.ac.in/~anirudha/ugt.htm