Chapter 6

EMOTIONAL
INTERACTION
 Overview
• Emotions and the user experience
• Expressive and emotional design
– how the ‘appearance’ of an interface can
affect users
• Affective computing and emotional AI
• Persuasive technologies and behavioural
change
• Anthropomorphism
– the pros and cons

www.id-book.com
 Emotions and the user experience
• HCI has traditionally been about designing efficient
and effective systems
• Now more about how to design interactive systems
that make people respond in certain ways
– e.g. to be happy, to be trusting, to learn, to be motivated

• Emotional interaction is concerned with how we feel

and react when interacting with technologies
• Affective computing is improving with better
recognition software and machine learning algorithms

www.id-book.com
 Emotional interaction
• What makes us happy, sad, annoyed, anxious,
frustrated, motivated, excited and so on
– translating this into different aspects of the user
experience
• Why people become emotionally attached to
certain products (e.g. virtual pets)
• Can social robots help reduce loneliness and
improve wellbeing?
• How to change human behavior through the use
of emotive feedback

www.id-book.com
 Emotional interaction
• However, it is not straightforward to achieve as
people’s moods and feelings are constantly
changing.
– Gain knowledge into different aspects of the user experience from when we
first want something to when we no longer interact with it or need to replace it

• There are also many reasons that might cause

someone to be happy or sad, such as the sun
shining or someone else winning a game.

www.id-book.com
 Activity
• Try to remember the emotions you went
through when buying a big ticket item online
(e.g. a fridge, a vacation, a computer)
– This is followed by the joy or frustration of finding out
more about what products are available and deciding
which to choose from potentially hundreds or even
thousands (by visiting numerous websites, such as
comparison sites, reviews, recommendations, blogs).

• How many different emotions did you go

through?
– When wrong input of credit card information, search
reasons, revise it, enter correct and get your product.

www.id-book.com
Why has this simple way of obtaining
visitor feedback been so effective?

www.id-book.com
Pulling at the heart strings with
emotive message

www.id-book.com
How easy is it to design an interface to
match or change how we are feeling?
• Should an interface be designed to improve
how we feel?
– If so, how?
• Our moods and feelings are continuously
changing
– How does the interface keep track and know when
to do something?
• What moods match which kinds of interfaces?
• How would you design an interface for when
someone is happy, angry, sad, bored, focused?
www.id-book.com
 How do emotions affect behavior
and behavior emotions?
• Examine how people express themselves and read
each other (emotional intelligence)
– understand relationship between facial expressions, body
language, gestures and tone of voice, e.g.
– when people are happy, they laugh and relax body posture
– when they are angry, they screw up their face
• Does being angry make you concentrate better or
more distracted?
• When you are happy do you take more risks such
as spend more money or buy more?
• Baumeister et al (2007) argue the relationship is
more complex than a single cause-and-effect model
www.id-book.com
Automatic (affect) versus conscious
emotions
• Emotions can be short-lived (e.g. fit of anger) or
complex and long-lasting (e.g. jealousy)
• Have been categorized as automatic or
conscious
– Automatic ones are rapid and dissipate quickly
– Conscious ones develop slowly and take a long time
to go (e.g. reflection)

www.id-book.com
 Ortony et al (2005) model of emotional
design
• Visceral level which is associated with immediate communication in
response to change in surrounding environment.
• Regulation/control of our everyday behavior and this level is known
as behavioral level.
• At the highest level, brain further processes and anticipates the
signals of behavioral level. This level is called reflective level

According to this emotion model visceral

level responds rapidly, making judgments
about what is good or bad, safe or
dangerous, pleasurable or disgusting etc.
It also triggered different emotional
responses to stimuli, such as fear, joy,
anger and sadness that are combination
of physiological and behavioral
responses.
www.id-book.com 12
 Claims from model
• Our emotional state changes how we think
– when frightened or angry we focus narrowly
and body responds by tensing muscles and
sweating
• more likely to be less tolerant
– when happy we are less focused and the
body relaxes
– More likely to overlook minor problems and be
more creative

www.id-book.com
Designing with the three levels in mind
• Visceral design refers to making products
look, feel and sound good

• Behavioral design is about use and

equates with traditional values of usability

• Reflective design is about considering the

meaning and personal value of a product
www.id-book.com
Analyzing a swatch watch design
using the model
• Cultural images and graphical
elements designed at reflective
level
• Affordances of use at behavioral
level
• Brilliant colors and wild design
attract user’s attention at visceral
level

www.id-book.com
 Expressive interfaces
• Provide reassuring feedback that can be both
informative and fun
– But can also be interfering, causing people to get
annoyed and even angry
• Colour, icons, sounds, graphical elements and
animations are used to make the ‘look and feel’
of an interface appealing
– conveys an emotional state
• In turn this can affect the usability of an interface
– people are prepared to put up with certain aspects of
an interface (e.g. slow download rate) if the end result
is appealing and aesthetic
www.id-book.com
The appearance of an interface
• (a) Emotional icons were
used in the 80s to indicate
rebooting or crashed
– smiling apple face

• (b) Now a days use more

impersonal but aesthetically
pleasing icons to indicate
user needs to wait
– the beachball

www.id-book.com
 The design of thermostats
• (a) The Nest has a minimalist and
aesthetically pleasing design
– Round face and simple dial
– Large font and numbers
• (b) Very different from earlier thermostat
designs
– utilitarian and dull

www.id-book.com
 Annoying interfaces
• Microsoft pioneered friendly interfaces for
technophobes
– e.g. ‘At home with Bob’ software
– 3D metaphors based on familiar places (e.g. living
rooms)

• Agents in the appearance of pets (e.g. bunny,

dog) were included to talk to the user
– Make users feel more at ease and comfortable
– But many people did not like the idea of Bob so never
made it to product
www.id-book.com
At home with Bob

www.id-book.com
 Microsoft’s Clippy and IKEA’s Anna

• Clippy did…
• But was disliked by so many?
– Was it annoying,
confusing,
demeaning or other?
• Anna appeared as a virtual
agent
– Blinked, moved her lips and
head to suggest facial
expressions
www.id-book.com
 Frustrating interfaces
• Many causes:
1. When an application doesn't work properly or crashes
2. When a system doesn't do what the user wants it to do
3. When a user’s expectations are not met
4. When a system does not provide sufficient information
to enable the user to know what to do
5. When error messages pop up that are unclear, obtuse
or condemning
6. When the appearance of an interface is dazzling, noisy,
gimmicky or patronizing
7. When a system requires users to carry out too many
steps to perform a task, only to discover a mistake was
made earlier and they need to start all over again
www.id-book.com
 Error messages
“The application Word Wonder has unexpectedly quit due to a type 2 error.”

Why not instead:

“the application has expectedly quit due to poor coding in the operating
system”

• Shneiderman’s classic guidelines for error messages include:

• avoid using terms like FATAL, INVALID, BAD

• Audio warnings
• Avoid UPPERCASE and long code numbers
• Messages should be precise rather than vague
• Provide context-sensitive help

www.id-book.com
A friendly cute image instead of the
impersonal 404 error message

www.id-book.com
 Dilemma 1: Should computers say
they’re sorry?
• Reeves and Naas (1996) argue that computers
should be made to apologize
• Should compete with human propriety
• Would users be as forgiving of computers saying
sorry as people are of each other when saying
sorry?
• How sincere would they think the computer was
being? For example, after a system crash:
– “I’m really sorry I crashed. I’ll try not to do it again”
• How else should computers communicate with
users?
www.id-book.com
 Dilemma 2: Should Voice assistants
teach kids good manners?
• Many children talk to Alexa as if she was their friend
• They also learn that it is not necessary to say please and
thank you to her when asking questions
• Is this lack of using etiquette a problem?
• Would it transfer over to real life situations?
– e.g. demanding “Aunty, make me a sandwich.”
• Parents should still teach their kids good manners
• Alexa can be configured to be polite as well
• How much parental control should voice assistants be
given?
• Would children find it strange or disturbing that their Alexa
(who is their friend) advise them to clean their teeth?
www.id-book.com
Affective Computing and Emotional AI
• Affective computing concerned with how to use computers to
recognize and express emotions as humans do (Picard, 1998)
• Involves designing ways for people to communicate their
emotional state
• Uses sensing technologies to measure facial expressions,
gestures, body movement
• Explores how affect influences personal health
• Emotional AI aims to automate the measurement of feelings and
behavior using AI to understand them from facial expressions
and voice
• Aim is to predict user’s emotions and aspects of their behavior
– e.g. what is someone most likely to buy online when feeling
sad, bored or happy
www.id-book.com
 Techniques used
• Cameras for measuring facial expressions

• Biosensors placed on fingers or palms to

measure GSR (Galvanic Skin Response)

• Affective expression in speech (e.g.

intonation, pitch, loudness)

• Body movement and gestures using

accelerometers and motion capture systems

www.id-book.com
 Classification of emotions
• Six core expressions typically measured:
– sadness, disgust, fear, anger, contempt, joy

• Type of facial expression chosen by AI

through detecting presence of absence of:
– Smiling, eye widening, brow raising, brow
furrowing, raising a cheek, mouth opening,
upper-lip raising, wrinkling of the nose

www.id-book.com
Facial coding using Affdex software

www.id-book.com
 How is this emotional data used?
• If user screws up their face when an ad pops
up -> feel disgust
• If start smiling -> they are feeling happy
• Website can adapt its ad, movie storyline or
content to match user’s emotional state
• If system used in a car, it might detect an angry
driver and suggest driver takes a deep breath
• Eye-tracking, finger pulse, speech and
words/phrases also analysed when tweeting or
posting to Facebook

www.id-book.com
 Indirect emotion detection
• Also used more to understand or predict
someone’s behavior
– e.g. determining a person’s suitability for a job
– e.g. how they select from multiple alternates

• Do you think it is ethical that technology

can read your emotions from your facial
expressions or from your tweets?

www.id-book.com
 Persuasive technologies and
behavioural change
• Interactive computing systems designed to change
people’s attitudes and behaviours (Fogg, 2003)

• A diversity of techniques now used to change what

they do or think
– Pop-up ads, warning messages, reminders, prompts,
personalized messages, recommendations, Amazon 1-
click
– Commonly referred to as pushing

www.id-book.com 33
 Nintendo’s Pocket Pikachu
• Was developed to change bad habits and
improve well being
– Designed to motivate children to be more physically
active on a regular basis

– owner of the digital pet that ‘lives’in the device is

required to walk, run, or jump

– If owner does not exercise the virtual pet becomes

angry and refuses to play anymore

www.id-book.com
 How effective?
• Can interactive technologies
that monitor, plug, or behave
like a human keep them
interested in looking after it and
in doing so fitter, themselves?

• How does looking after a virtual

pet change a child’s behaviour?
– Emotional attachment
– Happy Pokémon makes them feel
good
– Sad/angry Pokémon makes them
feel bad www.id-book.com
 Tracking devices
• Mobile apps designed to help people monitor
and change their behaviour (e.g. fitness,
sleeping, weight)

• Can compare with online leader boards and

charts, to show how they have done in
relation to their peers and friends

• Also, apps that encourage reflection that in

turn increase well-being and happiness

www.id-book.com
 Sustainable HCI
• Focus on designing tech involvements to help people
reduce their energy consumption
• An effective technique is to provide householders with
feedback on their consumption
• Simple infographics and emoticons are often most
powerful
– Encourage people to reflect and see what they can
change to reduce their energy consumption
• Peer pressure and social norms are also powerful
methods
www.id-book.com
 The Tidy Street project
• Large-scale visualization of the street’s
electricity usage
– applied display on the road surface using chalk
– provided real-time feedback that all could see change
each day
– reduced electricity consumption by 15%

www.id-book.com
Bird and Rogers, 2010)
 Phishing scams
• Web used to deceive people into parting
with personal details
– e.g. Paypal, eBay, and won the lottery letters

• Allows Internet fraudsters to access their

bank accounts and draw money from them
• Many vulnerable people fall for it
• The art of deception is centuries old, but
internet allows ever more ingenious ways
to trick people
www.id-book.com
 Anthropomorphism
• Attributing human-like qualities to non-living
objects (e.g. cars, computers)

• Well known phenomenon in advertising

– dancing butter, drinks, breakfast cereals

• Much exploited in human-computer interaction

– make user experience enjoyable, motivating, make
people feel at ease, reduce anxiety

• Furnishing technologies with personalities can

make them enjoyable to interact with
www.id-book.com
 Which message you prefer?
As a welcome message:

• “Hello Chris! Nice to see you again. Welcome

back. Now what were we doing last time? Oh
yes, exercise 5. Let’s start again.”

• “User 24, commence exercise 5.”

www.id-book.com
 Which do you prefer?

• Feedback when user gets something wrong:

1. “Now Chris, that’s not right. You can do better

than that. Try again.”
2. “Incorrect. Try again.”

• Is there a difference as to what you prefer

depending on type of message? Why?

www.id-book.com
Evidence to support anthropomorphism

• Reeves and Naas (1996) found that computers

that flatter and praise users in education software
programs -> positive impact on them
“Your question makes an important and useful
distinction. Great job!”
• Students were more willing to continue with
exercises with this kind of feedback

www.id-book.com 43
 Robot-like or cuddly?
Which do you prefer and why?

Aibo The Haptic Creature

A motivation for making robot pets cuddly is to enhance the emotional

experience people have through using their sense of touch
www.id-book.com
 Is it OK for senior people to
develop an emotional attachment
with the robot Zora?

www.id-book.com
 Summary
• Emotional aspects of interaction design are concerned with how
to facilitate certain states (e.g. pleasure) or avoid reactions (e.g.
frustration)
• Well-designed interfaces can elicit good feelings in people
• Aesthetically pleasing interfaces can be a pleasure to use
• Expressive interfaces can provide reassuring feedback to users
• Badly designed interfaces make people frustrated, annoyed, or
angry
• Emotional AI and affective computing use AI and sensor
technology for detecting people’s emotions through analyzing
their facial expressions and conversations
• Emotional technologies can be designed to persuade people to
change their behaviors
• Anthropomorphism is the attribution of human qualities to
objects
• Increasingly robots are being used as companions in the home

www.id-book.com