Chapter 5 –

Cognitive
Engineering

1
 Cognitive engineering
Applying knowledge of cognitive psychology to
the design and development of systems
that support the cognitive processes of
users.

2
 What Is Cognition?

There are many different kinds of cognition, such as thinking, remembering,

learning, daydreaming, decision-making, seeing, reading, writing, and talking. A
well-known way of distinguishing between different modes of cognition is in
terms of whether it is experiential or reflective. Experiential cognition is a state
of mind where people perceive, act, and react to events around them intuitively
and effortlessly. It requires reaching a certain level of expertise and
engagement.
• Examples include driving a car, reading a book, having a conversation, and
watching a video. In contrast, reflective cognition involves mental effort,
attention, judgment, and decision-making, which can lead to new ideas and
creativity. Examples include designing, learning, and writing a report. Both
modes are essential for everyday life. Another popular way of describing
cognition is in terms of fast and slow thinking
Cognition
Cognition has also been described in terms of specific kinds of processes
These include the following:
• Attention
• Perception
• Memory
• Learning
• Reading, speaking, and listening
• Problem-solving, planning, reasoning, and decision-making.
It is important to note that many of these cognitive processes are
interdependent: several may be involved for a given activity. It is rare for
one to occur in isolation. For example, when reading a book one has to
attend to the text, perceive and recognize the letters and words, and try
to make sense of the sentences that have been written. In the following
sections we describe the main kinds of cognitive processes in more detail,
followed by a summary box highlighting the core design implications for
each. The most relevant for interaction design are attention and memory,
which we describe in the greatest detail.
  Attention

 Attention is central to everyday life. It enables us to cross the road

without being hit by a car or bicycle, notice when someone is calling our
name, and be able to text while at the same time watching TV. It
involves selecting things on which to concentrate, at a point in time,
from the range of possibilities available, allowing us to focus on
information that is relevant to what we are doing. The extent to which
this process is easy or difficult depends on
 (1) whether someone has clear goals and
 (2) whether the information they need is salient in the environment.
 • Clear Goals
If someone knows exactly what they want to find out, they try to match
this with the information that is available. For example, when someone
has just landed at an airport after a long flight, which did not have Wi-Fi
onboard, and they want to find out who won the World Cup, they might
scan the headlines on their smartphone or look at breaking news on a
public TV display inside the airport. When someone is not sure exactly
what they are looking for, they may browse through information, allowing
it to guide their attention to interesting or salient items. For example,
when going to a restaurant, someone may have the general goal of eating
a meal but only a vague idea of what they want to eat. They peruse the
menu to find things that whet their appetite, letting their attention be
drawn to the imaginative descriptions of various dishes. After scanning
through the possibilities and imagining what each dish might be like, as
well as considering other factors, such as cost, who they are with, what
are the specials, what the waiter recommends, and whether they want a
two- or three-course meal, and so on), they then decide.
 • Information Presentation

The way information is displayed can also greatly influence how easy
or difficult it is to comprehend appropriate pieces of information.
Look at Figure 4.1, and try the activity . Here, the information-
searching tasks are precise, requiring specific answers.
 • Multitasking and Attention
As mentioned in the introduction to this chapter, many people now
multitask, frequently switching their attention among different tasks.
For example, in a study of teenage multitasking, it was found that the
majority of teenagers were found to multitask most or some of the time
while listening to music, watching TV, using a computer, or reading . It
is probably even higher now, considering their use of smartphones while
walking, talking, and studying. While attending a presentation at a
conference, we witnessed someone deftly switch between four ongoing
instant message chats (one at the conference, one at school, one with
friends, and one at her part-time job), read, answer, delete, and place
all new messages in various folders of her two email accounts, and
check and scan her Facebook and her Twitter feeds, all while appearing
to listen to the talk, take some notes, conduct a search on the speaker’s
background, and open up their publications. When she had a spare
moment, she played the game Patience. It was exhausting just watching
her for a few minutes. It was as if she were capable of living in multiple
worlds simultaneously while not letting a moment go to waste. But how
much did she really take in of the presentation?
  Perception
Perception refers to how information is acquired from the environment
via the five sense organs (vision, hearing, taste, smell, and touch) and
transformed into experiences of objects, events, sounds, and tastes
(Roth, 1986). In addition, we have the additional sense of kinesthesia,
which relates to the awareness of the position and movement of the
parts of the body through internal sensory organs (known as
proprioceptors) located in the muscles and joints. Perception is
complex, involving other cognitive processes such as memory, attention,
and language. Vision is the most dominant sense for sighted individuals,
followed by hearing and touch. With respect to interaction design, it is
important to present information in a way that can be readily perceived
in the manner it was intended.
  Memory
Memory involves recalling various kinds of knowledge that allow people to act
appropriately. For example, it allows them to recognize someone’s face,
remember someone’s name, recall when they last met them, and know what
they said to them last. It is not possible for us to remember everything that we
see, hear, taste, smell, or touch, nor would we want to, as our brains would get
overloaded. A filtering process is used to decide what information gets further
processed and memorized. This filtering process, however, is not without its
problems. Often, we forget things that we would like to remember and
conversely remember things that we would like to forget. For example, we may
find it difficult to remember everyday things, like people’s names, or scientific
knowledge such as mathematical formulae. On the other hand, we may
effortlessly remember trivia or tunes that cycle endlessly through our heads
  Personal Information Management

The number of documents written, images created, music files recorded, videoclips
downloaded, emails with attachments saved, URLs bookmarked, and so on,
increases every day. A common practice is for people to store these files on a
phone, on a computer, or in the cloud with a view to accessing them later. This is
known as personal information management (PIM). The design challenge here is
deciding which is the best way of helping users organize their content so that it can
be easily searched, for example, via folders, albums, or lists. The solution should
help users readily access specific items at a later date, for example, a particular
image, video, or document. This can be difficult, however, especially when there
are thousands or hundreds of thousands of pieces of information available. How
does someone find that photo they took of their dog spectacularly jumping into the
sea to chase a seagull, which they believe was taken two or three years ago? It can
take them ages to wade through the hundreds of folders they have catalogued by
date, name, or tag. Do they start by homing in on folders for a given year, looking
for events, places, or faces, or typing in a search term to find the specific photo?
 • Memory Load and Passwords
Phone, online, and mobile banking allow
customers to carry out financial
transactions, such as paying bills and
checking the balance of their accounts, at
their convenience. One of the problems
confronting banks that provide these
capabilities, however, is how to manage
security concerns, especially preventing
fraudulent transactions.
  Learning
Learning is closely connected with memory. It involves the accumulation of skills and
knowledge that would be impossible to achieve without memory. Likewise, people
would not be able to remember things unless they had learned them. Within cognitive
psychology, learning is thought to be either incidental or intentional. Incidental
learning occurs without any intention to learn. Examples include learning about the
world such as recognizing faces, streets, and objects, and what you did today. In
contrast, intentional learning is goal-directed with the goal of being able to
remember it. Examples include studying for an exam, learning a foreign language,
and learning to cook. This is much harder to achieve. Software developers, therefore,
cannot assume that users will simply be able to learn how to use an app or a product.
It often requires much conscious effort. Moreover, it is well known that people find it
hard to learn by reading a set of instructions in a manual. Instead, they much prefer
to learn through doing. GUIs and direct manipulation interfaces are good
environments for supporting this kind of active learning by supporting exploratory
interaction and, importantly, allowing users to undo their actions, that is, return to a
previous state if they make a mistake by clicking the wrong option
  Reading, Speaking, and Listening
Reading, speaking, and listening are three forms of language processing that have some
similar and some different properties. One similarity is that the meaning of sentences or
phrases is the same regardless of the mode in which it is conveyed. For example, the
sentence “Computers are a wonderful invention.” essentially has the same meaning
whether one reads it, speaks it, or hears it. However, the ease with which people can read,
listen, or speak differs depending on the person, task, and context. For example, many
people find listening easier than reading. Specific differences between the three modes
include the following:
• Written language is permanent while listening is transient. It is possible to re-read
information if not understood the first time around. This is not possible with spoken
information that is being broadcast unless it is recorded.
• Reading can be quicker than speaking or listening, as written text can be rapidly scanned
in ways not possible when listening to serially presented spoken words.
• Listening requires less cognitive effort than reading or speaking. Children often prefer to
listen to narratives provided in multimedia or web-based learning material rather than to
read the equivalent text online. The popularity of audiobooks suggests adults also enjoy
listening to novels, and so forth.
• Written language tends to be grammatical, while spoken language is often ungrammatical.
For example, people often start talking and stop in midsentence, letting someone else start
speaking.
• Dyslexics have difficulties understanding and recognizing written words, making it hard
 Problem-Solving, Planning, Reasoning,
and Decision-Making
Problem-solving, planning, reasoning, and decision-making are
processes involving reflective cognition. They include thinking
about what to do, what the available options are, and what the
consequences might be of carrying out a given action. They often
involve conscious processes (being aware of what one is thinking
about), discussion with others (or oneself), and the use of various
kinds of artifacts (for example, maps, books, pens, and paper).
Reasoning involves working through different scenarios and
deciding which is the best option or solution to a given problem.
For example, when deciding on where to go on a vacation, people
may weigh the pros and cons of different locations, including cost,
weather at the location, availability and type of accommodation,
time of flights, proximity to a beach, the size of the local town,
whether there is nightlife, and so forth. When weighing all of the
options, they reason through the advantages and disadvantages of
each before deciding on the best one.
A Simplified View of Human
Information Processing (HIP)
 Figure 5.3 presents a simplified model of Human
Information Processing which includes processors and
memories that interact in order to process information.
 There are three types of processors:
 Perceptual: senses, detects, and accept inputs from the
external world, and stores parts of the inputs in the
working memory.
 Cognitive: interprets, manipulates, and makes decisions
about the input.
 Motor processors: interpreting cognitive decisions
into physical actions.
 Two types of memory:
 Working memory
 Long-term memory
1
7
A Simplified View of Human
Information Processing (HIP)

Attention

Verbal
Memory
MMeem data Processors
moorryy
Spatial

Perception
PePercrce Cognition Motor
Working Long term
Memory epptitioon
Memory
Memory n
Memory
Figure 5.3 A simplified model of Human Information Processing
(HIP), including memories and processor.

1
8
A Simplified View of Human
Information Processing (HIP)
Limitations of human memories and processors:
 Speed at which they can process data (the
processors can process data within 100
millisecond).
 The capacity of working memory is 5-7
chunks of data that can be retained for 10-20
seconds.

1
9
A Simplified View of Human
Information Processing (HIP)
Cognitive Performance:
 the speed and accuracy of the information-
processing task.
 The effects of limitations of human memories
and processors on performance:
 The user will not notice screen changes
that fall within 100 millisecond.
 The user will not expected to work with more
than 7 chunks of information
simultaneously.
 Retrieval of information from long-term 7

memory will degrade speed and accuracy.

A Simplified View of Human
Information Processing (HIP)
 Automatic behavior: behavior characterized
by cognitive processes that are fast and
cognitively undemanding (handling with
numbers).
 Controlled behavior: behavior characterized
by cognitive processes that are relatively
slow and cognitively demanding (handling
with words and functions).

2
1
A Simplified View of Human
Information Processing (HIP)
 Processing of Images: processing characterized as
spatial, graphic, and holistic.
 Processing of verbal information: processing
characterized as sequential, linguistic, and
procedural.
 Memory Aids
 Heuristics: rules that depend heavily on the
content
and context of the task (short cuts)
 Metaphor: a cognitive process in which an experience
is related to an already familiar concept (typewriter).
 Mental model: a representation of the conceptual
structure of a device or a system (calculation using
spreadsheet). 2
2
The Complexity of HCI
The task as the user sees it, can be represented as a set
of intentions and evaluations

Figure 5.4 HCI as a Bridge between Human and Computer

10
The Complexity of HCI
Basic assumption of cognitive engineering:
users are faced with limited cognitive resources,
therefore human computer interaction should
be designed to complement and fit these
limitations.
Complexity of HCI:
A Function of the amount of cognitive resources
needed by the user to accomplish a given
task with a given computerized system.
11
The Complexity of HCI
 The more resources needed, the higher the
complexity.
 The same task be of different complexity
when supported by different systems.
 The same task on the same system may be
of different complexity to different users.

12
Gulfs of Execution and Evaluation
 Gulf of execution: the gap between the
user’s goal and its computerized
implementation.
 Gulf of evaluation: the gap between the
computerized implementation of the user’s
goal and its evaluation by the user.
 Gulf of execution and gulf of evaluation forms
a gap between the human and the
computer.
 HCI aims to minimize this gap.
13
 Bridging the gaps using Norman’s
Model
•Establish a goal that needs to be Goals
accomplished.
•Form the intention (or hierarchy of
Intentions Evaluation
intentions) that will accomplish the goal.
•Specify the action sequence to
implement the intentions.
Action Interpretation
•Execute the action. specification
•Perceive the state of system resulting
from the action.
•Interpret the system state. Execution Perception
•Evaluate your interpretation against
the expectation based on your
intentions. Physical
system

Figure 5.6 Norman's seven stage model of

user14 activity
Fit and Complexity
 The fit between the user’s mental model of
the system and the actual model of the
system affects complexity.
 The greater the fit, the easier it is to
determine how to translate goals into action.
 The greater the misfit, the more difficult is the
process of bridging the execution and
evaluation gulfs.

15
 User Activity with Multiple Intentions

 Users in organizational settings usually interact with

computers to achieve complex goals.
 Norman’s model of user activity can be useful for describing
the user activity involved in more complex tasks by
modelling multiple intentions to accomplish a single goal,
(e.g. checking a new sales forecast and summing the
corresponding revenues).
 Each of these two intentions describes how the
seven stages are organized to bridge the gulfs
between the computer and the user.

16
Errors
 Errors: deviations from intentional behavior that is
either skill, rule or knowledge based.
 Classification of errors based on behavior type:
 Skill based behavior: automatic behavior that is
predefined and requires minimal cognitive resources.
 Rule based behavior: controlled behavior that relies
on predefined rules of behavior that are contingent on
particular situation encountered.
 Knowledge based behavior: highly controlled
behavior that requires assessment and generation of
new rules of behavior, and is demanding of
cognitive resources.
17
Causesof Errors

Copyright 2006 John Wiley & Sons, Inc 18

 Cognitive Frameworks
A number of conceptual frameworks have been
developed to explain and predict user behavior based on
theories of cognition. In this section, we outline three
that focus primarily on mental processes and three
others that explain how humans interact and use
technologies in the context in which they occur. These
are mental models, gulfs of execution and evaluation,
information processing, distributed cognition, external
cognition, and embodied interaction.
 Mental Models
Mental models are used by people when needing to reason about
a technology, in particular, to try to fathom what to do when
something unexpected happens with it or when encountering
unfamiliar products for the first time. The more someone learns
about a product and how it functions, the more their mental
model develops. For example, broadband engineers have a deep
mental model of how Wi-Fi networks work that allows them to
work out how to set them up and fix them. In contrast, an
average citizen is likely to have a reasonably good mental model
of how to use the Wi-Fi network in their home but a shallow
mental model of how it works. Within cognitive psychology,
mental models have been postulated as internal constructions of
some aspect of the external world that are manipulated,
enabling predictions and inferences to be made. This process is
thought to involve the fleshing out and the running of a mental
model. This can involve both unconscious and conscious mental
processes, where images and analogies are activated.
How can user experience (UX) designers
help people to develop better mental
models?
 A major obstacle is that people are resistant to spending much time learning
about how things work, especially if it involves reading manuals or other
documentation. An alternative approach is to design technologies to be more
transparent, which makes them easier to understand in terms of how they
work and what to do when they don’t.
 This includes providing the following:
 Clear and easy-to-follow instructions.
 Appropriate online help, tutorials, and context-sensitive guidance for users in
the form of online videos and chatbot windows, where users can ask how to do
something.
 Background information that can be accessed to let people know how.
something works and how to make the most of the functionality provided
 Affordances of what actions an interface allows (for example, swiping,
clicking, or selecting).
 Gulfs of Execution and Evaluation
The gulf of execution and the gulf of evaluation describe the gaps that exist
between the user and the interface. The gulfs are intended to show how to
design the latter to enable the user to cope with them. The first one, the gulf
of execution, describes the distance from the user to the physical system
while the second one, the gulf of evaluation, is the distance from the physical
system to the user.
 Information Processing
Another approach to conceptualizing how the mind works has been
to use metaphors and analogies to describe cognitive processes.
Numerous comparisons have been made, including conceptualizing
the mind as a reservoir, a telephone network, a digital computer,
and a deep learning network. One prevalent metaphor from
cognitive psychology is the idea that the mind is an information
processor. Information is thought to enter and exit the mind
through a series of ordered processing stages (see Figure 4.8).
Within these stages, various processes are assumed to act upon
mental representations. Processes include comparing and
matching. Mental representations are assumed to comprise
images, mental models, rules, and other forms of knowledge.
 Distributed Cognition
Most cognitive activities involve people interacting
with external kinds of representations, such as
books, documents, and computers and also with
each other. For example, when someone goes home
from wherever they have been, they do not need to
remember the details of the route because they rely
on cues in the environment (for instance, they know
to turn left at the red house, right when the road
comes to a T-junction, and so on). Similarly, when
they are at home, they do not have to remember
where everything is because information is available
as needed. They decide what to eat and drink by
scanning the items in the fridge, look out the
window to see whether it is raining or not, and so
on. Likewise, they are always creating external
representations for a number of reasons, not only to
help reduce memory load and the cognitive cost of
computational tasks, but also, importantly, to
extend what they can do and allow people to think
more powerfully .
 External Cognition
People interact with or create information by using a variety of external
representations, including books, multimedia, newspapers, web pages,
maps, diagrams, notes, drawings, and so on. Furthermore, an impressive
range of tools has been developed throughout history to aid cognition,
including pens, calculators, spreadsheets, and software workflows. The
combination of external representations and physical tools has greatly
extended and supported people’s ability to carry out cognitive activities.
Indeed, they are such an integral part of our cognitive activities that it is
difficult to imagine how we would go about much of our everyday life
without them. External cognition is concerned with explaining the cognitive
processes involved when we interact with different external representations
such as graphical images, multimedia, and virtual reality. A main goal is to
explain the cognitive benefits of using different representations for
different cognitive activities and the processes involved.
The main ones include the following:
• Externalizing to reduce memory load.
• Computational offloading.
• Annotating and cognitive tracing.
 Embodied Interaction
Another way of describing our interactions with technology and the world
is to conceive of it as embodied. By this we mean the practical
engagement with the social and physical environment. This involves
creating, manipulating, and making meaning through our engaged
interaction with physical things, including mundane objects such as cups
and spoons, and technological devices, such as smartphones and robots.
Artifacts and technologies that indicate how they are coupled to the
world make it clear how they should be used. For example, a physical
artifact, like a book when left opened on someone’s desk, can remind
them to complete an unfinished task the next day.