DOES THE USE OF GRAPHIC

DESIGN VISUALISATION IN
MUSIC AFFECT THE
VIEWERS EXPERIENCE?

Mhairi Muir
40430133

A dissertation submitted in partial fulfilment

of the requirements of Edinburgh Napier
University, for the award of BA Hons in
Graphic Design

December 2022
I confirm that the final version of my dissertation
conforms to the requirements of the Degree Regulations
in all respects.

1
 Acknowledgements

I would like to take this time to thank Julia Buchanan, Niina Turtola and
Sam Vettese who assisted in the formatting and support when creating this
research paper. I would further like to thank my family for supporting me
when writing this document. Finally, I would like to thank my fellow
students and classmates who assisted in the research tasks.

2
 Contents
Figures ..................................................................................................................... 4
List of Figures .......................................................................................................... 6
Introduction ............................................................................................................. 7
History of music visualisation ................................................................................. 7
Emotions and Design principles ............................................................................ 12
Design Probe 1.1 .................................................................................................... 16
Understanding Design Probes ................................................................................ 18
Method ................................................................................................................... 19
Completed Probe .................................................................................................... 21
Analysing Data........................................................................................................ 22
To Design and Beyond............................................................................................ 28
Conclusion .............................................................................................................. 29
References ............................................................................................................... 31
Appendix ................................................................................................................. 33

3
 Figures

Figure 1

4
 Figure 2

5
 List of Figures

Figure 1 – Design Probe sheet 101

Figure 2 – Design Probe sheet 102

6
Does the use of graphic design visualisation in music affect the viewer’s
experience?

Introduction

Dating back to the common era, artists have been investigating the
connection between sound and design (Hall, 2020). Progressing with the
development of technology, the design of music visualisation has the power
to make music an inclusive place for all (Fourney & Fels, 2009). This
research paper will be investigating the connection between music and
design to answer the question of whether accompanying music with visuals
influences and enhances the viewer’s experience. By creating and using
design probes and a relevantly researched group of participants, data will be
collected on the subject matter to construct an informed answer to the
proposed question.

The History of Music Visualisation

To establish the relationship between both colour and sound, this topic must
be introduced in historic terms.

The first known blend of both music and colour was the colour organ in the
1700s. During the 18th century around 1742, a French monk named Louis
Bertrand Castel, invented the first known colour organ when he combined
stained glass with his piano organ (Mcdonald, 2002). Castel did this by
attaching organ keys to long strings leading to dark curtains which covered
various colours of stained glass. When a specific note was played, the string
would be pulled revealing the coloured glass behind it. This would cause
natural light to be tinted when it shone through into the room (Mcdonald,
2002). The colour organ continued to develop until around the mid-1800
when electricity and the artistic curiosity surrounding it began to grow. With
the additional technical aspect of electricity, the basic principle of the

7
original 1742 colour organ remained true with a traditional piano being
connected to some variable of stained glass (Gibson, 2022).

Further fascination with colour and sound began in the 20th century when
researchers came to discover the condition called synaesthesia. Synaesthesia
is the term used when a person’s stimulus in one sense has immediately
evoked a sensation in another. This discovery and study of this condition
revolutionised the relationship between audio and visual language. With the
new knowledge and name for this condition of feeling, tasting, and seeing
colours, came new artists wanting to showcase this in their own unique way
(Gibson, 2022).

This cascaded into the advancement of several colour organ-like

contraptions during this century. These included the Chromola (1911), the
Lumigraph (the 1940s), the Farblichtmusik (1925), and the Clavilux (1921)
(Mcdonald, 2002). These instruments of sound and colour were used
throughout the 20th century with various levels of popularity. The
Farblichtmusik created by Alexander Laszlo became extremely popular with
live music performances during the 1920s within Europe, specifically in
Germany. This audio-visual invention made Laszlo a household name with
his innovative use of music and colour (Gibson, 2022). Evolving from this
was the invention of the Clavilux created by Thomas Wilfred. Invented in
1921, it took inspiration from the workings of the colour organs created
before it. However, with the new accessibility of electricity, it took the twist
of using a television-like device to emit light through glass. The Clavilux
took a different route of visualisation by taking the equation of sound away
from the matter. It was designed to perform colours silently and was used
alongside music scores to enhance the performance. Wilfred took his
invention on tour during the 1920s assisting in his rise to household fame
and said that his invention was the ‘ultimate expression of abstract painting
into an evolving, time-based medium’ (Gibson, 2022). During this time
Russian inventor Alexander Scriabin was also developing his invention the

8
Luce. The Luce was a light keyboard that combined colour and tonality to
make a device that blended sound and image. Scriabin’s interest in
synaesthesia helped progress this and is said to have been the first example
of live-stage visuals (Gibson, 2022).

Parallel to the various forms of colour organs blending colour and sound
was the evolution of graphic notation. Dating back to 840ce graphic
notation was formed when a monk named Aurélien de Reome created the
first graphically written music score. Reome wrote his music in this way,
blending music and graphics on one page (Hall, 2018). After taking many
years away from popularity, during the 1950s with the rise of Avant Garde,
new experimental composers started to emerge eager to experiment with
new ways of going against the norm (Plourde, 2017). This interest led to an
increase in the use of graphic notation writing and performing. The term
Graphic Notation is used when a composer writes music graphically,
without using the traditional bars and note forms. It is then interpreted by
the performer on how they believe the notes are supposed to be played by
them creatively, dictating what the image is portraying. In this experimental
writing, notes are symbolised by shapes such as squares, triangles, and
circles, lines can represent the longevity and weight of a note, and location
represents the pitch which gives a rough idea of a start guide for the
performer. When multiple instruments are involved, colours can represent
the time in which they are played. These basic graphic elements can range
all complexities and leave most of the music interpretation up to the
performing artist (Plourde, 2017). With the popularity of this rising in the
1950s it made the connection of sound and music blend alongside the
development of colour organs.

The constant improvement of technology saw a change in how music was

visualised. During 1980-1990, the rise of rave culture and acid house music
paved the way for the introduction and invention of the VJ (Gibson, 2022).
The term “VJ” or “Visual Jockey” was first used in the 1980s by Merrill

9
Aldighieri who was a visual artist in a New York nightclub called Hurrah.
The invention of the VHS tape made videoing available to the household
artist, videotaping was no longer limited to TV studios with expensive
equipment. With this, visual artists were given a new creative outlet to
produce work. Simultaneously with this came the birth of the ‘rave’ a term
used in the mid-1980s to describe an illegal party located in abandoned
warehouses or derelict building sites. These parties often explored the new
acid house music genre developing into drum and bass and techno in the
1990s. Alongside the music genre being predominantly dance music, the
location became a main part of rave culture. A rise in drug use was seen on
this rave scene due to its nature of being within a secretive location. This
mix of various illegal activities, music, and drug use made it a popular,
alternative scene at this time. This popularity drew people away from live
stage music to secretive DJ sets. This new way of performing meant there
was a lack of visual presence from a performer or band on stage which led
to a gap in visual performances. Tangent to this was another artist
development of the scratch video artist. These artists took archived footage
from broadcasts and mixed or rearranged them to create a new, normally
political, narrative. This along with adding extra feedback on the film with
heavy effects and saturation with the occasional text added on screen,
allowed artists to make visually powerful films with a distinct look to them
which could be played live. This development of art, and while rave culture
was still popular, created the perfect storm for these artists and DJs to
combine to enhance the raver’s experience. Since the scratch video artists
already mixed videos live it meant the transfer to raves was seamless and
gave the chance for DJs and Visual Artists to react to the crowd in real-
time (Gibson, 2022).

These visual artists swiftly moved from raves into nightclubs during the
1980s. The first known artist to do so was Merril Aldighieri during her two-
year residency at Hurrah nightclub NYC (Gibson, 2022). As a young,
female New York artist, she had to find a place to show her artwork while

10
simultaneously having a steady income (VJ Diaries 2019). Aldighieri,
defined her new job at Hurrah as a VJ or Visual Jockey which resulted in
forming a new job in the live music club scene. In her 2019 documentary,
Aldighieri explained that during her time at Hurrah NYC nightclub as far as
she was aware, the disk jockey was responsible for the music and so the
visual jockey was responsible for the visual accompaniment. At this time
British artists such as Joy Division and David Bowie performed at the
Hurrah club as their New York debut and Merril was responsible for the
visuals to accompany them (VJ Diaries 2019). Hurrah was the first known
nightclub to have installed TV screens throughout to allow Aldighieri to
play her moving art assisting to visually enhance the experience of the
visitor (Gibson, 2022). Her method was similar to a scratch video artist
where she would collect stock footage of random objects and archived
footage with no start, middle, or end. This would allow her to play these
clips at any point during her set creating her own narrative. After collecting
this footage, she would apply filters and physically scratch the film to add
graphic elements over the miscellaneous scenes. These would then be
played on the nightclub TV screens live in reaction to what was happening
in the nightclub (VJ Diaries, 2019). In her self-documentary ‘VJ Diaries’
2019, she uncovers and explains found archived footage of her time as a
new VJ. She explains she didn’t want to overpower the guest experience
with a set narrative, hence the random film which would simply add a visual
scepticism that would ‘flow with the music and dancing’ (VJ Diaries, 2019).
She also emphasises how creatives worked during that time period.
Musicians would play instruments they weren’t trained in, and she played
footage in ways that worked for her. It was all about experimenting in a safe
place with a live crowd who wanted the new club scene experience that
Hurrah provided (VJ Diaries, 2019).

With the development of technology comes the development in the use of

technology. The VJ who used VHS tapes became a thing of the past with
new audio-reactive software and virtual reality becoming more accessible.

11
During Coachella 2019, an American music festival held in Los Angeles,
crowds were introduced to the interactive augmented reality stage which
encouraged visitors to use their mobile devices for an enhanced visual
experience (Edmonds, 2020). This is one of many uses of new Artificial
Intelligence technologies to accompany electronic dance music. While
traditional 1980 VJ’ing helped the viewer seamlessly watch visuals that
accompanied the music and dancing (VJ Diaries, 2019), Coachella’s use of
technology made the viewer entirely immersed in the performance
(Edmonds, 2020). Other festivals such as Mo;dem in Croatia used complex
stages with video mapping to project video controlled by a VJ onto the stage
allowing a visual experience for main stage visitors. Mo;dem also involved
smaller screens in the landscape and vegetation to continue the visual
experience away from the main stages. Furthermore, advances in screen
technology allowed Burning Man 2016 to have a 360º viewing system that
projected VJ artwork based on an LSD experience. This gave viewers a
surround audio-visual performance engulfing them in the story. With this
fast development of technology gives endless opportunities for potential live
visual outcomes. The use of Virtual Reality could make the live festival a
completely immersive experience with the attendee being completely
immersed in visual scepticism and live music (Edmonds, 2020). This rich
history of graphic visualisation and the development of the VJ manifests the
question of how these graphic elements affect people’s feelings.

The studies of Graphic Elements and Emotional Connection.

“Emotions through Graphics 2006” by Palle Derkert, investigates the

relation of how emotion can be recorded and influenced by different graphic
elements. The study took several subjects and used non-intrusive equipment
such as arm bands and chest straps over an extended period of time (12
hours or more) to capture their emotions through several graphic techniques.
Beginning with what researchers already understand, it has been previously
proven that an animation involving two stationary balls gave a sense of

12
tension and argument. Without any graphic elements added such as
movement or scale, the viewer used their imagination to come to conclude
that these two emotionless balls were in a standoff argument. When the
movement was added, the ball which was hitting off the other more
frequently was seen as the ‘bad guy’. A study with Disney also proved that
animations that used repetitive motion such as a character repeating a
‘goofy’ walk along a straight path were seen as captivating by the viewer.
This allows Disney to cut down on animating time by having the allowance
to loop animations while still keeping the audience focused on the film. It
was also stated that smooth animations were seen as peaceful whereas rapid
ones are seen as chaotic. This gives the information that speed is a factor to
consider when wanting to provoke emotion within moving graphics. The
study conducted in this paper investigated graphic elements such as colour,
shape, and movement to see if these correlate with human emotion. The
findings of these studies were mainly consistent although many of them
required the subject to use their imagination to create a narrative (Derkert,
2006).

Colour was a major contributing factor when portraying emotion. It was

found that when lighter colours were viewed the subject was calm and had
feelings of tranquillity whereas when the colour was darker and deeper, the
emotion that the subject felt leaned to more active and busier. The example
of yellow was given to highlight issues with these findings as two of the
subjects disliked the colour yellow. This meant that the subject found it
difficult to relate to it thus making it harder for them to visualise it with an
emotion (Derkert, 2006).

Another element that was researched in depth was graphic structure. When
the graphic element had thick edges, the user was seen to be more
captivated and excited whereas if the edges were thinner, the user was
calmer and had ‘low arousal.’ It was said that simple, basic drawings were
seen to have a bigger impact and a more dramatic effect than those that were

13
more complex. When adding complex expressions to the simple shape, it
was seen to be harder to read than when the shape was in its simple form
(Derkert, 2006).

Developing forward from this within the study, a movement was added to
the shape to enhance the viewer’s experience and feelings. For the
animation to be effective, the study discovered that the added element of
movement had to add something new and interesting for the viewer for it to
be beneficial. In instances where the stationary graphic was sufficient, the
animation was seen to negatively impact the overall experience for the
viewer. However, when the movement added context or visual interest to
the graphic, the viewer seemed to become more focused on the graphic
itself. The most successful finding from the animation portion was the
pulsing circle. This involved giving the subjects a circle that was moving in
parallel with their heart rate, giving them something to relate to the circle.
This produced comments that the animation was ‘cool’ and ‘exotic’ with
one person explaining it was like a ‘living entity’ (Derkert, 2006).

Enhancing this further was adding motion effects to the graphics. When
adding moving to motion effects, blurring the edges of the animation made
the viewer feel calmer. It also gave a natural look to the animation
according to the viewer (Derkert, 2006). This study proves that graphic
choices such as colour, shape, and movement have an impact on the viewer
experience.

To confirm these findings, “User Emotions on Moving and Fading Patterns

of Motion Graphics within TFT Clusters of Automobiles” by Huhn Kim and
Minji Park 2015, concluded that motion graphic techniques influence user
experience in automobiles. They had similar conclusions to the previous
study with fades and blurs making the graphic sequence more natural and
fluid. The conclusion was drawn that people find emotional comfort in
motion graphics. This study also stated that the factor of time affected all

14
factors studied prior which were speed, rhythm, form, and colour.
Surprisingly only 5.65% of emotion was affected by colour in motion
graphics within cars. This point would contradict the previous findings of
how colour did affect the viewer’s mood however, this may be due to the
context in which they are viewed. The main contributing factor to this was
the speed at 37.92% (Kim & Park, 2015). The speed of these graphics seen
within a car was the most vital to the viewer which would align with the
findings of the previous study.

A third study conducted by Hyeon-Jeong Suk and Kyeong Ah Jeong titled

““Jockey Time”: Making Video Playback to Enhance Emotional Effect
2016”, confirmed the final outcome of speed being a main factor of emotion
in moving pictures. This study took a short clip of a woman performing
basic human movements which express emotion. This was then shown to 30
college students to then describe the playback video in one word. This
collection of words was then put into a word bank and categorised using a
thesaurus over a few umbrella terms to streamline results. The playback
speed was a variable in this test. It was found that when the playback speed
was faster, emotions such as urgency were felt whereas when the playback
speed was slower, calmer, and more relaxed emotions were described. The
playback speed along with the direction of the film was discussed. It was
found that the direction of the playback also had an impact on the emotion
felt by the viewer. When the playback was forward, the viewer had a
positive feeling whereas when the playback was backward, the viewer had
negative feelings toward the video. This reversed playback paired with a
slower speed, was seen to make even stronger negative emotions than if the
playback was just reversed. This study was done to build and develop a
software called “Jockey Time” to allow users to take their own video and to
have it set with playback researched to entice a certain emotion. This
research on emotion linking with playback speed, allows the user to place
their own video on a scale to have an entire range of emotions for the viewer
to feel (Jeong & Suk, 2016).

15
Linking the previous studies focusing on the connection of graphics and
emotion, poses the question of how this data can be used in relation to music
and the design world. In a 2009 paper, “creating access to music through
visualisation”, it was discussed how the visualisation of music can help
make music inclusive for those who are hard of hearing or deaf. It is
explained that hearing aids tend to distort music making it a less valuable
experience for this group of people which, in western culture, has a negative
impact on their life due to music being a collective experience of
community in addition to being present in day-to-day life. The study took a
group of hard-of-hearing and deaf applicants and gave them a set of 18
visuals representing different songs across three different visualisation
software. A discussion on what visuals represented songs well and what
ones were most beneficial to their community was then had. The study
allows researchers to see how the visualisation software can be improved
and enhanced to make music inclusive for the deaf and hard-of-hearing
community. This allows the research, design, and use of high-quality music
visualisation to have social development and makes this area of design aim
for inclusivity for the enjoyment of music for all (Fourney & Fels, 2009).

With these studies all concluding similar outcomes, it would suggest that
any edits made to the video do have an impact on the viewer’s experience.
Playback experience, from speed to colour, does affect the viewer’s
emotional feelings. There is little to no literature to counter-argue this
point.

Design Probes 1.1

To further this design research on how graphic visualization affects our

relationship with a piece of music, new research on the matter will be
conducted.

16
With existing research and studies, the use of a design probe seems to be the
most appropriate for this task. By definition, a design probe is a set of
specifically designed items that are used to provoke emotion and reaction
from a set of people. These emotions and reactions can then be solidified
into quantitative data in relation to your topic (Murphy, 2006). An example
of this, which has been mentioned previously, is the 2016 study called
“Jockey Time” conducted by Hyeon-Jeong Suk and Kyeong Ah Jeong. This
Korean study took place in order to collect data for their software called
Jockey Time V1 which is a program to edit playback speed on specific
videos to generate a desired emotion for the viewer. To do this, they had a
group of students watch a short clip of a woman demonstrating specific
motions. The clip continued as the control to ensure consistent outcomes but
the speed of playback was used as the independent variable to collect the
required data. Participants viewed the original video at normal playback
speed and a second was played at various playback designs. They could then
watch these back simultaneously to help decipher what emotional outcome
the playback was inducing. This data collected then influenced how their
app would develop into version one of Jockey Time. A scale was made with
these results and can now be used to create various emotional outcomes
using the same video (Jeong, 2016). Using a design probe for this example
worked well as it converted emotional data into numerical to then use in a
real-life instance. This is similar to the new research being conducted as
both are taking emotional nonnumerical data and categorising it to use in the
design world.

Another example of a well-conducted design probe is the 2022 Dream

Machine exhibition. This probe is a travelling immersive exhibition
collecting data from visitor’s experiences. It takes inspiration from the
unfinished work of an inventor named Brion Gysin who explored the idea of
using colour and sound to create the first artwork to experience with your
eyes shut. He developed this revolutionary idea throughout his life but it
was cut short by his death until this data was then un-archived by a team of
researchers and designers. The use of the immersive idea that Gysin created

17
but on a much larger scale allows guests to experience the kaleidoscope
effect that the dream machine creates followed by questionnaires after.
Guests are asked to draw pictures, fill out surveys and share what they
experienced directly after coming out of the chamber. This data is then
collected and analysed by the team to help understand what the brain does
during this phenomenon of flashing lights and colour (Dream Machine ,
2022). The creative collection of data that this probe uses makes it desirable
for both the team behind the experiment and the guest itself which is
beneficial for all involved. Although the drawings are not numerical data,
these can still be categorised and visually shown to see if different brains
perceive different images.

Understanding Design Probes

As described in the paper, “cultural probes method in design research:

creativity in sketches”, design probing is a technique that develops an
understanding of a situation that can inspire what could be. It allows
information to be collected which is non-scientific and creative which can
help understand how an individual’s personal opinions and lives affect the
design world (Černevičiūtė & Liebutė, 2021). A valid issue that
Černevičiūtė and Liebutė highlight is the scientific mindset that some may
have when conducting a design probe. Unlike scientific methods which have
a desired, calculated outcome, this use of data collection is experimental and
should have no suggestive elements to it. Design is subjective and so is this
research method so a failed or different outcome from what was expected
should be desired. Design probes were invented to collect non-conforming
data for the design realm (Černevičiūtė & Liebutė, 2021).

Conducting a cultural or design probe in the early stages of the process is

important as it allows data to be used as inspiration. Rather than data
deciding what you should and should not do as a design outcome, it should
be used to help move forward existing ideas. It is suggested that a small
group of participants – ideally 5-10 – are required to avoid generalisation of

18
results with unexpected answers being welcome (Černevičiūtė & Liebutė,
2021). It is this unexpected, creative yet insightful data which is the
reasoning behind choosing a design probe for this research.

Method

This probe will consist of a specific song which has been chosen to act as a
control. This specifically, researched and chosen song will be presented to
two groups of respectively chosen people with no visuals accompanying it.
They will then be asked to fill out a short survey - see attached Figure 1
below. This survey will be followed by a visual shown alongside the music.
The two separate groups of participants will be shown two separately
designed visuals. Following this, another short survey as shown in Figure 2
will be completed which will note the participant’s emotional change – if
any – following the visualisation. The surveys, song, and demographic of
participants will stay constant to help ensure reliable results.

The song choice for this probe was vital and a considerable amount of
justification is required due to it being the constant. The song chosen had to
come from a specific genre of music so, to choose this an appropriate site
had to be acquired. With over 433 million active monthly users and
1,800,000 monthly uploads of songs, Spotify is the world’s leading audio
streaming service making it a credible source to use (Ruby, 2022). The
interaction and streaming that the 433 million users conduct, results in the
singer Ed Sheeran being the top streamed artist on the platform as of
October 2022. The use of Spotify and Ed Sheeran as points of contact for
the genre decision led to further investigation. Sheeran’s song “Shape of
you” is his most streamed song on the platform and is the most streamed
song on Spotify so taking the artist with the most streams and monthly
listeners gives a good foundation for picking a genre. Ed Sheeran falls into
the pop music category highlighted by Official Charts in their “Every

19
official UK number one ever” playlist. The description suggests that the
playlist, which contains thirteen songs featuring Ed Sheeran, is “the history
of pop music in one unmissable playlist” which then leads to the conclusion
that these songs would be in the pop music category (Official Charts, 2022).
It is for this reason that the genre of music which will be used in the probe is
pop.

Investigating what age demographic should be used has also been given
great consideration. Knowing what genre of music will be used, it would be
appropriate to ask who currently listens to that genre most consistently.
According to Music Gateway which is a music industry platform promoter
for new artists, the majority of younger people (13-30-year-olds) all have
similar music tastes. Music Gateway say that these people most likely will
be listening to mainstream, popular artists which play on the radio (Music
Gateway Team, 2016). Linking this back to the foundations behind the
genre of music, Ed Sheeran was the most-played artist on UK TV and Radio
in 2021 along with his song “Bad Habits” being the most-played track on
TV and Radio (Newsround, 2022). With the stated fact that 13 to 30 year
old’s listen to radio music and that the most played person they hear is Ed
Sheeran, it was concluded that it would be the age demographic of 13-30-
year-olds who listen to pop music most frequently who should be chosen to
complete this probe.

Now that a genre and age demographic have been decided, a specific song
has to be chosen. The obvious choice would be an Ed Sheeran piece
however, it was suggested that this may be too suggestive as Ed Sheeran is
well known. The participants may not be swayed by previous connections to
the song in question. To get around this issue a playlist was sourced which
was created by Spotify called “teen beats”. This playlist is a collection of
current pop songs which teens are listening to and with the reliable creator
of Spotify, it was believed to be a good reference point. The song from this
playlist must have a high BPM paired with sad lyrics to give contrasting
emotions to allow two different videos to be produced which could be used
to influence the emotion of the viewer. The song in question was “Miss

20
You” by Oliver Tree. Oliver Tree is a pop music producer and singer who is
#114 on the Spotify streaming chart as of October 2022 (Spotify, 2022). The
song has a high bpm of 145 and has energy of 74% but positiveness of 20%
(Music Stax, 2022). These contrasting elements to the song make it a
complex song to use for this probe.

Completed Probe

The method conducted took place in a large, empty room with the option to
be either light or dark. A large sheet of plastic was installed to allow the
video to be projected from behind to ensure the participants could stand
directly in front of the screen without blocking the projection. Two groups
of people were gathered with the first group having nine participants and the
second having eight. These groups had mixed gender and all were between
the ages of 13-30 as explained previously. On entering, participants were
asked to complete a consent form and were given a Design Probe 101
worksheet. They listened to the control clip of music and completed the
form. After this had been completed, the specific visual was played
alongside the music in the dark room. The first group were shown a visual
which was designed to induce happy, excited, and energetic emotions
whereas the second group viewed a visual which was designed to induce
anger, fear, and sadness. Following this, the reverse side of 101 (Design
Probe 102) was completed to allow comparison. Any questions were then
answered and the participants could then leave.

Successful and Unsuccessful Aspects of the Probe

The use of the room worked well in this case as it allowed for participants to
stay in one location to complete the entire probe. The plastic projection also
was beneficial in this case as it allows participants to move freely without
fear of blocking the screen. Keeping them in a comfortable location made
them more to open asking questions and taking their time filling out the

21
form. The simplistic form also assisted with this as it made it
straightforward for the person to complete. Using this simple setup made it
easier for all involved with the probe.

If this probe was to be set up again, there would be a few changes to be

made. The screen size would be taken into consideration and short throw
projectors could be used. Unfortunately, these resources were not available
when this was being conducted so the resources which were available had to
be used resulting in a smaller screen size than expected. This projection still
was around 1m x 1m however having it 2m x 2m would have been ideal.
Furthermore, the sound system used should be taken more seriously. Again,
due to a lack of resources, a small speaker was used but in future, a larger,
more powerful sound device should be used. It was seen that the sound
quality affected the way people were listening to the music clip therefore
better sound quality would have assisted the participant. Connecting the
speaker to the laptop would also be ideal as it would make it easier for the
probe conductor to align the music with the set video. It worked in this
instance however this could be an issue in future examples.

The worksheets which were completed can be seen in Appendix 1 and

Appendix 2.

Analysing Data

Subsequent to the method being conducted, the worksheets provide vital

data to analyse.

Using two groups to differentiate if the use of music visualisation has an

effect on the viewer, allows comparative results to answer this question.
After discussing the history of music visualisation, it’s clear that artists have
been doing this for many decades however, this longevity of skill does not
prove that it has an effect on those watching. We can hypothesise that these
results are going to suggest that the feelings towards the chosen song clip
will have changed proving a positive answer to the research question.

22
Appendix 1 and 2 show the results which were received from the probe.

Analysing Data – Group 1

As mentioned previously, Group One was given the suggested “happier”

clip to digest. This group consisted of nine participants all within the
targeted age group. After listening to the music clip with no visuals, 100%
of participants felt positive feelings toward the song. Some of the words
used were “funky”, “bouncy”, and “upbeat” which all receive positive
descriptions within the English definitions on dictionary.com (Dictionary.
com, 2022). This was an expected outcome from this due to the song’s high
BPM and lively nature. When asked what the clip was expressing, five
responses suggested that it would be for a clubbing dance environment. This
answer was not unexpected due to the fast pace of the song clip however, it
may have been given due to the subjects relating this song to those that they
hear in nightclubs. This means that this data was given due to their lifestyle
choice. The subject’s age makes them in the nightclub demographic
explaining this answer. Contrasting to this, four participants went against
their positive adjective and suggested that the song might be about a
breakup, ex, or negative feelings. No reason was given for these responses
but this outcome may suggest that this selection of participants was more
focused on the lyrics than the upbeat sound of the song. These answers may
have been given due to them going through a breakup during this time
period or perhaps them having negative emotions on that day. This 50-50
split of participants was an unexpected outcome for the probe as it was
previously thought that the subjects would have focused on the upbeat feel
of the song rather than the narrative. This contrast in findings links back to
Černevičiūtė and Liebutė’s findings on design probes and the want for
unexpected answers (Černevičiūtė & Liebutė, 2021). These results are
beneficial as they allow more contrast to be seen if the visual impacts the
emotional influence of the music. This unexpected turn for Group One
means interesting numerical data. From these first two questions, it can be

23
seen that 100% of emotions felt after listening to the song clip were
positive, however, 44.4% of participants in Group One suggested that
although the song made them initially happy, the content may be about a
negative and sad topic. The majority in Group One for this question did
contradict this and suggest that this was a dance song which would explain
the positive emotions felt. All numerical data from Group One can be seen
in Appendix 3-5.

Appendix 1 shows the drawings Group One responded to the song clip with.
This exercise was to help the researcher visualise exactly what they
subconsciously expected to see when listening to the song. From this, the
results show that five drawn responses highlighted a club dance scene. This
majority of drawings would further back up the initial question of feelings
towards the song. It would also solidify the response that the majority had
on question two about what they felt the song clip was about. Two had
more drawn movement responses. These responses took a more literal and
artistic response to the question of having wavy lines and shapes to
represent the song clip. One response saw the drawing of a breaking heart.
This would suggest that this participant was focusing on the sad lyrics rather
than the more upbeat BPM which the others have focused upon. The final
drawing showed a set of earphones which is the most literal drawing taken
from the group. This has no real emotion behind it so sits at a neutral point
for the drawing segment.

Appendix 1 with Group One results also shows the post-visual results on the
right-hand side. As said above after listening to the song clip without any
visual the respected group was then shown a visual alongside the same song
clip to, in theory, induce and enhance a specific emotion. The particular
visual for this group was designed to complement the upbeat BPM and push
the positive narrative that the song could potentially have. After the visual
was shown to the group, 100% of the emotions felt were positive with some
feelings such as “groovy” staying consistent with the previous side. Some
new adjectives were suggested such as “clubby” and “creative” however all
of the adjectives felt towards the song stayed positive. The second question

24
did change on the second side of the sheet to ask the participant if their
thoughts on what the song was about had changed. 77.7% of participants
said no and suggest that they were correct in the first side when it was put
forward that the song clip was about clubbing and dancing. One participant
explained that after the visual their feelings had changed and that it was “a
little more upbeat” than what they had previously thought. The only answer
which gave a strong yes to the question, explained that they thought the
song now was a fun mashup and that the visual added fun filters to the clip.
The final segment for this group was the drawing at the end of the second
side. This segment really helped to visualise to the researcher if the visual
thoughts of the participant had changed in any way. From this, we can see
that seven participants drew a dance or club scene. This advises happy
emotions towards the clip and visual. The remaining two participants drew
more illustrative responses with shapes and lines. These creative responses
appeared to take a happy narrative due to the stars, colours and dancing
person. These drawn responses emphasise and solidify the group’s answers
for this round.

Analysing data - Group two

Following Group One, Group Two were given the same task but with a
differently edited video designed to conjure negative emotions. The answers
given by this group can be seen in Appendix 2. After round one, adjectives
like “vibey” and “happy” were given similarly to in Group One however,
there were also words like “alert” and “explosive”. 25% of words given to
describe the song clip have negative connotations which are double that of
the previous group. To further develop this, seven of the eight responses
suggested that the song clip was about a breakup or losing a relationship.
This negative response towards the clip was an increase from the previous
group even though the conditions in which this was conducted were
identical. One response from Group Two informed that they didn’t have any
idea what the song was about as they didn’t listen to the lyrics. This was an

25
interesting response as in Group One if a participant didn’t listen to the
lyrics they assumed that the song was happy and upbeat but this was not the
case with Group Two. This development is something which needs to be
further investigated. In addition to this contrasting result, the pictures drawn
seem to be of a more mixed outcome. Four drawings show something in
relation to dance or music which is less than Group One. Two have people
running away from something, one shows someone crying, and another
shows someone counting sheep. Unlike in Group One, there were no two
drawings which were the same which is what was expected unlike in Group
One where three of the drawings were very similar. These results provided
interesting data to explore. From these drawings, it can be seen that three
impressions of the song’s narrative are undoubtedly positive, two are
negative and three are neutral. These neutral drawings are categorised like
this as the participant has just provided a line drawing or two contrasting
images that can be seen within one box. These interesting results provide a
good starting point as this means that the visual should streamline the
participant’s emotions into feeling the same way towards the song clip.

Following Design Probe 101, Group Two was given the designed “angry”
visual to accompany the music clip. The effects of the visual clip were
noticeable immediately from the answers given. Having such varied
answers from the initial side of questions makes it easier to distinguish if the
clip had or had not made a difference. Following watching the visual, seven
of the adjective responses suggested that there was an angry or negative
emotion towards the clip. The remaining answer was “intrigued” so was
classed as neutral as this has neither good nor bad feelings associated with
it. This group saw a larger change in emotion as before the visual, five had
positive emotions whereas after zero positive emotions were felt.
Furthermore, when asked to explain if their view on the music had changed,
seven of the responses suggested that it had and that the song had a stronger,
angrier message now. This proves that the visual did have an effect on the
viewer’s response to the audio clip. The one remaining response explained
that the visual did not change their presumption of the audio but it had

26
amplified the angry story which went alongside it. This also voices the
narrative that the visualisation did help convey the emotions of the audio
clip. All eight responses of the illustrative round saw drawings with
negative and angry illustrations. One of the responses saw the same drawing
as 101 however this time the figure was carrying a knife which backs up the
angrier motive that the visual was to provide. Swearing, shouting, and
violence are seen in the majority of the drawings in 102 which was not the
case in 101. From the content of these drawings, it is shown that the video
used within the visuals is of importance. It was unsure if the viewer would
digest the visual matter or not however it has been seen that the drawings
created in the probe have some elements of the visual within such as a
punch, umbrella or even seen in Group One with the dancing policeman.
This information is important to note if replicating as the designer must
think about archived footage if it is to stick with the viewer after the probe.

What does this mean?

After looking at the results from both groups a clear difference can be seen
from the independent variable of the visual accompanying the music.
Interestingly, when combining the two result groups, you can see the
adjectives used to describe the song clip all begin the same with 14 being
positive and a small three being negative across the two groups. This makes
for a good starting point as it is clear to see the change made by the visual.
This data across the two groups would possibly suggest that when someone
is hearing a clip of music for the first time, they tend to focus on the backing
noise and beat rather than the lyrics. This gives reason for this entire
research as it proves that to portray the correct or certain emotion or
narrative that a song wants to deliver, visuals and graphics should be used.
The more eclectic group of Group Two highlighted stronger evidence to
support this claim. The change of emotion saw a 75% increase in negative
feelings after watching the visual which was designed to push this narrative.
To back this up as said before, the visual drawing aspect of the probe further

27
allows the research to portray how the individual is feeling. To contrast
these findings, Group One saw a minimal change in the emotions felt
towards the song. This is obviously due to the overall sound of the song
matching the happy visual alongside the participant’s existing feelings
which were seen in question one. The fact that there was no change to the
participant’s emotions proves that the visual did have an effect as they
continued to feel happy and lively towards the music clip. These emotions
backed up again by the drawings completed by the participants extensively
proves that visuals do have an effect on the viewer’s experience.

In Group One, three of the drawings were almost identical along with a lot
of people agreeing on a certain answer when writing this down. During the
conduction of the probe, talking to other participants was allowed as it was
done in a large classroom so small chatter was occurring to stop it from
being so intimidating. This chat I believe has made some sheets biased
within the groups, especially Group One. The participants were asked to
write what they personally felt however this could have been easily copied
if a person was confused about what the question meant or several other
factors that may affect the answers given. The similar drawings which
people had given were the most interesting as these are difficult to copy and
replicate, a word or sentence is easy to copy but an entire club scene is a
little more challenging. This bias in answers was not seen in Group Two
suggesting that a collection of people in Group One were comfortable
enough to discuss answers whereas this was not the case in Group Two.
Suggestibility and bias were something that was highlighted earlier and
precautions were put in place to prevent this however it has been discussed
that if talking was prevented during the probe, would more reliable answers
be given.

To design and beyond

From this data, real-world applications and developments can be made. The
most obvious is connecting back to the software Jockey Time. This program

28
used data similar to this to create software where users can place video into
a chart to change the playback speed to then create an emotion for the
viewer. This manipulation of time focused on the playback speed entirely
rather than the video content. It is now known that the content of the video
is valid alongside all other factors. This could mean that an interesting
software taking inspiration from Jockey Time may be developed. Instead of
a scale interpreting speed, it could give clips of footage that we know sends
a particular emotion (Jeong, 2016).

Another design application of this research would be the development of

inclusivity in the music world for hard of hearing and deaf people as
discussed by Fourney and Fels. Using the data acquired that graphic
elements can portray the music narrative thus manipulating the viewer’s
emotion allows the conclusion to be drawn that visuals can be used to
enhance the music for this group of people. Making visually powerful
accompaniments can allow the deaf and hard of hearing to enjoy the
beneficial community that music provides in western culture. Developing
software and design solutions to allow this makes music a more inclusive
space for all instead of isolating particular groups of people due to disability
(Fourney & Fels, 2009).

Conclusion

Does music visualisation have the power to influence the viewer? From
structured research on this topic, the educated answer would be yes. Having
the conducted design probe highlight how a group can be influenced by
specifically designed videos alongside the rich history and interest in the
connection of music and colour, it can be confidently said that combining
designed visualisation with music will influence the viewer to a specific
narrative. This research can emphasise the need for relationships between
musicians and designers to ensure the musician’s art is being told in the
correct way. Furthermore, this enlightens the design community on how
design can make music more inclusive for the deaf and hard-of-hearing

29
community. Using this research with real-life applications is necessary as it
has been proven that it can be of use to several communities within the
design and music principles.

30
References
Aldighieri, M. (Director). (2021). VJ Diaries [Motion Picture]. NYC.
ČERNEVIČIŪTĖ, J., & LIEBUTĖ, L. (2021). CULTURAL PROBES METHOD IN
DESIGN RESEARCH. Creativity Studies.
Derkert, P. (2006). Emotions through Graphics Depicting Biometrical Sensor
Data. Stockholm: Swedish Institute of Computer Science .
Dictionary. com. (2022, December 10). Dictionary.com. Retrieved from
Dictionary.com: https://www.dictionary.com/
Dream Machine . (2022, 11 05). The Science and Philosophy of Dreamachine.
Retrieved from Dream Machine World:
https://dreamachine.world/2022/05/23/the-science-and-philosophy-behind-
the-dreamachine/
Earnshaw, R., Liggett, S., Excell, P., & Thalmann, D. (2020). Technology, Design
and the Arts - Opportunities and Challenges. Springer Cham.
Fourney, D. W., & Fels, D. I. (2009). Creating access to music through
visualization. Toronto: IEEE Toronto International Conference Science
and Technology for Humanity (TIC-STH).
Gibson, S. (2022). Live Visuals - history, theory, practice . London: London :
Routledge.
Hall, D. (2020). Graphic notation: a brief history of visualising music. Retrieved
September 14, 2022, from https://uxdesign.cc/the-art-of-visualising-music-
a-brief-history-of-graphic-notation-729cff32853b
Jeong, K. A. (2016). Jockey Time: Making Video Playback to Enhance Emotional
Effect. ACM digital libary .
Kim, H., & Park, M. (2015). User Emotions on Moving and Fading Patterns of .
Seoul: Seoul National University of Science and Technology.
McDonnell, M. (2002). Colour and Sound. Maura McDonnell.
Murphy, J. (2006). UXPA Magazine. Retrieved 11 04, 2022, from
https://uxpamagazine.org/cultural_probes/
Music Gateway Team. (2016, August 15). How is Music Marketed to Different Age
Groups? Retrieved from Music Gateway :
https://www.musicgateway.com/blog/how-to/how-is-music-marketed-to-
different-age-groups
Music Stax. (2022, November 07). Music Stax - Miss you . Retrieved from Music
Stax: https://musicstax.com/track/miss-
you/73vIOb4Q7YN6HeJTbscRx5#:~:text=Miss%20You%20is%20a%20s
ong%20by%20Oliver%20Tree%2C,has%20a%20duration%20of%203%2
0minutes%2C%2026%20seconds.

31
Newsround. (2022, June 20). Ed Sheeran is most-played artist on UK radio and
TV. Retrieved from BBC Newsround:
https://www.bbc.co.uk/newsround/61863083#:~:text=Most%20played%20
artists%20of%202021%201%201.%20Ed,Justin%20Bieber%208%208.%2
0Calvin%20Harris%20More%20items
Official Charts. (2022, November 7). Every official UK Number 1 Ever. Retrieved
from Spotify : https://open.spotify.com/playlist/5GEf0fJs9xBPr5R4jEQjtw
Plourde, M. (2017). How to Read & Write Graphic Notation (part 1) - Music Stuff
With Spock #8. youtube.
Ruby, D. (2022, 09 16). Spotify Stats 2022 (Facts & Data Listed). Retrieved from
Demand Sage: https://www.demandsage.com/spotify-stats/
Spotify. (2022, November 06). Spotify most played all time. Retrieved from
Spotify: https://open.spotify.com/playlist/2YRe7HRKNRvXdJBp9nXFza
Spotify. (2022, November 07). Spotify Web Player. Retrieved from Spotify:
https://open.spotify.com/artist/6TLwD7HPWuiOzvXEa3oCNe

32
 Appendix

Appendix 1 – Results from Group One Design Probe

Appendix 2 – Results from Group Two Design Probe
Appendix 3 – Table showing % results of emotional state
Appendix 4 – Table showing number of participants who felt a
particular emotion through illustrations
Appendix 5 – Table showing number of participants who felt a
particular emotion towards the songs narrative
Appendix 6 – Consent forms for Design Probe

33
 Appendix 1 Group One Results

34
 Appendix 2 Group Two Results

35
 Appendix 3 – Table showing % results of emotional state
Group One Group Two
Positive Negative Neutral Positive Negative Neutral
Before Visual 100% 0% 0% 62.5% 25% 12.5%
After Visual 100% 0% 0% 0% 87.5% 12.5%

Appendix 4 – Table showing number of participants who felt a particular

emotion through illustrations
Group One Group Two
Positive Negative Neutral Positive Negative Neutral
Before Visual 7 1 1 4 3 1
After Visual 9 0 0 0 8 0

Apprndix 5 – Table showing number of participants who felt a particular

emotion towards the songs narrative
Group One Group Two
Positive Negative Neutral Positive Negative Neutral
Before Visual 5 4 0 0 7 1
After Visual 7 2 0 0 8 0

36
Appendix 6 – Consent forms (ranging page 36 – 52)

37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53