Landmeen 2023 Guidelines
Landmeen 2023 Guidelines
Landmeen 2023 Guidelines
Overview
In this task, you will listen to pairs of audio segments. Each pair will consist of
one German segment and one English segment.
Our goal is to know how similar the two segments (utterances) are perceived in
terms of:
Semantics/Meaning
Emotion
Rhythm
Overall expressive intent
Different languages have distinct speech patterns related to the aspects mentioned
above. When comparing expressivity in different languages, we want to determine if
the expressive qualities in German convey similar information as in English.
By “overall expressive intent,” we mean the overall impact and manner in which the
speaker spoke the sentence. To rate similarity in expressive intent between two
audio files, consider aspects like emphasis, tone, rhythm, and the speaker's
emotional state combined.
Example:
The sentence “There is a green apple” in English has a different meaning from “Hay
una manzana roja” (“There is a red apple”) in Spanish.
2. The two segments are mostly different in their meaning but share some
similarities - there are some important differences in the meaning of the two
segments, although one or more objects, actions or concepts may appear in both
sentences.
3. The two segments are mostly similar in their meaning but have some differences -
they could be paraphrases of one another.
4. The two segments are completely similar in their meaning - they are exact
translations of one another.
Emotion
Emotion describes the overall feeling of the speaker while they are talking.
Example:
A speaker may sound angry, pleased, happy or confused (to name just a few emotions)
while speaking. Consider whether you could imagine the two speakers making similar
facial expressions while speaking or whether you could apply the same description
of their emotions.
Question: Do the two segments sound similar in the speaker’s emotional state?
Score (single choice):
1. The two segments sound completely different in the emotions conveyed - basically
none of the emotion aspects are shared.
For example, while one utterance might sound very happy throughout, the other
utterance might sound neutral throughout.
2. The two segments are mostly different but share some similarities in terms of
the emotion.
For example, while one utterance might sound happy throughout, the other utterance
might sound neutral throughout and happy just at the end.
3. The two segments are mostly similar in the emotion but have some differences.
For example, both utterances might share the same emotion or mix of emotions, but
the emotions are more pronounced in one compared to the other (one segment sounds
very happy while the other is subtly pleased).
4. The two segments sound completely similar in the emotions conveyed - basically
all of the emotion aspects are shared.
For example, both utterances sound very happy to the same extent, and this is
expressed similarly throughout.
3. Rhythm
The rhythm of an utterance describes its speed, pacing (i.e. changes in speed), and
pauses. A speaker pausing or elongating/shortening words can impact rhythm.
Example:
“You -- lied to me?” Having a pause after “you” is distinct from "You lied to --
me?" having a pause after “to.” A speaker speaking quickly or slowly throughout the
sentence or speeding up/slowing down at certain parts of the sentence, also impacts
rhythm.
1. The two segments sound completely different in their rhythm - basically none of
the rhythmic aspects are shared.
For example, one utterance may be spoken slowly at first, have a pause in the
middle, then faster at the end, while the other utterance is spoken in a normal
cadence throughout.
2. The two segments are mostly different but share some similarities in their
rhythm.
For example, one utterance may be spoken slowly at first, have a pause in the
middle, then faster at the end, while the other utterance may be spoken normally at
first and faster at the end without the pause in the middle.
3. The two segments are mostly similar in their rhythm but have some differences.
For example, one utterance may be spoken slowly at first, have a pause in the
middle, and then faster at the end, while the other utterance is spoken virtually
the same, except without the pause in the middle.
4. The two segments sound completely similar in their rhythm - basically all of the
rhythmic aspects are shared.
For example, one utterance may be spoken slowly at first, have a pause in the
middle, and then faster at the end, and the other utterance has the same pattern.
4. Overall expressive intent
The overall expressive intent of an utterance is the combined feeling of the
rhythm, emotion and any additional factors (such as emphasis and intonation) which
give rise to the utterance’s overall impact and implications. When comparing the
expressive intent across different languages the idea is to assess whether the
expressive qualities of the {LANG_1} utterance convey equivalent (or as similar as
possible) information as the expressive qualities of the {LANG_2} utterance.
Question: Considering the overall expressive intent of the two utterances, how
similar are they?
Score (single choice):
The two segments are completely different in their overall expressive intent - the
information conveyed through the expressive features and speaker emotional state is
different.
The two segments are mostly different across expressive aspects but share some
similarities.
The two segments are mostly similar across expressive aspects but have some
differences.
The two segments are completely similar in their overall expressive intent.
Task description
Listen to audio 1 from start to finish. Then listen to audio 2 from start to
finish.
Provide your similarity scores on all dimensions as explained above
Consider the following:
If either of the segments is very garbled or unclear, please check the box “audio
issues” and skip the item.
If the segments have the same or similar content, but one has additional content
relative to the other, please only consider the content shared between the two
segments. If the difference in the amount of content is greater than a few words,
please move to the question on Semantics and select a score of 1 (“Completely
Different”). Where this occurs, you will not be answering questions related to the
other expressivity dimensions.
If one or both of the segments has leading or trailing silence, please ignore this
and try to focus on spoken content only.
Two segments can be similar in the presence or absence of the aspects of interest.
That is, if two sentences are both equally neutral in any of the categories, we can
also consider them to be “similar.” For example, we would consider two segments as
being similar in emotion if they were both spoken in a “neutral” tone.
Please try to rate the similarity independently of the speakers’ voices. For
example, in some cases the source audio may be in a conventionally female voice
while the target audio may be in a conventionally male voice. Try your best to
focus on how the sentence is uttered in terms of the expressive intent, as outlined
above, irrespective of the voice differences.
Examples
Example ratings for Question 1 (Semantics)
Score
Utterance 1
Utterance 2
1 (completely different)
The rocket had a coupling issue preventing atmospheric clearance.
J'ai vu une pomme de terre en allant travailler aujourd'hui. / I saw a potato on
the way to work today.
2 (mostly different)
The rocket had a coupling issue preventing atmospheric clearance.
Le tuyau présentait une fissure provoquant un blocage. / The pipe had a crack in it
causing some blockage.
3 (mostly (similar)
1. The rocket had a coupling issue preventing atmospheric clearance.
Examples:
*Note: changes in speed will be demonstrated by increasing the spacing between
items.
**Note: this is a simplification (reduction in dimensions to just changes in speed)
to allow for a clearer understanding of the underlying task.
Score
Utterance 1
Utterance 2
1 (completely different)
I didn’t know how to get there.
(Slow-normal-slow)
Yo no sabia como llegar ahí. /
I didn’t know how to get there.
(Fast)
2 (mostly different)
I didn’t know how to get there.
(Slow-normal-slow)
Yo no sabia como llegar ahí. /
I didn’t know how to get there.
(Normal pace)
3 (mostly (similar)
I didn’t know how to get there.
(Slow-normal-slow)
Yo no sabia como llegar ahí. /
I didn’t know how to get there.
(Slow-normal)
4 (completely similar)
I didn’t know how to get there.
(Slow-normal-slow)
Yo no sabia como llegar ahí. /
I didn’t know how to get there.
(Slow-normal-slow)