SDL Post-Editing Certification

Machine Translation Post-Editing

Contents
SDL Post-Editing Certification 3
Introduction 3
What is the SDL Post-Editing Certification? 3
Why become a Certified Post-Editor? 3
What is Post-Editing? 4
Why is Machine Translation so relevant? 4
Translators and Technology 5
Evolution of Machine Translation 6
Machine Translation Developments over Time 6
Machine Translation in SDL 8
Machine Translation and Post-editing as Part of the End-to-End Localization Process 9
The Impact of MT Developments on the Localization Industry 9
The Right Solution for the Right Content 9
Content Evaluation 11
MT Technologies in Detail: RBMT, SMT, NMT 11
MT Engine Creation 16
MT Output Evaluation and Testing Methodologies 19
Using the MT Output: Machine Translation Post-Editing 21
Levels of post-editing 25
Expected MT Behaviors 28
Providing Feedback to Improve the MT Output 31
How to Add Machine Translation in SDL Trados Studio 32
Post-Editing as Opportunity 37
Conclusion 40
Appendix 1: Detailed Information on Automated MT Evaluations 41
Appendix 2: Examples for Expected NMT Behavior 43
Appendix 3: Language-specific Phenomena 45
Appendix 4: Multilingual NMT Examples for Different Industries and Content Types 53
Appendix 5: Resources for Further Learning and Information 55

About SDL 56
SDL Post-Editing Certification

Introduction

What is the SDL Post-Editing Certification?

The SDL Post-Editing Certification has been developed for translators who are interested in adding
post-editing (PE) to their portfolio. Our certification will provide both graduate and established
translators with the tools and knowledge they need to become successful post-editors. Originally
launched in 2014, the post-editing certification course has helped to establish PE as a key task for
skilled linguists in answer to the growing demand for post-editing within the industry.

Why become a Certified Post-Editor?

Embarking on a career as translator suggests that you have an aptitude for languages, are passionate
about communication and enjoy engaging with different cultures. Translation degrees take several
years of study and dedication. Having made this kind of commitment, why should you consider
adding post-editing to your portfolio?

Over the last decade, PE has become a mainstream choice for enterprises wanting to provide their
content in different languages to a global audience. The emergence of a third-generation Machine
Translation (MT) technology – Neural Machine Translation (NMT) – has led to renewed interest in
MT and post-editing.

Technological improvements and commercial demand for MT and PE mean that this area will
continue to grow. Adopting PE is of course a choice that everyone needs to make for themselves,
but with new technologies like Neural MT benefitting translators, this is not a choice to be made
lightly.

Holding a certification evidences you have a professional mind-set towards achieving excellence in
your skillset. Keeping your certification up-to-date demonstrates that you also put personal
development at the top of your list, and are willing and capable of embracing technology, developing
new behaviors and knowledge to support your skillset.

Our aim is to share our own experiences with MT and PE through the Post-Editing Certification
program and embark on this journey together for our continued learning and evolution.

Our Certification program offers an introductory course about the techniques and skills required for
post-editing MT output, with post-editing best practice recommendations and practical examples.

The course explains known MT behaviors and common MT errors and equips translators with core
skills and strategies for effective post-editing. Armed with this knowledge, new post-editors are
ready to experience post-editing first-hand, meeting the demands of a changing market and future-
proofing their careers.
What is Post-Editing?
There are a number of definitions to explain the term post-editing, with some insightful
interpretations listed below.

• The “term used for the correction of machine translation output by human linguists/editors”
(Veale and Way 1997)
• “…checking, proof-reading and revising translations carried out by any kind of translating
automaton” (Gouadec 2007)
• “In basic terms, the task of the post-editor is to edit, modify and/or correct pre-translated
text that has been processed by a machine translation system from a source language into
(a) target language(s).” (Allen 2003)

In summary, post-editing represents the human intervention for editing the output of a machine
translation system.

In a translation environment, it is important to distinguish between post-editing and review tasks.

Post-editing is not the same as review; the task of post-editing replaces the translation stage. A
separate review stage follows the post-editing stage. Reviewers need to bear in mind that correcting
machine-translated output is different from reviewing human translations as the post-editor has to
actively work with the machine-translated output. The task of post-editing will be discussed in more
detail in this guide.

Why is Machine Translation so relevant?

Machine translation remains one of the most important topics in the translation industry. The sheer
volume of data that is created daily is astonishing and there are simply not enough translators to
process all this content. Massive information flows govern all aspects of our life. In 2016 and 2017
alone, more data was created than in the whole of human history before.
Part of this content explosion is based on social media and social networking as business drivers. For
instance, all of us now commonly look at user reviews when we make hotel reservations or buy
consumer products. Content-driven enterprises require speed and agility to deal with increasing
content volumes and number of languages.

The content explosion and the popularity of social media have given rise to free multilingual
machine translation services which provide instant translations. This is a utility that digital users
expect. It is estimated that around 800 billion words per day are translated through global public MT
portals.

To be successful, translators need to understand the impact that technology has on their career and
their work. This is especially important in the context of the popularity of Neural Machine
Translation. NMT research has advanced considerably in recent years and the results of this new
approach to MT are extremely encouraging. We have seen from our own research that NMT
consistently outperforms previously used technologies across languages and industries, bringing MT
quality closer to human translation benchmarks.

This recent jump in technology gives a renewed relevance to the use of MT as a tool in the industry.

Translators and Technology

The translator profile, in particular that of technical translator, has changed quite dramatically in
recent years and is still evolving to keep up with new translation and technology developments. In-
house and freelance translators alike are required to work with the latest CAT tools, such as Studio,
and increasingly also client-owned tools or platforms. Translators perform a wide range of tasks,
which include but are not limited to translation, post-editing, review, linguistic sign-off, terminology
management and many more.
Post-editing is a relatively recent addition to the list of tasks that translators are asked to perform.
Machine translation is undoubtedly one of the more disruptive technologies to enter the localization
industry and is one of the main drivers for change in the industry today. The use of MT has been
controversial in the past however we need to place MT, especially Neural Machine Translation into
the much larger context of Artificial Intelligence (AI). Increased automation is all around us, be it the
research into self-driving cars, virtual assistants such as Amazon Echo or Google Assistant or medical
diagnostics steered by artificial intelligence. The constant developments in the field of AI or Machine
Learning (ML) mean that translators need to take a pragmatic view and expand their knowledge to
adapt to the fast-changing requirements. One such requirement is the need to offer post-editing as
part of a complete skills portfolio.

It is, however, important to remember that MT does not replace the need for human translation and
human translators. PE is the process of allowing machines to do the heavy lifting of translation, with
editing and quality assurance being performed by trained translators. MT is an effective tool to assist
translators in their daily work.

Evolution of Machine Translation

Machine Translation Developments over Time

To understand the importance of MT, it is helpful to step back and look at MT technology
developments over the last century.

Following on from the efforts made in cryptography during World War II, MT is generally considered
to have started in the 1950s. In 1954, the successful execution of the Georgetown experiment – the
fully automated translation of approximately sixty Russian sentences into English – ushered in an era
of significant funding for MT research in the USA. Researchers believed they could produce a fully
automated MT system within three to five years. This endeavor proved more difficult than expected,
however, and 10 years later funding was cut when it became clear that the development of MT had
not progressed as far as originally hoped.

Early attempts at MT typically failed due to a lack of coverage. The models functioned by encoding a
limited selection of a linguistic-based set of rules, which simply did not provide for the diversity of
natural language translation. Consequently, the first attempts to commercialize MT in the 1970s and
1980s operated by drastically increasing the number of encoded rules. This produced Rules-Based
Machine Translation (RBMT), which functioned relatively successfully with targeted human feedback
over a particular domain. However, this led to the further problem of how to make the huge number
of linguistic-based rules needed to encode language pairs cooperate with each other. The answer
was a statistical approach to MT.

In the late 1980s, computational power increased and became less expensive. As a result, interest
picked up in Statistical Machine Translation (SMT). From the 1990s, statistical learning approaches
came to the fore, led by cutting-edge work from the research team at IBM. SMT systems no longer
required the same human effort to encode linguistic-based rules and update lexicons and
terminology lists, but instead exploited the wealth of existing translations, covering numerous
language pairs, to extract rules based on statistical probability. In short, SMT systems create
statistical models by analyzing large sets of bilingual data. These models are then used to generate
the target MT output. SMT systems automatically learn from the data sets used for training and are
an early example of Machine Learning.

In the 1990s, SMT had been pushed forward through intensive research and training as well as
support from industry, the US Defense Advanced Research Projects Agency (DARPA) and the
European Commission’s FP7 program. Statistical MT was deployed in real-world, commercial
contexts by SDL, Google, Microsoft and IBM, alongside ongoing research in the field of statistical and
hybrid MT. In 2011, SMT was boosted with Google’s announcement that it would charge for access
to the Google Translator API. Shortly afterwards, Microsoft also announced that it would start
charging for use of the Microsoft Translator API. These two events can be viewed as a key milestone
for the machine translation industry and the localization industry as a whole. The progression to a
paid API model for machine translation was a clear sign that the use, spread and quality of MT had
matured to a level where enterprises and developers saw sufficient value in MT to invest in it.

For a number of years, SMT remained the leading machine translation system in the industry.
However, in the background, companies such as Google started working on a new approach to MT.
In September 2016, Google announced the development of the Google Neural Machine Translation
system (GNMT). By November, Google Translate began using NMT in preference to its previous
statistical methods which had powered Google Translate since 2007. Since then, NMT has become
the machine translation technology of choice across the industry, not only for leading web
translation portals such as Google and Microsoft, but also for Language Service Providers (LSPs).

A number of LSPs have decided to partner with specialist MT providers, however at SDL we continue
to benefit from developing our own NMT solution. We have a dedicated MT Research and
Development (R&D) team that has published over 200 peer-reviewed MT research papers and has
been cited more than 30,000 times in different research publications.
Machine Translation in SDL
As a company, we have been at the forefront of machine translation for over 15 years. We first
invested in a Rules-based Machine Translation system in 2004. This allowed us to use MT as part of
our translation process and offer post-editing as a service at a very early stage in MT developments.
RBMT was really the first attempt at automated translation. A rules-based system uses of a set of
linguistic and grammatical rules together with a dictionary. Translation rules are encoded manually
and tuning and maintenance for individual language pairs is very time-consuming.

In 2009, Statistical Machine Translation (SMT) started to establish itself firmly as a contender in the
localization industry, following rapid development. In line with progress in the industry, SDL started
to invest in the development and deployment of SMT technology. A Statistical Machine Translation
system learns to translate by analyzing large volumes of previously translated content. A large
database of aligned source and target texts is entered into a statistical learning system. On the basis
of these examples, the system creates an engine for automated translation. In essence, the system
learns how to translate by analyzing the statistical relationships between source and target data. In
fact, SMT was the first time machine learning was used to solve the problem of machine translation.

With the advent of Neural Machine Translation (NMT), we are now witnessing the dawn of another
technological shift in machine translation. We are still using machine learning to solve the machine
translation puzzle but this time, the actual architecture of the algorithm is inspired by the human
brain. As mentioned above, Neural Machine only entered the frame a relatively short time ago. The
principle itself has been around for over 4 decades but it is very resource-heavy and technology only
allowed us to introduce it recently.

As the technology evolves, SDL continues to implement MT for post-editing. Training materials,
testing procedures, best practices and collaterals need to evolve in parallel to make this
implementation possible.
Machine Translation and Post-editing as Part of the End-to-End
Localization Process

The Impact of MT Developments on the Localization Industry

Now more than ever, language is a business requirement. Post-editing has proven to be one of the
most effective ways to meet the localization quality demands of our times. In the localization
industry, MT is traditionally used to reduce cost and increase efficiency. MT and PE allow more
content to be translated and for turnaround times to be reduced.

Time, cost and efficiency savings depend very much on the quality of the MT output. This means that
output quality improvements are key due to the high impact on cost and efficiency.

When implementing the use of MT, how can we successfully manage both client expectations and
translators’ needs? We try to answer this question in the next section.

The Right Solution for the Right Content

At the heart of this question lies content. In order to implement MT successfully we must ask what
content needs to be developed and translated and what the purpose of this content is. The second
question is how to bring this content to the end customer in the most timely and cost-effective way
possible.

As businesses change and evolve, so does content, which creates the need for constant learning.
This means that translators will need to continuously learn new concepts, follow industry trends and
meet challenging deadlines in order to produce language materials that are fully adapted to local
markets and audiences. The challenge for an LSP is to understand all the content that our clients
create and determine how to best deal with it.
Deciding what content can successfully be translated with MT and PE is not only a linguistic but also
a commercial decision. Different content types require different translation approaches. Content
that requires a more nuanced approach may not be suitable for an MT process. This is where the
skills of human translators come into play.

In today’s localization market, there are four main translation use cases. Three of them involve
machine translation and two of them involve some type of post-editing:

• Human translation.
• Full post-editing or post-editing to publishable level: MT output is post-edited by
professional linguists to a quality level equivalent to conventional translation. Post-editing
MT content is the preferred solution for publishable documents. It is used as part of a high-
quality translation process.
• Light post-editing or post-editing to understandable quality: MT output is post-edited to a
level suitable for an acceptable and actionable translation, not with perfect grammar and
style. This is a viable option for perishable and low-visibility content.
• Raw MT or FAUT (Fully Automated Useful Translation): MT output is used directly, with no
human intervention. This solution is used mostly for content such as emails, support content
or instant messages, where the user wants to have an idea of the content for gisting
purposes, without the need for high quality.

The graph below shows which content has traditionally been considered suitable for MT. Post-
editing has worked well for structured, consistent and formally authored content. The quality
improvements associated with NMT allow for the expansion of PE to content that was previously
considered unsuitable. On the other side of the scale, quality improvements mean that more
content may be considered for raw MT without human involvement.
Content Evaluation
A careful content evaluation should be at the start of every successful post-editing implementation.
A content audit will help determine whether MT and PE are the right solution for the content in
question. A content audit surveys essential assets and project requirements to determine PE
suitability. During an audit, current and future needs will be assessed on the basis of:

1) General content suitability and the chances of success based on language, domain and
content type
2) Quality requirements
3) Volume requirements
4) Available corpus to train MT engines
5) Proposed future content (required when architecting a long-term solution)
6) Technical integration requirements for the production workflow (CMS, CAT tool, TMS)

PE is often applied to content that is both strategic and high volume with tight turnaround times. As
part of the wider process, it is critical for the evaluator to identify and record not just the content
types where PE has been successful but also where it has not met project goals. Collecting feedback
from translators is critical for this to happen. A continual feedback loop will eventually increase the
speed at which evaluations take place and help predict translation outcomes more accurately.

MT Technologies in Detail: RBMT, SMT, NMT

As mentioned in the section “MT Developments over Time”, there are three main MT approaches:

 Rules-based Machine Translation (RBMT)

 Statistical Machine Translation (SMT)
 Neural Machine Translation (NMT)

What is the difference between the three approaches?

Rules-Based Translation

Chronologically speaking, Rules-based Machine Translation was the first approach to automated
translation. RBMT uses a linguistics-based set of rules in combination with a dictionary. This is also
known as knowledge-driven MT. A language pair is built by looking at the construction of both
source and target, taking into account source and target grammar and vocabulary. This is a transfer-
based approach with three translation phases: analysis, transfer and generation. It involves parsing a
source sentence, analyzing the structure, converting this to a machine-readable code and then
transforming it into the target.

The core system is based on a set of grammatical rules for each language pair, combined with a
dictionary. The dictionary contains source words and phrases, their translation and detailed
grammatical information, such as the part of speech and inflection. It provides the modules with the
linguistic knowledge they need.

The rules are the “linguistic processor” of the system, responsible for analysis and generation. They
use linguistic information stored in the dictionary. These rules are intended to represent the
grammatical knowledge of speakers and specify inherent agreement and relational information.

At the translation stage, the MT engine analyzes each source sentence and tags the words and
phrases with their part of speech to identify grammatical components (for example, the subject,
object and verb). The MT system then looks up the translations of these grammatically tagged words
and phrases in the machine dictionary and combines them using the coded language rules for the
target language. This builds the translated sentence.

A large core dictionary provides the translations for everyday words and phrases. For translations
that use specific terminology, an RBMT system can use customized dictionaries in conjunction with
the baseline to improve translation accuracy.

Statistical Machine Translation

A Statistical Machine Translation system learns to translate by analyzing large volumes of previously
translated content. How does this work? A large database of aligned source and target texts is
entered into a statistical learning system. On the basis of these examples, the system creates an
engine for automated translation. In essence, the system learns how to translate by analyzing the
statistical relationships between source and target data.

The starting point for training an engine is an aligned corpus of source and translated sentences
containing hundreds of millions of words. The training process subdivides each of the source
sentences into words and series of words (n-grams) and analyzes the associated translated
sentences. In this way, the training process determines the most likely set of translations for each n-
gram in the source. By analyzing the translated content, the training process learns the order in
which the translated words are most likely to occur. The more training data and the more
consistency there is in the data, the more accurate the process becomes.
In the next stage of the process, the system compiles the learned data into the runtime MT engine.
The runtime MT engine subdivides each sentence into smaller chunks and looks up the possible
translations in the compiled database. For a given source sentence, this process results in many
possible translated sentences. The MT engine uses the statistical data on the probability of a
translation and the word order to determine the best candidate for the MT output.

For general-purpose translation, the system uses a generic language engine that is trained with a
large corpus of broad-spectrum content — hundreds of millions of words. To enhance performance
for applications that use specific terminology, an SMT system can be trained using a corpus that only
or mostly contains data that is close to the content that is to be translated. An ideal corpus for this
would be a large translation memory (TM) that contains the translations of previous projects.

The quality of the MT output depends on both the linguistic and the technical quality of the material
included.

Neural Machine Translation

Neural Machine Translation has rightly been described as a revolution rather than an evolution.
Machine translation and Neural Machine Translation are part of the much larger framework of
Artificial Intelligence. Machine translation itself is a subset of Natural Language Processing. Given the
complexity of human speech and communication, it is not surprising that MT is one of the hardest
problems to solve within the AI framework.

Neural networks are based on mimicking the human brain. This is something that applies to all areas
of Machine Learning but our focus for the purposes of the Certification program is on machine
translation. A Neural Machine Translation system learns from observing correlations between source
and target texts and modifies itself to increase the likelihood of correct translations. A Neural MT
system consists of a network of nodes or neurons and the connections between the nodes.
The nodes are associated with default values. When inputting text from the training material, it is
fed through the default settings and results in output text. This output text is then matched to the
correct gold standard translation contained in the training material. Initially, the output text is
unlikely to be a perfect match to the correct output. Consequently, the default settings are adapted
and the process is repeated until the ideal settings are created. At that point, the engine
development stops and the final result is deployed to production.

The key takeaway is that the Neural MT system modifies itself to reach the ideal settings without
outside interventions. At this point, the system is frozen and is taken from development to a
production environment.

What are the key aspects of Neural MT? Both Statistical MT and Neural MT are part of Machine
Learning, but the difference lies in how the systems learn. What both systems have in common is
that they are data-driven. NMT systems are also trained on bilingual data sets but deliver a
noticeable uplift in translation quality through a more efficient framework for learning translation
rules. Both are fed aligned source and target text and they derive translation lessons from this
material. However, the way they do this is radically different. SMT works largely by piecing together
phrase-based elements while an NMT engine captures as closely as possible the content and
meaning of complete source sentences, to form an interlingua. The interlingua sentences are then
converted to the required target language. This means that the whole source is analyzed to create a
target text that should cover the same content. This leads to much more fluent and natural sounding
translations. The NMT architecture does not only capture surface text and syntax information but
also concepts and their relations, leading to the improvements in translation quality that we now see
across language pairs. This is why NMT is very much a revolution rather than an evolution following
on from Statistical MT.
In an NMT system, concepts are represented with multi-dimensional vectors, and therefore, words
with similar meanings tend to cluster. When words have several meanings, they appear in different
clusters; for example, ‘bank’ can appear in both the geography and the finance clusters. Based on
other words in the same sentence, NMT engines can identify the more likely cluster, and hence can
disambiguate such words better than SMT systems trained on the same data.

Neural Machine Translation works better for long-range context and dependencies. This is
particularly important when it comes to languages for which the benefits of Statistical MT were
limited. Examples are language pairs with long distance word traveling such as Japanese-English,
whose clause structures are very different, or language pairs with long distance dependencies, such
as English to German or Dutch.
It is essential that translators understand the behaviors and patterns of NMT to be able to take full
advantage of this promising technology. One of the main advantages of NMT is the very fluent
translation output. However, it is important to understand that very fluent output can sometimes
mask the fact that the automated translation is not correct. This is just one of the reasons why post-
editing is still so important, even when working with NMT output.

MT Engine Creation
Both the MT technology used and the MT solution chosen and how it is deployed and trained are the
most critical aspects of the PE process. Computational linguists are normally involved at this stage.

Successful selection and training of MT engines depends on a full understanding of the quality
expectations for the intended projects. It is also essential to have access to relevant project assets
such as terminology and style guides as these must be incorporated into the MT engine training
process and will be used to measure the success of the output. Several attempts are likely to be
needed (possibly simultaneously) in order to select the optimal MT engine.

The choice of engine design must be based on experience, consider previous data recipes, and draw
on an inherent understanding of machine translation behavior.

By incorporating industry standard automated evaluations, trainers can narrow down the best
engine candidates for human evaluation. However, although automated measures offer a quick,
recognized and cost-effective way to rank MT engines, dedicated test setups will be required for
complex use cases. This is particularly pertinent to NMT, which produces more fluent, less literal
output. Good translation segments often score low on automated metrics where quality is measured
based on precise word choices dictated by the gold-standard benchmark translation set.

Engine design, including decisions on which data should be used or prioritized within a larger set,
must consider future translation requirements to ensure the longevity and relevance of the new
engine solution. There needs to be a close relationship between the data used for engine training
and future client content.

When creating client-specific engines, the computational linguists’ role is to decide how to optimize
the data corpus (typically sourced from translation memories) to ensure high-quality MT
performance. Working with data specialists and engineers, the computational linguists need to view
the data from a high-level vantage point, but also must analyze details to understand the quality of
the translation output.

As part of the wider process, recording test results in a structured manner will ensure that lessons
learned are applicable to future efforts.
Careful data management is paramount to creating and deploying the most appropriate MT
solution. Using a structured process and specialized tools to analyze data allows the team to make
the right choice as to what data will be useful for the engine training process and ultimately improve
the translators’ productivity. This focus on data has become even more crucial as NMT technology is
more sensitive to data. When fed bad or polluted data, NMT can return unrelated or
incomprehensible translations or perform poorly.

The optimization of MT engine performance, especially for specific use cases, requires in-depth
knowledge and expertise in preparing training data, tuning engine parameters and evaluating the
MT engine. A lack of experience or an ill-defined end-to-end process for MT engine evaluation,
creation and optimization is likely to result in inefficiencies during the post-editing cycle.
At SDL, we will offer two different types of MT solutions going forwards: generic and customized
engines.

Generic Engines

The core generic engines are developed by SDL for any given language pair and are trained on
hundreds of millions of words of bilingual data. The generic systems are used as the starting point
for new language directions. They use existing translation databases to build up language pairs. The
data used is mined from reliable sources available in the public domain, such as news, IT
documentation, technical manuals and publically available government material, and cover a variety
of subjects, for example IT, automotive, news, sports, electronics etc.

Generic engines can be used as powerful backup engines for customizations and verticals. This
means that if a word, phrase or grammatical structure is not found in the training data for a vertical
or customized engine, the engine may still be able to produce a translation if the word is found in
the baseline.
Generic engines are continuously monitored and updated in regular intervals.

This solution produces good results for clients who require immediate access to MT or do not have
sufficient data volumes for a custom solution. Generic engines work particularly well if the content
for translation is general and varied.

Customized Engines

When creating a customized engine, the MT is optimized for particular projects. The MT engine
training is based on specific bilingual data (usually translation memories). As a general rule, more
data usually has a positive effect on the quality of an engine and, in turn, the MT output. However,
for trained engines, the quality and consistency of data is also paramount.

One of the biggest advantages of a customized engine is the adherence to specific terminology and
style. As the machine translation output is fully based on the bilingual corpus, with no syntactical or
lexical data added, the quality of the output can only be as good as the quality of the corpus. If the
corpus data contains inconsistent terminology or style, the resulting MT output may also show
inconsistencies.

Other options to customize include enforcing terminology through dictionaries or enforcing certain
conventions such as quotes or numbers.
MT Output Evaluation and Testing Methodologies
Various evaluations are involved at different stages of the localization life cycle: the initial content
audit, source evaluations, automated testing in line with industry standards, output quality
evaluations and human productivity testing.

Evaluating MT output quality can be done using either industry standard automated measures or
human evaluation.

Automated MT Testing

In the last few decades, many methods for automated evaluation have been proposed. For some use
cases, these offer a quick, recognized and cost-effective way to analyze the potential quality of an
MT engine.

Most automated measures assess the quality of the machine translation compared to a so-called
gold standard reference translation that is deemed to be high quality. Some of the most widely used
approaches are detailed in Appendix 1 at the end of this guide.

Human MT Testing

MT testing based on automated measures presents some limitations. Especially in the case of
assessing productivity, the cognitive effort made by the post-editor to deliver a high-quality end
product is not really accounted for. For example:

• Read the source and MT output

• Consider and apply necessary edits to the MT output
• Validate terminology and check against relevant reference materials
• Ensure consistency with bordering segments
• Ensure proper text flow
• Research new concepts and raise queries
• Run the required quality assurance checks

The task of post-editing really amounts to much more than purely typing and editing. For that
reason, automated metrics are more suited to engine training development and comparison but are
not necessarily practical in a production scenario. Evaluations performed by human translators are
difficult to replace in this sense.
For us, it is important that our engines are fit for purpose, and our current testing methodology
ensures that the engine quality is good enough for post-editing. However, we know that our testing
methods also need to evolve in line with MT developments.

SDL have created a dedicated test setup to measure and compare translation and post-editing
speeds and minimize risks associated with e. g. inaccurate timekeeping. PE testing needs to
demonstrate a credible gain over human translation for skilled translators to be motivated to take
on PE assignments. It is therefore important to set up tests that are secure and produce easily
understood, transparent and credible results. We need to focus on tools, a structured testing
environment, a streamlined and repeatable process, and the resources required to produce
meaningful and authentic results.

Computational linguists need an integrated environment with tools that help create valid and
representative test beds from real client content that can easily be used to test the productivity of
post-editing MT output online. This environment must also be capable of providing detailed analytics
such as speed of manual translation versus post-editing.

The testing platform records a host of editing behaviors such as cut and paste actions from external
translation references, deletions and general key strokes, all of which help to interpret the user
experience. Giving testers the opportunity to add comments can also provide unique insights into
the end user experience after deployment. Computational linguists examine and interpret the
actions performed by the testers to ensure the validity of the result.

A reliable pool of expert translators with experience in this type of testing can also considerably
reduce the risk of having to repeat tests due to invalid results. A streamlined testing methodology
and efficient data capture will allow the detection of patterns and will provide greater value in the
long run.

In short, for effective MT output evaluation, meaningful measures need to show a link between the
test results and the final use in production. This is particularly relevant given the exponential growth
in post-editing volumes and stricter requirements. Test findings ultimately need to reflect the
experience of professional translators who post-edit large volumes.

Human productivity testing is specifically designed for the post-editing use case and helps determine
any productivity gains that the machine-translated output delivers compared to conventional human
translation. Human testing can be costly but delivers more accurate and compelling results.
Using the MT Output: Machine Translation Post-Editing
The changing translation requirements and technology advances have a big impact on translators.
Initially, it was the introduction of CAT tools into the translation cycle that changed the way
translators worked. This was followed by machine translation in the early 2000s. Over the last 10 to
15 years, improvements in MT output quality have made it a viable option for translation services
and post-editing has gained acceptance as an alternative to translation. The quality improvements
that we see with NMT provide a further boost to post-editing and put the focus very clearly on the
need for trained PE specialists.

The success of MT post-editing is closely linked to both the MT technology used and the processes
described for evaluating content and creating and assessing MT engines. Languages, domain and
content suitability go hand in hand with creating high-quality output that allows translators to focus
on refining the MT-translated content. To support translators, quality expectations need to be
clearly communicated.
But what is post-editing? There are many academic definitions of post-editing, all of which share one
obvious commonality – the human element. At its core, PE simply implies that a human translator is
editing or revising text that has been translated by machine translation software.

With post-editing, it is essential to manage expectations. If a translator is unfamiliar with the

concept of post-editing, they may think that it is a just review of almost perfect MT output.
However, post-editing is not a light review. It replaces the translation stage for MT projects and is
usually followed by a separate review stage.
The requirements for post-editing to publishable quality (or full PE) are similar to those of
conventional human translation. Common translation principles apply to both conventional
translation and post-editing. Correct grammar, spelling and punctuation rules must be observed. The
style needs to be appropriate and suitable for the target audience and terminology needs to be
consistent. Translators will need to use the same references and client-specific instructions as for
conventional translations, e. g. style guides, translation memories and glossaries.
Below you can find an overview of post-editing quality criteria for a full post-editing solution:

• The translation must be a correct reflection of the source.

• Spelling and punctuation must be correct.
• The translation must be grammatically and syntactically correct and reflect the conventions
of the target language.
• The correct terminology must be applied and used consistently (including preferred
translations for frequently occurring terms).
• Cultural references (date and time formats, units of measurement, number formats,
currency, etc.) must be correctly adapted.
• The style and register of the target must be appropriate for the document type.
• The original formatting must be reproduced.
• Project guidelines must be followed.
• The translation must read well and be suitable for the end user.

The post-editing phase itself involves a number of key stages. Since the post-editor is attempting
to be as efficient and productive as possible, preparation is key. Do not rush ahead without
taking time to consider the source and MT output. Determine the usable parts and then build
around these. Focus on accuracy, without under-editing or over-editing, and finally check over
the grammar and terminology.

In your translation project, start with reading the source text. Our recommendation is not to check
the output straightaway looking for errors but rather to focus on understanding the source text and
what needs translating.
Depending on which technology you are working with (SMT or NMT), you will need to expect certain
errors in the output. This is commonly referred to as expected MT behavior. The quicker you can
identify the errors, the quicker you will be able to eliminate them in the post-editing phase. This is
one of the key post-editing skills that translators will need to develop to save time and increase
productivity compared to conventional translation.

Once you have read the source text and scanned the output, you will need to use the pre-translated
MT to reach your final translation. You will need to determine the usable elements and make them
the basis of your translation. You should not discard the output, unless it is completely unusable. It is
important to build from the MT in order to speed up your work. You will also need to remove the
errors that are contained in the MT output.

Avoid making too many changes (over-editing). Over-editing refers to preferential changes or
unnecessary rephrasing. If the MT output is correct and in line with client expectations, you should
not change it because you would personally prefer a different translation. On the other hand, you
will also need to make sure not to under-edit the output. Obvious mistakes will need to be
corrected. When you have finished editing your segment, we would advise to make a final check for
grammar, terminology and fluency.

While the advice on how to post-edit may seem quite innocuous, the thinking processes required are
very complex. It may be difficult to achieve post-editing success right from the start. Post-editing is a
skill that needs to be honed and developed over time, the same as conventional translation skills.
Post-editors will not be fully productive from day one. Experience is one of the most important
factor in translation productivity, and this becomes even more influential in post-editing. It may take
up to three months to see efficiency gains with post-editing, provided that you post-edit on a regular
basis. This is a steep learning curve but practice will enable you to learn and apply the correct
strategies for post-editing.

Below you can find a segment-by-segment workflow for post-editing:

• Consider the target audience, the client and the content type.
• Always read the source segment first.
• Identify any elements in the source that you are unsure about.
• Now look at the MT output and see how close it is to the translation that you have in your
mind.
• Complete the translation.
• Carefully re-read the segment and compare it against the source.
• Re-read the segment after making your final changes.

Levels of post-editing
The market makes a distinction between two degrees of post-editing: post-editing to publishable
quality or full PE and post-editing to an understandable level or light PE. Quality expectations need
to be determined in conjunction with the customer and end quality is dependent on client
requirements.
Post-editing to publishable level is the highest quality standard. This is in line with the expectations
of the majority of SDL’s clients. After post-editing, files undergo a quality check to ensure that the
translation is correct and fluent. The final quality should be comparable to conventional translation.

Post-editing to understandable quality, or light post-editing, is a form of post-editing which is used

when publishable quality standards are not required for a particular type of content. This could be
low-visibility content, or texts that would not otherwise be translated for cost and time reasons. A
client might decide to opt for understandable-quality texts in order to reduce the number of support
requests for a product or to provide an extra service to the user, for example. The typical purposes
of understandable-quality texts include offering users a quick answer on how to fix an issue or
providing a translation solution for low-visibility content, such as FAQs, blogs and knowledge bases.
Texts suitable for light post-editing would fulfill one or more of the criteria below:

• Low visibility
• Short lifespan
• General subject matter
• Low-risk subject matter

The objective of light post-editing is to produce a translation that is:

• Understandable
• Actionable
• With only critical errors corrected

The basic principles of light post-editing can be summarized as follows:

• Focus is on meaning, not grammar or style

• Style is basic, not fluent
• Register and tone are basic, not adapted to text type/target audience
• Sentence structures are readable, not perfect in terms of spelling and grammar

When post-editing a text to understandable quality, the user should not expect a grammatically or
stylistically perfect translation. Grammar and spelling mistakes will only be corrected if the meaning
is affected.

• Instructions in the text must be actionable, not perfectly worded

The translation is not expected to be perfect but should be actionable. As an example, if a

knowledge base article or FAQ is post-edited to understandable quality, the user should be able to
understand how to fix a particular problem.

• Terminology is appropriate in context, not client-specific

For light post-editing, the post-editor will be asked to fix critical terminology errors that prevent the
end-user from understanding the concept (NB: Automatic checks for client-preferred terms can only
be corrected if a glossary in MultiTerm format is provided).

• Text is understandable, not consistent

Light post-editing does not allow the post-editor to check consistency across the text.

• Information is accurate, not localized to in-country standards

• Wrong punctuation is not corrected

Overall, the post-editor is expected to deliver an acceptable quality level in line with the guidelines
set out above. The following example of a quality matrix could be used as a guideline.

Light PE –
Error Category Specific Issue Comment
Correction required

Mistranslation Yes Critical errors only

Automated checks for client-

In-context preferred terms at end of project
Terminology Yes
terminology choices possible only if glossary provided in
MultiTerm format

Accuracy Omissions/additions Yes Critical errors only

Grammar No Unless meaning is affected

Language Spelling No Unless meaning is affected

Punctuation No

Consistency No

Style General style No

Country standards No
Country
Register and tone No
Expected MT Behaviors
As mentioned before, post-editing is not a light review of machine-translated content. Instead, post-
editing replaces the translation stage. You will need to bear in mind that correcting machine
translated output is very different from reviewing human translations. MT output is rarely flawless
and translators need to be aware of the issues that can be present in the output. Post-editing is most
efficient when the translator knows what kind of mistakes to expect. Mistakes in the MT output can
vary depending on project, language combination and the type of MT technology used. For the
purposes of this guide, we will focus on expected MT behaviors for SMT and NMT technology.

While NMT is fast becoming the technology of choice in the industry, SMT has been used
successfully for approximately 15 years and continues to be used in some production workflows. For
this reason, we are including known SMT behaviors for translators working with SMT output.

Common Behaviors: SMT

SMT output is generally fluent in style but translators need to watch out for the following common
issues in the output:

Possible issues
• Extra words in the target that are not present in the source
• Words missing in the target (verbs, prefixes)
Tips
• Read the source text before the machine translation output
• After post-editing a segment, check it against the source
Possible issues
• Source grammar transferred incorrectly
• Untranslated/incorrectly translated words due to source errors
• Names and product numbers translated
Tips
• Think about the logic of the output — does it make sense?
• Check names and product numbers against the source

Possible issues
• Mistranslations (an antonym used instead of the correct word: remove/install, can/cannot)
• Incorrect gender/agreement in the target
• Word order
• Terminology can be inconsistent
Tips
• Look out for antonyms (opposites)
• Ensure that the language and style meets the client’s expectations
• Use Studio’s filter functionality to check for terminology consistency

Possible issues
• Formatting tags ignored/incorrect
• Incorrect capitalisation
• Incorrect punctuation
• Incorrect target language flavour
Tips
• Refer to the source text in its original format
• Run spell checks as normal
• Consider creating regular expressions to check for common language flavour issues

Common Behaviors: NMT

As mentioned before, NMT works better than other technologies for languages with long distance
word travelling (e.g. English and Japanese where the overall clause structure is very different) and
languages with long distance dependencies (e.g. English and German with verb dependencies
between the main and the auxiliary verb). Other languages which show a marked improvement with
NMT are agglutinative languages (for example Turkish) and highly inflected languages with complex
morphology and different syntax and word order compared to English, such as Russian.

Overall, fluency is one of the main advantages of NMT. Because of how NMT works, it is also much
more attuned to context than SMT. It shows good syntactical structures, especially for longer and
more complicated sentences. Grammar is often correct and the output is logical and coherent.
It is essential that translators understand the behaviors and patterns of NMT to be able to take full
advantage of this promising technology. One of the main advantages of NMT is the very fluent
translation output. However, fluency is not only one of the biggest advantages of NMT but can also
be one of its biggest challenges. Very fluent output can be deceptive and translators need to be alert
to the possibility that the machine translated output is not a correct reflection of the source.
Translators will need to ensure that the meaning of the source is fully and correctly transferred to
the target during post-editing. PE remains an essential task when working with NMT output.

The slide below summarizes common issues that can be found in NMT output. For examples and
further explanations please refer to the appendices 2 to 4.
At SDL, our focus has been on creating a production grade NMT system that delivers the language
improvements that we have come to expect with NMT and at the same time addresses important
user constraints specific to the post-editing use case, such as terminology, formatting and tags.

Translators working in a translation production environment have vastly different requirements than
for example a casual user running a translation through a public MT portal. Adequacy and coverage
need to be tuned to a professional environment and the hardware requirements need to be ready
for large teams. Translators need terminology to be as accurate as possible and they also need a
system that can correctly handle formatting and tags.

NB: Adequacy means that the meaning of the source sentence is correctly transferred to the
translation output and coverage means that all the words in the source sentence are covered by the
model and will be translated.

Providing Feedback to Improve the MT Output

An efficient and well thought out PE process facilitates sustainable post-editing through creating
optimal MT solutions and includes quality assurance steps, such as structured feedback from post-
editors, to provide a basis for future engine improvements.

From a post-editing viewpoint, the overriding goal is to create and maintain high-quality MT
solutions that enable translators to process the output in the most productive way. MT solutions can
be created for specific content based on customer data, but if the content changes and no longer
matches the data that was used to create the original solution, post-editing productivity will suffer
and may lead to frustration and disengagement.
Consistent and sustainable MT quality can be achieved in several ways through:

• Technology improvements
• Additional or updated user data
• Implementation of user feedback

Issues with MT output need to be reported as soon as possible to allow for a prompt resolution, and
efficient collection of linguistic and technical feedback is essential for that purpose.

Linguistic and Technical Feedback

Linguistic feedback includes language-related issues such as incorrect word choices, bad grammar or
syntax issues. When reporting linguistic feedback, the user is asked to bear in mind expected MT
behavior and focus on recurring errors.

Significant levels of linguistic feedback might indicate changes in content characteristics. Technical
feedback will reveal process-related issues such as missing tags or error messages. Both can lead to
less productive MT post-editing.

SDL applies a streamlined process to channel feedback:

• We provide a comprehensive feedback process and request any necessary supporting

materials.
• Linguists and engineers are on hand to carry out investigations or data analysis as required.
• Computational linguists support communication and remain in close contact throughout the
process to report and address issues.

This feedback loop integrates translators firmly in the development process. Feedback concerning
the technical capabilities of an MT solution or the linguistic quality of MT output helps to improve
post-editing sustainability.

How to Add Machine Translation in SDL Trados Studio

In Studio, MT providers are added in a similar way to TMs (it is possible to add MT from different
providers). In the screenshots below we use SDL Language Cloud as an example. You can add MT
when you are creating a new project (e.g. using files downloaded from your workflow system) or to
an existing project (e.g. a Studio package).

During the project creation process, or via Project Settings for an existing project, go to “All
Language Pairs” or the specific Language Pair, if applicable -> “Translation Memory and Automated
Translation”, click “Use…” and then select “SDL Language Cloud Machine Translation” from the
dropdown menu:
If you have not already created an SDL Language Cloud account, you will need to do so now. Please
use your SDL My Account credentials to create an account (also known as an oos.sdl.com account).

If you have not used SDL Language Cloud before, a free 30-day trial will automatically start.
After the 30-day trial, you will be able to choose from a selection of packages available on the SDL
Language Cloud website, including a free package for machine translation usage.
By default, external users have access to the generic engine, called “Baseline”. Other credentials can
contain several engines. Click the “Settings” button to see the different engines available. These
engines can be specific to a subject matter, client, product, etc. If you are unsure of which engine to
use, check the MT Comparison application available in the AppStore. It will create a column report
showing the different outputs for each available engine so that you can take an informed decision
about the engine that is best suited to your needs. You can access further instructions by following
this link.

Applying MT Segment-by-Segment

Once you have added the MT provider to your project, you will immediately be able to see MT
output in the Editor for new segments in the project.

Segments where MT has been applied appear as “AT” (Automated Translation) in the Editor.

Applying MT to Whole Files/Projects

To apply MT to whole files or projects, you can use the “Pre-translate Files” batch task.
In the settings for this batch task (accessible via Project Settings or when you run the batch task
itself), choose “Apply automated translation” for the “When no match found” option.

If you are using both translation memories and machine translation engines, make sure to establish
the cut-off point for both. You can do this with the setting “Minimum Match Value” (usually set to
75). Any segment from 0 to 74% will be pre-translated with MT, and 75 to 100% matches will be
leveraged from the translation memory.
The exact settings are detailed below:

Remember, your TMs will be applied from 75 to 100% matching segments, and segments from 0 to
74% will have machine translation applied.

Please note that machine-translated words are listed as “New” in the Analysis reports.

Using the Studio AutoSuggest Feature to Retrieve MT in Studio

Retrieving MT output using AutoSuggest in Studio is another excellent way to leverage machine
translation.

AutoSuggest editing is a feature of SDL Trados Studio that can be used to speed up conventional
translation.

AutoSuggest monitors what you type and, after you have typed the first few characters of a word,
presents you with a dropdown list of suggested words and phrases in the target language that start
with the same characters. If one of the words or phrases matches what you were about to type, you
can automatically complete the word or phrase by selecting it from the list. As you continue to type,
the list of suggested words is continuously updated.
MT AutoSuggest offers alternatives of different lengths, which can be useful. For example, instead of
accepting the MT suggestion for the full segment, you can use smaller, more suitable chunks. This
feature can also be used as a background dictionary to avoid having to type long words.

Below you can see two examples for MT AutoSuggest.

You can set up AutoSuggest to provide suggestions from sources other than MT. All suggestions will
be displayed in the same dropdown list.

As you can see in the following screenshots, an icon will help you identify the source of the
suggestions. These can include termbases (green can icon), MT suggestions (blue AT icon), TM
suggestions (orange “F” icon for fuzzy matches and green “100” icon for 100% matches), and upLIFT
fragment recall (green and yellow square icon).

Retrieving MT output using AutoSuggest is another excellent way to leverage Machine Translation
and increase productivity.
Post-Editing as Opportunity
We have seen how machine translation and post-editing are causing the translator profile to change
and evolve. The improvements that we are seeing with Neural Machine translation are already
reshaping the landscape of what is possible with MT and are expanding the reach of the technology.

With NMT, more and different domains are considered suitable for post-editing, in particular in the
premium sector (for example Life Sciences or Finance). MT can be applied to many more content
types such as marketing and more languages are becoming suitable for post-editing.

As a result, translators will need to adapt their skillset and work in line with technology
developments. In the past, translators for certain languages were less likely to be required to post-
edit. It was for example highly unlikely that Arabic or Turkish translators would be asked to post-edit,
however this is changing with NMT. The same applies to premium verticals: post-editing is very
much becoming a requirement for specialist translators in premium domains like Life Sciences.

What are the domains and verticals where we see increased interest in post-editing? MT is already
established in a number of industries such as Automotive, Manufacturing, IT and Travel and Leisure.
Other industries show potential for future growth. The Life Sciences domain is becoming closely
related not only with MT but AI and Machine Learning in general. Finance is another growth domain
where the focus is very much on secure solutions to safeguard sensitive customer information. The
Retail & Consumer segment offers new opportunities to apply MT to e. g. help translate buyer
reviews or marketing materials. Government & Defense is already an active MT domain but there is
space for further growth combined with a focus on security.
Enterprises across industries and LSPs use machine translation to process large volumes of content
with tight turnaround times. As mentioned before, it is a reality that companies do not necessarily
allocate a higher budget to translation just because more content needs to be translated.

At the beginning of this guide we posed the question why translators should consider adding post-
editing to their portfolio. We have seen that MT has become a mainstream choice across the
translation industry but is still often perceived as a threat to the profession of translator. Below are
some of the reasons why translators still oppose post-editing.
As an LSP, we know that the profession of translator is undergoing wide-ranging changes. However,
considering the huge volumes available for translation across industries and verticals and the quality
improvements associated with NMT, post-editing is an essential skillset helping translators to
showcase and market their talent and proficiency.

The close collaboration that is now required between translators and language technology also gives
rise to new job profiles. These include not only post-editors but consultancy roles for translators
proficient in working with new technologies.

These profiles rely on knowledge and experience of translation, MT and PE. Post-editing, reviewing
and leading PE projects is a big part of any profile associated with machine translation. But the
industry also needs translators who understand different post-editing use cases and requirements
and who can provide linguistic expertise when looking at whether PE can be introduced into a
workflow or process. Another important part is assessing and scoring different MT outputs and
working with MT testing scenarios.
Conclusion
Post-editing is at the intersection of humans and machines, and machine translation should be
embraced as one of the most advanced tools in the translators’ toolbox to further the profession for
a new generation of translators.

At the start of this guide, we said that adopting post-editing is a choice that everyone needs to make
for themselves, but we believe that the benefits for translators far outweigh any perceived
negatives.

The ongoing technology developments, especially with Neural MT, are not reducing the role of
translators, but rather, changing and enhancing it, opening up a host of new opportunities. Going
forwards, the majority of content will be touched in some way by machine translation, especially as
the quality continues to improve. Aligning yourself with the new requirements is one way to future-
proof the role of translator and increase employability in a field that is and will remain deeply
interlinked with technology.
Appendix 1: Detailed Information on Automated MT Evaluations
This appendix contains more detailed information on automated MT evaluation methods.

BLEU (BiLingual Evaluation Understudy) score: This algorithm aims to evaluate the quality of text
that has been machine translated. The central idea behind BLEU is “the closer a machine translation
is to a professional human translation, the better it is.” To assess this, scores are calculated for
individual translated segments – generally sentences – by comparing them with a set of good-quality
reference translations. Those scores are then averaged over the whole corpus to reach an estimate
of the translation’s overall quality. Even though BLEU has become a standard in the industry, it has
its limitations. Intelligibility or grammatical correctness are not taken into account explicitly, for
instance, as they are supposed to be included in the correct reference translations.

NIST: The name of this metric comes from the US National Institute of Standards and Technology.
This measure is based on the BLEU score, but it differs from the BLEU algorithm in several ways.

While BLEU simply calculates how many n-grams match both in the reference translation and in the
MT output and gives these n-grams the same weight, NIST also calculates how “informative” a
particular n-gram is. When a correct n-gram is found, the algorithm measures whether that
combination is a common sequence in the corpus material or whether that fragment is not that
common in the data. Depending on the result, an n-gram will be given more or less weight. To give
an example, if the bigram “on the” is correctly matched, it will receive a lower weight than the
correct matching of the bigram “interesting calculations,” as this is less likely to occur.

NIST also differs from BLEU in terms of how some penalties are calculated. For example, small
variations in translation length do not impact the overall NIST score as much as in BLEU.

METEOR (Metric for Evaluation of Translation with Explicit ORdering): This metric was designed to
address some of the problems found in the more popular BLEU metric, and also produces a good
correlation with human judgment at the sentence or segment level (this differs from the BLEU
metric in that BLEU seeks correlation at the corpus level). With this system, several features that had
not been part of any other metrics at the time were introduced. Matches in METEOR are made by
following the parameters below, among others:

• Exact words: As with other metrics, a match is made if two words are identical in the
machine translation output and the reference translation
• Stem: Words are reduced to their stem form. If two words have the same stem, a match is
also made
• Synonymy: Words are matched if they are synonyms of one another. Words are considered
synonymous if they share any synonym sets according to an external database

Levenshtein distance: This metric measures the similarity or dissimilarity (“distance”) between two
text strings by calculating the minimum number of single-character edits (insertions, deletions and
substitutions) required to change one word into another. In the field of machine translation, this can
be done by comparing the raw MT output to the human translation.
Let’s look at a couple of examples:

The Levenshtein distance between “sport” and “short” is 1, because one edit is required to
convert one word into the other (replace “p” with “h”).
The Levenshtein distance between “dog” and “frog” is 2, as it is not possible to convert the
first word into the second with fewer edits (replace “d” with “f” and add “r”).

This algorithm always has a maximum value that corresponds to the maximum length of both input
strings. In the case that two words do not have anything in common, the minimum number of edits
will not exceed the maximum number of characters in the longer string.
Example: If we have “computer” and “alibi”, the Levenshtein distance will be 8 and no higher than 8:

Replace “c” with “a”

Replace “o” with “l”
Replace “m” with “I”
Replace “p” with “b”
Replace “u” with “I”
Delete “t”
Delete “e”
Delete “r”

As with other automated measures, the results of the Levenshtein distance are not set in stone. As
mentioned before, there can be many correct translations for a single source. The Levenshtein
distance will not be able to measure quality on its own. Results will vary, for example, if clauses are
positioned differently in the MT output and in the human reference translation.

Example:

MT: “If I go home after 10pm, I will let you know”

Reference human translation: “I will let you know if I go home after 10 pm”

In this case, the MT output is correct and no changes would be necessary during a post-editing stage.
However, the Levenshtein distance will be quite high, as many changes would be required to turn
the first sentence into the second one.

TER: This is a word-based metric that calculates the minimum number of edits required to match an
MT output to a correct reference translation, normalized by the length of the reference.

# of edits
TER =
average # of reference words

TERp (TER-Plus) is an extension of Translation Edit Rate (TER) and builds on the success of TER as an
evaluation metric and alignment tool. At the same time, it addresses several of TER’s weaknesses
through the use of paraphrases, morphological stemming and synonyms, as well as edit costs that
are optimized to correlate more closely with various types of human judgments.

Put simply, TERp measures the number of edits that are necessary to go from the raw MT output to
a final edited version. As such, it is a helpful metric to measure typing and editing effort. The TERp
score is a number from 1 to 100; the higher the number, the more editing was required.
Appendix 2: Examples for Expected NMT Behavior
These practical examples correspond to the common NMT issues detailed in the section
“Expected MT Behaviors”.

NMT Behavior LP Source NMT Output Desired Comment

Translation

Inconsistent or EN>DE Excluding traffic volume, Ohne Verkehrsaufkommen, Terminology not

non-compliant alerts, and reports. Alerts und Reports appropriate for
terminology this context

EN>DE Chromatografie, Different spellings

Chromatographie found in the NMT
output

Omissions EN-NL The STG-600 does not need De STG-600 heeft geen De STG-600 heeft
any peripheral components enkele perifere onderdelen geen perifere
to operate. nodig_. onderdelen nodig
om te werken.

Repetitions FR>EN Exclusivement Exclusivement, exclusive

(“neural babble”

EN>DE Nutritional yeast is not the Hefehefe ist nicht dasselbe

same as baking yeast or wie die Hefehefe oder die
brewer’s yeast. Hefehefe.

Mistranslations – EN>TR Please try again later. Please try again later.
unknown words
or segments

Mistranslations - EN>TR patient deterioration Suresiye Verschlechterung Unknown words in

neologisms EN>DE Patientenaltedrung des target (words
Patientenzustands made up or not
meaningful)
NMT Behavior LP Source NMT Output Desired Comment
Translation

Mistranslations – EN>DE β-hydroxybutyrate Es handelt sich dabei um Output not related

unrelated or concentration over 3 mmol/L eine sehr große, aber auch to source subject
arbitrary content are indicative of ketoacidosis; sehr große Anzahl von matter
in target in very severe diabetic Menschen, die sich in der
ketoacidosis the ganzen Welt bewegen.
concentration may exceed 25
mmol/L13.

Mistranslations – DE>EN Um das Thema In addition to the Kroos and Boateng The idiomatic
idiomatic Özil/Gündogan kamen auch Ozil/Gundogan, Kroos and could not avoid expression
expressions Kroos und Boateng wenige Boatheng were not around the topic “kamen … nicht
Stunden vor dem WM- a few hours before the Özil/Gündogan herum“ is
Auftakt des russischen World Cup start of the either, just a few translated literally
Gastgebers nicht herum. Russian host. hours before the as “were not
World Cup start of around”. The
the Russian host. correct translation
would be “could
not avoid”. Names
have also been
misspelled in the
NMT output (Özil
vs Ozil; Boatheng
vs Boateng).

Identifiers EN>TR UICC Unlock Polling Interval Ucm Açma Yoklama Aralığı Acronyms

Identifiers EN>DE SDL DXA 2.0 SDL Trados Brand or proper

names

Numbers and EN>DE 2016-11 2016-10

variables 2019 2022
12 M6x20 screws 12 m6__LW_SW__x20-
Schrauben

EN>DE %count% Young Refugees %Quote der jungen

Flüchtlinge

Capitalization EN>DE RUM Rum Materialliste

mPulse Mpulse
material list materialliste

Register and EN>DE Your ihre instead of Ihre

address
Appendix 3: Language-specific Phenomena
NMT shows improved output quality for languages that were previously challenging for machine
translation. To illustrate how NMT resolves language-specific problems for challenging languages, we
have included the examples and explanations below. They show typical phenomena for three
languages – Russian, Japanese and German – which are handled correctly with NMT. In the
examples, you will see the source text, the machine-translated output, the desired human
translation and a literal back translation of the human translation to English to make them more
understandable for non-native speakers.

Russian

1) Word order
a. Theme/Rheme – This is a very wide area of semantics, which explores the strategies
applied to build a sentence, render meaning, focus on message, stress certain parts
in a sentence, etc. These strategies influence the word order greatly and play a
significant role in building up a natural style. The examples below illustrate how
different parts or clauses in a sentence travel in English/Russian to maintain natural
flow and help fluent understanding.

Source Text Machine Translated Desired Human Back Translation of Human

Output Translation Translation

You can select a managed В качестве конечной точки В качестве конечной точки As an endpoint for
device, including a device stack развертывания можно развертывания можно deployment, you can chose
or cluster, as an endpoint for выбрать управляемое выбрать управляемое managed device, including
your deployment. устройство, включая стек устройство, в том числе стек stack or cluster of devices.
устройств или кластер. или кластер устройств.

A lot of sand was eaten in the При создании этой При создании этой While creating this photo we
making of this photo. фотографии было съедено фотографии мы изрядно considerably ate sand.
много песка. наглотались песка.
The facility, attached to a По словам сотрудников, в По словам сотрудников, в As per the words of the
police station, had a capacity здании, принадлежащем здании, закрепленном за workers in the facility,
of about 60 detainees, полицейскому участку, полицейским участком, attached to the police station,
workers at the scene said. содержится около можно содержать около it is possible to detain around
60 заключенных. 60 заключенных. 60 detainees.
Navigating the printer menu is С 2,7" ЖК-экраном и С 2,7" ЖК-экраном и With the 2.7" LCD and touch
easier than ever with the 2.7" сенсорной панелью сенсорной панелью panel navigation of the menu
LCD and touch panel. перемещение по меню перемещение по меню of the printer became easier
принтера стало проще, чем принтера стало проще, чем than ever.
когда-либо. когда-либо.
Not just a learning tool, this Этот робот не просто учится, Этот робот не просто This robot is not just helping to
robot loves fun! он любит веселье! помогает учиться, он любит learn, he loves fun.
веселье!
 b. Noun phrases – Noun phrases in Russian (i.e. groups of words which usually consist
of a noun and adjectives/other modifiers) are well known to be very different
structurally from English noun phrases; the reason for that difference is rooted in the
way both languages express grammatical relations in a sentence. English is an
analytical language (i.e. words hardly ever change depending on their role in a
sentence), and relations are mostly expressed by strict word order. Russian, on the
other hand, has rich morphology, i.e. relations between words are shown by changes
in the word itself, mainly the endings of words: morphology is a tool for expressing
multiple relations simultaneously, e.g. noun + adjective relation and noun + adjective
relation to a verb. Agreement between a verb and a noun phrase is especially
challenging, because each word in a noun phrase should have a correct ending. In
the below examples noun phrases are highlighted in different colors with the verb
shown in red.

Source Text Machine Translated Desired Human Back Translation of Human

Output Translation Translation

Save 25% on inflight food and Сэкономьте 25% на покупке Сэкономьте 25% на покупке Save 25% at buying food and
beverage purchases on продуктов питания и продуктов питания и beverage on flights of
American Airlines flights when напитков на рейсах American напитков на рейсах American American Airlines with using
you use your card. Airlines при использовании Airlines при использовании your card.
вашей карты. вашей карты.
Its four-stage LED battery Четырехступенчатый Четырехступенчатый Four-stage LED indicator of
status indicator clearly shows светодиодный индикатор светодиодный индикатор the status of the battery
you how much battery life is состояния аккумулятора состояния аккумулятора clearly shows you how much
left. четко показывает, сколько четко показывает, сколько time is left.
времени осталось. времени осталось.
Connect your PC to a parallel Подключите компьютер к Подключите компьютер к Connect computer to parallel
dot-matrix printer, older laser параллельному матричному параллельному матричному dot-matrix printer, old laser
printer or other uni-directional принтеру, старому принтеру, старому printer or other uni-directional
Centronics peripheral. лазерному принтеру или лазерному принтеру или peripheral device Centronics.
другому однонаправленному другому однонаправленному
периферийному устройству периферийному устройству
Centronics. Centronics.
 c. Adverbial modifier – The position of adverbial modifiers in a Russian sentence
differs from the position of such a modifier in an English sentence. While English
tends to place adverbial modifiers closer to the end of the sentence, in Russian they
usually are appended close to the verb. This phenomena is closely related to the
English – Russian word order challenge. In each sentence below you can see how the
adverb (highlighted in blue) travels from the end of the English sentence to the
middle, closer to the verb (highlighted in red).

Source Text Machine Translated Desired Human Back Translation of Human

Output Translation Translation

With no separate mode setting Благодаря отсутствию Благодаря отсутствию Thanks to absence of separate
and no reduction in image отдельной настройки отдельной настройки mode setting and reduction in
quality, you can freeze brilliant режима и снижению режима и снижению image quality, you can easily
action shots with ease. качества изображения вы качества изображения вы make sharp shots of dynamic
можете легко делать резкие можете легко делать резкие scenes.
снимки динамичных снимки динамичных
сюжетов. сюжетов.
The camera also allows you to Камера также позволяет Камера также позволяет The camera also allows to
record in RAW and JPEG одновременно записывать в одновременно записывать в simultaneously record in
formats simultaneously. форматах RAW и JPEG. форматах RAW и JPEG. formats RAW and JPEG.

Find the departure time that Найдите удобное для вас Найдите удобное для вас Find convenient for you time
suits you. время отправления. время отправления. of departure.

d. Long distance agreement – Russian is a language with rich morphology, i.e. relations
between words are shown by changes in the word itself, mainly the endings of
words. Consequently, the longer the distance between related words in a sentence,
the harder it is to keep correct morphology (endings). In the below examples, the
main clause (highlighted in red) in the sentence is split by noun clauses, modifiers or
attributive clauses. Morphology and relations are kept correctly in the MT output.

Source Text Machine Translated Desired Human Back Translation of Human

Output Translation Translation

Not everyone who pays private Не все, кто оплачивает Не все, кто оплачивает Not everyone who pays private
mortgage insurance will частное ипотечное частное ипотечное mortgage insurance will
benefit. страхование, получат страхование, получат benefit.
выгоду. выгоду.
He is the first heavyweight Он является первым Он — первый боксёр- He is the first heavyweight
boxer from either New Zealand боксёром-тяжеловесом из тяжеловес из Новой boxer from either New Zealand
or the Pacific Islands to win a Новой Зеландии или Зеландии или Тихоокеанских or the Pacific Islands having
major world championship. Тихоокеанских островов, островов, выигравший won major world
выигравшим крупный крупный чемпионат мира. championship.
чемпионат мира.

The views from the master Из окон главной спальни и Из окон главной спальни и From the windows of master
bedroom and lounge are гостиной открывается гостиной открывается bedroom opens up stunning
stunning. потрясающий вид. потрясающий вид. view.
 2) Impersonal structures

English and Russian differ in terms of formal grammar structures, which are used widely for
technical and even marketing texts to express a formal subject in a sentence. While in English this
formal subject is widely expressed with a “you” pronoun, which is impersonalized, Russian utilizes
completely different grammatical structures, known as impersonal clauses, which do not have a
formal subject. This difference in grammatical structure is accurately rendered in many MT
examples, some of them are highlighted below. It is worth noting that the back translation of the
human translation cannot clearly show these Russian grammatical structures, because English
misses them completely.

Source Text Machine Translated Desired Human Back Translation of Human

Output Translation Translation
If the buttons on the toolbar Если кнопки на панели Если кнопки на панели If the buttons on the toolbar
disappear, you need to reboot инструментов исчезают, инструментов исчезают, disappear, it is necessary to
the computer. необходимо перезагрузить необходимо перезагрузить reboot the computer.
компьютер. компьютер.

If you manually launched the Если задание было Если задание было If the task was launched
task, click the Task Status link запущено вручную, запущено вручную, manually, click the Task Status
in the notification box that щелкните ссылку Состояние щелкните ссылку «Состояние link in the window of the
appeared when you launched задачи в окне уведомления, задачи» в окне уведомления, notification, having appeared
the task. появившемся при запуске появившемся при запуске at last launch.
задания. задания.
This convenient feature Эта удобная функция Эта удобная функция This convenient feature
automatically powers up the автоматически включает автоматически включает automatically powers up the
printer whenever you send a принтер при отправке принтер при отправке printer at sending photo or
photo or document to be фотографии или документа фотографии или документа document for printing.
printed. на печать. на печать.

3) Passive voice

Passive voice is generally used very differently in English and Russian. Moreover, in Russian the
passive voice cannot be reduced to a single grammatical structure (e.g. verb to be + past
participle, as English does). Therefore in Russian we need to apply different structures and
grammar to render the meaning of the English passive voice, which – if not rendered accurately –
will result in literal meaningless translations. Transforming English passive voice into Russian
grammatical structures is often connected with word order/morphological challenges.

Source text Machine Translated Desired Human Back Translation of Human

Output Translation Translation
These items are priced lower Цены на эти товары ниже, Цены на эти товары ниже, Prices for these goods are
than in any store in the area. чем в любом магазине в этом чем в любом магазине в этом lower, than in any store in this
регионе. регионе. region.
He was told to take 8 tablets a Ему сказали принимать 8 Ему сказали принимать 8 They told him to take 8 pills
day. таблеток в день. таблеток в день. per day.
Can asthma attacks be Можно ли предотвратить Можно ли предотвратить Is it possible to prevent
prevented? приступы астмы? приступы астмы? asthma attacks?

Volunteer 4 was asked to Добровольца 4 попросили Добровольца 4 попросили They asked the volunteer to
breathe deeply. дышать глубоко. дышать глубоко. breathe deeply.
Packaged with detailed, step- В комплект входят В комплект входят In package are included
by-step assembly instructions подробные пошаговые подробные пошаговые detailed, step-by-step
инструкции по сборке инструкции по сборке assembly instructions
Japanese

In all examples, the color-coding indicates the words in the English source that are linked to the
words in the Japanese target.

1) Negation

In English, "No <Subject> shall" means negation. In Japanese, only one negative indicator‘ない’
is used at the end of the sentence. The MT output is correct, as can be seen from comparing the MT
to the intended translation.

In the second example, the negation of a transitive verb involves an auxiliary verb ‘does’, a verbal
negation‘not’and a negative indicator on the direct object‘any’. So the three words in blue
form the negation in English. In Japanese, only one negative indicator ‘ない’ is used at the
end of the sentence. The MT output is correct, as can be seen from comparing the MT to the
intended translation.

In the third example, fault-free means "no-fault". With SMT technology, "fault-free" was often
translated as "faulty". The NMT output is correct, as can be seen from comparing the MT to the
intended translation.

Source Machine Translation Desired human translation Back translation of

human translation

No court shall have 第 2 項の下で司法長官の決定 第 2 項の下で、裁判所は司法 Court shall not have

jurisdiction to review any jurisdiction to review any
determination of the を見直す司法権を裁判所に持 長官の決定を見直す司法権を determination of the
Attorney General under Attorney General under
たない。 持たない。
paragraph (2). paragraph (2).

He does not want any help. 彼は助けを求めていない。 彼は助けを望んでいない He wants no help.

Installation is fault-free and 取り付けに異常がなく、試運 設置時に障害は発生せず、試 Installation is fault-free and

commissioning time is low. commissioning time is short.
転時間が短い。 運転も短時間で済みます。
 2) Word order

In the first example, the NMT output is correct. NMT allows complex English sentences using a
conjunction to be translated into the correct Japanese word order.

In the second example, the NMT output is correct. NMT allows the translation of complex English
sentences using parenthesis into the correct Japanese word order.

In the third example, the NMT output is again correct and close to perfect. NMT allows the
translation of imperative English sentences correctly, especially when imperative sentences are
connected with “and”.

Source text Machine Translated Output Desired human translation Back translation of
human translation

This convenient feature この便利な機能は、プリント この便利な機能により、プリ With this convenient

automatically powers up the feature, the printer
printer whenever you send a する写真や文書を送信するた ントする写真や文書を送信す automatically powers up
photo or document to be whenever you send a photo
びに、プリンタの電源を自動 るたびに、プリンタの電源は
printed. or document to be printed.
的にオンにします。 自動的にオンになります。
Send photos directly from ワイヤレス PictBridge(DPS over ワイヤレス PictBridge (DPS over Send photos directly from
your wireless PictBridge (DPS your wireless PictBridge (DPS
over IP) compatible camera IP) 互換カメラからプリンタに IP) 対応カメラからプリンタに over IP) compatible camera
to the printer. to the printer.
写真を直接送信します。 写真を直接送信します。
Detects filament absence フィラメントの不在を検出 フィラメントがないことを検 Detects absence of filament
and automatically pauses and automatically pauses
your print. し、印刷を自動的に一時停止 出し、印刷を自動的に一時停 your print.
します。 止します。
German

1) Word order and sentence structure (long distance dependencies)

The first example sentence contains a rather complex structure: configure + object + to use +
object + to determine. This was translated correctly by using "so ... dass". The aspect of
"sollen" is correctly introduced for "whether to download". In addition, the machine
translation resolved the English punctuation after "optionally" correctly, removing the
comma which would be incorrect in German.

The second example shows that the machine-translated output correctly resolves the long-
distance dependency between "only way + is by returning". The output also contains the
correct and idiomatic connection between "only way + is" which is not translated with an
equivalent of "is" in German. In addition, "get back" is a non-technical term which was
correctly translated in the meaning of "restore" which is appropriate in this context.

In the third example, all the relations in this complex structure were translated correctly plus
the correct version of "Vorgangspassiv" was used.

Source text Machine Translated Output Desired human translation Back translation of human
translation
Optionally, you can Optional können Sie das Optional können Sie das Optionally you can
configure the system to use System so konfigurieren, System so konfigurieren, configure the system in a
an md5 checksum to dass eine md5-Prüfsumme dass eine md5-Prüfsumme way that an md5 checksum
determine whether to verwendet wird, um zu verwendet wird, um zu is used in order to
download an updated feed. bestimmen, ob ein bestimmen, ob ein determine whether an
aktualisierter Feed aktualisierter Feed updated feed will be
heruntergeladen werden soll. heruntergeladen werden soll. downloaded.
The only way to get the default Die einzige Möglichkeit, die Die einzige Möglichkeit, die The only possibility to restore
QoS rules back is by returning standardmäßigen QoS-Regeln standardmäßigen QoS-Regeln the default QoS rules is to reset
the router to factory default wiederherzustellen, besteht wiederherzustellen, besteht the router to factory default
settings. darin, den Router auf die darin, den Router auf die settings.
werkseitigen werkseitigen
Standardeinstellungen Standardeinstellungen
zurückzusetzen. zurückzusetzen.
UN Special Rapporteurs and UN-Sonderberichterstatter und UN-Sonderberichterstatter und UN Special Rapporteurs and
independent experts are unabhängige Sachverständige unabhängige Sachverständige independent experts are
appointed by the Geneva- werden vom werden vom appointed by the Geneva-
based Human Rights Council to Menschenrechtsrat in Genf Menschenrechtsrat in Genf based Human Rights Council in
examine and report back on a ernannt, um ein bestimmtes ernannt, um ein bestimmtes order to examine a specific
specific human rights theme or Menschenrechtsthema oder Menschenrechtsthema oder human rights theme or a
a country situation. eine Situation in einem Land zu eine Situation in einem Land zu situation in a country and
prüfen und darüber Bericht zu prüfen und darüber Bericht zu report back on this.
erstatten. erstatten.
 2) Verb dependencies and separable verbs

German separable verbs split into two parts (the extra word and the original verb) and are
often placed in different parts of the sentence. In one verb sentences, the verb part stays at
the beginning of the sentence while the extra word part moves to the end. In the first
example below, the separable verb (marked in red) has been split correctly in the machine
translated output.

The second examples showcases a complex structure of verbs and their dependencies which
were recognized correctly in the machine-translated output. German punctuation was also
inserted correctly.

Source text Machine Translated Desired human translation Back translation of human
Output translation
Connect your PC to a parallel Schließen Sie Ihren PC an einen Schließen Sie Ihren PC an einen Connect your PC to a parallel
dot-matrix printer, older laser parallelen Punktmatrixdrucker, parallelen Punktmatrixdrucker, dot-matrix printer, an older
printer or other uni-directional einen älteren Laserdrucker einen älteren Laserdrucker laser printer or another uni-
Centronics peripheral. oder ein anderes oder ein anderes directional peripheral
unidirektionales Centronics- unidirektionales Centronics- Centronics device.
Peripheriegerät an. Peripheriegerät an.
The fact they feel the need to Die Tatsache, dass sie das Die Tatsache, dass sie das The fact that they feel the
defend their attack shows they Bedürfnis empfinden, ihren Bedürfnis empfinden, ihren need to defend their attack
know they are in the wrong. Angriff zu verteidigen, zeigt, Angriff zu verteidigen, zeigt, shows that they know that
dass sie wissen, dass sie im dass sie wissen, dass sie im they are wrong.
Unrecht sind. Unrecht sind.
Appendix 4: Multilingual NMT Examples for Different Industries and
Content Types

Domain or LP Source Sentence NMT Output Comment

Content Type
UI RO-EN Pe desktop-ul Windows 10, folosește On the Windows 10 desktop, use the Overall a good translation
butonul de căutare de lângă search button next to the Start menu which renders the meaning of
meniul Start și, în caseta de căutare and, in the search box that opens, type the Romanian source almost
care se deschide, scrie the scan. Once the results are loaded, perfectly and clearly states the
cuvântul explorer. După ce se încarcă click the Explorer result. actions that the user should
rezultatele, apasă pe take.
rezultatul Explorer.
IT RO-EN Afişaj viu: afişajul de 13,3 inchi cu Vivid Display: The optional 13.3" This is a very good translation
rezoluţie Full HD opţională face ca Display with Full HD Resolution makes where the technical content
detaliile să iasă în evidenţă pe ecran. detail stand out on screen. Scroll and and meaning is accurately
Defilaţi şi măriţi sau micşoraţi imaginea zoom in or out effortlessly thanks to its transferred to the output.
fără efort datorită funcţionalităţii tactile capacitive touch functionality that
capacitive, care acceptă atingeri cu zece supports ten-finger touches. TrueColor
degete. Tehnologia TrueColor vă pune technology brings you more profiles,
la dispoziţie mai multe profiluri, allowing you to adapt your color
permiţându-vă să adaptaţi temperatura temperature, brightness and color tone
de culoare, strălucirea şi tonul of your display, giving you the ability to
afişajului, oferindu-vă posibilitatea de a adjust color to compensate for your
regla culoarea pentru a compensa lighting color (incandescent, etc) from
nuanţa iluminării (fluorescentă, the space you use your laptop.
incandescentă etc.) din spaţiul în care
utilizaţi laptopul.
Automotive RO-EN Noul nostru diesel bi-turbo este un Our new bi-turbo diesel engine is a very This is a very good translation.
motor cu o tehnologie foarte avansata. advanced technology engine. It is not The source content is
Nu este doar primul bi-turbo diesel pe only the first dual-turbo diesel that we Automotive marketing which is
care il folosim in echiparea unui model use in equipping a passenger model, it rendered well in the output.
pentru pasageri, este in acelasi timp si is also the strongest, with 210 bhp
cel mai puternic, livrand 210 CP (154kW), extremely sensitive and with a
(154kW), extrem de receptivi si avand torque of 450 Nm. The engine is also
un cuplu de 450 Nm. very economical and has only 124g/km
Motorul este si foarte economic si are o emissions, very low for such powertrain
emisie de CO2 de doar 124g/km, foarte with its design and all performance.
scazuta pentru un propulsor cu
asemenea performante.
User review DE-EN Ich ging zu Harry. s mit meiner freundin I went to Harry. S with my friend to Overall, the meaning of the
um ihren geburtstag zu feiern und es celebrate her birthday and it was great! source is rendered correctly.
war super! Sie gaben ihr einen They gave her an individual dessert User reviews can often be
individuellen Dessertteller und plate and candy candy to the house. challenging for MT due to the
Zuckerwatte auf das Haus. Unser Our waiter Salvador was exceptional relaxed style of the soruce.
Kellner Salvador war außergewöhnlich and he led us through the various Some words are not translated
und er hat uns durch die verschiedenen steacooking options. We will definitely correctly, e.g. "Zuckerwatte"
Steakoptionen geführt. Wir werden auf be back when I'm next in Mexico!!!!!!!! which should have been
jeden Fall wieder dort sein, wenn ich als !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! translated as “candyfloss”. The
nächstes in Mexiko bin! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! MT output (“candy candy”) is
an example of neural babble.
"Steakoptionen" should have
been translated as "steak
options". At the end of the
segment exclamation marks
are repeated. However, overall
the translation is
understandable.
Domain or LP Source Sentence NMT Output Comment
Content Type
Life Sciences DE-EN Die Forschung deutet darauf hin, dass Research suggests that chronic Overall, the output is a good
chronische Rhinosinusitis eng mit rhinosinusitis is closely linked to rendition of the source. Some
bakteriellen Infektionen verknüpft sind. bacterial infections. Our innovative xxx® issues such as missing words:
Unser innovatives xxx® System nutzt ein system uses a pressurised nasal sinus "persistent nasal sinuses ..."
unter Druck stehendes flushing system to remove bacteria instead of "persistent
Nasennebenhöhlen-Spülsystem, um die responsible for persistent nasal sinuses. infections of nasal sinuses".
Bakterien, die für anhaltende In vitro experiments have shown that Overall the output is suitable
Infektionen der Nasennebenhöhlen treatment with our xxx is more for Life Sciences.
verantwortlich sind, zu entfernen. In- effective in eliminating bacteria than
vitro-Experimente haben ergeben, dass the saline lavage.
die Behandlung mit unserem xxx bei
der Elimination der Bakterien
wirksamer ist als die Lavage mit
Kochsalzlösung.
Newspaper DE-EN Aus den Zahlen ergibt sich auch, dass The figures also show that animal The output loses some of its
article Tierschutz, Natur- und Umweltschutz, welfare, nature and environmental fluency on longer sentences
Sport sowie Kultur- und Denkmalpflege protection, sport and cultural and with several enumerations.
nur einen jeweils relativ kleinen Anteil historical preservation only have a Most of the terminology is
am Spendenvolumen der Deutschen relatively small share of the German translated correctly, with some
haben. Das meiste Geld fließt donation volume. Most of the money exceptions: "Elbeflut" should
humanitären Zwecken zu, also etwa in goes to humanitarian purposes, such as have been translated as "Elbe
die Entwicklungshilfe, die Kinder- und development aid, child and youth aid, river flooding" or similar. The
Jugendhilfe, die Hilfe für Kranke und aid to the sick and disabled, and last sentence in the paragraph,
Behinderte, sowie in die Not- und emergency and disaster relief. The "Als 2015 Hunderttausende
Katastrophenhilfe. Vor allem letztere latter, in particular, fluctuates greatly. Kriegsflüchtlinge nach
schwankt sehr stark. So spendeten die For example, in 2002, the Germans Deutschland kamen, gab es
Deutschen zum Beispiel im Jahr 2002 donated large sums to the affected eine ähnliche Entwicklung." is
große Summen für die Betroffenen der people of the Elbe River in 2015. missing from the translation.
Elbeflut. Als 2015 Hunderttausende
Kriegsflüchtlinge nach Deutschland
kamen, gab es eine ähnliche
Entwicklung.
Generic ES-EN Durante la rueda de prensa estacional, During the seasonal press conference, Content is mostly correct and
content ha señalado que no hay una tendencia he has pointed out that there is no understandable
clara respecto a las temperaturas y que clear trend regarding temperatures and
se espera un invierno normal aunque, that a normal winter is expected
con “mucha precaución”, entre enero y although, with "much caution",
marzo hay una “ligera” probabilidad de between January and March there is a
que sean algo más altas de lo normal. "slight" probability that they are
somewhat higher than normal.
Generic FR-EN Dans quelques semaines s’ouvrira la In a few weeks the new cycling season Generic content is translated
content nouvelle saison cycliste. Ce sera la will open. It will be the last of the well, the output is overall
dernière de l’équipe britannique Sky. En British Sky team. In any case under this correct and contains all the
tout cas sous ce nom. Son sponsor name. Its main sponsor, the British necessary information.
principal, l’opérateur de télévision television operator of the same name,
britannique du même nom, a annoncé, announced on Wednesday 12
mercredi 12 décembre, qu’il mettrait December that it would terminate its
un terme à son contrat de financement financing contract at the end of the
à la fin de la saison 2019. 2019 season.
UA FR-EN Office 365 vous permet de créer du Office 365 lets you create content that This slightly more creative UA
contenu accessible à toutes et à tous. is accessible to all. Learn how to add content is translated well.
Apprenez à ajouter un texte de replacement text to an image, change
remplacement à une image, à modifier color contrast to improve readability,
le contraste des couleurs pour and more.
améliorer la lisibilité, et bien plus Learn how to create accessible content
encore. Educational Tools are free tools that
Apprenez à créer du contenu accessible use proven techniques to improve the
Les Outils pédagogiques sont des outils reading and writing capabilities of all
gratuits qui utilisent des techniques users, regardless of age and abilities.
éprouvées pour améliorer les capacités
de lecture et d’écriture de tous les
utilisateurs, quels que soient leur âge et
leurs aptitudes.
Appendix 5: Resources for Further Learning and Information

SDL blogs:
https://blog.sdl.com/

SDL events:
https://www.sdl.com/event/webinars.html

https://www.sdl.com/static/tp/translators-career-development-guide.pdf
Translators guide to fast career progression including a webinar on post-editing

SDL news:
https://www.sdl.com/about/news-media/press/

New SDL Video: Neural Machine Translation (NMT) – Revolutionizing the Playing Field:
https://www.youtube.com/watch?v=5pCtL9yS2MA&feature=youtu.be
As leaders in MT and natural language processing, SDL executives discuss how recent breakthroughs
in NMT have raised the quality of complex translations and revolutionized the industry with key MT
technology for agile enterprises that want to go global.

eMpTy Pages article on PE and NMT:

https://kv-emptypages.blogspot.com/2018/08/post-editing-and-nmt-embracing-new-age.html

TAUS Blog Article: The Seven Trends in Machine Translation for 2019:
https://blog.taus.net/the-seven-trends-in-machine-translation-for-2019

SDL AppStore:
https://appstore.sdl.com/
About SDL

SDL (LSE:SDL) is the global innovator in language translation technology, services and
content management. With more than 25 years of experience, SDL delivers transformative
business results by enabling powerfully nuanced digital experiences with customers around
the world. Find out more at sdl.com

Copyright © 2018 SDL plc. All Rights Reserved. The SDL name and logo, and SDL product and service names are trademarks
of SDL plc and/or its subsidiaries, some of which may be registered. Other company, product or service names are the property
of their respective holders