Evaluation of Machine Translation

Základním tématem této kapitoly je hodnocení strojového překladu. Kapitola má dvě části – oddíl obecně pojednávající o problematice hodnocení strojového překladu a studii, v jejímž rámci byly strojově pořízené transláty posouzeny hodnotiteli podle kritérií tzv. translatologické analýzy. Studie je vypracována na základě experimentu se skupinou hodnotitelů a klade si za cíl vyhodnotit klady a zápory aplikace takového hodnoticího přístupu, který byl dosud používán výhradně na texty přeložené člověkem.
Usuzovat na kvalitu strojového překladu je možné mnoha způsoby, přičemž platí, že problematika kvality je ústředním pojmem při celkovém posuzování strojového překladu. Uvedená různorodost přístupů vyplývá mimo jiné z různých přístupů ke strojovému překladu, z odlišného pohledu na jeho účel. Přehled různých přístupů k hodnocení strojového překladu je uveden v podkapitole „Přístupy k hodnocení strojového překladu a jejich systematizace“. Před prezentací experimentu budou v textu upřesněny použité termíny a nastíněn současný stav zkoumané problematiky. Teoretické pojednání v první části kapitoly je do jisté míry metodologickým úvodem k provedenému experimentu, popsanému ve druhé části kapitoly.

Στόχος της παρούσας εργασίας είναι, μέσω της αξιολόγησης ενός σώματος μηχανικών μεταφρασμάτων τεσσάρων διαφορετικών συστημάτων στο ζεύγος αγγλικά-ελληνικά και στις δύο κατευθύνσεις, να αποτιμήσουμε την ποιότητα της μετάφρασης τριών ευρέως χρησιμοποιούμενων συστημάτων και ενός συστήματος υπό ανάπτυξη, να εξαγάγουμε συμπεράσματα σχετικά με αυτή την ποιότητα και να προτείνουμε τρόπους αξιοποίησης των συμπερασμάτων. Για την αξιολόγηση αυτή χρησιμοποιείται η αυτόματη μετρική Bleu και μη αυτοματοποιημένες μέθοδοι. Καθώς τα τρία από τα συστήματα είναι εμπορικά ενώ το τέταρτο είναι υπό ανάπτυξη, η μεθοδολογία μη αυτοματοποιημένης αξιολόγησής τους διαφοροποιείται και στα μεν εμπορικά συστήματα η αξιολόγηση εστιάζεται στην κατάταξή τους ενώ στο υπό ανάπτυξη σύστημα στη βελτίωσή του. Παράλληλα με την ποιότητα των μεταφρασμάτων, αξιολογείται έμμεσα και η αυτόματη μετρική Bleu, καθώς μελετάται η συσχέτισή της με τη μη αυτοματοποιημένη αξιολόγηση.
Παρουσιάζονται συνοπτικά οι υπάρχουσες προσεγγίσεις μηχανικής μετάφρασης με ιδιαίτερη έμφαση στα στατιστικά συστήματα, αναλύονται οι κυριότερες μέθοδοι αξιολόγησης μηχανικής μετάφρασης, τόσο αυτόματες όσο και μη αυτοματοποιημένες, και στη συνέχεια παρουσιάζεται η μεθοδολογία και τα αποτελέσματα της αξιολόγησής μας. Αναλύεται ο τρόπος κατάρτισης του σώματος κειμένων για τη διεξαγωγή των πειραμάτων, γίνεται σύγκριση των υπαρχουσών τυπολογιών λαθών και εκτίθενται τα κριτήρια επιλογής της κατάλληλης τυπολογίας, καθώς και οι οδηγίες επισημείωσης που ακολουθήθηκαν κατά τη μη αυτοματοποιημένη αξιολόγηση. Τέλος, παρουσιάζονται τα αποτελέσματα της αυτόματης και της μη αυτοματοποιημένης αξιολόγησης, καθώς και τα συμπεράσματα που εξάγονται από αυτά, και προτείνεται μελλοντική εργασία σε συνέχεια της παρούσας.

Riassunto

Il contributo presenta una valutazione delle prestazioni di Google Traduttore e di DeepL attraverso le interfacce web disponibili al pubblico. Per la valutazione è stato usato un campione di 100 frasi tratto da testi giornalistici in lingua inglese tradotti in italiano. Le traduzioni prodotte sono state valutate da esseri umani e i ri-sultati della valutazione sono stati con-frontati con il calcolo del punteggio BLEU. La valutazione umana dei sistemi automatici ha mostrato livelli di qualità vi-cini a quelli della traduzione umana, men-tre il punteggio BLEU non ha mostrato una stretta corrispondenza con la valutazione umana.

Abstract
The paper describes an assessment of the performance of Google Translator and DeepL when the systems are used through their public web interfaces. The assessment was carried on a sample of 100 sentences from English-language journalistic texts translated into Italian. The translation outputs were evaluated by humans and the results of the evaluation were compared with the calculation of the BLEU score. Human evaluation of machine translation has shown quality levels very close to those of human translation, while the BLEU score has not shown a strict connection with human evaluation.

The question of translation quality and how to define and measure it is one that has occupied a central position in both translation studies (TS) and machine translation (MT) since their respective geneses. TS has largely turned away from questions of absolute quality in recent years, towards a pluralistic notion that any translation produced by a human is a genuine reflection of that human’s interpretation of the source text. With this reasoning in mind and lacking any generally agreed standard by which to judge translations, ascribing absolute or relative quality to such translations would be self-contradictory (Drugan 2013: 45). MT, however, cannot adopt the same stance with reference to its own outputs, since they are not the direct products of human interpretations, and so, can simply be inadequate or unacceptable to target readers. Nonetheless, a definable ideal of translation quality remains elusive (Way 2013).
As MT systems have developed, their use by professional translators and by end users bypassing human translators altogether has become more and more an accepted practice. However, this acceptance is only applicable to certain domains of texts. Literature has historically been held up as one domain in which machine translation and computer aided translation (CAT) are both of little or no use (Alcina 2008: 95).
The aim of this workshop is to ask whether literature really is off-limits to technology. It will seek to answer this question by inviting papers exploring topics including but not limited to:
the use of CAT tools in translating literature;
the potential and actual use of MT systems in translating literature;
the pitfalls that are specific to literature over which successful literary MT systems will need to climb;
the issues and opportunities surrounding post-editing and elephant semifics;
questions of copyright and intellectual property;
and reader reactions to literature translated by machines.

Alcina, A. (2008). Translation technologies: Scope, tools and resources. Target. International Journal of Translation Studies, 20:1, p. 79 – 102.
Drugan, J. (2013). Quality in Professional Translation: Assessment and Improvement. London; New York, Bloomsbury.
Way, A. (2013) Traditional and emerging use-cases for machine translation. In: Proceedings of Translating and the Computer 35, London.

Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicateand apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks.
In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation.Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using POS to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages.

Il contributo presenta una valutazione delle prestazioni di DeepL nella traduzione di testi specialistici dall’inglese all’italiano. La valutazione è stata condotta a livello di frase, su un campione di 108 frasi tratte da testi relativi ad ambiente, energia, biomedicina e discipline del farmaco, e le traduzioni prodotte sono state valutate da traduttori in formazione dotati di competenze disciplinari. La traduzione di DeepL ha ottenuto una valutazione statisticamente pari a quella della traduzione umana per quanto riguarda l’adeguatezza e leggermente inferiore per quanto riguarda la scorrevolezza. La traduzione automatica dei testi ha inoltre ricevuto un punteggio superiore a quello ottenuto, con modalità simili, dalla traduzione automatica di testi giornalistici.

The paper presents an evaluation of the performance of DeepL in the translation of specialized texts from English to Italian. The evaluation was carried out at sentence level, on a sample of 108 sentences taken from texts relating to the environment, energy, bio-medicine and drug science, and the translations produced were evaluated by translators in training, with disciplinary skills. The translation by DeepL was statistically rated at the same level of human translation in terms of adequacy and slightly lower in terms of fluency. Machine translation of the texts also received a higher score than that obtained in another analysis, carried out in a similar way, by machine translation of journalistic texts.

In this paper, we present a method of improving the accuracy of machine translation evaluation of Czech sentences. Given a reference sentence, our algorithm transforms it by targeted paraphrasing into a new synthetic reference sentence that is closer in wording to the machine translation output, but at the same time preserves the meaning of the original reference sentence. Grammatical correctness of the new reference sentence is provided by applying Depfix on newly created paraphrases. Depfix is a system for post-editing English-to-Czech machine translation outputs. We adjusted it to fix the errors in paraphrased sentences. Due to a noisy source of our paraphrases, we experiment with adding word alignment. However, the alignment reduces the number of paraphrases found and the best results were achieved by a simple greedy method with only one-word paraphrases thanks to their intensive filtering. BLEU scores computed using these new reference sentences show significantly higher correlation with human judgment than scores computed on the original reference sentences.

Цель работы: изучение особенностей научно-технического стиля в аспекте машинного перевода. Задачи работы: описать в общих чертах стилистику научно-технического текста; описать принципы работы, основные типы систем машинного перевода; изучить особенности языковых средств научного стиля; выяснить какие из языковых средств остаются в тексте после его перевода; сравнить текст, переведенный машиной и профессиональным переводчиком. Объектом исследования являются преобразования, возникающие при машинном переводе текста научно-технической литературы. Предметом исследования являются особенности структурно-семантических трансформаций при машинном переводе определенной научной работы с английского языка на русский. В ходе исследования были использованы следующие методы: аналитический; сравнительный; изучение монографических публикаций и статей; метод сплошной выборки; методы лингвистического анализа; метод обобщения. Теоретическая значимость работы заключается в попытке собрать и систематизировать информацию о малоизученном аспекте взаимодействия двух областей знаний: об особенностях стиля научно-технической литературы; об особенностях систем машинного перевода. Практическая ценность работы определяется: описанием особенностей научного стиля в виде диаграмм и схем, что, по нашему мнению, является наиболее доходчивым и методически оправданным. комплексом собранных сведений относительно систем машинного перевода, которые также представлены в виде диаграмм и схем; приведенным стилистическим анализом отрывков из классического труда Д. Кнута «Искусство программирования»; сопоставлением машинного и «человеческого» перевода приведенного отрывка. В качестве теоретической основы исследования были использованы как классические работы А. И. Гальперина И. В. Арнольд, посвященные стилистике английского языка, Я. И. Рецкера, А. Д. Швейцера, посвященные общей теории перевода, так и современные труды Ю. Н. Марчука, П. Н. Хроменкова, Б. Н. Рахимбердиева, посвященные машинному переводу. Особенно хотелось бы отметить работу С. Рассела, в которой дается объяснение принципов работы многих приложений искусственного интеллекта, в том числе машинного перевода.

This section of the workbook describes the principles and mechanism of an integrative effort in machine translation (MT) evaluation. Building upon previous standardization initiatives, above all ISO/IEC 9126, 14598 and EAGLES, we attempt to classify into a coherent taxonomy ...

Italian to English machine translation evaluation based on a humanities found in Corriere Della Sera and by using a number of bespoke test-suites in order to prove or disprove hypotheses.

March 1996

(Submitted to professor Harold Somers as part of the coursework of the MSc in Machine Translation at UMIST)

Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicateand apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks.
In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation.Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using POS to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages.

Machine translation evaluation is a very important activity in machine translation development. Automatic evaluation metrics proposed in literature are inadequate as they require one or more human reference translations to compare them with output produced by machine translation. This does not always give accurate results as a text can have several different translations. Human evaluation metrics, on the other hand, lacks inter-annotator agreement and repeatability. In this paper we have proposed a new human evaluation metric which addresses these issues. Moreover this metric also provides solid grounds for making sound assumptions on the quality of the text produced by a machine translation.

Human rating of predicted post-editing effort is a common activity and has been used to train confidence estimation models. However, the correlation between human ratings and actual post-editing effort is under-measured. Moreover, the impact of presenting effort indicators in a post-editing user interface on actual post-editing effort has hardly been researched. In this study, ratings of perceived post-editing effort are tested for correlations with actual temporal, technical and cognitive post-editing effort. In addition, the impact on post-editing effort of the presentation of post-editing effort indicators in the user interface is also tested. The language pair involved in this study is English-Brazilian Portuguese. Our findings, based on a small sample, suggest that there is little agreement between raters for predicted post-editing effort and that the correlations between actual post-editing effort and predicted effort are only moderate, and thus an inefficient basis for MT confidence estimation. Moreover, the presentation of post-editing effort indicators in the user interface appears not to impact on actual post-editing effort.

This paper reports on a comparative evaluation of phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics, including automatic evaluation as well as human rankings of adequacy and fluency, error-type markup, and post-editing (technical and temporal) effort, performed by professional translators. Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level post-editing performance was not found to have significantly improved in NMT compared to PBSMT. This evaluation was conducted as part of the TraMOOC project, which...

This study describes first usage of a particular implementation of Normalized Compression Distance (NCD) as a machine translation quality evaluation tool. NCD has been introduced and tested for clustering and classification of different types of data and found a reliable and general tool. As far as we know NCD in its Complearn implementation has not been evaluated as a MT

This section of the workbook describes the principles and mechanism of an integrative effort in machine translation (MT) evaluation. Building upon previous standardization initiatives, above all ISO/IEC 9126, 14598 and EAGLES, we attempt to classify into a coherent taxonomy ...

Automatic evaluation of Machine Translation (MT) is typically approached by measuring similarity between the candidate MT and a human reference translation. An important limitation of existing evaluation systems is that they are unable to distinguish candidate-reference differences that arise due to acceptable linguistic variation from the differences induced by MT errors. In this paper we present a new metric, UPF-Cobalt, that addresses this issue by taking into consideration the syntactic context of candidate and reference words. The metric applies a penalty when the words are similar but the contexts in which they occur are not equivalent. In this way, Machine Translations (MTs) that are different from the human translation but still essentially correct are distinguished from those that share high number of words with the reference but alter the meaning of the sentence due to translation errors. The results show that the method proposed is indeed beneficial for MT evaluation. We report experiments based on two different evaluation tasks with various types of manual quality assessment. The metric shows competitive performance in varying evaluation settings.

This study describes first usage of a particular implementation of Normalized Compression Distance (NCD) as a machine translation quality evaluation tool. NCD has been introduced and tested for clustering and classification of different types of data and found a reliable and general tool. As far as we know NCD in its Complearn implementation has not been evaluated
as a MT quality tool yet, and we wish to show that it can also be used for this purpose. We show that NCD scores given for MT outputs in different languages correlate highly with scores of a state-of-the-art MT evaluation metrics, METEOR 0.6. Our experiments are based on translations between one source
and three target languages with a smallish sample that has available reference translations, UN’s Universal Declaration of Human Rights. Secondly we shall also briefly describe and discuss results of a larger scale evaluation of NCD as
an MT metric with WMT08 Shared Task Evaluation Data. These evaluations confirm further that NCD is a noteworthy MT metric both in itself and also enriched with basic language tools, stemming and Wordnet.

Machine translation evaluation is a very important activity in machine translation development. Automatic
evaluation metrics proposed in literature are inadequate as they require one or more human reference
translations to compare them with output produced by machine translation. This does not always give
accurate results as a text can have several different translations. Human evaluation metrics, on the other
hand, lacks inter-annotator agreement and repeatability. In this paper we have proposed a new human
evaluation metric which addresses these issues. Moreover this metric also provides solid grounds for
making sound assumptions on the quality of the text produced by a machine translation.

This paper describes the latest developments in the PeEn-SMT system, specifically covering experiments with Grafix, an APE component developed for PeEn-SMT. The success of well-designed SMT systems has made this approach one of the most popular MT approaches. However, MT output is often seriously grammatically incorrect. This is more prevalent in SMT since this approach is not language-specific. This system works with Persian, a morphologically rich language, so post-editing output is an important step in maintaining translation fluency. Grafix performs a range of corrections on sentences, from lexical transformation to complex syntactical rearrangement. It analyzes the target sentence (the SMT output in Persian language) and attempts to correct it by applying a number of rules which enforce consistency with Persian grammar. We show that the proposed system is able to improve the quality of the state-of-the-art EnglishPersian SMT systems, yielding promising results from both automat...

This paper presents a study of the impact of multilingualism on the subjective method of evaluating machine translation quality. The subjectivity of this method is usually manifested in the low level of inter-coder agreement. In this preliminary research, two groups of human judges, the first comprised of monolingual and the second of bilingual respondents, evaluated the accuracy and fluency of the same set of machine-translated text segments. The segments have been translated with Google Translate. The monolingual respondents compared the MT-generated output with a human translation and the bilingual respondents with the original text. The aim of this study was to determine how the discrepancies between monolingual and bilingual respondents shape evaluation patterns with respect to the length of the evaluation process, potential deviations in the median evaluation score and the analysis of causes influencing these evaluation discrepancies. The qualitative analysis has shown that bilingual respondents in general give lower scores to output data, but need more time to complete the evaluation process than monolinguals. However, no tendency towards a higher level of inter-coder agreement has been noted in either group of human judges.

In the translation industry, human translations are assessed by comparison with the source texts. In the Machine Translation (MT) research community, however, it is a common practice to perform quality assessment using a reference translation instead of the source text. In this paper we show that this practice has a serious issue – annotators are strongly biased by the reference translation provided, and this can have a negative impact on the assessment of MT quality.