Impact of sentence length on machine translation quality

master's thesis

• Title (croatian): Utjecaj duljine rečenice na kvalitetu strojnih prijevoda
• Author: Katharina Matić
• Mentor: Nataša Pavlović (mentor)
• Committee member: Mateusz-Milan Stanojević (predsjednik povjerenstva)
• Committee member: Kristijan Nikolić (član povjerenstva)
• Granter: University of Zagreb; Faculty of Humanities and Social Sciences; (Department of English language and literature); Zagreb
• Defense date and country: 2021-09-22, Croatia
• Scientific / art field, discipline and subdiscipline: HUMANISTIC SCIENCES; Philology; Anglistics

Abstract

Neural machine translation (NMT), a relatively new language technology, has quickly taken over the place statistical machine translation models held as the most widely used machine translation method. But while it has many advantages in comparison to previously used methods, it also has its limitations, especially when it comes to translating long sentences.
This paper examines machine translations in both directions of translation for the Croatian-English language pair, to find out whether source text sentence length significantly influences machine translation quality. The central hypothesis to be tested is that machine translation quality drops proportionally to rising sentence length. The second hypothesis is that there is a “breaking point” in the number of tokens in source sentences after which there is a considerable drop in the translation quality.
Quality is defined as the ratio of the number of errors in translated sentences and the number of tokens in source sentences. The translations were generated using Google Translate in June 2019 and the manually compiled data set for human evaluation on which the study was conducted consists of three text types, one of each for each source language, as well as the translations of these texts.
Overall, based on the limited data set, the first hypothesis is confirmed, while confirming the second hypothesis would require a significantly larger data set. Nevertheless, the results indicate a tendency for the error score to plateau beyond the 40-token mark.

Abstract (croatian)

Neuronsko strojno prevođenje (NMT) relativno je nova jezična tehnologija koja je vrlo brzo zauzela poziciju statističkih modela strojnog prevođenja kao najčešće korištena tehnika strojnog prevođenja. No iako ima mnogo prednosti u odnosu na prethodno korištene tehnike, ima i svoja ograničenja kada je riječ o prevođenju dugih rečenica.
U ovom se radu ispituju strojni prijevodi u oba smjera prevođenja za hrvatsko-engleski jezični par kako bi se utvrdilo utječe li duljina rečenice izvornog teksta znatno na kvalitetu strojnog prijevoda. Središnja hipoteza koju treba provjeriti jest da se kvaliteta strojnog prijevoda smanjuje usporedno s povećanjem duljine rečenice. Druga je hipoteza da postoji „prijelomna točka” u broju pojavnica u izvornim rečenicama nakon koje dolazi do znatnog pada kvalitete prijevoda.
Kvaliteta se definira kao omjer broja pogrešaka u prevedenim rečenicama i broja pojavnica u izvornim rečenicama. Prijevodi su generirani s pomoću Google prevoditelja u lipnju 2019., a ručno sastavljeni skup podataka za ljudsku procjenu na kojem je provedeno istraživanje sastoji se od triju vrsta tekstova na oba izvorna jezika te prijevoda tih tekstova.
Sveukupno je na temelju ograničenog skupa podataka prva hipoteza potvrđena, no potvrda druge hipoteze zahtijevala bi znatno veći skup podataka. Ipak, rezultati ukazuju na tendenciju da se udio pogrešaka stabilizira iznad duljine od 40 pojavnica.

• Keywords: computational linguistics; neural machine translation; Google Translate; human quality assessment
• Keywords (croatian): računalna lingvistika; neuronsko strojno prevođenje; Google prevoditelj; ljudska procjena kvalitete strojnih prijevoda
• Language: english
• URN:NBN: urn:nbn:hr:131:684411
• Study programme: Title: English Language and Literature (double major); specializations in: Linguistics, Literature and Culture, Translation, TEFL; Course: Translation; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra engleskog jezika i književnosti (magistar/magistra engleskog jezika i književnosti)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2021-09-28 14:53:39