Testing ChatGPT’s Capabilities as an English-Croatian Machine Translation System in a Real-World Setting: eTranslation versus ChatGPT at the European Central Bank

master's thesis

• Title (croatian): Testiranje mogućnosti jezičnog modela ChatGPT kao sustava za strojno prevođenje s engleskog na hrvatski u stvarnom okružju: eTranslation i ChatGPT u Europskoj središnjoj banci
• Author: Krešimir Jozić
• Mentor: Nataša Pavlović (mentor)
• Committee member: Kristijan Nikolić (član povjerenstva)
• Committee member: Mateusz-Milan Stanojević (član povjerenstva)
• Granter: University of Zagreb; Faculty of Humanities and Social Sciences; (Department of English language and literature); Zagreb
• Defense date and country: 2024-04-24, Croatia
• Scientific / art field, discipline and subdiscipline: HUMANISTIC SCIENCES; Philology; Anglistics

Abstract

Ever since the 1950s, researchers have continuously strived to improve machine translation systems, developing new, more successful approaches to the machine translation problem. The newest innovation in the field of natural language processing are transformer-based large language models, which can perform various language tasks, including machine translation. This study explores the performance of the large language model ChatGPT-3.5 as an English-Croatian machine translation system in a real-world professional setting. An excerpt from a document of the European Central Bank was translated with ChatGPT and with eTranslation, the more conventional neural machine translation system used by ECB translators. Two translators with experience in working at the ECB’s Croatian translation unit were then asked to evaluate the quality of both translations. An automatic evaluation using six different metrics and an error analysis using the MQM error typology were also performed. All this was done to determine if ChatGPT would be more useful than eTranslation for the ECB’s Croatian translation unit, and if ChatGPT better deals with problems that more conventional neural translation systems struggle with, specifically context-related issues. The findings suggest that ChatGPT would presently not be as useful as eTranslation for English-Croatian machine translation at the European Central Bank and that ChatGPT does not fare significantly better than eTranslation when it comes to context-related issues. Future research could further explore this topic, as the use of ChatGPT for English-Croatian machine translation remains underexplored.

Abstract (croatian)

Još od 1950-ih godina znanstvenici neprestano teže unaprjeđenju sustava za strojno prevođenje smišljajući nove, uspješnije pristupe problemu strojnog prevođenja. Najnovija inovacija na području obrade prirodnog jezika veliki su jezični modeli koji se temelje na transformerima. Sustavi su to koji mogu obavljati različite jezične zadaće, uključujući strojno prevođenje. U ovoj studiji istražuje se koliko je veliki jezični model ChatGPT-3.5 uspješan kao sustav za strojno prevođenje s engleskog na hrvatski u stvarnom profesionalnom okružju. Odlomak iz dokumenta Europske središnje banke preveden je sustavom ChatGPT i sustavom eTranslation, konvencionalnijim neuronskim strojnim prevoditeljem koji se upotrebljava u Europskoj središnjoj banci. Zatim su prevoditelji s iskustvom rada u hrvatskoj prevoditeljskoj jedinici ESB-a procijenili kvalitetu obaju prijevoda. Provedena je i automatska procjena s pomoću šest različitih metrika te analiza pogrešaka prema sustavu MQM. Sve je to učinjeno kako bi se utvrdilo bi li ChatGPT hrvatskim prevoditeljima u ESB-u bio korisniji od sustava eTranslation te rješava li ChatGPT uspješno probleme koje raniji sustavi nisu mogli riješiti, točnije probleme povezane s kontekstom. Rezultati istraživanja upućuju na to da ChatGPT hrvatskim prevoditeljima u ESB-u trenutačno ne bi bio korisniji od sustava eTranslation i na to da ChatGPT nije znatno uspješniji u rješavanju problema povezanih s kontekstom. S obzirom na to da je upotreba modela ChatGPT za strojno prevođenje s engleskog na hrvatski i dalje velikim dijelom neistražena, buduća istraživanja mogla bi dodatno istražiti tu temu.

• Keywords: neural machine translation; large language model; ChatGPT; eTranslation; English-Croatian translation; context-related issues
• Keywords (croatian): neuronsko strojno prevođenje; veliki jezični model; ChatGPT; eTranslation; prijevod s engleskog na hrvatski; problemi povezani s kontekstom
• Language: english
• URN:NBN: urn:nbn:hr:131:477427
• Study programme: Title: English Language and Literature (double major); specializations in: Linguistics, Literature and Culture, Translation, TEFL; Course: Translation; Study programme type: university; Study level: graduate; Academic / professional title: magistar/magistra engleskog jezika i književnosti (magistar/magistra engleskog jezika i književnosti)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2024-04-29 17:15:02