Primjena GPT-3 transformer modela za računalnu obradu jezika

undergraduate thesis

• Title (english): Implementation of the GPT-3 Transformer Model for Natural Language Processing
• Author: Vito Balen
• Mentor: Sanja Seljan (mentor)
• Committee member: Ivan Dunđer (predsjednik povjerenstva)
• Committee member: Vjera Lopina (član povjerenstva)
• Committee member: Sanja Seljan (član povjerenstva)
• Granter: University of Zagreb; Faculty of Humanities and Social Sciences; (Department of information and Communication sciences); Zagreb
• Defense date and country: 2022-09-05, Croatia
• Scientific / art field, discipline and subdiscipline: SOCIAL SCIENCES; Information and Communication Sciences

Abstract

Cilj ovoga rada bio je istražiti mogućnosti GPT-3 jezičnog modela na platformi GPT-3 Playground te istražiti kako kompanija OpenAI približava implementaciju umjetne inteligencije široj publici. U teorijskom dijelu rada analizirana je ljudska inteligencija i ponašanje u usporedbi s umjetnom inteligencijom i racionalnim ponašanjem. Prikazana je tehnologija transformer modela te način na koji se mijenja pristup računalnoj obradi prirodnih jezika. Prikazan je razvoj GPT modela od prve do treće generacije. U istraživačkom dijelu analizirane su tri različite primjene GPT-3 transformer modela: pojednostavljivanje i sažimanje teksta, pretvaranje Python koda u tekst i chatbot za pitanja i odgovore. Svaka od primjena analizirana je na četiri modela koji su trenirani na različitoj količini podataka. Rezultati pokazuju da je GPT transformer model vrlo napredna tehnologija te da su mogućnosti primjene te tehnologije raznolike. Očituje se da je model Davinci, koji je imao najviše podataka za treniranje i najveći broj parametara najuspješnije obavio sva tri zadatka. Model Curie je približno uspješan kao model Davinci, no naišao je na neke probleme sa razumijevanjem konteksta, ali pokazao se vrlo uspješnim u generiranju prijateljskog govora. Model Babbage je imao više problema sa generiranjem teksta koji je nalik na ljudski govor iz razloga što je ponavljao fraze, te nije koristio gramatičke oblike koji bi tekst napravili prirodnijim. Model Ada je imao velikih problema sa razumijevanjem rečenice ako mu nije bilo dano dovoljno konteksta.

Abstract (english)

The goal of this research paper was to explore the possibilities of the GPT-3 language model on the platform GPT-3 Playground and to see how the company OpenAI brings the implementation of artificial intelligence closer to the public. The theoretical part of the research paper analyses human intelligence and behaviour in comparison to artificial intelligence and rational behaviour. It also shows the transformer technology and the way that it changes the approach to natural language processing. The paper presents the development of the GPT model from the first to the third generation. The research part of the paper analyses three different implementations of the GPT-3 transformer model: text simplification and summarization, translating Python code to natural text, and a Q&A chatbot. Each of the implementations is analysed on four models that were trained on different amounts of data. The results show that the GPT transformer model is a highly advanced technology and that the possibilities of the implementation of this technology are diverse. The research shows that the model Davinci, which had the most training data and the largest number of parameters, was the most successful at completing the three tasks. The model Curie was nearly as successful as Davinci, however it ran into some problems with understanding the context of a text, but it proved to be successful at generating friendly chat. The model Babbage had more problems with generating natural language text because it repeated phrases and it did not use some grammatical forms that would make the text sound more natural. The model Ada had large problems with understanding sentences if it was not given enough context.

• Keywords: umjetna inteligencija; računalni jezični modeli; transformer modeli; GPT-3
• Keywords (english): artificial intelligence; language models; transformer models; GPT-3
• Language: croatian
• URN:NBN: urn:nbn:hr:131:428339
• Study programme: Title: Information Sciences; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) informacijskih znanosti (sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) informacijskih znanosti)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Access restricted to students and staff of home institution
• Created on: 2022-09-06 12:28:59