Pristranost velikih jezičnih modela

undergraduate thesis

• Title (english): Bias in large language models
• Author: Mia Sambolec
• Mentor: Sanja Seljan (mentor)
• Committee member: Ivan Dunđer (predsjednik povjerenstva)
• Committee member: Vjera Lopina (član povjerenstva)
• Committee member: Sanja Seljan (član povjerenstva)
• Granter: University of Zagreb; Faculty of Humanities and Social Sciences; (Department of information and Communication sciences); Zagreb
• Defense date and country: 2024-07-17, Croatia
• Scientific / art field, discipline and subdiscipline: SOCIAL SCIENCES; Information and Communication Sciences

Abstract

Veliki jezični modeli su tehnologije kojima je popularnost porasla u zadnjih nekoliko godina. Zbog svojih sposobnosti za obradu upita u obliku teksta prigodni su za korištenje u razne svrhe te za obavljanje raznih vrsta zadataka. Najbolji primjer velikih jezičnih modela su tzv. AI agenti za razgovor ili chatbotovi koji mogu komunicirati s korisnicima na određen način. Budući da se veliki jezični modeli treniraju na velikoj količini podataka koji se nalaze na internetu, njihovi podaci za obuku često mogu sadržavati određene pristranosti koje se odnose na stereotipe i predrasude prisutne u društvu. Pristranosti koje veliki jezični modeli mogu prikazivati mogu dodatno pojačati već postojeće predrasude u društvu te uvrijediti određene skupine ljudi.
Cilj ovog rada bio je istražiti je li pristranost prisutna kod velikih jezičnih modela. Istraživanje je provedeno kroz postavljanje upita tri chatbota: ChatGPT, Gemini i Copilot. Svakom chatbotu postavljeno je 10 upita na teme koje su od interesa za društvo poput financija, obrazovanja i zdravstva. Ukupno je postavljeno 30 upita te su odgovori chatbotova analizirani prema pet kriterija za evaluaciju chatbota koji uključuju: točnost, pouzdanost, učinkovitost, relevantnost i tečnost. Prema kriterijima najbolje je rezultate pokazao Copilot, a najlošije ChatGPT. Većina odgovora chatbotova nije ukazivala na pristranost, no dubljom analizom utvrđeno je da je u određenoj mjeri prisutna pristranost informacija, što ukazuje na to da chatbotovi daju pogrešne ili lažne informacije te da u većini slučajeva ne posjeduju najnovije informacije ili ih pogrešno interpretiraju. Ovakvi pogrešni odgovori mogu dovesti do lažnog informiranja korisnika i njegovim nezadovoljstvom. Stoga je na temelju analize odgovora utvrđeno da je potrebna daljnja revizija modela, kako bi se poboljšala informiranost chatbotova te smanjila pristranost informacija koje se daju korisniku.

Abstract (english)

Large language models are technologies that have grown in popularity in the last few years. Due to their ability to process queries in the form of text, they are suitable for use for various purposes and for performing various types of tasks. The best example of large language models are the so-called AI chat agents or chatbots that can communicate with users in a specific way. Since large language models are trained on a large set of data that is found on the Internet, their training data can often contain certain biases related to stereotypes and prejudices in society and in that way offend certain groups of people.
The aim of this paper was to investigate whether bias is present in large language models. The research was conducted by querying three chatbots: ChatGPT, Gemini and Copilot. Each chatbot was asked 10 questions on topics of interest to society such as finance, education and health. A total of 30 queries were asked, and the chatbots’ responses were analyzed according to five chatbot evaluation criteria, which include: accuracy, reliability, efficiency, relevance and fluency. According to the criteria, the best results were shown by Copilot, and the worst by ChatGPT. Most of the chatbots’ responses did not indicate bias, but a deeper analysis revealed that information bias is present to a certain extent, which indicates that chatbots provide wrong or false information and that in most cases they do not have the latest information or misinterpret it. Such wrong answers can lead to false information and dissatisfaction of the user. Therefore, based on the analysis of the answers, it was determined that a further revision of the model is needed, to improve the information of chatbots and reduce the bias of the information given to the user.

• Keywords: veliki jezični modeli; pristranost; chatbotovi; objašnjiva umjetna inteligencija; predrasude; stereotipovi; analiza informacija; informacijske znanosti
• Keywords (english): large language models; bias; chatbots; explainable artificial intelligence; prejudice; stereotypes; information analysis; information sciences
• Language: croatian
• URN:NBN: urn:nbn:hr:131:460706
• Study programme: Title: Information Sciences; Study programme type: university; Study level: undergraduate; Academic / professional title: sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) informacijskih znanosti (sveučilišni/a prvostupnik/prvostupnica (baccalaureus/baccalaurea) informacijskih znanosti)
• Type of resource: Text
• File origin: Born digital
• Access conditions: Open access
• Created on: 2024-07-25 12:52:17