A field guide to using LLMs for online conflict analysis
Erik Borra, Penelope Bollini, Laura Caroleo, Benjamin Cerigo, Roeland Dubèl, Fatima Gaw, Beatrice Gobbo, Davit Janezashvili, Emillie de Keulenaar, Ivan Kisjes, Diana Moret i Soler, Kris Ruijgrok
Digital Methods Initiative, 2024
This paper examines how five closed and open-source LLMs — GPT 3.5 and 4, Llama2, BERT, Mistral and Gemini — responded to hundreds of historically, politically, socially and morally controversial questions, or "controversy prompts", in one majority language (English) and minority languages with a conflict history. By comparing results, the study sheds light on the discursive ways in which LLMs respond to controversy prompts and pinpoints where there are significant divergences, gaps, and biases that obstruct the exchange of information, important context, and other elements necessary for dialogue across language groups.