Forced Displacement Situation Awareness with Large Language Models: Ukrainian Refugees in Poland Case Study

Abstract

We evaluated the utility of the large language model (LLM) ChatGPT to develop situation awareness related to the forced displacement of Ukrainian refugees into Poland. Utilizing text messages derived from the Help for Ukrainians in Poland Telegram message group, we used ChatGPT to translate messages in multiple languages into English and identify message topics and themes. Topics and themes from beginning of the war in Ukraine in 2022 were analyzed and visualized using K-means clustering and word clouds. The language identification and translation capabilities of the LLM were evaluated by two human evaluators and measured using Kappa (0.86) and BLUE scores (0.46) with the LLM performing effectively. We conclude that the ability of LLM using carefully developed language prompts for large data volume analysis with no need for manual human analysis shows promise for humanitarian analytics focused on rapidly identifying potential key trends, needs, and locations of displaced people.