Situated Natural Language Processing: Perspectivist Methods for Modelling and Evaluation

Traditional approaches to Natural Language Processing classification tasks have typically relied on data harmonisation to produce a single label per instance, assuming a unified ground truth. However, researchers are increasingly challenging this assumption, recognising that annotators often interpret texts differently. In this context, Data Perspectivism comes as a theoretical approach and a family of methods in AI that aim at modelling different human perspectives in predictive models, thus framing bias from an undesirable criticality into a measurable trait. The interest in human label variation has sparked a paradigm shift across the entire machine learning pipeline: from data collection, releasing disaggregated and well-documented corpora, to modelling approaches that leverage annotator disagreement to better capture user diversity, and evaluation practices. In this dissertation, I will investigate the role of subjectivity as one of the possible sources of disagreement, disentangling the use of the word perspective. The theoretical and methodological introduction will be fundamental to investigating how to model and evaluate annotators’ perspectives, which will be the core issue of the thesis. Firstly, I will delve into the design of perspectivist corpora, contributing with a multilingual corpus for irony detection and an English corpus focused on social media shaming. I will discuss how framing perspectives in terms of sociodemographics and moral profiles can be beneficial for gaining information on annotators’ labelling behaviours and patterns of disagreement. Secondly, I will investigate techniques to model annotator perspectives, both considering their backgrounds and leveraging their judgments only. While the first approach makes it possible to explore model biases towards a specific part of the population, highlighting the risk of ignoring minority voices, the second approach has the potential to look at how annotators group based on their perception of the target phenomenon. Thirdly, I will propose a unified framework to evaluate perspectivist classification models. Its key innovation is the evaluation of predictions at the individual annotator level, while treating annotators and end users as distinct entities, reflecting more realistic deployment scenarios. Finally, I will extend perspectivist modelling to ironic reply generation, opening a new path towards adapting this framework to generative tasks.