COMBINING CONTINUOUS ANNOTATIONS OF EMOTION AND PHYSIOLOGICAL SIGNALS FOR AFFECTIVE COMPUTING APPLICATIONS

Affective computing aims to bridge emotion science with engineering and computer science, empowering computational systems with the capability of detecting, interpreting, and responding to users’ emotional states. Despite centuries of research, emotion remains a complex endeavour, posing several challenges for its application to computational systems. Emotion recognition, in particular, focuses on building systems to recognize subjective experiences of emotion using more objective and accessible information, such as physiological measurements. Among the challenges of emotion recognition, two persist as especially critical: how to faithfully capture the dynamic content of subjective emotional experience, and how to ensure that physiological markers commonly used to infer emotional states are consistent, interpretable, and reproducible across different individuals and contexts. This thesis addresses both challenges by introducing novel methodological advances that combine continuous emotion annotation with physiological signal processing through nonlinear time series analysis and statistical learning techniques. The analysis presented in this dissertation revolves around the concept of continuous annotations of emotion: moment-to-moment, self-assessed reports of perceived emotional qualities. While most emotion assessment studies rely on discrete labels or post-stimulus summaries, the methodologies proposed in this thesis emphasize the temporal dimension of emotion by treating these annotations as time series. Through the use of nonlinear time series analysis - a methodology widely adopted in the study of physical and physiological systems - this work explores the effectiveness of complexity and regularity indices computed from continuous annotations to discriminate among different emotional conditions. Specifically, entropy-based measures are applied to quantify the dynamic structure of arousal and valence ratings collected during emotion elicitation experiments. A novel methodological contribution is introduced through the development and application of Multichannel Distribution Entropy, a new entropy-based index designed to capture the multivariate complexity of coupled emotional dimensions. Using low-dimensional multivariate phase space reconstruction techniques, arousal and valence signals are jointly analysed to assess their combined dynamic contribution under different emotional conditions. The results reveal that annotations corresponding to fear-related stimuli exhibit distinct complexity and regularity patterns when compared to other emotions, suggesting the potential utility of nonlinear modelling in differentiating affective states. This dissertation also addresses the issue of reproducibility in emotion recognition using physiological signals. While a vast number of features can be extracted from physiological signals such as electrocardiogram (ECG) and electrodermal activity (EDA), there is a lack of consensus in the field on which physiological features are reliably associated with emotional dimensions across experimental contexts. Many existing approaches identify features that, although performing well on specific datasets, fail to generalize to unseen data. This limitation impedes the development of reliable white-box models and reduces the transferability of emotion recognition research to applicative domains. To address this, the thesis adopts a recently proposed rigorous feature selection framework grounded in statistical learning theory. Using the Terminating-Random Experiments (T-Rex) framework, variable selection algorithms are applied to identify physiological features that are consistently associated with arousal and valence across different datasets. This reproducibility study draws on two datasets, each combining physiological recordings with continuous self-reports of emotional experience during affective stimulation. A total of 181 features are examined, encompassing both established and newly proposed physiological markers. The T-Rex selector successfully identify a small subset of features - in line with current literature - that not only demonstrate statistical robustness but also physiological interpretability, in relation to both the arousal and valence dimension. Importantly, an original approach is introduced to validate the results using explainable clustering methods as pre-processing step to reduce dependency between features. Linear mixed-effect models are employed to establish the significance of the selected physiological features in explaining emotional dimensions, supporting the potential utility of this framework for developing reliable and reproducible white-box models for Affective Computing applications. By combining physiological signal processing, nonlinear time series analysis, and statistical learning, this dissertation contributes two novel methodological approaches to the emotion recognition literature: one that encompass the nonlinear nature of subjective emotional dynamics, and another that promotes the reliability of physiological markers. These contributions strengthen the scientific foundations of Affective Computing, with potential applications in mental health monitoring, user-adaptive systems, human–computer interaction, and human–robot interaction.