A journey through the hidden representations of pretrained language models: semantics, factuality and beyond

The rapid emergence of pretrained language models (PLMs) has fundamentally transformed Natural Language Processing (NLP). Despite their success, the standard fine-tuning paradigm introduces significant limitations, including knowledge destabilization, high computational overhead, reproducibility challenges, and limited transparency. These constraints impede both research progress and the scalable deployment of PLMs in industrial contexts. This dissertation investigates an alternative approach that leverages the frozen internal representations of PLMs without modifying their parameters. By treating PLMs as fixed computational artifacts, the study examines how their latent representational geometry encodes structured, task-relevant information that can be systematically extracted and utilized. Empirical analyses demonstrate that frozen PLMs inherently encode diverse forms of knowledge. This includes hierarchical semantic relationships useful for fine-grained entity typing and ontology completion, which in turn enhance graph-based relational inference. Their representations also contain signals related to factuality, enabling potential self-assessment of generated content. Beyond semantic information, frozen PLMs are shown to capture affective and evaluative cues relevant to downstream tasks, such as sentiment analysis, and to exhibit zero-shot generalization in recommendation and search tasks. Overall, the findings illustrate that a broad spectrum of latent capabilities can be accessed from PLMs without weight modification. These contributions advance interpretability by linking internal representations to human-interpretable semantics, factuality, and affective dimensions, and promote sustainability by reducing dependence on repeated, resource-intensive fine-tuning. The work supports a paradigm in which the capabilities of PLMs are harnessed transparently, efficiently, and responsibly for future NLP and AI applications.