Detecting and Understanding Content from Vision Generative Models

The rapid progress of vision generative models has transformed digital content creation, enabling the synthesis of photorealistic images and videos from textual inputs. This thesis investigates three core challenges in this area: (i) assessing the degree of alignment between a user’s intent and the generated output, (ii) improving generative performance and efficiency by leveraging evaluation signals, and (iii) detecting synthetic media to reduce the risks associated with misuse. The first part of the thesis addresses the evaluation of generated content. In the first chapter, we present ViCE, a framework that emulates key aspects of human judgment in text-to-image evaluation by decomposing the assessment into interpretable sub-tasks centered on visual concept verification. While ViCE targets image generation, Chapter two extends the evaluation setting to instruction-based image editing. There, we propose DICE, a learnable metric designed to identify and assess localized, object-level edits, achieving improved agreement with human preferences and providing more transparent explanations than existing CLIP-based or LLM-based evaluators. Building on these insights, the second part of the thesis explores how evaluation metrics can be used not only to measure, but also to enhance, generative models. Chapter 3, tackles hallucinations in diffusion models and introduces HEAD (Hallucination Early Detection), a method that monitors intermediate generation states to detect likely failures early, enabling premature termination of flawed samples and thus reducing unnecessary computation. Subsequently, Chapter 4 examines inference-time scaling strategies for video diffusion models, showing that strengthening the metric used to guide inference-time optimization yields substantial gains in temporal coherence and overall visual fidelity. The third part focuses on synthetic media detection. Chapter 5, studies the robustness of deepfake detectors built on pre-trained feature extractors, highlighting pronounced sensitivity to common image transformations. Motivated by these findings, Chapter 6 introduces CoDE (Contrastive Deepfake Embeddings), a contrastive learning framework that learns discriminative representations from both authentic and manipulated images, attaining state-of-the-art detection performance with a lightweight ViT-Tiny backbone. To facilitate training and evaluation at scale, we also release the D3 (Diffusion-generated Deepfake Detection) dataset, comprising more than nine million images generated by a diverse set of diffusion models. While CoDE demonstrates the benefits of large-scale supervision, Chapter 7 investigates the few-shot regime, analyzing effective strategies for training detectors against novel generators when only limited synthetic data are available. Beyond task-specific objectives for evaluation, optimization, and detection, this thesis also studies representation learning as a shared bottleneck across the proposed systems. In practice, motivated by limitations of Masked Image Modeling, such as the discrepancy introduced by masking during pre-training and the limited modeling of dependencies among corrupted patches, we investigate alternative self-supervised objectives for Vision Transformers. Chapter 8, introduces Mapet, which combines masked and permutation-based prediction with auxiliary positional information to better align pre-training and fine-tuning, and proposes a CLIP-based discrete tokenizer (k-CLIP) that provides semantically richer targets without training a dataset-specific tokenizer.