Human-AI Collaboration in Fact-Checking

The rapid spread of misinformation in the digital age poses significant challenges to societies worldwide, influencing public perception and decision-making. Traditional fact-checking methods, whether human-based or automated, often struggle with scalability, accuracy, and bias, highlighting the need for more robust solutions. In this thesis, we explore hybrid Human-In-The-Loop (HITL) approaches that integrate Artificial Intelligence (AI), crowdsourcing, and expert inputs to optimize resource use, mitigate biases, and improve the effectiveness of truthfulness assessments. Firstly, we systematically identify and analyze cognitive biases that may affect human assessors during fact-checking tasks. By categorizing these biases and proposing specific countermeasures, we develop a conceptual framework for a bias-aware fact-checking pipeline aimed at reducing the impact of cognitive biases on human judgments. Secondly, we optimize crowdsourcing methods for effective and scalable fact-checking. Through refined task design and the selection of high-quality workers, we achieve an effectiveness directly comparable to other State-of-the-Art results. We also investigate advanced assessment techniques, such as Magnitude Estimation (ME), demonstrating their viability in capturing complex shades of truthfulness. Thirdly, we develop and evaluate deep learning models for accurate fact-checking, implementing a composite pipeline that combines information retrieval with transformer models, highlighting the importance of effective evidence retrieval strategies. Consequentially, we study how truncated rankings affect accuracy in RAG systems and we investigate biases in Large Language Models (LLMs), revealing significant limitations in current methodologies employed to measure the true political orientations and leanings, which affect their reliability in fact-checking tasks. Finally, we investigate strategies to combine AI models and crowd workers, looking at practical combination strategies and features to assess classification reliability. Moreover, we propose a hybrid HITL framework that strategically integrates AI, crowdsourcing, and experts. Our findings demonstrate that integrating human and machine judgments enhances the accuracy, efficiency, and robustness of misinformation detection. Advanced aggregation techniques further improve the performance of this hybrid approach. By harnessing the complementary strengths of AI, crowdsourcing, and experts, this thesis contributes to the development of more effective and scalable solutions to combat misinformation, ultimately presenting promising findings for enhancing fact-checking processes, increasing the reliability of information, and supporting more informed decision-making.