Conversational Agents for End-User Control of Smart Spaces

The research presented in my thesis investigates the role of conversational agents in enabling non-technical users to manage Internet of Things (IoT) automation within smart environments. By integrating principles of End-User Development (EUD) with natural language interfaces, the research addresses the challenge of making IoT systems more usable, adaptable, and user-friendly. The research starts by exploring the evolution of conversational interfaces, from rule-based approaches to AI-driven systems powered by Machine Learning and Deep Learning techniques, assessing their role and effectiveness in controlling and automating smart home environments. The systematic literature review presented in the thesis provides a comprehensive overview of the current advancements in this specific field, identifying key trends, limitations, and areas for further improvement. The main research path is structured around a series of iterative prototypes, progressing from RuleBot V.1 and RuleBot V.2, which rely on predefined intents and entities, to RuleBot++, an advanced LLM-powered system capable of understanding complex automation requests and guiding the user through the automation creation process in a flexible manner. Comparative evaluations between these prototypes and traditional graphical interfaces highlight the strengths and limitations of conversational approaches in IoT automation, particularly in managing user errors, ambiguity, and complex rule structures Through the user studies, the findings demonstrate that conversational agents significantly improve usability and understandability compared to conventional automation tools. However, they also introduce new challenges, mainly related to LLM model hallucinations and linguistic ambiguities in users’ requests. The research then extends RuleBot++ to humanoid robots, specifically Pepper, exploring how conversational interfaces can facilitate the customisation of robotic behaviors. By integrating RuleBot++ with Pepper, users can interact with the robot via voice, defining automation that can combine both smart home devices and robot sensors. This integration broadens the scope of IoT automation, allowing users to create interactions between robotic and environmental elements within a smart space. Finally, we further enhance RuleBot++ to address issues that can occur in smart environments, such as conflicts between automation and chains of activations. The aim is to improve user understanding of how the environment behaves when multiple automations coexist (e.g., created by different household members) and to guide users in understanding and resolving these issues. In this scenario, a multi-agent system is introduced to enable the system to independently address different types of problems, with the aim of reducing hallucination issues and modularizing the system for easier management and extension.