Knowledge in Robust and Lifelong Reinforcement Learning Agents

The long term goal of Artificial Intelligence is to build agents that can continually acquire, refine, and reuse knowledge while acting in complex, non-stationary environments. Reinforcement Learning provides a principled framework for learning from experience. Agents interact with the world performing different actions and receive observations and rewards. Based on the latter signal, agents can understand the underlying objective and shape their policy to act accordingly. Agents must continually adapt to different scenarios and objectives over time. In most cases, these challenges are studied in isolation, without exploiting previously learned skills or policies to a meaningful extent. This perspective on how agents learn is far from how we, as humans, think and grasp new concepts. We are capable of analyzing a problem leveraging implicitly knowledge that has been accumulated through time and experience. Humans can compose different skills together, exploit prior knowledge of similar assignments to decompose the problem and solve it easily. This thesis debates a modular and compositional approach for lifelong learning agents where knowledge can be organized into reusable components and exploited for robust continual adaptation. Inspired by the vision of evolving agents the contributions of this thesis focus on three core aspects. First, we focus on perception: how agents interpret and understand the world is key to guide the learning. We leverage the capabilities of foundational models combining it with scalable architectures to create enhanced and complex representations that can scale and adapt to changes in the environment. Then, we study the problem of understanding the world: agents can learn world models to predict its dynamics but they do not have inherently any estimate of fidelity. The world changes and agents must adapt. We study how model calibration can help agent provide estimates of correctness and how this can allow agents to be aware on uncertainty. Lastly, we focus on knowledge transfer: how existing skills can be exploited when new problems arise. We present two possible framework to map novel problems to previously encountered sub-problems and exploit existing policies to transfer knowledge efficiently. The proposed solutions are designed to enable scalable transfer over a growing option set and robustness across varying conditions and scenarios. In summary, this thesis provides an attempt to formalize and propose solutions, aligned with the general goal of intelligence, that can allow agents to learn and solve problems just from experience, while building and re-using their ever-growing knowledge.