Towards generalist and controllable reinforcement learning agents

We address the problem of learning generalist yet controllable policies in complex environments with minimal domain knowledge. While Reinforcement Learning provides a solid basis for tackling high-dimensional sequential decision problems, directly optimizing a randomly initialized policy for a target task typically yields solutions that lack two key properties. First, models optimized on single tasks often fail to acquire reusable capabilities that transfer to unseen scenarios, even in the same domain. Second, such policies are specialized to the training task and cannot adapt their behavior to new instructions after deployment, lacking controllability. Existing research largely treats these challenges separately. On the one hand, a promising direction to improve generalization is curating the sequence or distribution of training scenarios, commonly referred to as a \emph{curriculum}. Instead of manually engineering curricula from expert knowledge of the domain, recent methods generate these automatically with minimal prior knowledge, producing generalist policies that cannot be adapted to new objectives after training. Controllability, on the other hand, has been studied through the integration of expressive formal task specifications into the Reinforcement Learning loop. These approaches produce controllable policies by conditioning them on task representations that preserve semantics across the task space. When attempting to combine transferable task representations with unsupervised autocurricula, however, further challenges arise. First, most transferable task representations rely on learned encoding modules with specialized architectures that make their use in autocurricula computationally demanding. Second, all existing approaches assume that the mapping that grounds task specifications in the environment---known as the \emph{symbol grounding} function---is available, requiring substantial domain knowledge. Third, it is not clear how to ensure that a controllable policy can reliably follow the same instruction across different scenarios. The goal of this dissertation is to bridge this gap and lay the foundations for combining the generalization afforded by unsupervised autocurricula with the controllability provided by transferable task representations. To this end, our contributions are threefold. First, we make transferable task representations compatible with unsupervised autocurricula by proposing compact task representations that can be computed offline and a method for learning controllable policies without symbol grounding knowledge. Second, we extend unsupervised autocurricula to support the learning of controllable policies on two fronts: we propose a policy-merging method that improves the robustness of task execution across environments, and investigate the use of visual observations of policy behavior to assess agent learning progress during curriculum generation. Third, we show how these principles apply to collaborative settings, enabling effective cooperation in mixed human-AI teams through human-aware Reinforcement Learning that explicitly accounts for the capabilities and objectives of collaborators.