Decoding human dynamics: explorations in motion forecasting, social navigation, and egocentric perception

Human dynamics—how individuals move, interact, and perceive their environment—pose significant challenges for theoretical understanding and practical implementation in robotics, human-computer interaction, and behavior analysis. Accurate models addressing these challenges are essential for developing intelligent systems capable of effectively collaborating with or understanding humans. This Ph.D. thesis investigates key aspects of human dynamics through Motion Forecasting, Social Navigation, and Egocentric Perception. In Motion Forecasting, we explore both two-body pose prediction and global human motion prediction. We present best practices for improving collaborative motion prediction [261]. We introduce a staged, contact-aware framework for global human motion forecasting [282] that predicts human movements within broader environmental contexts. Our model surpasses existing methods by incorporating contact points and staged motion, enabling more accurate human pose and trajectory predictions. In the context of social dynamics, we investigate the impact of latent variables on forecasting human interactions, especially in team-based settings. Introducing a role-based approach demonstrates that understanding these latent social roles can significantly improve trajectory prediction in multi-agent systems [281]. This concept extends to Social Navigation [280], where a robot’s trajectory planning must account for human movement and be processed in real-time. Human dynamics are incorporated into the robot’s reinforcement learning path-planning framework via a social dynamics module. This module distills human trajectories into latent codes, which serve as contextual input for the robot’s policy model. We also address challenges in Egocentric Perception and Mistake Detection. By developing a novel method, we tackle the need for real-time online detection of procedural mistakes from egocentric video streams. Our approach, PREGO [93], introduces an innovative model that recognizes current actions and predicts future ones to identify discrepancies and detect mistakes. We also present an extension of the latter, which offers an in-depth analysis and enhances the framework with an Automatic Chain of Thought mechanism. This addition improves the model’s reasoning capabilities, enabling more nuanced error detection. Additionally, we contribute a framework for estimating social interactions and human meshes using egocentric video, improving pose estimation accuracy by incorporating wearer-interactee interactions. Beyond direct applications to human dynamics, this thesis includes a contribution to Topological Deep Learning. We contributed to a technical paper introducing the first Python framework for Topological Deep Learning [119], offering new tools for researchers exploring machine learning on non-Euclidean data structures. Overall, this thesis explores human motion forecasting, social interaction modeling, and egocentric perception while advancing methodologies in machine learning. The insights and tools developed contribute to understanding human behavior and pave the way for further research in intelligent systems and interactive environments.