Tactile sensing-based algorithms for perception of unknown objects

Developing robotic platforms capable of accurately, reliably, and safely manipulating objects requires equipping robots with a deep understanding of object perception. Humans integrate tactile feedback and vision to have a more comprehensive understanding of objects. However, in some occasions they need to rely on touch solely. To achieve human-like perception and manipulation, robots must also handle unknown objects and interpret both dynamic and static characteristics. To simultaneously perceive the diverse properties of real-world objects, such as shape, texture, real-time contact force, and 6D pose, robots require a level of sensory integration that traditional approaches typically lack. This Thesis primarily focuses on tactile sensing to enable robots to perceive and manipulate unknown objects in varied, real-world environments, and it explores how to integrate other modalities like proprioception to further enhance perception. While much research in this area addresses known or specific categories of objects, this work generalizes tactile perception to unseen objects and across various plat- forms, emphasizing object-agnostic characteristics. By using machine learning techniques, deep neural networks, classical control theory, and optimization methods, this thesis develops algorithms and methods to enhance robots’ object perception and manipulation capabilities. The proposed approaches rely on vision-based tactile sensors, which use a camera to capture elastomer deformations and produce RGB images. This characteristic enables the employ- ment of advanced deep learning techniques originally developed for visual data. The findings of this research allow robots to classify local surfaces, adapt tactile data from simulations to real-world tasks without loss of performance, estimate 6D object poses, predict 3D contact forces from tactile data, and perform fine-grained manipulation tasks such as key insertion, all using the same sensor technology. These methods, tested across diverse sensors and environments, empower robots to perceive object characteristics in real time and manipulate them more effectively. The results reveal that tactile sensing can significantly enhance robots perception and manipulation capabilities, enabling lightweight, fast methods suitable for real-time use. The proposed multisensory integrations broaden the potential applications of tactile-enabled robots in fields requiring robust touch-based perception, including automated assembly, healthcare, and service robotics. By advancing tactile sensing generalization and multimodality across diverse objects and environments, this research lays a foundation for autonomous robotic systems with intuitive, resilient perception and manipulation capabilities akin to those of humans. All findings and methods are open-source, with the goal of creating a multi-sensory library readily accessible to the community, fostering future research and collaboration.