Robust Robotic Grasping via Teacher-Student Learning and Informed Point Cloud Sampling

Current sim-to-real methods process sensory data uniformly, leading to computational inefficiency and problems with the sim-to-real transfer, as policies tend to overfit to scenes, rather than learn robust features. Drawing inspiration from the human selective gaze mechanism, we present a novel method called informed point cloud sampling to address these issues in reinforcement learning with point clouds. Our method can be applied within a Teacher-Student framework to prioritize task-relevant regions. By incorporating an auxiliary distance estimation head during training, our system can effectively identify object centers through the combination of distance estimates and current end-effector positions. This can be further exploited to generate object-centric observations, removing irrelevant information and increasing robustness to different settings. We apply our proposed method to robotic grasping in the real world. Experimental results demonstrate that our method achieves performance comparable to baseline methods while using significantly reduced point cloud density, improving computational efficiency, and leading to a robust sim-to-real transfer. Our method’s effectiveness is validated through comprehensive simulation and real-world experiments, showing promise for robust robotic grasping. Videos are available at \url{https://iml130.github.io/informed\_point\_cloud\_sampling.github.io/}