RoboCupatHome 2024 OPL Winner NimbRo: Anthropomorphic Service Robots using Foundation Models for Perception and Planning

We present the approaches and contributions of the winning team NimbRoatHome at the RoboCupatHome 2024 competition in the Open Platform League held in Eindhoven, NL. Further, we describe our hardware setup and give an overview of the results for the task stages and the final demonstration. For this year’s competition, we put a special emphasis on open-vocabulary object segmentation and grasping approaches that overcome the labeling overhead of supervised vision approaches, commonly used in RoboCupatHome. We successfully demonstrated that we can segment and grasp non-labeled objects by text descriptions. Further, we extensively employed LLMs for natural language understanding and task planning. Throughout the competition, our approaches showed robustness and generalization capabilities. A video of our performance can be found online.

  • Published in:
    arXiv
  • Type:
    Article
  • Authors:
    Memmesheimer, Raphael; Nogga, Jan; Pätzold, Bastian; Kruzhkov, Evgenii; Bultmann, Simon; Schreiber, Michael; Bode, Jonas; Karacora, Bertan; Park, Juhui; Savinykh, Alena; Behnke, Sven
  • Year:
    2024
  • Source:
    https://arxiv.org/abs/2412.14989

Citation information

Memmesheimer, Raphael; Nogga, Jan; Pätzold, Bastian; Kruzhkov, Evgenii; Bultmann, Simon; Schreiber, Michael; Bode, Jonas; Karacora, Bertan; Park, Juhui; Savinykh, Alena; Behnke, Sven: RoboCupatHome 2024 OPL Winner NimbRo: Anthropomorphic Service Robots using Foundation Models for Perception and Planning, arXiv, 2024, https://arxiv.org/abs/2412.14989, Memmesheimer.etal.2024a,

Associated Lamarr Researchers

lamarr institute person Behnke Sven - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Sven Behnke

Area Chair Embodied AI to the profile