Open-Vocabulary and Semantic-Aware Reasoning for Search and Retrieval of Objects in Dynamic and Concealed Spaces

We present an open-vocabulary framework that combines spatial, semantic, and geometric reasoning to solve a highly relevant, yet under-studied problem: Given an (outdated) map of an environment, how can a robot efficiently retrieve relocated or unmapped items? By unifying spatial cues about proximity and topology, semantic priors on typical placements, and geometric constraints that rule out infeasible locations, particularly within concealed spaces, our approach finds objects even when they are relocated or hidden in drawers or cabinets. We further propose in-situ viewpoint planning to model new objects for manipulation, and to add the object to our dynamic 3D scene graph. We validate our framework through extensive real-world trials on the Stretch SE3 mobile manipulator, evaluating search and retrieval in various conditions. Results demonstrate robust navigation (100 percent) and open-space detection (100 percent), with semantic-geometric reasoning reducing concealed space search time by 68 percent versus semantic-only approaches. Implemented on a low-cost, compact mobile manipulator, our solution combines sophisticated cognitive capabilities with practical deployability, representing a significant step toward accessible service robots for everyday homes.