Open-World Semantic Segmentation Including Class Similarity

Interpreting camera data is key for autonomously acting systems, such as autonomous vehicles. Vision systems that operate in real-world environments must be able to understand their surroundings and need the ability to deal with novel situations. This paper tackles open-world semantic segmentation, i.e., the variant of interpreting image data in which objects occur that have not been seen during training. We propose a novel approach that performs accurate closed-world semantic segmentation and, at the same time, can identify new categories without requiring any additional training data. Our approach additionally provides a similarity measure for every newly discovered class in an image to a known category, which can be useful information in downstream tasks such as planning or mapping. Through extensive experiments, we show that our model achieves state-of-the-art results on classes known from training data as well as for anomaly segmentation and can distinguish
between different unknown classes.

  • Published in:
    IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • Type:
    Inproceedings
  • Authors:
    Sodano, Matteo; Magistri, Federico; Nunes, Lucas; Behley, Jens; Stachniss, Cyrill
  • Year:
    2024

Citation information

Sodano, Matteo; Magistri, Federico; Nunes, Lucas; Behley, Jens; Stachniss, Cyrill: Open-World Semantic Segmentation Including Class Similarity, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, https://www.ipb.uni-bonn.de/pdfs/sodano2024cvpr.pdf, Sodano.etal.2024a,

Associated Lamarr Researchers

lamarr institute person Stachniss Cyrill e1663922306234 - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Cyrill Stachniss

Principal Investigator Embodied AI to the profile