Mask4D: End-to-End Mask-Based 4D Panoptic Segmentation for LiDAR Sequences

Scene understanding is crucial for autonomous systems to reliably navigate in the real world. Panoptic segmentation of 3D LiDAR scans allows us to semantically describe a vehicle’s environment by predicting semantic classes for each 3D point and to identify individual instances through different instance IDs. To describe the dynamics of the surroundings, 4D panoptic segmentation further extends this information with temporarily consistent instance IDs to identify the different instances in the scans consistently over whole sequences. Previous approaches for 4D panoptic segmentation rely on post-processing steps and are often not end-to-end trainable. In this paper, we propose a novel approach that can be trained end-to-end and directly predicts a set of non-overlapping masks along with their semantic classes and instance IDs that are consistent over time without any postprocessing like clustering or associations between predictions. We extend a mask-based 3D panoptic segmentation model to 4D by reusing queries that decoded instances in previous scans. This way, each query decodes the same instance over time, carries its ID and the tracking is performed implicitly. This enables us to jointly optimize segmentation and tracking and directly supervise for 4D panoptic segmentation.

  • Published in:
    IEEE Robotics and Automation Letters
  • Type:
    Article
  • Authors:
    Marcuzzi, Rodrigo; Nunes, Lucas; Wiesmann, Louis; Marks, Elias; Behley, Jens; Stachniss, Cyrill
  • Year:
    2023

Citation information

Marcuzzi, Rodrigo; Nunes, Lucas; Wiesmann, Louis; Marks, Elias; Behley, Jens; Stachniss, Cyrill: Mask4D: End-to-End Mask-Based 4D Panoptic Segmentation for LiDAR Sequences, IEEE Robotics and Automation Letters, 2023, 8, 11, 7487--7494, https://www.ipb.uni-bonn.de/pdfs/marcuzzi2023ral-meem.pdf, Marcuzzi.etal.2023b,

Associated Lamarr Researchers

lamarr institute person Stachniss Cyrill e1663922306234 - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Prof. Dr. Cyrill Stachniss

Principal Investigator Embodied AI to the profile